Tender Lovemaking

I’ve been refactoring Mechanize for an 0.7.0 release. Basically I’m trying to clean the code up and there are a few features that I think are unnecessary, but I would like to ask people first.

1. REXML as a parser.

I want to remove support for REXML. I don’t use it. Hpricot seems to do everything I need.

2. 1.8.2 thru 1.8.4 support

I’ve got a bunch of monkey patches for 1.8.2 thru 1.8.4. I’d like to remove these because I think most people are on 1.8.5 or up.

3. WWW::Mechanize::Page#watch_for_set

I am going to remove this method. It made sense when REXML was the main parser, since REXML was so slow. I think that Hpricot is fast enough that this method is not so useful.

I’m going to make 0.7.0 lazily build up form and link objects, which should give everyone a slight speed increase but makes watch_for_set obsolete (sort of).

I’m changing around the class names to be better organized, but they should all have the same methods. Also, if there are any feature requests, let me know!

The Mechanize library is used for automating interaction with websites.
Mechanize automatically stores and sends cookies, follows redirects,
can follow links, and submit forms. Form fields can be populated and
submitted. Mechanize also keeps track of the sites that you have visited as
a history.

Changes:

= Mechanize CHANGELOG

== 0.6.5

* Copying headers to a hash to prevent memory leaks
* Speeding up page parsing
* Aliased fields to elements
* Adding If-Modified-Since header
* Added delete_field! to form. Thanks to Sava Chankov
* Updated uri escaping to support high order characters. Thanks to Henrik Nyh.
* Better handling relative URIs. Thanks to Henrik Nyh
* Now handles pipes in URLs

http://rubyforge.org/tracker/?func=detail&aid=7140&group_id=1453&atid=5709

* Now escaping html entities in form fields.

http://rubyforge.org/tracker/?func=detail&aid=7563&group_id=1453&atid=5709

* Added MSIE 7.0 user agent string

mechanize version 0.6.5 has been released!

http://mechanize.rubyforge.org/

The Mechanize library is used for automating interaction with websites.
Mechanize automatically stores and sends cookies, follows redirects,
can follow links, and submit forms. Form fields can be populated and
submitted. Mechanize also keeps track of the sites that you have visited as
a history.

Changes:

= Mechanize CHANGELOG

== 0.6.5

* Copying headers to a hash to prevent memory leaks
* Speeding up page parsing
* Aliased fields to elements
* Adding If-Modified-Since header
* Added delete_field! to form. Thanks to Sava Chankov
* Updated uri escaping to support high order characters. Thanks to Henrik Nyh.
* Better handling relative URIs. Thanks to Henrik Nyh
* Now handles pipes in URLs

http://rubyforge.org/tracker/?func=detail&aid=7140&group_id=1453&atid=5709

* Now escaping html entities in form fields.

http://rubyforge.org/tracker/?func=detail&aid=7563&group_id=1453&atid=5709

* Added MSIE 7.0 user agent string

http://mechanize.rubyforge.org/

I was debugging Mechanize the other day, and thought it would be handy to have a graph of objects in memory and they’re relationship with each other. So I put together a simple script that outputs a Graphviz file illustrating what the object points to. Here’s the code:

require 'ograph'
require 'rubygems'
require 'mechanize'

mech = WWW::Mechanize.new
mech.get('http://google.com/')

puts ObjectGraph.graph(mech, /^WWW/)

and the output:

Right now it only supports Arrays, but should be easily extensible to support all other Enumerable types. Here’s the code for ObjectGraph:

class ObjectGraph
  def self.graph(target, class_name = /./)
    stack         = [target]
    object_links  = []
    seen_objects  = []
    seen_hash     = {}

    while stack.length > 0
      object = stack.pop
      next if seen_hash.key? object.object_id
      seen_hash[object.object_id] = 1

      if object.is_a?(Enumerable) && ! object.is_a?(String)
        object.each { |iv|
          if iv.class.to_s =~ class_name || object.is_a?(Enumerable)
            object_links.push([object.object_id, iv.object_id])
            stack.push(iv)
          end
        }
      else
        object.instance_variables.each do |iv_sym|
          iv = object.instance_variable_get iv_sym
          if iv.class.to_s =~ class_name || iv.is_a?(Enumerable)
            object_links.push([object.object_id, iv.object_id])
            stack.push(iv)
          end
        end
      end
      seen_objects.push([object.object_id, object.class])
    end

    s = <<END
digraph g {
    graph [ rankdir = "LR" ];
    node [ fontsize = "8"
           shape = "ellipse"
    ];
    edge [ ];
END
    seen_objects.each { |id, klass|
      s += <<END
      "#{id}" [
      label = "<f0> #{id}|#{klass}"
      shape = "record"
      ]
END
    }
    object_links.each_with_index { |(from, to), i|
      s += "\"#{from}\":f0 -> \"#{to}\":f0 [ id = #{i} ]\n"
    }
    s += "}\n"
    s
  end
end

Update: added Enumerable support, so Hashes are now graphed.

I thought that while I was traveling in Europe I would be able to work on WWW::Mechanize, but then I found out that I can’t do anything when I’m travling. Traveling is too much work for me to do anything productive. Luckily I took a day off after my travels, and I was able to complete my work on Mechanize only a couple days late. I think Mechanize is getting more popular, judging from the number of bug submitted. I’ve fixed more bugs this release than any other. Check out the CHANGELOG for more details on the bug fixes.

As far as new features are concerned, I’ve added a submit method to forms. Before, you had to call submit on the mechanize object and pass in the form like this:

page = agent.submit form

Now you can just call the submit method on the form like this:

page = form.submit

Along the same line, I’ve added a click method to links. Instead of passing a link to the agent object, you can just call the click method on the link like so:

page = link.click

I wonder when people will figure out where the release names come from. Nobody has said anything about them to me….. So far I have:

• Twan
• Sylvester
• Rufus
• Chuck

I’ve been working pretty hard on merging Mechanize with Hpricot in to what I call Mechpricot, or Hprichanize. Things are going quite well.

I’ve totally cleaned up the Cookie code to use more cookie code from WEBrick, and in the process I found a bug in the way that WEBrick parses set-cookie headers. Unfortunately no one has replied to my patch on ruby-talk, so I think I’ll try ruby-core next…..

I removed the dependency on mime-types in favor of using code in WEBrick. I’d like to have as few dependencies as possible. I would like to make a few more performance tweaks before I release. I’ll see if I can get this completely done before this weekend. Otherwise everyone will have to wait until I’m back from Spain!

I was able to write a pretty quick mechanize script that grabbed the playlist from kexp and put it in a SQLite database. From that, I came up with the top 4 songs played on KEXP for July:

1. Tv On The Radio – Playhouses
2. Midlake – Roscoe
3. Johnny Cash – Gods Gonna Cut You Down
4. Silversun Pickups – Lazy Eye

I also used Gruff Graphs to make a graph: