Tender Lovemaking

Shit. This is a boring topic. Just writing the title made me cry a little bit out of boredom. Unfortunately this topic is something I feel compelled to write about because I think that most Ruby developers dealing with XML know very little about the topic, and yet XML namespaces are crucial when dealing with XML documents. So, in order to curb the boredom, I will attempt to demonstrate why we need namespaces and how they affect you when dealing with XML in the shortest amount of time. I will also try to sprinkle in a few swear words and innuendos just to make sure you’re paying attention.

A tale of two companies

One day, long ago, when XML was written on punch cards, Alice’s Auto Supply decided that they would distribute their inventory as XML so that other people would know what they had in stock. They came up with an XML document that looked like this:

<?xml version="1.0"?>
<inventory>
  <tire name="super slick racing tire" />
  <tire name="all weather tire" />
</inventory>

Excellent! Programmers started consuming the inventory for Alice’s shop. They could pull a list of tires from the document like this:

doc.xpath('//tire')

Bob’s Bike Shop also wanted to get on the XML broadcast bandwagon. So they followed suit and produced an XML document as well:

<?xml version="1.0"?>
<inventory>
  <tire name="narrow street tire" />
  <tire name="mountain trail tire" />
</inventory>

Again, programmers were happy. They started consuming inventory for Bob’s Bike Shop and getting a list of bike tires like this:

doc.xpath('//tire')

Everything was going well until someone decided to consume inventory from both sources. Fuck. There was no way to tell the difference between a car tire from Alice’s Auto Supply, or a bike tire from Bob’s Bike Shop. The search criteria used for both documents was the same:

doc.xpath('//tire')

How was one suppose to tell the difference between a bike tire and a car tire? There is a naming conflict, and this code would return both! Systems began crashing, punch cards were lit on fire, massive power outages occurred. Needless to say, our society was at it’s lowest point.

Fortunately, a very smart group of people (much smarter than me) came along and said “let’s associate these things to something unique, then we can tell them apart”. That unique bit of information was a URL. They also said “let’s have the ability to name the url so that we can easily reference it in our XML documents”. Fortunately, the names for a URL does not have to be unique since it is tied to the unique URL! Yay!

Alice got wind of these updates to the XML spec, and wanted to make sure that everyone could tell the difference between her car tires and some other tire. So she updated her inventory document, adding her URL with a name, and naming all of her inventory:

<?xml version="1.0"?>
<inventory xmlns:car="http://alicesautosupply.example.com/">
  <car:tire name="super slick racing tire" />
  <car:tire name="all weather tire" />
</inventory>

Alice’s new inventory document was pushed out. The developers had not yet updated their code. They thought they had a new bug, the code was only returning tires from Bill’s Bike shop. But how to get the car tires? They had to inform the parser they were looking for tires associated with Alice’s url, and changed their code to look like this:

doc.xpath('//tire')
doc.xpath('//aliceAuto:tire',
  'aliceAuto' => 'http://alicesautosupply.example.com/'
)

The first query returned tires that have no namespace and the second query returned tires that belong to Alice’s shop.

Alice’s inventory grew and grew (owing it all to namespacing her document of course). Prefixing everything in her document with “car” was taking a toll on her fingers as well as her puchcard supply. The XML superheros had a trick up their sleeves for Alice. They said that “URLs could be declared as a default” that way every tag could be associated with a URL but not explicitly declare a name.

Armed with this knowledge, Alice was able to change her XML to this:

<?xml version="1.0"?>
<inventory xmlns="http://alicesautosupply.example.com/">
  <tire name="super slick racing tire" />
  <tire name="all weather tire" />
</inventory>

Her XML now stated that “inventory” and all tags inside “inventory” belonged to her URL, but did not need a prefix. Everything was still associated with her URL so that developers could tell the difference between her tires and all other tires, but she did not have to add prefixes.

As for the developers consuming her XML, they never noticed the change. Their code asked for all tires belonging to Alice’s URL, and Alice’s document still declared that those tires belonged to her URL. Punchcards were saved, carpel tunnel was cured, and the world rejoiced.

Conclusion

I hope this story hasn’t been too boring. Here are the key parts I wanted to explain:

1. Namespaces prevent tag name collision.
   We would not be able to deal with colliding tag names without namespaces.
2. When you don’t specify a namespace in your search, that means you want tags with no namespace.
3. Namespaces are tied to URLs. Only the URL must be unique which is why you must use the URL in your XPath queries.

Remember that there is a difference between asking for tags that belong to a namespace and ones that do not. Also remember that a default namespace in a document means that tags which do not explicitly have a namespace belong to the default. So even though those tags do not have a prefix you must use a namespace when querying for them.

Bonus Round

Even though using namespaces is essential when searching an XML document, Nokogiri tries to help out. If there are namespaces declared on the root node of a document, Nokogiri will automatically register those for you. You will still have to use the prefix when searching the document, but the URL registration is done for you.

Let’s modify Alice’s XML a little to demonstrate:

<?xml version="1.0"?>
<inventory xmlns="http://alicesautosupply.example.com/" xmlns:bike="http://bobsbikes.example.com.">
  <tire name="super slick racing tire" />
  <tire name="all weather tire" />
  <bike:tire name="skinny street" />
</inventory>

Using Nokogiri, these two statements are equivalent:

doc.xpath('//xmlns:tire',
  'xmlns' => 'http://alicesautosupply.example.com/'
)
doc.xpath('//xmlns:tire')

We can specify the namespace ourselves, or use the same name that Nokogiri picks for us.

Similarly, if we want to find bike tires, these two statements are equivalent:

doc.xpath('//bike:tire',
  'bike' => 'http://bobsbikes.example.com/'
)
doc.xpath('//bike:tire')

The other day at LA RubyConf during the Johnson presentation, I showed a few slides which I don’t think were given the time that they deserve. Not that we didn’t have enough time, I just don’t think I made as big a deal about them as I should have. Those particular slides demonstrated HTML Document Object manipulation executed in JavaScript outside any web browser. Those particular slides, and that particular code, is the culmination of over a year worth of work (and Yak Shaving) and I would like to talk about it a little more in detail here.

Since I started doing any sort of non-trivial browser dependent JavaScript, I’ve wanted to be able to test the code which I wrote. Hitting refresh on a webpage seems like a hack. Setting up a special browser to refresh the page for me also seems like a hack. I want to run “rake test” and have my JavaScript DOM manipulations tested right along with everything else, no browser dependence required. As far as I could tell, we need three things to make that happen:

1. A JavaScript runtime that can be used in Ruby
2. A parser with browser-like HTML correction schemes
3. A DOM interface that mirrors a browsers DOM interface

Over the weekend, I think we’ve come a lot closer. John has finally released Johnson. Johnson solves problem number 1. Johnson provides a JavaScript runtime that is fully accessible in Ruby. Watch our RubyConf 2008 presentation about Johnson for more details about that project.

Number 2, I believe, has been solved by nokogiri. As far as I can tell, the tree generated inside libxml2 is very similar to one found in the browser. Nokogiri was partly a Yak Shave for number 3. Since I had writing a DOM interface in mind, nokogiri’s api lends well to writing a DOM api.

Number 3 was partly solved this weekend. I’ve been working on a DOM api called taka. Taka sits a DOM api on top of nokogiri. The goal of the project is to mirror a browser’s DOM api in Ruby.

With these three tools in place, I believe that we have a good start on a browserless JavaScript testing environment.

Codes

Enough talk. Let’s look at some codes. Take this HTML page for example:

<html>
  <head>
    <script>
      function populateDropDown() {
        var select = document.getElementById('colors');
        var options = ['red', 'green', 'blue', 'black'];
        var i;
        for(i = 0; i < options.length; i++) {
          var option = document.createElement('option');
          option.appendChild(document.createTextNode(options[i]));
          option.value = options[i];
          select.appendChild(option);
        }
      }
    </script>
  </head>
  <body onload="populateDropDown()">
    <h1>Behold the Johnson</h1>
    <form>
      <select id="colors">
      </select>
    </form>
  </body>
</html>

The JavaScript in this HTML will add a few option tags as children of the select tag. Effectively populating the drop down for our user. It would be nice if we could write a test to assert that when this JavaScript executes, the option tags are actually added as children of the select tag.

With Johnson and Taka, it is possible to write such a test:

require 'rubygems'
require 'taka'
require 'johnson'
require 'test/unit'

class OptionTagsAppendedTest < Test::Unit::TestCase
  def setup
    # Create our DOM object
    @document = Taka::DOM::HTML(DATA.read)

    # Create a new JavaScript runtime
    @rt = Johnson::Runtime.new

    # Set the document in the runtime
    @rt['document'] = @document

    # Execute any script tags
    @document.getElementsByTagName('script').each do |script|
      @rt.evaluate(script.textContent);
    end
  end

  def test_options_populated_by_onload
    # 0 option tags before onload is executed
    assert_equal 0, @document.getElementsByTagName('option').length

    # Execute the onload body attribute
    @rt.evaluate(@document.getElementsByTagName('body')[0].onload)

    # 4 option tags after onload is executed
    assert_equal 4, @document.getElementsByTagName('option').length
  end
end

There. It’s done. This test executes the JavaScript and manipulates your HTML the same way the browser would. You can run this code today, just make sure to install the johnson and taka gems first.

Problems

There are at least a few problems. This HTML code is, admittedly, carefully crafted. So far, taka only implements the DOM 1 interface. That means taka is missing many methods that are available in browsers. The good news though is that Taka is pure ruby and open source. As soon as you find methods that are missing, fork the repo, add a test, and send a pull request. I will be sure to merge it.

Conclusion

We are making progress towards testing JavaScript without a browser. We have to do it a step at a time. The solution I have presented to you, while not complete, has promise. I think that the only thing standing in our way right now is time and man power. The methods that need to be implemented on Taka to make it mirror a browser are not hard (take a look at the taka source). These methods just need to be written.

Oops! When I released nokogiri version 1.0.7, I totally forgot to talk about Nokogiri::Slop() feature that was added. Why is it called “slop”? It lets you sloppily explore documents. Basically, it decorates your document with method_missing() that allows you to search your document via method calls.

Given this document:

doc = Nokogiri::Slop(<<-eohtml)
<html>
  <body>
    <p>hello</p>
    <p class="bold">bold hello</p>
  <body>
</html>
eohtml

You may look through the tree like so:

doc.html.body.p('.bold').text # => 'bold hello'

The way this works is that method missing is implemented on every node in the document tree. That method missing method creates an xpath or css query by using the method name and method arguments. This means that a new search is executed for every method call. It’s fun for playing around, but you definitely won’t get the same performance as using one specific CSS search.

My favorite part is that method missing is actually in the slop decorator. When you use the Nokogiri::Slop() method, it adds the decorator to a list that gets mixed in to every node instance at runtime using Module#extend. That lets me have sweet method missing action, without actually putting method missing in my Node class.

Here is a simplified example:

module Decorator
  def method_a
    "method a"
  end

  def method_b
    "method b: #{super}"
  end
end

class Foo
  def method_b
    "inside foo"
  end
end

foo = Foo.new
foo.extend(Decorator)

puts foo.method_a # => 'method a'
puts foo.method_b # => 'method b: inside foo'

foo2 = Foo.new
puts foo2.method_b # => 'inside foo'
puts foo2.method_a # => NoMethodError

Module#extend is used to add functionality to the instance ‘foo’, but not ‘foo2′. Both ‘foo’ and ‘foo2′ are instances of Foo, but using Module#extend, we can conditionally add functionality without monkey patching and keeping a clean separation of concerns. You can even reach previous functionality by calling super.

But wait! There’s more! You can stack up these decorators as much as you want. For example:

module AddAString
  def method
    "Added a string: #{super}"
  end
end

module UpperCaseResults
  def method
    super.upcase
  end
end

class Foo
  def method
    "foo"
  end
end

foo = Foo.new
foo.extend(AddAString)
foo.extend(UpperCaseResults)

puts foo.method # => 'ADDED A STRING: FOO'

Conditional functionality added to methods with no weird “alias method chain” involvement. Awesome!

I love ruby!

While I was developing nokogiri, I had to learn how to cross compile gems for win32. I don’t have a compiler on windows, so I had to do this on OS X. I just want to dump a few notes here so that other people might benefit, and so that I won’t forget in the future.

As far as I can tell, there are 4 major steps to getting your native gem cross compiled for windows:

1. Get a cross compiler (mingw)
2. Cross compile ruby
3. Cross compile your gem
4. Building your gemspec

Step 1, The Cross Compiler

This step is pretty easy. I used Mac Ports to install mingw32. I just did:

$ sudo port install i386-mingw32-binutils i386-mingw32-gcc i386-mingw32-runtime i386-mingw32-w32api
[/sourcecode]

namespace :build do
  file "stash/ruby-1.8.6-p287.tar.gz" do |t|
    puts "downloading ruby"
    FileUtils.mkdir_p('stash')
    Dir.chdir('stash') do
      url = ("ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p287.tar.gz")
      system("wget #{url} || curl -O #{url}")
    end
  end
end

namespace :build do
  namespace :win32 do
    file 'cross/bin/ruby.exe' => ['cross/ruby-1.8.6-p287'] do
      Dir.chdir('cross/ruby-1.8.6-p287') do
        str = ''
        File.open('Makefile.in', 'rb') do |f|
          f.each_line do |line|
            if line =~ /^\s*ALT_SEPARATOR =/
              str += "\t\t    " + 'ALT_SEPARATOR = "\\\\\"; \\'
              str += "\n"
            else
              str += line
            end
          end
        end
        File.open('Makefile.in', 'wb') { |f| f.write str }
        buildopts = if File.exists?('/usr/bin/i586-mingw32msvc-gcc')
                      "--host=i586-mingw32msvc --target=i386-mingw32 --build=i686-linux"
                    else
                      "--host=i386-mingw32 --target=i386-mingw32"
                    end
        sh(<<-eocommand)
          env ac_cv_func_getpgrp_void=no \
            ac_cv_func_setpgrp_void=yes \
            rb_cv_negative_time_t=no \
            ac_cv_func_memcmp_working=yes \
            rb_cv_binary_elf=no \
            ./configure \
            #{buildopts} \
            --prefix=#{File.expand_path(File.join(Dir.pwd, '..'))}
        eocommand
        sh 'make'
        sh 'make install'
      end
    end

    desc 'build cross compiled ruby'
    task :ruby => 'cross/bin/ruby.exe'
  end
end

namespace :build
  task :win32 do
    dash_i = File.expand_path(
      File.join(File.dirname(__FILE__), 'cross/lib/ruby/1.8/i386-mingw32/')
    )
    Dir.chdir('ext/nokogiri') do
      ruby " -I #{dash_i} extconf.rb"
      sh 'make'
    end
  end
end

HOE = Hoe.new('nokogiri', Nokogiri::VERSION) do |p|
  p.developer('Aaron Patterson', 'aaronp@rubyforge.org')
  p.developer('Mike Dalessio', 'mike.dalessio@gmail.com')
  p.clean_globs = [
    'ext/nokogiri/Makefile',
    'ext/nokogiri/*.{o,so,bundle,a,log,dll}',
    'ext/nokogiri/conftest.dSYM',
    GENERATED_PARSER,
    GENERATED_TOKENIZER,
    'cross',
  ]
  p.spec_extras = { :extensions => ["Rakefile"] }
end

namespace :gem do
  namespace :win32 do
    task :spec => ['build:win32'] do
      File.open("#{HOE.name}.gemspec", 'w') do |f|
        HOE.spec.files += Dir['ext/nokogiri/**.{dll,so}']
        HOE.spec.platform = 'x86-mswin32-60'
        HOE.spec.extensions = []
        f.write(HOE.spec.to_ruby)
      end
    end
  end
end