2009-05-06 @ 23:33

[ANN] nokogiri 1.3.0rc1 has been released!

= nokogiri version 1.3.0rc1 has been released!

Thanks to herculean efforts by my nokogiri partner in crime, Mike Dalessio, nokogiri now works on JRuby 1.3.0RC1 via FFI.

To install this prerelease gem do this: $ jgem install nokogiri -s http://tenderlovemaking.com/ ~~~

Then you should be able to do this:

$ jirb irb(main):001:0> require 'open-uri' => true irb(main):002:0> require 'rubygems' => true irb(main):003:0> require 'nokogiri' => true irb(main):004:0> doc = Nokogiri::HTML(open('http://www.google.com/search?q=tenderlove')) => # irb(main):005:0> doc.css('h3.r a.l').length => 10 irb(main):006:0> ~~~

== CAVEATS!

  • The JRuby FFI gem only works with JRuby 1.3.0RC1
  • You MUST install it from my gem server
  • The gem version will say 1.2.4, that is actually because I couldn’t get pre release gem versions working. Don’t worry, it’s actually the 1.3.0 release candidate.
  • You can get an MRI version and the JRuby version from my gem server, no windows support yet.

== ACCOLADES

  • Mike made this FFI monster happen! I can’t thank him enough.
  • Thanks to the JRuby team for making FFI work!

== CHANGELOG

  • hahahahahaha
  • hahahahahahaha
  • hahahaha
  • hahahahahahahha
  • You’ll get to see the acutal changes when this isn’t a release candidate
  • Or check out the git repository

== More information

read more »

2009-05-07 @ 11:26

Fat binary gems make the rockin' world go round

Right now people who publish native gems targeting the windows platform have a problem. Our problem is supporting ruby 1.8 and 1.9 at the same time. Right now, we can’t build one gem targeting 1.8 and one gem targeting 1.9, and have rubygems differentiate the two. I have a solution: fat binary gems. We can build a gem that contains dynamic libraries that target ruby 1.8 and ruby 1.9 on windows, with no changes to rubygems whatsoever. I’ve put together a proof of concept that I want to share. I will walk through the steps for building a fat binary gem with the tools we have today. The steps I am going to present are not necessarily the best steps, they are just the steps I took to get this idea working.

The tools I will use are MinGW for cross compiling, hoe and rake-compiler for their packaging and compiling tasks, multiruby for cross compiling 1.8 and 1.9, and use nokogiri as the target gem to be built.

Here is the basic strategy for making dreams happen:

1. Gem entry point must be written in Ruby

When someone does “require ‘whatever’” on your library, that ‘whatever.rb’ file must be written in ruby and work with both 1.8 and 1.9. The reason is because we will:

2. Dynamically determine the correct SO file to load

We can determine at runtime the current ruby version, then load the appropriate SO file at runtime.

Let’s get down to business and see it in action.

Getting our hands dirty

The first thing we need to do is make sure that the so file from the ruby 1.8 build and the ruby 1.9 build are in a different place. The way I accomplished this was by customizing my Rake::Extension task (from rake-compiler), and adding a prerequisite to the cross task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
RET = Rake::ExtensionTask.new("nokogiri", HOE.spec) do |ext|
  ext.lib_dir = "ext/nokogiri"
end

task :muck_with_lib_dir do
  RET.lib_dir += "/#{RUBY_VERSION.sub(/\.\d$/, '')}"
  FileUtils.mkdir_p(RET.lib_dir)
end
if Rake::Task.task_defined?(:cross)
  Rake::Task[:cross].prerequisites << "muck_with_lib_dir"
end[/sourcecode]

This code will make sure the so file goes to "ext/nokogiri/1.8" when compiling with ruby 1.8, and "ext/nokogiri/1.9" when compiling with 1.9.  Then, all you have to do is compile your extension twice:

<code>
$ ~/.multiruby/install/1.8.6-p114/bin/rake cross compile
$ rm -rf tmp
$ $ ~/.multiruby/install/1.9.1-rc2/bin/rake cross compile

WARNING! Watch out for that “rm -rf”. That is removing the tmp directory that rake-compiler made. rake-compiler doesn’t seem to know that I switched ruby versions. In order to get the two different compilations working, I had to manually remove the already compiled objects.

Dynamic loading

So we’ve got our compiled so files in two different locations. What about loading? This step is very easy. Since our entry point will be in ruby, we can just write this in our entry point file:

1
2
3
4
5
6
7
8
9
10
11
12
13
if RUBY_PLATFORM =~/(mswin|mingw)/i
  # Fat binary gems, you make the Rockin' world go round
  require "nokogiri/#{RUBY_VERSION.sub(/\.\d+$/, '')}/nokogiri"
else
  require 'nokogiri/nokogiri'
end[/sourcecode]

Basically all this code says is "if we're running windows, load the shared object from a path that contains the ruby version".  When a windows user requires this file, the path to the shared object is <strong>determined by the version of ruby</strong> that they are using.  If they're running 1.8, the path will be "nokogiri/1.8/nokogiri", if they're running 1.9, "nokogiri/1.9/nokogiri".

<h3>Packaging</h3>
We've got one more hurdle to overcome, and that is packaging.  We need to make sure that when we're building the windows gem, our custom so files are added to the gem.  To do this, I just added another task:

{:lang="ruby"}

task :add_dll_to_manifest do HOE.spec.files += Dir[‘ext/nokogiri/.{dll,so}’] HOE.spec.files += Dir[‘ext/nokogiri/{1.8,1.9}/.{dll,so}’] end

if Rake::Task.task_defined?(:cross) Rake::Task[:cross].prerequisites « :add_dll_to_manifest end[/sourcecode]

This makes sure that any extra dll or so files in our ext directories are added to the gem. Now we can run our packaging task: $ ~/.multiruby/install/1.8.6-p114/bin/rake cross native gem ~~~

If everything went well, we can examine the content of our packaged gem and find two different so files: $ gem spec pkg/nokogiri-1.2.4-x86-mswin32.gem files | grep nokogiri.so - ext/nokogiri/1.8/nokogiri.so - ext/nokogiri/1.9/nokogiri.so $ ~~~

Conclusion

There we have it, a fat binary gem. This gem will work with Ruby 1.8 OR Ruby 1.9 on windows. If you’re a windows user, and you’d like to try using this fat binary gem, I have it on my gem server. Just do: $ gem install nokogiri -s http://tenderlovemaking.com/ ~~~ The next full release of nokogiri will be using this technique for windows builds. Also, the rake tasks that I've presented were somewhat simplified. If you'd like to get very specific, check out the nokogiri source.

Next Steps

I would like to work with Luis on integrating this functionality in to rake-compiler. I’m not sure the best way to go about it, but I know that he and I can simplify these steps even further.

read more »

2009-05-18 @ 19:13

Autotest and Vim integration

Yay! I got vim and autotest integration working. When I run autotest, if there is an error, I can have Vim read the errors from autotest and jump me to the right place.

Here is a video of me using it:

Please note that I’m not copying and pasting anything. In vim, I hit a command and Vim automatically picks up errors from autotest and jumps me to the line where the error occurred.

You too can impress your friends with this trick! Here’s how:

  1. Make sure you have vim-ruby installed
  2. Use this as your .autotest file: {:lang="ruby"} ~~~ require 'autotest/restart' Autotest.add_hook :initialize do |at| at.unit_diff = 'cat' end Autotest.add_hook :ran_command do |at| File.open('/tmp/autotest.txt', 'wb') { |f| f.write(at.results.join) } end ~~~
  3. Add this to your .vimrc: compiler rubyunit nmap <Leader>fd :cf /tmp/autotest.txt<cr> :compiler rubyunit<cr>
  4. Now when you get an error in autotest, just type "\fd" in Vim to jump straight to your first error. The contents of /tmp/autotest.txt will be used in your errorfile. In Vim do ":help quickfix" for more info on what you can do with your new found power. Caveat: You don't get unit_diff. I'm working on that. Any help would be much appreciated (I suck at errorformat in Vim).
read more »