Tenderlove Making

Fat binary gems make the rockin' world go round

May 7, 2009 @ 11:26 am

Right now people who publish native gems targeting the windows platform have a problem. Our problem is supporting ruby 1.8 and 1.9 at the same time. Right now, we can’t build one gem targeting 1.8 and one gem targeting 1.9, and have rubygems differentiate the two. I have a solution: fat binary gems. We can build a gem that contains dynamic libraries that target ruby 1.8 and ruby 1.9 on windows, with no changes to rubygems whatsoever. I’ve put together a proof of concept that I want to share. I will walk through the steps for building a fat binary gem with the tools we have today. The steps I am going to present are not necessarily the best steps, they are just the steps I took to get this idea working.

The tools I will use are MinGW for cross compiling, hoe and rake-compiler for their packaging and compiling tasks, multiruby for cross compiling 1.8 and 1.9, and use nokogiri as the target gem to be built.

Here is the basic strategy for making dreams happen:

Let’s get down to business and see it in action.

RET = Rake::ExtensionTask.new("nokogiri", HOE.spec) do |ext|
  ext.lib_dir = "ext/nokogiri"
end

task :muck_with_lib_dir do
  RET.lib_dir += "/#{RUBY_VERSION.sub(/\.\d$/, '')}"
  FileUtils.mkdir_p(RET.lib_dir)
end
if Rake::Task.task_defined?(:cross)
  Rake::Task[:cross].prerequisites << "muck_with_lib_dir"
end[/sourcecode]

This code will make sure the so file goes to "ext/nokogiri/1.8" when compiling with ruby 1.8, and "ext/nokogiri/1.9" when compiling with 1.9.  Then, all you have to do is compile your extension twice:

<code>
$ ~/.multiruby/install/1.8.6-p114/bin/rake cross compile
$ rm -rf tmp
$ $ ~/.multiruby/install/1.9.1-rc2/bin/rake cross compile

WARNING! Watch out for that “rm -rf”. That is removing the tmp directory that rake-compiler made. rake-compiler doesn’t seem to know that I switched ruby versions. In order to get the two different compilations working, I had to manually remove the already compiled objects.

if RUBY_PLATFORM =~/(mswin|mingw)/i
  # Fat binary gems, you make the Rockin' world go round
  require "nokogiri/#{RUBY_VERSION.sub(/\.\d+$/, '')}/nokogiri"
else
  require 'nokogiri/nokogiri'
end[/sourcecode]

Basically all this code says is "if we're running windows, load the shared object from a path that contains the ruby version".  When a windows user requires this file, the path to the shared object is <strong>determined by the version of ruby</strong> that they are using.  If they're running 1.8, the path will be "nokogiri/1.8/nokogiri", if they're running 1.9, "nokogiri/1.9/nokogiri".

<h3>Packaging</h3>
We've got one more hurdle to overcome, and that is packaging.  We need to make sure that when we're building the windows gem, our custom so files are added to the gem.  To do this, I just added another task:

```ruby
task :add_dll_to_manifest do
  HOE.spec.files += Dir['ext/nokogiri/**.{dll,so}']
  HOE.spec.files += Dir['ext/nokogiri/{1.8,1.9}/**.{dll,so}']
end

if Rake::Task.task_defined?(:cross)
  Rake::Task[:cross].prerequisites << :add_dll_to_manifest
end[/sourcecode]

This makes sure that any extra dll or so files in our ext directories are added to the gem.  Now we can run our packaging task:
<code>
$ ~/.multiruby/install/1.8.6-p114/bin/rake cross native gem

If everything went well, we can examine the content of our packaged gem and find two different so files: $ gem spec pkg/nokogiri-1.2.4-x86-mswin32.gem files | grep nokogiri.so

  • ext/nokogiri/1.8/nokogiri.so
  • ext/nokogiri/1.9/nokogiri.so $

<h3>Conclusion</h3>
There we have it, a fat binary gem.  This gem will work with Ruby 1.8 OR Ruby 1.9 on windows.  If you're a windows user, and you'd like to try using this fat binary gem, I have it on my gem server.  Just do:
<code>
$ gem install nokogiri -s http://tenderlovemaking.com/

The next full release of nokogiri will be using this technique for windows builds. Also, the rake tasks that I’ve presented were somewhat simplified. If you’d like to get very specific, check out the nokogiri source.

« go back