Tenderlove Making

Fat binary gems make the rockin' world go round

Right now people who publish native gems targeting the windows platform have a problem. Our problem is supporting ruby 1.8 and 1.9 at the same time. Right now, we can’t build one gem targeting 1.8 and one gem targeting 1.9, and have rubygems differentiate the two. I have a solution: fat binary gems. We can build a gem that contains dynamic libraries that target ruby 1.8 and ruby 1.9 on windows, with no changes to rubygems whatsoever. I’ve put together a proof of concept that I want to share. I will walk through the steps for building a fat binary gem with the tools we have today. The steps I am going to present are not necessarily the best steps, they are just the steps I took to get this idea working.

The tools I will use are MinGW for cross compiling, hoe and rake-compiler for their packaging and compiling tasks, multiruby for cross compiling 1.8 and 1.9, and use nokogiri as the target gem to be built.

Here is the basic strategy for making dreams happen:

1. Gem entry point must be written in Ruby

When someone does “require ‘whatever’” on your library, that ‘whatever.rb’ file must be written in ruby and work with both 1.8 and 1.9. The reason is because we will:

2. Dynamically determine the correct SO file to load

We can determine at runtime the current ruby version, then load the appropriate SO file at runtime.

Let’s get down to business and see it in action.

Getting our hands dirty

The first thing we need to do is make sure that the so file from the ruby 1.8 build and the ruby 1.9 build are in a different place. The way I accomplished this was by customizing my Rake::Extension task (from rake-compiler), and adding a prerequisite to the cross task:

RET = Rake::ExtensionTask.new("nokogiri", HOE.spec) do |ext|
  ext.lib_dir = "ext/nokogiri"

task :muck_with_lib_dir do
  RET.lib_dir += "/#{RUBY_VERSION.sub(/\.\d$/, '')}"
if Rake::Task.task_defined?(:cross)
  Rake::Task[:cross].prerequisites << "muck_with_lib_dir"

This code will make sure the so file goes to "ext/nokogiri/1.8" when compiling with ruby 1.8, and "ext/nokogiri/1.9" when compiling with 1.9.  Then, all you have to do is compile your extension twice:

$ ~/.multiruby/install/1.8.6-p114/bin/rake cross compile
$ rm -rf tmp
$ $ ~/.multiruby/install/1.9.1-rc2/bin/rake cross compile

WARNING! Watch out for that “rm -rf”. That is removing the tmp directory that rake-compiler made. rake-compiler doesn’t seem to know that I switched ruby versions. In order to get the two different compilations working, I had to manually remove the already compiled objects.

Dynamic loading

So we’ve got our compiled so files in two different locations. What about loading? This step is very easy. Since our entry point will be in ruby, we can just write this in our entry point file:

if RUBY_PLATFORM =~/(mswin|mingw)/i
  # Fat binary gems, you make the Rockin' world go round
  require "nokogiri/#{RUBY_VERSION.sub(/\.\d+$/, '')}/nokogiri"
  require 'nokogiri/nokogiri'

Basically all this code says is "if we're running windows, load the shared object from a path that contains the ruby version".  When a windows user requires this file, the path to the shared object is <strong>determined by the version of ruby</strong> that they are using.  If they're running 1.8, the path will be "nokogiri/1.8/nokogiri", if they're running 1.9, "nokogiri/1.9/nokogiri".

We've got one more hurdle to overcome, and that is packaging.  We need to make sure that when we're building the windows gem, our custom so files are added to the gem.  To do this, I just added another task:

~~~ ruby
task :add_dll_to_manifest do
  HOE.spec.files += Dir['ext/nokogiri/**.{dll,so}']
  HOE.spec.files += Dir['ext/nokogiri/{1.8,1.9}/**.{dll,so}']

if Rake::Task.task_defined?(:cross)
  Rake::Task[:cross].prerequisites << :add_dll_to_manifest

This makes sure that any extra dll or so files in our ext directories are added to the gem.  Now we can run our packaging task:
$ ~/.multiruby/install/1.8.6-p114/bin/rake cross native gem

If everything went well, we can examine the content of our packaged gem and find two different so files: $ gem spec pkg/nokogiri-1.2.4-x86-mswin32.gem files | grep nokogiri.so - ext/nokogiri/1.8/nokogiri.so - ext/nokogiri/1.9/nokogiri.so $ ~~~


There we have it, a fat binary gem. This gem will work with Ruby 1.8 OR Ruby 1.9 on windows. If you’re a windows user, and you’d like to try using this fat binary gem, I have it on my gem server. Just do: $ gem install nokogiri -s http://tenderlovemaking.com/ ~~~ The next full release of nokogiri will be using this technique for windows builds. Also, the rake tasks that I've presented were somewhat simplified. If you'd like to get very specific, check out the nokogiri source.

Next Steps

I would like to work with Luis on integrating this functionality in to rake-compiler. I’m not sure the best way to go about it, but I know that he and I can simplify these steps even further.

« go back