Cross Compiling Ruby Gems for win32

While I was developing nokogiri, I had to learn how to cross compile gems for win32. I don’t have a compiler on windows, so I had to do this on OS X. I just want to dump a few notes here so that other people might benefit, and so that I won’t forget in the future.

As far as I can tell, there are 4 major steps to getting your native gem cross compiled for windows:

1. Get a cross compiler (mingw)
2. Cross compile ruby
3. Cross compile your gem
4. Building your gemspec

Step 1, The Cross Compiler

This step is pretty easy. I used Mac Ports to install mingw32. I just did: $ sudo port install i386-mingw32-binutils i386-mingw32-gcc i386-mingw32-runtime i386-mingw32-w32api ~~~

After a while, I could run i386-mingw32-gcc to compile stuff. Next up, cross compiling ruby.

Step 2, Cross Compile Ruby

This seemed like the hardest step to me. I was able to get ruby cross compiling to work after studying documentation at eigenclass, and reading Matt’s excellent notes in Johnson.

First, you have to download ruby, so I wrote a rake task to do just that. This rake task downloads ruby in to a “stash” directory: ~~~ ruby namespace :build do file “stash/ruby-1.8.6-p287.tar.gz” do |t| puts “downloading ruby” FileUtils.mkdir_p(‘stash’) Dir.chdir(‘stash’) do url = (“ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p287.tar.gz”) system(“wget #{url} || curl -O #{url}”) end end end ~~~ Next you have to apply a patch to Makefile.in so that it will work with the cross compiler. Once that patch is applied, you can compile ruby with mingw32. Here is my rake task to do that, and unfortunately the strange Makefile.in patch is very necessary: ~~~ ruby namespace :build do namespace :win32 do file ‘cross/bin/ruby.exe’ => [‘cross/ruby-1.8.6-p287’] do Dir.chdir(‘cross/ruby-1.8.6-p287’) do str = ‘’ File.open(‘Makefile.in’, ‘rb’) do |f| f.each_line do |line| if line =~ /^\s*ALT_SEPARATOR =/ str += “\t\t “ + ‘ALT_SEPARATOR = “\\"; \’ str += “\n” else str += line end end end File.open(‘Makefile.in’, ‘wb’) { |f| f.write str } buildopts = if File.exists?(‘/usr/bin/i586-mingw32msvc-gcc’) “–host=i586-mingw32msvc –target=i386-mingw32 –build=i686-linux” else “–host=i386-mingw32 –target=i386-mingw32” end sh(«-eocommand) env ac_cv_func_getpgrp_void=no \ ac_cv_func_setpgrp_void=yes \ rb_cv_negative_time_t=no \ ac_cv_func_memcmp_working=yes \ rb_cv_binary_elf=no \ ./configure \ #{buildopts} \ –prefix=#{File.expand_path(File.join(Dir.pwd, ‘..’))} eocommand sh ‘make’ sh ‘make install’ end end

desc 'build cross compiled ruby'
task :ruby => 'cross/bin/ruby.exe'   end end ~~~ After executing that task (which will take a while), you should have a cross compiled ruby that you can link against.

Step 3, Cross compiling your extension

The final part is cross compiling the extension. Now that you have your cross compiled ruby, you just need to cross compile your extension. The only thing special you need to do here is change the ‘-I’ flag you send to ruby when executing ‘extconf.rb’. Here is a slightly simplified version of my task to do that: ~~~ ruby namespace :build task :win32 do dash_i = File.expand_path( File.join(File.dirname(FILE), ‘cross/lib/ruby/1.8/i386-mingw32/’) ) Dir.chdir(‘ext/nokogiri’) do ruby “ -I #{dash_i} extconf.rb” sh ‘make’ end end end ~~~ Once that is completed, it is time to package the gem. In order to do that, you need to generate your gemspec.

Step 4, generating the gemspec

I typically use Hoe for packaging my gems. Hoe makes generating my gemspecs pretty easy. One little problem though is that hoe makes assumptions for your gemspec based on the system you are currently running. Since we’re cross compiling, we need to muck with the gemspec in order to package our win32 gem.

To modify the gemspec, what I do is assign the new Hoe object to a constant like so: ~~~ ruby HOE = Hoe.new(‘nokogiri’, Nokogiri::VERSION) do |p| p.developer(‘Aaron Patterson’, ‘aaronp@rubyforge.org’) p.developer(‘Mike Dalessio’, ‘mike.dalessio@gmail.com’) p.clean_globs = [ ‘ext/nokogiri/Makefile’, ‘ext/nokogiri/*.{o,so,bundle,a,log,dll}’, ‘ext/nokogiri/conftest.dSYM’, GENERATED_PARSER, GENERATED_TOKENIZER, ‘cross’, ] p.spec_extras = { :extensions => [“Rakefile”] } end ~~~ Then when I’m building my win32 gemspec, I modify the gemspec with win32 specific bits and write out the gemspec. This task modifies the gemspec file list to include any binary files such as dll’s and so files that I’ve built, assigns the platform to mswin32, and tells the gemspec that there are no extensions to be built: ~~~ ruby namespace :gem do namespace :win32 do task :spec => [‘build:win32’] do File.open(“#{HOE.name}.gemspec”, ‘w’) do |f| HOE.spec.files += Dir[‘ext/nokogiri/**.{dll,so}’] HOE.spec.platform = ‘x86-mswin32-60’ HOE.spec.extensions = [] f.write(HOE.spec.to_ruby) end end end end ~~~ We have to modify the file list and remove any extension building tasks because the gem is going to be shipped with the pre-built windows binaries. Setting the platform to that hardcoded string is a total hack, but I couldn’t figure out a different way. If you were building this spec on windows, you should use “Gem::Platform::CURRENT” instead of that string. After executing this task, you should end up with a file named “packagename.gemspec”. Just run “gem build packagename.gemspec”, and you’ll have your win32 gem, completely windows free!

Final Notes

Unfortunately just because it compiled, doesn’t mean it will run. My workflow for testing was to package the gem, transfer it to a windows machine, run “gem unpack” on the gem. After unpacking the gem, I could go in to the directory and run my tests. Once I was satisfied that all tests passed, I would release the gem.

One final thing…. Nokogiri ships with the libxml and libxslt dll files. In order to get those files to be found with dlopen (or whatever it is that windows uses), they must be in your PATH. Yes. Your PATH. So Nokogiri changes the environment’s PATH to include the directory where the DLL’s are located. You can see the hot PATH manipulation code here.

If you want to see all of the uncensored nitty gritty of the cross compilation action, check out the Nokogiri Rakefile located here.

Good luck, and don’t forget about those windows people.