Getting started

I use a Nesco American Harvest dehydrator. I found mine on sale for about $50, but it looks like the going rate on Amazon is about $65.

I like this dryer because the fan is on top, it’s easy to clean, and you can buy and use extra trays. I think it goes up to 10 or 15 trays, but I only have 8.

Step 1: Remove the top nob

In order to disconnect the heating element, we must remove the plastic bar that goes across the top of the fan. The first step is to remove the knob:

Just pull the knob straight up, and it should come off:

Step 2: Disconnect the bar

Flip the lid over. There should be four screws outside the ring. Remove just those four screws:

Flip the lid back over, and you should be able to remove the center plastic bar with the Nesco logo on it:

Step 3: Disconnect the heater

These two leads go to the potentiometer that controls the heater:

Disconnect one. The circuit will open, and the heater disabled:

I taped mine down with electrical tape:

That’s it!

Just follow the steps backwards to reassemble your dryer. The electrically inclined may want to enhance their dryer with a switch. I (on the other hand) am lazy. I bought a second dryer so that I could make jerky and do normal food dehydrating at the same time.

You can see more pictures of the process here. I plan to write a follow up post where I actually make jerky. Yum!

Lately I’ve been working on adding DTrace probes to Ruby 2.0. I am interested in using them to profile and improve rails startup time. Today I want to look at building a profiler with DTrace, finding slow method calls, and making some incremental improvements.

The examples I’m going to show will only work on systems where DTrace is available, and with either stock Apple MRI 1.8.7, or my special fork of Ruby. Likely you don’t have my fork, so if you’d like to play along, you’ll have to use Apple’s Ruby 1.8.7. I’ve published my branch here, but I will be adding and moving probes, so please don’t depend on it. Eventually it will be merged to ruby trunk (hopefully).

All of these examples will be executing in the context of a Rails application built from Rails master.

What to profile?

It’s easy to say that application boot time is not important. When we start our applications in production, they are only booted once, so the cost is only paid once. But as an application developer, the “boot once” argument doesn’t hold water. Every time we run a rake task, we boot the app. Every time we run tests, we boot the app. Every time we run migrations, we boot the app. Every time we restart our webserver, we boot the app. In the context of application development, class reloading helps to some extent, but not when someone runs rake test.

There are good efforts surrounding the improvement of Rails test speeds, but I would like to suggest that these are just working around a larger problem: booting Rails is slow. If we can make Rails boot times fast, I think it would eliminate the need for testing outside the rails context.

So where do we start? Typically, we begin at the beginning. Today we’ll start profiling the rake environment command. This task simply requires all of the files for your application, readying everything for execution against a Rails context. Many things can impact the startup time of our application: file parse time, method call time, GC, etc. Today we’re going to focus on slow methods, and using DTrace to find those methods.

Viewing method calls

The DTrace probes in Ruby allow us to detect when method calls are made. We can tell when Ruby enters a method by watching the function-entry probe. This probe is given:

• recipient class name
• method name
• file name and line number of the call site

Here is an example DTrace script that prints out all of this information each time a function-entry probe is activated:

ruby*:::function-entry
{
  printf("-> %s::%s (%s:%d)\n", copyinstr(arg0), copyinstr(arg1), copyinstr(arg2), arg3);
}

Save this to a file, and run the file along with rake environment like so:

$ sudo dtrace -q -s calls.d -c'rake environment'

You’ll see all of the method call entries, and what file and line made the call. Here is a sample of the output:

-> Module::local_constant_names (/Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb:85)
-> Module::local_constants (/Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb:78)
-> Module::constants (/Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb:79)
-> Array::map (/Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb:86)
-> Symbol::to_s (/Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb:86)

Counting method calls

We were able to see all of the method calls, but this doesn’t give us a good understanding of where the program is spending time. Instead of just viewing method calls, let’s use the aggregation functionality in DTrace to count the number of times a method is called. To do this, we use @ variables in DTrace. @ variables are known as “aggregation variables” and let us, well, aggregate things.

Aggregate variables are associative arrays that allow arbitrary keys. We can modify our DTrace program to aggregate the count of the recipient class name and method name pairs like so:

ruby*:::function-entry
{
  @[copyinstr(arg0), copyinstr(arg1)] = count();
}

When the program finishes executing, DTrace will automatically output the aggregate variable. Running the program yields this output (I’ve truncated the output to the last 10 lines or so):

$ sudo dtrace -q -s calls.d -c'rake environment'

  Class                 core#define_method           6332
  Symbol                ==                           6612
  Module                method_added                 9329
  Kernel                hash                         9437
  Array                 hash                        11966
  Array                 each                        13101
  Fixnum                ^                           15540
  Module                ===                         20267
  String                hash                        37878
  Symbol                to_s                       214691
$

Great! We can see that when rake environment is run, we’re calling Symbol#to_s over 200,000 times. Certainly this will put a burden on the Garbage Collector. If we change our program to aggregate by class, method, and call site, we can find out who is making so many calls. Let’s modify the program to add file and line number information:

ruby*:::function-entry
{
  @[copyinstr(arg0), copyinstr(arg1), copyinstr(arg2), arg3] = count();
}

Rerun the program, and we’ll see output like this (again I’ve truncated the output):

$ sudo dtrace -q -s calls.d -c'rake environment'

  Gem::Dependency        requirement           /Users/aaron/.local/lib/ruby/1.9.1/rubygems/dependency.rb               98             4641
  Array                  hash                  /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb             1308             4697
  Bundler::DepProxy      method_missing        /Users/aaron/.local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/rubygems_ext.rb              194             4809
  BasicObject            ==                    /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb             1334             5610
  Kernel                 ===                   /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb             1334             5610
  Module                 ===                   /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb             1334             5940
  Array                  hash                  /Users/aaron/.local/lib/ruby/1.9.1/rubygems/requirement.rb              135             6001
  Fixnum                 ^                     /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb             1308            11078
  String                 hash                  /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb             1308            31209
  Symbol                 to_s                  /Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb               86           208900
$

This shows us that introspection.rb line 86 is calling Symbol#to_s over 208k times. If we take a look at that method, we’ll see that on 1.9 it will call Symbol#to_s many times:

def local_constants #:nodoc:
  constants(false)
end

# Returns the names of the constants defined locally rather than the
# constants themselves. See <tt>local_constants</tt>;.
def local_constant_names
  local_constants.map { |c| c.to_s }
end

Knowing the number of times a method was called and who called it is pretty good. But we can’t tell if that’s why our program is slow. It could be that the method is very fast, so calling it many times doesn’t matter. Certainly we need to keep an eye on this method, but let’s try to find out where this program is spending the most time.

Sampling method calls

It would be best if we could time how long ruby spends in any particular method. But in order to do that, we would have to keep track of the Ruby stack. DTrace doesn’t provide us with a good way to access the Ruby stack. We could write our own program to manually keep track, but there are problems. For example, if something raises an exception, how far up the stack did it go? Which method caught the exception? What about catch / throw? It’s Friday and I don’t feel like figuring out all the edge cases for dealing with that.

Rather than keeping track of the stack, we can use a timer probe in DTrace. We will set up two probes. One probe will record function entries in a global variable, the other probe will run at fixed intervals and record the last function that was called. This approach assumes that the more time we spend in any particular function, the more likely we’ll see that function recorded. Every time a method is entered, this DTrace script will record the method call. Then we have a timer that runs at 5000hz taking a sample of the last method entered.

ruby*:::function-entry
{
  k = copyinstr(arg0);
  m = copyinstr(arg1);
}

tick-5000hz
/k != 0/
{
  @[k, m] = count();
}

Running this script against rake environment yields these results:

$ sudo dtrace -q -s calls.d -c'rake environment'

  Module            method_added               329
  Fixnum            ^                          330
  Mutex             unlock                     355
  String            hash                       477
  Array             -                          671
  Module            constants                  779
  Kernel            require                   2262
  IO                set_encoding              3713
  Symbol            to_s                      4095
  Kernel            `                         4122
$

The number on the right is the number of times the timer executed while ruby was executing the method on the left. Symbol#to_s is definitely in our top 10, but it seems we’re actually spending more time in Kernel#`. Let’s find all places where Kernel#` is called but using DTrace. We’ll use predicates to ensure our probe only fires when that method is called. Predicates are declared below the probe name and above the curly brace within slashes. Our probe body will simply aggregate based on the file where the call happened:

ruby*:::function-entry
/copyinstr(arg0) == "Kernel" && copyinstr(arg1) == "`"/
{
  @[copyinstr(arg2)] = count();
}

Execute this script:

$ sudo dtrace -q -s calls.d -c'rake environment'

  /Users/aaron/.local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler.rb                1
  /Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/coffee-rails-2120408a4063/coffee-rails.gemspec                3
  /Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/sass-rails-3c24e4fd8dd3/sass-rails.gemspec                3
  /Users/aaron/.local/lib/ruby/gems/1.9.1/gems/execjs-1.2.9/lib/execjs/external_runtime.rb                5
$

On the right side, we have the number of calls made to Kernel#`, and on the left we have the file where that call was made. We can see that running rake environment will create 12 subshells. Let’s see how much time we’re spending by creating new subshells:

ruby*:::function-entry
/copyinstr(arg0) == "Kernel" && copyinstr(arg1) == "`"/
{
  self->time = timestamp;
}

ruby*:::function-return
/copyinstr(arg0) == "Kernel" && copyinstr(arg1) == "`"/
{
  this->took = (timestamp - self->time) / 1000000;

  @[copyinstr(arg2)] = sum(this->took);
  @total = sum(this->took);
}

END
{
  printf("Time per file:\n");
  printa(@);
  printf("\nTOTAL: ");
  printa(@total);
}

Here we keep track of the total time spent in Kernel#` in the @total variable, and the aggregate per file in the @ variable. The report lists the amount of time in milliseconds that we’re spending. If we run this along with the time command, we can get a rough idea of what percentage we’re spending in Kernel#`:

$ time sudo dtrace -q -s calls.d -c'rake environment'
Time per file:

  /Users/aaron/.local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler.rb  15
  /Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/coffee-rails-2120408a4063/coffee-rails.gemspec  246
  /Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/sass-rails-3c24e4fd8dd3/sass-rails.gemspec  267
  /Users/aaron/.local/lib/ruby/gems/1.9.1/gems/execjs-1.2.9/lib/execjs/external_runtime.rb  398

TOTAL:
              926

real  0m7.942s
user  0m4.052s
sys   0m2.909s
$

With DTrace enabled, the total run time took about 8000ms and of that time 926ms was spent inside Kernel#`. So let’s say about 11% of our time running rake environment is spent in subshells.

Removing the Subshells

First we need to figure out where these subshells are being created. Since we’re studying DTrace, let’s write a script to aggregate call sites of Kernel#`:

ruby*:::function-entry
/copyinstr(arg0) == "Kernel" && copyinstr(arg1) == "`"/
{
  @[copyinstr(arg2), arg3] = count();
}

Running this, we can find the locations of our subshells:

/Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/coffee-rails-2120408a4063/coffee-rails.gemspec       19        1
/Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/coffee-rails-2120408a4063/coffee-rails.gemspec       20        1
/Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/coffee-rails-2120408a4063/coffee-rails.gemspec       21        1
/Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/sass-rails-3c24e4fd8dd3/sass-rails.gemspec       22        1
/Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/sass-rails-3c24e4fd8dd3/sass-rails.gemspec       23        1
/Users/aaron/.local/lib/ruby/gems/1.9.1/bundler/gems/sass-rails-3c24e4fd8dd3/sass-rails.gemspec       24        1
/Users/aaron/.local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler.rb      209        1
/Users/aaron/.local/lib/ruby/gems/1.9.1/gems/execjs-1.2.9/lib/execjs/external_runtime.rb      150        5

We’ll tackle fixing the gemspecs first because they are most boring to fix.

Fixing the gemspecs

Each gemspec shells out and runs git in order to populate various parts of the gemspec. Here are the relevant lines from one of the gemspecs:

s.files         = `git ls-files`.split("\n")
s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }

To fix this, we’re going to generate the gemspec in advance, then check in the generated gemspec. The downside of this solution is that whenever anybody adds or removes a file, the gemspec must be regenerated. The upside to this solution is that you see exactly what files are packaged in your gemspec. A more optimal solution would be to change bundler such that when a git repository is checked out, a gemspec for that repo is automatically built. If you’re looking for an opportunity to contribute to open source, this is it!

The gemspecs are now generated in advance. Next we should look at fixing subshells Bundler.

Fixing Bundler

In bundler.rb around line 209, we find that Bundler is trying to locate the sudo executable:

def requires_sudo?
  return @requires_sudo if defined?(@checked_for_sudo) && @checked_for_sudo

  path = bundle_path
  path = path.parent until path.exist?
  sudo_present = !(`which sudo` rescue '').empty?

  @checked_for_sudo = true
  @requires_sudo = settings.allow_sudo? && !File.writable?(path) && sudo_present
end

Let’s refactor this to use a which method that searches the PATH environment variable:

def which(cmd)
  if File.executable? cmd
    cmd
  else
    path = ENV['PATH'].split(File::PATH_SEPARATOR).find { |path|
      File.executable? File.join(path, cmd)
    }
    path && File.expand_path(cmd, path)
  end
end

def requires_sudo?
  # ...
  sudo_present = which "sudo"
  # ...
end

Our which implementation isn’t 100% perfect, (it can’t handle input like “./sudo” correctly) but it is good enough to find the sudo executable on a person’s system. Benchmarks reveal that the implementation that doesn’t shell out is over 10x faster than the version that does:

require 'rubygems'
require 'benchmark/ips'

def sand_which cmd
  `which #{cmd}`
end

def blair_which cmd
  if File.executable? cmd
    cmd
  else
    path = ENV['PATH'].split(File::PATH_SEPARATOR).find { |path|
      File.executable? File.join(path, cmd)
    }
    path && File.expand_path(cmd, path)
  end
end

Benchmark.ips do |x|
  x.report("sand") { sand_which 'sudo' }
  x.report("blair") { blair_which 'sudo' }
end

$ ruby lol.rb
                sand      251.0 (±4.8%) i/s -       1265 in   5.051489s (cycle=23)
               blair     3222.0 (±8.2%) i/s -      16218 in   5.076616s (cycle=306)
$

Now that we have bundler fixed, let’s take a look at ExecJS.

Fixing ExecJS

Our last place that shells out is inside ExecJS. The DTrace probes specifically list execjs/external_runtime.rb line 150. If we look there, we’ll find another implementation of which:

def which(command)
  Array(command).each do |name|
    name, args = name.split(/\s+/, 2)
    result = if ExecJS.windows?
      `#{shell_escape("#{ExecJS.root}/support/which.bat", name)}`
    else
      `#{shell_escape('command', '-v', name)} 2>/dev/null`
    end

    if path = result.strip.split("\n").first
      return args ? "#{path} #{args}" : path
    end
  end
  nil
end

When fixing performance problems, it’s helpful to understand the task the code is trying to accomplish, as well is how the method is called. Here is the stack trace for this method inside ExecJS:

lib/execjs/external_runtime.rb:134:in `locate_binary'
lib/execjs/external_runtime.rb:97:in `initialize'
lib/execjs/runtimes.rb:37:in `new'
lib/execjs/runtimes.rb:37:in `'
lib/execjs/runtimes.rb:9:in `'
lib/execjs/runtimes.rb:8:in `'
lib/execjs.rb:2:in `require'
lib/execjs.rb:2:in `'

There are a couple important things we can learn from this stack trace: 1) this method is called immediately when the file is required, and 2) it’s called from inside a constructor (on line 97). Let’s look at locate_binary, the initializer, and how the objects are constructed in runtimes.rb:

external_runtime.rb:

def initialize(options)
  @name        = options[:name]
  @command     = options[:command]
  @runner_path = options[:runner_path]
  @test_args   = options[:test_args]
  @test_match  = options[:test_match]
  @encoding    = options[:encoding]
  @binary      = locate_binary
end

protected
def locate_binary
  if binary = which(@command)
    if @test_args
      output = `#{shell_escape(binary, @test_args)} 2>&1`
      binary if output.match(@test_match)
    else
      binary
    end
  end
end

runtimes.rb:

module ExecJS
  module Runtimes
    # ...

    Node = ExternalRuntime.new(
      :name        => "Node.js (V8)",
      :command     => ["nodejs", "node"],
      :runner_path => ExecJS.root + "/support/node_runner.js",
      :encoding    => 'UTF-8'
    )

    # ...
  end
end

The initializer calls locate_binary which calls which which shells out. This is a great example of a hidden dependency in a constructor, our hidden dependency being the ability to shell out, and the availability of the command command. This isn’t to say that the command command will sometimes not be available, the problem is that people constructing this object will not know that it depends on these things. It’s “hidden” from the caller. The first thing we should do is stop the constructor from immediately shelling out. We’ll do this by initializing @binary to nil, adding a binary method, and changing all places that access the instance variable to use the method instead. Our diff looks like this:

diff --git a/lib/execjs/external_runtime.rb b/lib/execjs/external_runtime.rb
index a8180d1..65f46f9 100644
--- a/lib/execjs/external_runtime.rb
+++ b/lib/execjs/external_runtime.rb
@@ -94,7 +94,7 @@ module ExecJS
       @test_args   = options[:test_args]
       @test_match  = options[:test_match]
       @encoding    = options[:encoding]
-      @binary      = locate_binary
+      @binary      = nil
     end

     def exec(source)
@@ -113,16 +113,21 @@ module ExecJS

     def available?
       require "multi_json"
-      @binary ? true : false
+      binary ? true : false
     end

+    private
+      def binary
+        @binary ||= locate_binary
+      end
+
     protected
       def runner_source
         @runner_source ||= IO.read(@runner_path)
       end

       def exec_runtime(filename)
-        output = sh("#{shell_escape(*(@binary.split(' ') << filename))} 2>&1")
+        output = sh("#{shell_escape(*(binary.split(' ') << filename))} 2>&1")
         if $?.success?
           output
         else

Now the object will only shell out if the value is absolutely necessary. This technique is called “extracting a getter”, and you can read more about it in Michael Feathers’s book Working Effectively with Legacy Code (IMO this book is a mandatory read for any software engineer). Now we’re lazily shelling out, but if we look at the impact, we’ve gone from ExecJS creating 5 subshells down to 3 subshells:

$ sudo dtrace -q -s calls.d -c'rake environment'
/Users/aaron/.local/lib/ruby/gems/1.9.1/gems/execjs-1.2.9/lib/execjs/external_runtime.rb      155      3
$

The target is 0 subshells, so we should continue. This code is trying to locate executables on the file system, similar to what bundler was doing. We can change the which method in ExecJS to be a Ruby implementation similar to the implementation we put in bundler:

diff --git a/lib/execjs/external_runtime.rb b/lib/execjs/external_runtime.rb
index 65f46f9..88a08ce 100644
--- a/lib/execjs/external_runtime.rb
+++ b/lib/execjs/external_runtime.rb
@@ -121,6 +121,22 @@ module ExecJS
         @binary ||= locate_binary
       end

+      def which_windows(name)
+        result = `#{shell_escape("#{ExecJS.root}/support/which.bat", name)}`
+        result.strip.split("\n").first
+      end
+
+      def which_unix(name)
+        if File.executable? cmd
+          cmd
+        else
+          path = ENV['PATH'].split(File::PATH_SEPARATOR).find { |path|
+            File.executable? File.join(path, cmd)
+          }
+          path && File.expand_path(cmd, path)
+        end
+      end
+
     protected
       def runner_source
         @runner_source ||= IO.read(@runner_path)
@@ -147,19 +163,14 @@ module ExecJS
       end

       def which(command)
-        Array(command).each do |name|
+        Array(command).find do |name|
           name, args = name.split(/\s+/, 2)
-          result = if ExecJS.windows?
-            `#{shell_escape("#{ExecJS.root}/support/which.bat", name)}`
-          else
-            `#{shell_escape('command', '-v', name)} 2>/dev/null`
-          end
+          path = ExecJS.windows? ? which_windows(name) : which_unix(name)

-          if path = result.strip.split("\n").first
-            return args ? "#{path} #{args}" : path
-          end
+          next unless path
+
+          args ? "#{path} #{args}" : path
         end
-        nil
       end

       if "".respond_to?(:force_encoding)

It isn’t clear what the utility of the Windows batch file is, so it’s been extracted to it’s own method in order to maintain the current functionality. After these patches (which I have submitted here), the number of subshells is now down to zero.

Total Impact

Before:

$ time rake environment

real    0m2.377s
user    0m2.028s
sys 0m0.320s

After:

$ time rake environment 

real    0m2.224s
user    0m1.946s
sys 0m0.260s

After eliminating subshells, we’ve saved 153 milliseconds for approximately 6% savings. Not as much as the 11% we were hoping for, but still greater than 0. If we re-run our sampling profiler, we’ll find no more references to Kernel#`:

$ sudo dtrace -q -s calls.d -c'rake environment'

  Module           ===                306
  Mutex            unlock             324
  Fixnum           ^                  353
  Module           method_added       368
  Array            -                  509
  String           hash               603
  Module           constants          859
  Kernel           require           2292
  IO               set_encoding      3371
  Symbol           to_s              4561

Conclusion

Removing subshells from the Rails startup process wasn’t a silver bullet. From my experience in doing performance work, there is rarely a silver bullet that will fix performance problems. That said, even 2.2 seconds is still too slow for Rails startup time. If we collate the sampling profiler by file and method call, I think we can get a more clear picture about our runtime cost:

$ sudo dtrace -q -s calls.d -c'rake environment'

  Array            hash                  /Users/aaron/.local/lib/ruby/1.9.1/rubygems/requirement.rb              255
  Mutex            unlock                                                              365
  Fixnum           ^                     /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb              493
  Array            -                     /Users/aaron/git/rails/activesupport/lib/active_support/dependencies.rb              507
  IO               set_encoding          /Users/aaron/.local/lib/ruby/1.9.1/rubygems/custom_require.rb              636
  Module           constants             /Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb              841
  IO               set_encoding          /Users/aaron/git/rails/activesupport/lib/active_support/dependencies.rb             1098
  String           hash                  /Users/aaron/.local/lib/ruby/1.9.1/rubygems/specification.rb             1188
  Kernel           require               /Users/aaron/git/rails/activesupport/lib/active_support/dependencies.rb             1253
  Symbol           to_s                  /Users/aaron/git/rails/activesupport/lib/active_support/core_ext/module/introspection.rb             8981

Of the top ten slow methods, five of them are inside Rails. These particular methods are all associated with the way that Rails handles missing constants and code reloading. The total amount of time spent dealing with code reloading is likely far greater than the subshelling issue we fixed. I knew this before writing this article, but tackling the reloading code in Rails is much longer than a single blog post. The goal of this blog post was to show that:

• Startup performance can be improved
• Even small changes can make an impact
• DTrace is pretty awesome

I’d like to write an article with a more comprehensive analysis of where Rails spends it’s time on boot (GC, file parsing, method calls), but I don’t have all the information I need yet. My next steps are to:

• Try and understand Rails code reloading better
• Add more GC related probes to Ruby
• Add source compile probes to ruby

Once I’ve made more progress, I’ll write another article.

<3<3<3<3

When you require a file, rubygems must find the file. If the file isn’t on the load path, it must find it in your installed gems. To do this, it loads all gemspecs and searches for the file. This means all gemspecs on your system are evaluated, so the more gems you have, the longer it takes. You can read more in the rubygems comments.

The second problem is when bundler comes in to the picture. When rubygems loads the gemspec files, it turns them in to Gem::Specification objects and caches them. Sometimes bundler must modify your GEM_HOME and GEM_PATH environment variables. These environment variables tell rubygems where to locate gems on your file system. Unfortunately, if these variables are modified after the specification cache is built, that means the cache must be cleared. Until recently, bundler always cleared the rubygems cache. Since rubygems can load gemspecs before bundler gets activated (or may even while bundler is being activated!), this means it’s not unusual for all gemspecs on your system to be evaluated twice. Earlier today, I submitted a patch that will only clear the cache if the environment variables are modified. Hopefully this will mean fewer gemspec evals in the next version of bundler.

To help reduce boot time, just remember:

• Use gemsets (if you’re on RVM)
• Clean unused gems often

To clean up your gems, just use the command gem cleanup.

Hope that helps!

See the full announcement here!

I’m going to get this over. To me, this is like removing a bandage, I must do it all at once. I’ve increased the Love Bucks, so I’ll list the winners along with their prizes:

• First place of 500 Love Bucks: mnutt
• Second place of 400 Love Bucks: fredwu
• Third place of 300 Love Bucks: WanderingMatt
• Tied for third place with 300 Love Bucks: ejc’s matzilla!

Winners, please send me your paypal address and I’ll send you the Love Bucks.

Okay, you can stop reading now

No, seriously. Stop reading. The rest of this post is actually for myself. I need to record this so that I can remind my future self. This contest turned out to be more than I had expected. Every entry was above and beyond anything I could have hoped or wished for.

However, there is a downside for me. I do not want to choose. Each entry was amazing. If I randomly came across rubycommitters.org with any of the designs that had been submitted, I would think “wow, that’s a great website”. My training through school was very analytical. I remember when I realized that judgement of work in English / Art classes was more arbitrary compared to that of my Science and Math classes. It was a turning point in my life. I wanted to focus on Sciences because the path seemed more clear.

Now, I’ve forced myself to judge something that is very arbitrary. Yes, there are tangible goals to this project. Things like “which design communicates the information most effectively?”, “which will be easiest to maintain?”. But even these questions are difficult for me to answer. Communicating information is important, but I also want to express my feelings. Every one of the designs submitted shows my feelings in some way or another. I don’t know how to measure my feelings, so I feel like I can’t accurately measure the submissions. How can I choose which of my feelings is correct if there is no measurement?

This being said, I will probably not hold a contest again. I like sticking to code. Code is something I know how to measure.

Why are you still reading?

I started this blog for entertaining my friends and myself. I’m glad I can use my hobby and work to teach, help, and entertain others. I feel like many people I don’t know read this, so sharing my feelings on it is now difficult because I am actually a shy person. I wish I could repay the kindness of the people that submitted designs to this contest. I feel like Love Bucks isn’t enough.

Hey folks, somehow I made a mistake and one entry was omitted from the index. Please take a look at static mirror, and vote for it here if you like it! Apologies to the people that submitted this entry. I didn’t mean to miss this!

Non Update:

The deadline for entry was last night. I’ve compiled a static mirror of all of the entries here.

I am humbled, and incredibly impressed by the number and quality of all of the entries. I can’t thank everyone enough for doing this. The quality of the results is so good, that I would really appreciate feedback from the community.

Please check out the entries list. Carefully review all entries. Then, leave feedback by either voting up the entry on github, or leaving a comment on my blog. I will take all feedback in to consideration tomorrow when I decide the winner.

Remember that when you judge these designs, please judge just the design. If links are broken, or encoding looks wrong, or something is a little bit off, it’s probably my fault. Pick the designs you like the most, and we’ll polish them up after declaring a winner.

Please tell me about all of the designs you like. If you like 5 of them, let me know. It’s also helpful to know why you like those designs.

As I said, I’ll declare the winner tomorrow before 23:59:59 PST.

Again, I really can’t thank everyone enough for doing this. The results are so much more than I could have hoped for, or expected. You have all outdone yourselves! No amount of Love Bucks can repay for the awesomeness I’ve witnessed, the best I can do is keep fixing bugs on Ruby, Rails, and any other project I get my hands on!

Thank you!

Alright, now that the formalities are out of the way, LET’S GET DOWN TO BUSINESS! Some of you may or may not know, I joined the Ruby core team in October 2009. I am very proud to be a member of ruby-core. However, I have noticed that the Ruby core team does not have an awesome website like the Rails core team.

To rectify this situation, I registered a domain: rubycommitters.org. Unfortunately, my design skills are… sub-par. I would like to fist fix this by having a contest. I would like you to code the design for rubycommitters.org.

How do I enter?

Just fork the rubycommitters.org project on github, follow the instructions in the README, and make the site look good! When you’re done, send me a pull request.

How many times can I enter?

As many as you want. However, you can only win one place.

When is the due date?

You must send me the pull request by January 19th, 23:59:59 PST.

When will winners be announced?

I will announce winners by January 21st, 23:59:59 PST. Once I’ve decided, I’ll email the winners to get their PayPal information and transfer the money Love Bucks.

What are the prizes?

My employer pays me in Tenderlove Cash (from here on referred to as “Love Bucks”), so the prize will be in Love Bucks. Fortunately for all of us, Love Bucks exchange with the US Dollar at a 1:1 ratio. So I will send you your prize via PayPal in the form of US Dollars.

First Place: 200 300 Love Bucks (in the form of US Dollars)
Second Place: 100 Love Bucks (in the form of US Dollars)

All entrants will win a hug from me that is redeemable next time we meet each other!

How will entries be judged?

By me, however I want. I’ll probably ask the intertubes for help, but I have the final say.

Why is the prize so low?

These Love Bucks are coming out of my own wallet! Give me a break!

Conclusion

Since I announced I needed help last weekend, already 9 people have started working, so you’d better get cracking!

I am proud to be a member of the Ruby core team, I’m proud to be involved in the Rails development team, and most of all I’m proud to be a member of the best development community in the world. Thanks to everyone that makes being a member of the Ruby community so awesome!

EDIT!!!!

@mariozig has graciously offered to up the ante. He has donated 100 Love Bucks to the first place winner! That’s a total of 300 Love Bucks for the person who wins first place! Awesome!!!

In Part 2, we’ll modify our extconf.rb file to find important files in libstree, then we’ll create a Ruby class that is backed by a C structure.

The final code associated with this part of of my Writing Ruby C Extensions series can be found here.

Using mkmf to find libraries

As I mentioned in the previous post, extconf.rb is used when installing a native gem to locate libraries, header files, and test various things about the target system before installing. We’re going to teach our extconf.rb file to locate the libstree dynamic library along with the header files. We’re also going to allow people to tell the gem where to find libstree, and set up our extconf.rb with some sensible defaults.

mkmf configuration with dir_config

The first thing we’ll do is tell mkmf where to look for libstree files by default. We do this using the dir_config method. dir_config takes three arguments:

• An arbitrary string, but usually the library name (like “stree”)
• A list of paths to search for header files
• A list of paths to search for library files

The dir_config method also allows users installing our gem to configure where mkmf should look for various files. Let’s take a look at our call to dir_config and talk about what it does:

LIBDIR      = Config::CONFIG['libdir']
INCLUDEDIR  = Config::CONFIG['includedir']

HEADER_DIRS = [
  # First search /opt/local for macports
  '/opt/local/include',

  # Then search /usr/local for people that installed from source
  '/usr/local/include',

  # Check the ruby install locations
  INCLUDEDIR,

  # Finally fall back to /usr
  '/usr/include',
]

LIB_DIRS = [
  # First search /opt/local for macports
  '/opt/local/lib',

  # Then search /usr/local for people that installed from source
  '/usr/local/lib',

  # Check the ruby install locations
  LIBDIR,

  # Finally fall back to /usr
  '/usr/lib',
]

dir_config('stree', HEADER_DIRS, LIB_DIRS)

First, this code builds a two lists of sensible defaults for finding header files and library files. The HEADER_FILES and LIB_DIRS constants contain lists of common places to find libraries. These settings will be nice for our users because if they have libstree installed in /opt/local/ or /usr/local/ it will find the library without any user intervention.

Finally, we call dir_config with the string “stree” and two lists. This call to dir_config only configures mkmf with directories to search. We actually haven’t done any searching at this point. The dir_config call also allows users to configure the gem on installation. The call sets up the following flags for our user to configure:

• --with-stree-dir
• --with-stree-include
• --with-stree-lib

Finding headers and libraries

Now that we’ve configured mkmf with where we can find libraries and headers, we need to search for required header files and libraries. We’ll do that with two functions: find_header and find_library.

We need to find the stree/lst_string.h header file, so we’ll just supply that to the find_header method like so:

unless find_header('stree/lst_string.h')
  abort "libstree is missing.  please install libstree"
end

This code will tell mkmf to find the header file we need. If the header file can’t be found, find_header will return false, and we can abort installation and provide some instructions. If the find_header method is a success, the directory where the header file was found will be added to the -I flags that get passed to your compiler.

Next, we need to find the libstree dynamic library. For this task, we’ll use the find_library function call:

unless find_library('stree', 'lst_stree_free')
  abort "libstree is missing.  please install libstree"
end

The find_library function takes two arguments. The first argument is the library that we need to link against. This string will be passed to the -l flags. The second argument is a symbol we need to find in the library.

In this code example, mkmf will create a test C program that tries to link against stree and find the function lst_stree_free. If linking is successful, the path will be added to the -L flags provided to your compiler. If it fails, we abort installation and provide an error message.

Creating the Makefile

Just like the last article we still need the call to create_makefile in our extconf.rb:

create_makefile('stree/stree')

You can find the complete extconf.rb here.

Wrapping LST_String from libstree

libstree defines a String type structure. We’re going to define a class in Ruby to wrap up this string type structure. Eventually, we’ll have some Ruby code that looks like this:

string = STree::String.new 'foo'
assert_equal 3, string.length

In fact, since we’re doing TDD let’s start with a test for the length method. We’ll also add a test to ensure that objects other than String objects will raise a TypeError:

require 'stree'
require 'test/unit'

module STree
  class TestString < Test::Unit::TestCase
    def test_length
      string = STree::String.new 'foo'
      assert_equal 3, string.length
    end

    def test_type_error
      assert_raises(TypeError) do
        STree::String.new Object.new
      end
    end
  end
end

File structure

In my C projects, I like to make one C file per class. We have to make an entry point though, so we’ll keep stree.h and stree.c from our previous project. Then we’ll write stree_string.h and stree_string.c to keep our String class.

Library entry point

The entry point to our C code will be in stree.c. The stree.c file will initialize the String class. Here is the new stree.h file that includes libstree:

#ifndef RUBY_STREE
#define RUBY_STREE

#include <ruby.h>;
#include <stree/lst_string.h>;

#include <stree_string.h>;

extern VALUE mSTree;

#endif

We include header files from libstree, we include the header file for the string class, then we declare a global variable which will hold a reference to our Ruby “STree” module.

The new stree.c file looks like this:

#include <stree.h>

VALUE mSTree;

void Init_stree()
{
  mSTree = rb_define_module("STree");

  Init_stree_string();
}

When our library is required, Init_stree is called, then we’ll define the STree module (assigning it to the global module variable) and initialize our String class. Now we need to define Init_stree_string in stree_string.h and stree_string.c.

Defining the String class

First we’ll create the header file for our string class. We’ll only have one public function called Init_stree_string, so our header file will look like this:

#ifndef RUBY_STREE_STRING
#define RUBY_STREE_STRING

#include <stree.h>

void Init_stree_string();

#endif

We include the main stree.h header file, then define our public initialize function. Now we need to define the body of the Init_stree_string function in stree_string.c:

#include <stree_string.h>

void Init_stree_string()
{
  VALUE cSTreeString = rb_define_class_under(mSTree, "String", rb_cObject);
}

The rb_define_class_under function will define a class “String” in the module pointed to by mSTree with a parent class of Object. This C code is equivalent to the following Ruby code:

module STree
  class String
  end
end

At this point, you should be able to compile the project and run the tests. We haven’t defined any methods on the STree::String class in Ruby yet, but our project should compile, and the tests should execute. If you’re following along, you should see test output like this:

  1) Error:
test_length(STree::TestString):
ArgumentError: wrong number of arguments (1 for 0)
    ./test/test_stree_string.rb:7:in `initialize'
    ./test/test_stree_string.rb:7:in `new'
    ./test/test_stree_string.rb:7:in `test_length'

Allocating the String class

The first thing we’re going to do is teach Ruby how to allocate our String class. Ruby gives us a hook when the allocate method is called where we can allocate internal structures (we’re actually defining the allocate method on the STree::String class).

First, let’s modify the init function to tell ruby about our allocate function:

void Init_stree_string()
{
  VALUE cSTreeString = rb_define_class_under(mSTree, "String", rb_cObject);

  rb_define_alloc_func(cSTreeString, allocate);
}

rb_define_alloc_func tells Ruby to call a function pointer allocate when this class gets allocated. New we need to define our allocate function:

static VALUE allocate(VALUE klass)
{
  LST_String * string = malloc(sizeof(LST_String));

  return Data_Wrap_Struct(klass, NULL, deallocate, string);
}

In our allocate function, we allocate enough memory to hold an LST_String struct. Then we call Data_Wrap_Struct to return our actual Ruby object. Data_Wrap_Struct takes four arguments:

• The Ruby class we’re dealing with (in this case it’s cSTreeString
• A function pointer that is called when the object is marked
• A function pointer that is called with the object is freed
• A void pointer of the data we want to wrap

You’ll notice we’re referencing a function deallocate that isn’t defined yet. Let’s define that function now:

static void deallocate(void * string)
{
  lst_string_free((LST_String *)string);
}

The deallocate function is called with the pointer we passed to Data_Wrap_Struct, in this case an LST_String pointer. We’ll use the lst_string_free function from libstree to free our pointer.

Defining STree::String#initialize

Now we need to define the initialize method. This method will take one argument (a string), and we’ll populate the underlying LST_String struct with information from the Ruby string.

To define the initialize method, first we call rb_define_method:

void Init_stree_string()
{
  VALUE cSTreeString = rb_define_class_under(mSTree, "String", rb_cObject);

  rb_define_alloc_func(cSTreeString, allocate);
  rb_define_method(cSTreeString, "initialize", initialize, 1);
}

rb_define_method takes 4 arguments:

• The class on which we want to define a method
• The name of the method we’re defining
• A function pointer that will be called when our method is called
• The number of parameters passed to that function

Next we need to define our initialize C function:

static VALUE initialize(VALUE self, VALUE rb_string)
{
  LST_String * string;
  void * data;

  Check_Type(rb_string, T_STRING);

  Data_Get_Struct(self, LST_String, string);

  data = calloc(RSTRING_LEN(rb_string), sizeof(char));
  memcpy(data, StringValuePtr(rb_string), RSTRING_LEN(rb_string));

  lst_string_init(
      string,
      data,
      sizeof(char),
      RSTRING_LEN(rb_string));

  return self;
}

The initialize function has two parameters, the first is the instance of our STree::String object, the second is the single parameter for our method.

After declaring our variables we check the type of the required argument. Check_Type is a macro provided by Ruby to let us perform type checking on objects. We use this macro to ensure that the user passed us a Ruby string. If not, the Check_Type macro will automatically raise a type error.

Next we make a call to a macro provided by Ruby: Data_Get_Struct. Our LST_String pointer is stored inside the Ruby VALUE object, and Data_Get_Struct will extract our pointer. We give this macro the ruby object self, followed by the struct type we want to extract (LST_String), followed by the pointer where it will be assigned (string).

We need to copy the contents of the Ruby string to a buffer that our LST_String can keep. To do that, we use:

• calloc to allocate the memory
• RSTRING_LEN to get the number of bytes in our string
• memcpy to copy the memory contents
• StringValuePtr to get the underlying character pointer from Ruby

We give the data to libstree by calling lst_string_init, then finally return self.

At this point, we should have one passing test and one failing test:

  1) Error:
test_length(STree::TestString):
NoMethodError: undefined method `length' for #
    ./test/test_stree_string.rb:8:in `test_length'

Next we need to define the length method.

Defining STree::String#length

The hard part is over. Defining the length method should be much easier than the initialize method. Just like the initialize method, we need to call rb_define_method:

void Init_stree_string()
{
  VALUE cSTreeString = rb_define_class_under(mSTree, "String", rb_cObject);

  rb_define_alloc_func(cSTreeString, allocate);
  rb_define_method(cSTreeString, "initialize", initialize, 1);
  rb_define_method(cSTreeString, "length", length, 0);
}

This time, we’re defining a function length that takes 0 arguments. Now lets define the length C function:

static VALUE length(VALUE self)
{
  LST_String * string;

  Data_Get_Struct(self, LST_String, string);

  return INT2NUM(lst_string_get_length(string));
}

Just like the initialize function, we declare our variables, then unwrap our struct. We use the lst_string_get_length function from libstree to get the string length as an integer. Then we use a macro provided by Ruby, INT2NUM, that converts the integer to a Ruby Numeric object and return that object.

After we’ve defined this method, all of our tests should pass:

Loaded suite -e
Started
..
Finished in 0.000873 seconds.

2 tests, 2 assertions, 0 failures, 0 errors

Yay!

Conclusion

OMG! C CODE WRAPPED WITH RUBY!

We’ve scratched the surface for writing C extensions in Ruby. In this part, we:

• taught our system how to find the library we want to use
• (briefly) dealt with memory management of our objects
• defined modules and classes
• defined methods on our classes

You can grab the code for part 2 here.

Happy holidays to EVERYONE! I hope you liked Part 2 of Writing Ruby C Extensions!

<3<3<3<3<3<3<3<3 –tenderlove

What is the Ruby Advent Calendar?

Each day leading up to December 25th, one person posts an article to their blog and adds a link to their blog on the Advent Calendar. So, 25 blog posts total. The posts can be about anything related to Ruby.

In my opinion, the Ruby Advent Calendar is about encouraging people to blog about Ruby and help others participate in the Ruby community.

How I can participate?

In past years, the articles have mainly been in Japanese. This year, we’re trying to do an English language Ruby Advent Calendar.

@yhara_en was kind enough to add English instructions. I will add screen shots with notes here so that you can more easily participate.

Step 1: Get some Coffee

As I’ve said in previous blog posts, I love it when instructions tell you to get coffee because I always do. So here is your opportunity. Go get some coffee!

Step 2: Click a Button

First, go to this link. Then click the button that says “このエベントに参加登録する”:

Once you’ve clicked that button, you need to register.

Step 3: Register

The easiest way to register is with your Twitter account. Click the link that says “Twitterでログイン”:

You’ll be taken through the normal Twitter authorization path. That part is in English, so I’m not going to cover it.

Step 4: Edit account details

Enter your name and website and click save (the button that says “保存する”):

Step 5: Register for the Ruby Advent Calendar

Go back to the Ruby Advent Calendar and click the giant red button again:

This time it will ask you to enter a comment:

So that we can coordinate better, I just said which day I would like to participate.

Step 6: Publish a Blurgh Post

On your day, publish your blog post, then head back to the advent calendar, scroll to the bottom of the page, and link to your blog in the comments section:

That’s it!

Happy Holidays, and Happy Blurghing!!!

<3<3<3<3<3<3 –tenderlove

Anyway, let’s talk RDoc, iPad, and epub! I like documentation. I especially like consuming documentation. I thought it would be neat if I could read documentation on my iPad. As it turns out, getting RDoc documentation on your iPad isn’t that hard!

According to Wikipedia, iBooks is an EPUB reader. EPUB is a standard format for making books. The EPUB format is basically a zip file that contains a bunch of XHTML and XML documents. The XHTML documents are the “meat” of your book, where the XML documents tell the reader where to find everything, and the order in which to put things. RDoc already emits HTML, so our job is to make sure it emits XHTML along with the special XML files. How do we do that?

RDoc supports a plugin system where we can hook in and emit anything we want. To hook in to RDoc, we just add a special file to our gem (“lib/rdoc/discover.rb”), and register with the RDoc plugin system. So I wrote a gem called paddle that plugs in to RDoc, emits the documentation as XHTML along with the supporting XML files. It even comes with a nice Ruby logo! I encourage you to take a look at the source. The code is quite short, but could be refactored even smaller!

Using the Paddle

Creating your own books with Paddle is really easy. First, install paddle:

  $ sudo gem install paddle

Then find a project for which you want to create a book. For this example, I’ll generate a book for one of my gems called texticle. From the project root, use the rdoc command. Make sure to tell rdoc to use the “paddle” formatter, and supply a title (very important to supply a title!):

  $ cd git/texticle
  $ rdoc -f paddle -t 'Texticle Documentation' -o epub lib

Now there should be an “epub” directory that contains your book. But we’re not quite done yet. There is one more step. The book must be in a zipfile, and the zipfile requires a particular format. Let’s create the zipfile now using the “zip” command:

  $ cd epub
  $ zip -Xr9D texticle.epub mimetype *

You should end up with a file named “texticle.epub”. Just drag that file to iTunes, sync up your iPad, and boom!

Problems

I hacked this out in an evening, so there are a few problems. I’ll mention them here, just so you’re not surprised, and to give you ideas for patches to submit!

• Right now, the links don’t work:

I haven’t figured out why, but they don’t. That will come soon.

• The author field isn’t filled out in the book:

I need to teach RDoc to take more command line options so we can tell Paddle what to use for the book’s author field

• Only classes, modules, and the things they contain are documented:

Right now, your README file won’t show up in the book. That is just missing right now. It should be easy to add, I just haven’t done it.

A couple books to get you started

• Here’s the Rails Documentation
• Here’s the Nokogiri Documentation

THE END!

Thanks for reading! Have fun making books for your iPad, and don’t forget to send patches back to me!