Anyway, I want to talk about database connection management in ActiveRecord. I am not too pleased with its current state of affairs. I would like to describe how ActiveRecord connection management works today, how I think it should work, and steps towards fixing the current system.

TL;DR: database connection API in ActiveRecord should be more similar to File API

Thinking in terms of files

It’s convenient to think of our database connection as a file. Dealing with files is very common. When we work with files, the basic sequence goes something like this:

• Open the file
• Do some work on the file handle
• Close the file

We’re very used to doing these steps when dealing with files. Typically our code will look something like this:

File.open('somefile.txt', 'wb') do |fh| # Open the file
  fh.write "hello world"                # Do some work with the file
end                                     # Close file when block returns

We don’t want to share open files among threads because dealing with synchronization around reading and writing to the file is too difficult (and time consuming). So maybe we’ll store the handle in a thread local or something until we’re ready to close it.

Our basic requirements for dealing with a database connection are essentially the same as when dealing with files. We need to open our database connection, do some work with the connection (send and receive queries), and close the connection. We have these similarities, yet the API for dealing with database connections in ActiveRecord is vastly different. Let’s look at how each of these steps are performed in ActiveRecord today.

Opening a connection

Opening a connection to the database is very easy. First we configure ActiveRecord with the database specification, then we call connection to actually get back a database handle:

ActiveRecord::Base.establish_connection(
  :adapter  => "sqlite",
  :database => "path/to/dbfile")

connection_handle = ActiveRecord::Base.connection

The main difference between this API and the File API is that we’ve separated the connection specification from actually opening the connection. In the case of opening a file, we call open along with a “specification” which includes the file name and how we want to open it. In this case, we’ve separated the two; essentially storing the specification in a global place, then opening the connection later.

This leads to two questions:

1. Where is the specification stored?
2. When I call connection, what specification is used?

The answer to the first question can be found by reading the establish_connection method. Specifically if we look at line 63 we’ll find a clue. Since this method is a class method, the call to name returns the class name of the recipient. This name (along with our actual spec) is passed in to the connection handler object. If we jump through a few more layers of indirection, we’ll find that what we have is essentially a one to one mapping of class name to connection specification.

Armed with this information, we can tackle the second question. If we look at the implementation of connection, it calls retrieve_connection on itself, which calls retrieve_connection on the connection handler with itself. A few more method calls later, and we see that each ActiveRecord subclass walks up the inheritance tree looking for a connection:

def retrieve_connection_pool(klass)
  pool = @connection_pools[klass.name]
  return pool if pool
  return nil if ActiveRecord::Base == klass
  retrieve_connection_pool klass.superclass
end

If we read this code carefully, we’ll notice that not only are connection specifications mapped to classes so are database connections!

Why is this bad?

This behavior smells bad to me. The reason is because we’re tightly coupling classes along with database connections when really this relationship doesn’t need to exist.

How can it be improved?

If this tight coupling is removed, the complexity of ActiveRecord can be reduced and at the same time increasing the features available! The way we can reduce this coupling is by passing the connection specification to the method that actually opens the connection. Specifications can be stored on each class as a convenience, but nothing more.

What if opening a connection looked more like this?

spec = ActiveRecord::Base.specificiation
ActiveRecord::ConnectionPool.open(spec) do |conn|
  ...
end

We could maintain the current behavior by storing specifications on each class, but eliminate the coupling between connection and class. We would be able to delete all of the code that looks up connections by class hierarchy, and open the doors to having features like this:

spec = database_a
ActiveRecord::ConnectionPool.open(spec) do |conn|
  User.find_all
end

spec = database_b
ActiveRecord::ConnectionPool.open(spec) do |conn|
  User.find_all
end

Working with the connection

Working with our connection should remain the same. We have one place to retrieve our connection and work with it. Woo!

Dealing with thread safety

Sharing open file handles among threads probably isn’t a good idea and the same can be said about open database connections. So how does ActiveRecord keep connections localized to one thread? If we jump through many, many, method calls, we’ll find where the connection is actually checked out of the connection pool. It is here we see how thread safety is handled:

# Retrieve the connection associated with the current thread, or call
# #checkout to obtain one if necessary.
#
# #connection can be called any number of times; the connection is
# held in a hash keyed by the thread id.
def connection
  @reserved_connections[current_connection_id] ||= checkout
end

A hash is kept where the key is the current_connection_id. The implementation of current_connection_id looks up the current id. If the id isn’t set, it sets it to the object id of the current thread:

def current_connection_id #:nodoc:
  ActiveRecord::Base.connection_id ||= Thread.current.object_id
end

Next we look at the implementation of connection_id to find that it just gets and sets a thread local:

def connection_id
  Thread.current['ActiveRecord::Base.connection_id']
end

def connection_id=(connection_id)
  Thread.current['ActiveRecord::Base.connection_id'] = connection_id
end

These methods ensure that we have a one to one relationship of open connection and thread.

Closing the connection

Finally we reach our last step: closing the connection. How many of you have closed your connection to the database in ActiveRecord? My guess is that it’s very few. I think the reason people don’t typically close their connections with ActiveRecord is twofold. One, you don’t have to because it just does it for you, and two, the API to close a particular connection is pretty convoluted.

So how is the connection closed today? There are two ways, the easy way and the hard way.

The easy way

The easy way is good enough in a non-threaded application. A rack middleware clears out all of the connections at the end of the request. The source for clear_active_connections! is pretty simple. For each connection pool in the system (remember it’s one pool per AR class and connection spec), release that connection:

# Returns any connections in use by the current thread back to the pool,
# and also returns connections to the pool cached by threads that are no
# longer alive.
def clear_active_connections!
  @connection_pools.each_value {|pool| pool.release_connection }
end

Each pool releases the connection it has using the current_connection_id (which happens to be the current thread id):

# Signal that the thread is finished with the current connection.
# #release_connection releases the connection-thread association
# and returns the connection to the pool.
def release_connection(with_id = current_connection_id)
  conn = @reserved_connections.delete(with_id)
  checkin conn if conn
end

Not bad. But what if our system has multiple threads?

The hard way

Believe it or not, the connection pool in ActiveRecord will check in connections in the checkout method. Let me say that again: the checkout method checks in connections and checks out connections. If you’re not facepalming yet, let’s look at a small part of the checkout method:

@queue.wait(@timeout)

if(@checked_out.size < @connections.size)
  next
else
  clear_stale_cached_connections!
  if @size == @checked_out.size
    raise ConnectionTimeoutError, "could not obtain a database connection#{" within #{@timeout} seconds" if @timeout}. The max pool size is currently #{@size}; consider increasing it."
  end
end

This bit of the checkout method is not called unless our connection pool has become full. First we wait for other threads to check in their connection. While we’re waiting, if other threads checked in their connection, the first branch of the if statement executes, and a connection is returned. If no threads have checked in their connection, we call clear_stale_cached_connections!:

def clear_stale_cached_connections!
  keys = @reserved_connections.keys - Thread.list.find_all { |t|
    t.alive?
  }.map { |thread| thread.object_id }
  keys.each do |key|
    checkin @reserved_connections[key]
    @reserved_connections.delete(key)
  end
end

This method walks through every thread in your system, looking for connections that were allocated to threads that no longer exist. Then it checks in connections associated with those dead threads. Since there is really no easy way for users to check in their own connections, this is actually a common code path for systems that use threads.

Why is this bad?

It should be pretty clear why this behavior is bad. Walking through every thread in the system, and asking if it’s alive isn’t very cheap. Even worse is that we’re coupling ourselves to the threading system. We cannot change the connection pool to work with other concurrency solutions (like Fibers) because those solutions may not give us the introspection we need to perform this operation!

But really, this is treating a symptom. The real problem is that checking in connections is too difficult, so people don’t do it.

How can we fix this?

I think the best solution for this is to mimic the File API. If we do this, it will become natural for people dealing with the database connection to actually close the connection.

We should make ActiveRecord::Base.connection consult a thread local. That thread local is set in the rack middleware where the connection is opened. If someone creates a new thread, they must populate that thread local, and close the connection at the end of the thread.

Simplified, our middleware would become something like this:

class ConnectionManagement
  def call env
    spec       = ActiveRecord::Base.spec
    connection = ActiveRecord::ConnectionPool.open spec
    ActiveRecord::Base.connection = connection

    @app.call env

    connection.close
  end
end

When people create a new thread, it would look something like this:

Thread.new do
  spec = ActiveRecord::Base.spec
  ActiveRecord::ConnectionPool.open(spec) do |connection|
    ActiveRecord::Base.connection = connection

    # do some stuff
  end
end

What does this buy us?

This buys us two important things: simple connection pool management, and freedom of choice on our concurrency model.

omg the end.

I hope I’ve convinced you that by simply learning to treat our database connection like a file, we can reduce code complexity and at the same time increase the features available. I think I can add this feature to Rails 3.2 and mostly maintain backwards compatibility. I think we can keep 100% backwards compatibility if we add some sort of flag like config.i_suck_and_will_not_close_my_database_connections = true or, config.my_app_is_awesome = true.

Anyway, I’m totally sick and I’ll stop blllluuurrrrggghhhing now.

<3 <3 <3 <3 <3

The benchmark program

Here is the program I was using for benchmarking. It’s slightly modified from the original program that @paul so kindly submitted:

require 'active_record'
require 'benchmark'

p ActiveRecord::VERSION::STRING

ActiveRecord::Base.establish_connection(
  :adapter  => "sqlite3",
  :database => ":memory:"
)

ActiveRecord::Base.connection.execute("CREATE TABLE active_record_models (id INTEGER UNIQUE, title STRING, text STRING)")

class ActiveRecordModel < ActiveRecord::Base; end
ActiveRecordModel.new

N = 100_000
Benchmark.bm { |x| x.report('new') { N.times { ActiveRecordModel.new } } }

I could tell from perftools.rb output we were calling new methods, and those new methods took plenty of time.

Here is the call graph for this benchmark on Rails 3.0:

Compared to the call graph on Rails 3.1:

After some work, I was able to remove the method calls to scope when allocating new AR objects, but the benchmark still showed 3.1 to be slower than 3.0. I noticed we were spending more time in the Garbage Collector on 3.1 than on 3.0. This lead me to believe that we were creating more objects in the newer version.

Counting Object Allocations

Counting object allocations with Ruby 1.9 is actually pretty easy. We can call ObjectSpace.count_objects to get a list of the live objects in the system. That method returns a hash where the keys are the object type and the values are the number of that object in the system (there are other keys, be we don’t care about them for now).

Knowing this information, we can write a function that will calculate a rough difference in object allocations:

def allocate_count
  GC.disable
  before = ObjectSpace.count_objects
  yield
  after = ObjectSpace.count_objects
  after.each { |k,v| after[k] = v - before[k] }
  GC.enable
  after
end

p allocate_count { 100.times { {} } }

The first thing we do in this function is disable the garbage collector. It’s possible that the code we’re investigating could trigger GC, and that will impact our allocated object count.

Then we grab our current list of objects, yield to the code we want to profile, grab the list again, calculate our delta, enable GC and return!

If you run this code, you should see output similar to this (I’ve removed keys I don’t care about):

{ ... :T_HASH=>101, ... }

You can see from the output that we’ve allocated 101 hashes. 100 came from our 100.times call, and one came from calling the method on ObjectSpace.

Since inspecting our code can impact object allocations, these counts can’t be 100% accurate. Another annoying thing is that we can’t determine where our code is allocating these hashes. It’s easy enough to see the hash literals in this code, but imagine a project like rails. You may be able to find hash literals in your source, but determining which ones are being called is difficult.

After running this against the profile, I found that ActiveRecord in 3.1 was allocating 2x the number of hashes that 3.0 allocated. Now the problem is figuring out where these hashes were being allocated.

DTrace

DTrace gives us hooks in to our processes. Many of the hooks provided are for system calls, or other various functions. DTrace requires that we write probes in our code, and unfortunately Ruby does not have DTrace probes built in.

Adding DTrace to Ruby

To find these Hash allocations, I decided to add DTrace support. I usually run Ruby built from Ruby trunk, so running a modified Ruby seemed acceptible. It turns out adding DTrace support is pretty easy.

First I added a probe definition file. This file describes the probes that I want to add to Ruby. It contains the probe names and the signatures of the probes:

provider ruby {
  probe hash__alloc(const char *, int);
};

This definition says we’re declaring a probe called hash-alloc and it will take a string (the file name) and an int (the line number).

Next I used this definition file to generate a header file that ruby would use:

$ dtrace -o probes.h -h -s probes.d

Finally, I modified hash.c to trigger the probe:

diff --git a/hash.c b/hash.c
index b49aff8..c40d94d 100644
--- a/hash.c
+++ b/hash.c
@@ -15,6 +15,7 @@
 #include "ruby/st.h"
 #include "ruby/util.h"
 #include "ruby/encoding.h"
+#include "probes.h"
 #include <errno.h>

 #ifdef __APPLE__
@@ -221,6 +222,9 @@ hash_alloc(VALUE klass)
     OBJSETUP(hash, klass, T_HASH);

     RHASH_IFNONE(hash) = Qnil;
+    if(RUBY_HASH_ALLOC_ENABLED()) {
+       RUBY_HASH_ALLOC(rb_sourcefile(), rb_sourceline());
+    }

     return (VALUE)hash;
 }

After this, I built and installed ruby. Then I wrote a dtrace script to display the filename and line number where hash allocations were happening:

ruby*:::hash-alloc
{
  printf("%s:%d", copyinstr(arg0), arg1);
}

Then I ran the benchmark with dtrace:

$ sudo dtrace -s x.d -c 'ruby -I lib test.rb'

Here is an excerpt of the output:

  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1528
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1529
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1534
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1535
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1525
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/persistence.rb:322
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1527
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1528
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1529
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1534
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1535
  1 126930            hash_alloc:hash-alloc /Users/aaron/git/rails/activerecord/lib/active_record/base.rb:1525

I was able to compare the output from this script on Rails 3.0 vs Rails 3.1. From this output, I was able to determine:

• The number of hash allocations
• Where the hashes were allocated
• Which allocations were new in Rails 3.1

Armed with this information, I was able to eliminate some allocations and return speed in Rails 3.1 to that of Rails 3.0.

DTrace vs gdb vs memprof

I probably could have accomplished this task with memprof, but it requires that I use 1.8.7 from rvm. Working with RVM on my machine is difficult (don’t ask, let’s just say I’m a “special needs” user), and I wanted to use 1.9. So memprof was out the window.

I was able to get the same information by scripting gdb. I set a breakpoint at the correct function, called rb_sourcefile() and rb_sourceline() and redirected to a file. The problem with gdb is that it seemed very slow, and scripting it was a pain (though I am not a gdb expert!).

DTrace was fast, relatively easy to use, and very scriptable. It made me happy!

OMG!!!

I would like to see dtrace probes officially added to ruby trunk. The ruby that ships with OS X has them built in, but they don’t give us information like hash literal allocations. It looks like they were in ruby trunk at one point, but were then reverted. I would like to see them added again.

If you want to play with them on ruby trunk, here is my full patch. Make sure to run make probes.h before make && make install.

HAPPY WEDNESDAY!!!! <3 <3 <3 <3

(it feels good to blog on a non “professional” blog)

Today I discovered that it’s OK for to_ary to return nil. Ruby spec tests this behavior, and the
ruby implementation supports it. But why would you want to implement this?

Array#flatten and to_ary

Consider the following code:

class Item
  def respond_to?(name, visibility = false)
    p "respond to: #{name}"
    false # we don't respond to anything!!
  end

  def method_missing(name, *args)
    p "method missing: #{name}"
    super # if something?
    # do something else
  end
end

[[Item.new]].flatten

In Ruby 1.9, flatten will actually call to_ary on each of the items in the collection. If the method table for this object contains to_ary, Ruby will call to_ary, and if method_missing is implemented, it will call method_missing regardless of how your respond_to? returns. The reason ruby will call method_missing is because it could implement to_ary. (As for why it ignores the return value of respond_to?, I don’t know)

Naturally, if we were to call to_ary directly on this object it would raise a NoMethodError exception. When calling flatten, ruby will swallow the exception. If we enable $DEBUG by running ruby with -d, we can see the exception:

[aaron@higgins ~]$ ruby -d test.rb
Exception `LoadError' at /Users/aaron/.local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1215 - cannot load such file -- rubygems/defaults/operating_system
Exception `LoadError' at /Users/aaron/.local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1224 - cannot load such file -- rubygems/defaults/ruby
"method missing: to_ary"
Exception `NoMethodError' at test.rb:9 - undefined method `to_ary' for #<Item:0x00000101079e80>
"respond to: to_ary"
[aaron@higgins ~]$

Dispatching to method_missing can be expensive, and using exceptions for flow control even more problematic. The way we can get around this issue is by implementing to_ary and having it return nil:

class Item
  def respond_to?(name, visibility = false)
    p "respond to: #{name}"
    false # we don't respond to anything!!
  end

  def method_missing(name, *args)
    p "method missing: #{name}"
    super # if something?
    # do something else
  end

  private
  def to_ary
    nil
  end
end

[[Item.new]].flatten

Run the code again with -d, and you’ll see no more calls to respond_to?, no more calls to method_missing, and no more exceptions raised:

[aaron@higgins ~]$ ruby -d test.rb
Exception `LoadError' at /Users/aaron/.local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1215 - cannot load such file -- rubygems/defaults/operating_system
Exception `LoadError' at /Users/aaron/.local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1224 - cannot load such file -- rubygems/defaults/ruby
[aaron@higgins ~]$

Array() and to_a

The Array() function exhibits a similar behavior, but with the to_a method. Try this same code, but rather than using Array#flatten, do this:

Array(Item.new)

We can fix the error produced by this code with a slight change to our class:

class Item
  def respond_to?(name, visibility = false)
    p "respond to: #{name}"
    false # we don't respond to anything!!
  end

  def method_missing(name, *args)
    p "method missing: #{name}"
    super # if something?
    # do something else
  end

  private
  def to_ary
    nil
  end
  alias :to_a :to_ary
end

[[Item.new]].flatten
Array(Item.new)

Hmmmm

Are warnings annoying? Yes. Is this strange behavior? I tend to think so. But if I’m forced to deal with a class the implements method_missing, I’d like to reduce the number of calls to method_missing.

Anyway. I hope you found this informative. Have a Happy Tuesday!!!!

<3 <3 <3 <3

Small Side Note

The reason there are exceptions coming from rubygems when -d is enabled is because rubygems attempts to require files that are not shipped with rubygems. These files are for packagers to provide. For example someone packaging rubygems for debian, may need to do customizations and those files are where that happens.

ZOMG!!!! HAPPY THURSDAY!!!! Maybe I shouldn’t be so excited now. I want to talk about stuff I’ve been working on in Rails 3.1, and problems I’m encountering today. I want to use this blllurrrggghhh blog post to talk through through the problems I’ve been having, and to share the pain with others.

Pie is delicious!

One feature that would be useful to add to Rails is having a streaming response body. When Rails processes a response, the entire response is buffered in memory before it can be sent to the user. Some information like Content Length (among other things) is derived, and the response is sent.

Sometimes buffering a response is less than ideal. It would be nice if we could send the head tag along with any css or script includes to the browser as quickly as possible. Then the browser can download external resources while we’re still processing data on the server. If this were possible, total response time may remain the same, but the time to first byte would be decreased and the page would load faster as external resource can be downloaded in parallel.

This feature sounds great, but there are many things to think about before it can be implemented. We need to support infinite streams, chunked encoding, prevent header manipulation, ensure database connections, blah, blah blah.

Rack interface

I’m getting ahead of myself. Before we get to our ultimate “pie in the sky” streaming solution, let’s take a look at the Rack API. Rack defines an interface for writing web applications. A rack handler must respond to call which takes one parameter, the request environment. call must return a three item list of:

• Response code
• Headers
• Body

The response code should be a number (like 200), the headers are a hash (like { ‘X-Omg’ => ‘hello!’ }). The body must respond to each and take a block. The body must yield a string to the block, and the string will be output to the client. Optionally, the body may respond to close, and rack will call close when output is complete.

An Example Rack application

Let’s write an example application. Our sample application will simulate an ERb page. We’ll add some sleep statements to simulate work happening during the ERb rendering process:

class FooApplication
  class ErbPage
    def to_a
      head = "the head tag"
      sleep(2)
      body = "the body tag"
      sleep(2)
      [head, body]
    end
  end

  def call(env)
    [200, {}, ErbPage.new.to_a]
  end
end

For the purposes of demonstration, we’ll be using a fake implementation of rack:

class FakeRack
  def serve(application)
    status, headers, body = application.call({})
    p :status  => status
    p :headers => headers

    body.each do |string|
      p string
    end

    body.close if body.respond_to?(:close)
  end
end

If we feed our application through FakeRack like this:

app  = FooApplication.new
rack = FakeRack.new

rack.serve app

We’ll see output from the rack application, and the total program run time is about 4 seconds:

$ time ruby foo.rb
{:status=>200}
{:headers=>{}}
"the head tag"
"the body tag"

real    0m4.008s
user    0m0.003s
sys     0m0.003s

Great! So far, no problem. Why don’t we add a middleware to time how long the response takes.

Rack Middleware

Rack Middleware is simply another Rack application. With Rack, we set up a linked list of middleware that eventually point to the real application. We give the head of the linked list to Rack, Rack calls call on the head of the list, and it is the list’s responsibility to call call on it’s link.

Here, we’ll write a Rack middleware to measure how long the “ERb render” takes and add a header indicating the response time.

class ResponseTimer
  def initialize(app)
    @app = app
  end

  def call(env)
    now                        = Time.now
    status, headers, body      = @app.call(env)
    headers['X-Response-Took'] = Time.now - now

    [status, headers, body]
  end
end

When we construct the ResponseTimer, we pass it the real application. Then we pass the response timer instance to rack:

app   = FooApplication.new
timer = ResponseTimer.new app
rack  = FakeRack.new

rack.serve timer

When rack calls call on the response timer, it records the current time, then calls call on the real application. When the real application returns, the response timer then adds a header with the time delta. The output of this program will look like this:

$ time ruby foo.rb
{:status=>200}
{:headers=>{"X-Response-Took"=>3.999937}}
"the head tag"
"the body tag"

real    0m4.010s
user    0m0.004s
sys     0m0.004s

Speeding up our response time

We’ve noticed a problem with our Rack application. When a client connects, it takes 4 seconds before they receive any data! It would be nice if we could feed our client the head tag ASAP so they can download external resources.

We know that Rack will call each and (depending on your webserver) immediately send data to the client. Rather than computing values in ERb ahead of time, we’ll compute them when Rack asks for them (when each is called).

Let’s refactor the ERb page to be lazy about calculating values:

class FooApplication
  class ErbPage
    def each
      head = "the head tag"
      yield head

      sleep(2)

      body = "the body tag"
      yield body

      sleep(2)
    end
  end

  def call(env)
    [200, {}, ErbPage.new]
  end
end

Now no values are calculated until rack calls each on our body. If we run the program, we’ll see output from the application more quickly than before.

However, the output is somewhat strange:

$ time ruby foo.rb
{:status=>200}
{:headers=>{"X-Response-Took"=>1.1e-05}}
"the head tag"
"the body tag"

real    0m4.032s
user    0m0.027s
sys     0m0.016s

The time command reports that our response was about 4 seconds. But our response header says that the response took nearly 0 seconds! Why is this?

If we look closely at our timer middleware, we can see it is only timing how long it took for call to return.

We cannot guarantee that any processing happened during the call method.

Let me say that again:

We cannot guarantee that any processing happened during the call method.

We wanted our response timer to time how long the ERb took to render, but really it is just timing how long the call method took.

ZOMG HOW FIX?!?

Iterating over the body

One way to fix is to iterate over the body. If the timer iterates over the body, then we can calculate the real time:

class ResponseTimer
  def initialize(app)
    @app = app
  end

  def call(env)
    now                        = Time.now
    status, headers, body      = @app.call(env)

    newbody = []
    body.each { |str| newbody << str }
    headers['X-Response-Took'] = Time.now - now

    [status, headers, newbody]
  end
end

But this solution is no good! Our response timer now buffers the response, and our client ends up waiting for 4 seconds before they get any data.

We know that Rack calls close on the body after it’s done processing the request. Why don’t we try hooking on that method?

Introducing a Proxy Object

One way we can hook on to the close method is by wrapping the response body in a proxy object. Then we can intercept calls made on the body and perform any work we need done:

class ResponseTimer
  class TimerProxy
    def initialize(body)
      @now     = Time.now
      @body    = body
    end

    def close
      @body.close if @body.respond_to?(:close)

      $stderr.puts({'X-Response-Took' => (Time.now - @now)})
    end

    def each(&block)
      @body.each(&block)
    end
  end

  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body = @app.call(env)

    [status, headers, TimerProxy.new(body)]
  end
end

Wow! Suddenly our middleware is not so simple. This proxy solution is sub-optimal for a few reasons. We’re required to make a new object for every request, and our proxy object will add another stack frame between calls from rack to the response body. Even worse, every middleware that needs to do work after the response is finished must define this proxy object.

This solution does get the job done. If we look at the output from the program, we’ll see that the TimerProxy in fact measures ERb processing time correctly:

$ time ruby foo.rb
{:status=>200}
{:headers=>{}}
"the head tag"
"the body tag"
{"X-Response-Took"=>4.000268}

real    0m4.044s
user    0m0.029s
sys     0m0.015s

Diligent readers will note that the response time is no longer part of the response headers. This is because when the body is flushed, the headers must be flushed too. We no longer have the opportunity to add extra headers when each is called on the body.

Our solution isn’t too bad, but it actually isn’t complete. The full awkwardness of this API along with a complete solution can actually be felt (and read) in the Rack source itself.

Lady Gaga Solution

Another possible solution is to decorate the body using a module. We can define a module, then simply call extend on the body with the module:

class ResponseTimer
  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body      = @app.call(env)
    body.extend(Module.new {
      now = Time.now

      define_method(:close) do
        super if defined?(super)

        $stderr.puts({'X-Response-Took' => (Time.now - now)})
      end
    })

    [status, headers, body]
  end
end

The body is extended with an anonymous module. During module definition, the time is recorded. We use define_method because it uses a lambda which will keep a reference to the previously calculated time. In the close method, we call super if it’s defined, then output our time.

This example also works, but has a few downsides. It is different than previous examples because we are timing only the ERb rendering and not call plus ERb rendering. Using this solution, we’re required to create a new module on every request, and also break method caching on every request. Similar to the proxy object solution, we must create a new module and extend for every middleware that must to processing after the response is finished.

ZOMG YOUR EXAMPLE IS CONTRIVED

Yup. But I merely simplified a real world problem. As I mentioned earlier, you can see the awkwardness of this API in rack.

But now that we know about this problem, we can identify middleware that will break streaming responses. For example, Rails defines a middleware that checks connections back in to the connection pool. If our ERb in Rails was streaming, we would lose the database connection during ERb render. The same is true with the query cache in active record. Surely, these cannot be the only middleware that will break when a streaming body is used!

Lifecycle hooks

I think a good solution to this problem would be if Rack provided lifecycle hooks. A Place where we can say “run this when the response is done”. We can define something like that today using middleware:

class EndOfLife
  attr_reader :callbacks

  def initialize(app)
    @app       = app
    @callbacks = []
  end

  def call(env)
    status, headers, body = @app.call(env)
    body.extend(Module.new {
      attr_accessor :eol

      def close
        super if defined?(super)
        eol.callbacks.each { |cb| cb.call }
      end
    })
    body.eol = self

    [status, headers, body]
  end
end

app = FooApplication.new
eol = EndOfLife.new app
eol.callbacks << lambda { puts "it finished!" }

rack  = FakeRack.new

rack.serve eol

This keeps us from defining many proxy objects or module extensions during a response. We only define one module extension, and hook any “end of life” hooks on to this instance. The downside is that we cannot guarantee the position of this middleware in the middleware linked list. That means that the “end of life” middleware may not actually execute at the end of the response!

A “real” solution

Rack’s interface is simple, and I like that. The simplicity is attractive, but the API seems to fall on it’s face when we start talking about streaming web servers. If I remember correctly, Apache 1.0 modules suffered the same problems that Rack is presenting us today. Maybe we should look at Apache 2.0 buckets and filters and design our API using patterns from a project that has already solved this problem.

ZOMG I AM TIRED OF TYPING!!

I’m not happy with any of the solutions I’ve presented. All of them have downsides that I find unattractive. We can live with the downsides, but life will suck. If any of you dear readers have better solutions for me, I am all ears!

Thanks for listening, and HAVE A GREAT DAY!!!!

<3 <3 <3 <3 <3

Edit: I just noticed that Rack contains a “timer” middleware similar to the one I’ve implemented in this blog post. You can view the broken middleware here.

Alright, now that the formalities are out of the way, LET’S GET DOWN TO BUSINESS! Some of you may or may not know, I joined the Ruby core team in October 2009. I am very proud to be a member of ruby-core. However, I have noticed that the Ruby core team does not have an awesome website like the Rails core team.

To rectify this situation, I registered a domain: rubycommitters.org. Unfortunately, my design skills are… sub-par. I would like to fist fix this by having a contest. I would like you to code the design for rubycommitters.org.

How do I enter?

Just fork the rubycommitters.org project on github, follow the instructions in the README, and make the site look good! When you’re done, send me a pull request.

How many times can I enter?

As many as you want. However, you can only win one place.

When is the due date?

You must send me the pull request by January 19th, 23:59:59 PST.

When will winners be announced?

I will announce winners by January 21st, 23:59:59 PST. Once I’ve decided, I’ll email the winners to get their PayPal information and transfer the money Love Bucks.

What are the prizes?

My employer pays me in Tenderlove Cash (from here on referred to as “Love Bucks”), so the prize will be in Love Bucks. Fortunately for all of us, Love Bucks exchange with the US Dollar at a 1:1 ratio. So I will send you your prize via PayPal in the form of US Dollars.

First Place: 200 300 Love Bucks (in the form of US Dollars)
Second Place: 100 Love Bucks (in the form of US Dollars)

All entrants will win a hug from me that is redeemable next time we meet each other!

How will entries be judged?

By me, however I want. I’ll probably ask the intertubes for help, but I have the final say.

Why is the prize so low?

These Love Bucks are coming out of my own wallet! Give me a break!

Conclusion

Since I announced I needed help last weekend, already 9 people have started working, so you’d better get cracking!

I am proud to be a member of the Ruby core team, I’m proud to be involved in the Rails development team, and most of all I’m proud to be a member of the best development community in the world. Thanks to everyone that makes being a member of the Ruby community so awesome!

EDIT!!!!

@mariozig has graciously offered to up the ante. He has donated 100 Love Bucks to the first place winner! That’s a total of 300 Love Bucks for the person who wins first place! Awesome!!!

But you said YAML and JSON! wtf?

I know! In the YAML 1.2 spec, JSON is a subset of YAML. Psych supports YAML 1.1 right now, so *much* (but not all) JSON is supported. Once libyaml is upgraded to YAML 1.2, it will have full JSON support!

Why do we want to do an event based parser?

Twitter streams are a never ending flow of user status updates, and if we want a process to live forever consuming these updates, it would be nice if that process kept a low memory profile. Psych is built in such a way that we can hand it an IO object, it will read from the IO object, then call callback methods as soon as possible. It buffers as little as possible, sending events as soon as possible. If you are familiar with SAX based XML parsing, this will be familiar to you. Plus it is a fun problem!

Let’s start by writing an event listener for some sample JSON.

Event Listener

Our event listener is only going to listen for scalar events, meaning that when Psych parses a string, it will send that string to our listener. There are many different events that can happen, so Psych ships with a handler from which you can inherit. If you check out the source for the base class handler, you can see what types of events your handler can intercept.

For now, let’s write our scalar handler, and try it out.

require 'psych'

class Listener < Psych::Handler
  def scalar(value, anchor, tag, plain, quoted, style)
    puts value
  end
end

listener = Listener.new
parser   = Psych::Parser.new listener
parser.parse DATA

__END__
{"foo":"bar"}

If you run this code, you should see the strings “foo” and “bar” printed.

In this example, our handler simply prints out every scalar value encountered. We created a new instance of the listener, pass that listener to a new instance of the parser, and tell the parser to parse DATA. We can hand the parser an IO object or a String object. This is important because we’d like to hand the parser our socket connection, that way the parser can deal with reading from the socket for us.

Hooking up to Twitter

It would be convenient for us if Twitter’s stream was one continuous JSON document. Why? If it was, we could feed the socket straight to our JSON parser and start consuming events immediately. Unfortunately, Twitter’s stream is not so kind for us event based consumers. We’ll need to trick our JSON parser to think the feed is one continuous document. We’ll get tricky with our data in a minute, but first let’s deal with authentication.

Authentication

Twitter requires us to authenticate before we can consume a feed. Stream authentication is done via Basic Auth. Let’s write a class that can authenticate and read from the stream. Once we do that, we’ll concentrate on parsing the stream.

require 'socket'

class StreamClient
  def initialize user, pass
    @ba = ["#{user}:#{pass}"].pack('m').chomp
  end

  def listen
    socket = TCPSocket.new 'stream.twitter.com', 80
    socket.write "GET /1/statuses/sample.json HTTP/1.1\r\n"
    socket.write "Host: stream.twitter.com\r\n"
    socket.write "Authorization: Basic #{@ba}\r\n"
    socket.write "\r\n"

    # Read the headers
    while((line = socket.readline) != "\r\n"); puts line if $DEBUG; end

    # Consume the feed
    while line = socket.readline
      puts line
    end
  end
end

StreamClient.new(ARGV[0], ARGV[1]).listen

This class takes a username and password and calculates the basic auth signature. When “listen” is called, it opens a connection, authorizes, reads the response headers, and starts consuming the feed.

Processing the Feed

If we look at the output from the previous script, we’ll see that the Twitter stream looks something like this:

512
{"in_reply_to_screen_name":null,...}

419
{"in_reply_to_screen_name":"tenderlove"...}

Which isn’t valid JSON. Instead, it’s a header (the number) indicating the length of the JSON chunk, the JSON chunk, then a trailing “\r\n”. We would like the stream to look something like this:

---
{"in_reply_to_screen_name":null,...}
...
---
{"in_reply_to_screen_name":"tenderlove"...}
...

This chunk is two valid YAML documents. If the stream looked like this, we could feed it straight to our YAML processor no problem. How can we modify the stream to be suitable for our parser?

Fun with Thread and IO.pipe

If we create a pipe, we can have have one thread process input from Twitter and feed that in to the pipe. We can then give the other end of the pipe to our JSON processor and let it read from our processed feed. Let’s modify the “listen” method in our client to munge the feed to a pipe, and hand that off to our YAML processor. I only care about the text of people’s tweets, so let’s modify our listener too.

Here is our completed program:

require 'socket'
require 'psych'

class StreamClient
  def initialize user, pass
    @ba = ["#{user}:#{pass}"].pack('m').chomp
  end

  def listen listener
    socket = TCPSocket.new 'stream.twitter.com', 80
    socket.write "GET /1/statuses/sample.json HTTP/1.1\r\n"
    socket.write "Host: stream.twitter.com\r\n"
    socket.write "Authorization: Basic #{@ba}\r\n"
    socket.write "\r\n"

    # Read the headers
    while((line = socket.readline) != "\r\n"); puts line if $DEBUG; end

    reader, writer = IO.pipe
    producer = Thread.new(socket, writer) do |s, io|
      loop do
        io.write "---\n"
        io.write s.read s.readline.strip.to_i 16
        io.write "...\n"
        s.read 2 # strip the blank line
      end
    end

    parser = Psych::Parser.new listener
    parser.parse reader

    producer.join
  end
end

class Listener < Psych::Handler
  def initialize
    @was_text = false
  end

  def scalar value, anchor, tag, plain, quoted, style
    puts value if @was_text
    @was_text = value == 'text'
  end
end

StreamClient.new(ARGV[0], ARGV[1]).listen Listener.new

Great! In 30 lines, we’ve been able to provide an event based API for consuming Twitter streams. Were it not for the feed munging, we could reduce that by 9 lines!

Problems

So far, there have only been two problems for me with this script. The first is that we are forced to buffer the response from Twitter, but we cannot help that. The second is that sometimes the JSON emitted from Twitter is not parseable by Psych. I think this is just due to Psych only supporting YAML 1.1.

Conclusion

It’s true that we could have implemented this same interface without a pipe and a thread. Rather than munging the stream, we could create a new parser instance for each status update. But why create so many objects for parsing the stream when we only need one?

Anyway, have fun playing with this code, and I encourage you to try out Ruby 1.9.2. I think it’s really fun! PEW PEW PEW! HAPPY SATURDAY!

Lately I’ve been trying to compile my ruby extensions with Clang. One reason I like trying out my extensions with Clang is because it catches some errors that GCC doesn’t. If you know the right things to set, it’s pretty easy to get your extension to compile with Clang. Unfortunately finding the right thing isn’t always easy, but I found the right bits to flip and I want to share!

Here’s how to do it. Add this line to your extconf.rb right after you require mkmf:

require 'mkmf'

RbConfig::MAKEFILE_CONFIG['CC'] = ENV['CC'] if ENV['CC']

# ... rest of your extconf goes here

Then when you compile your extension, just set CC to point at clang:

$ CC=/Developer/usr/bin/clang rake compile

You can see it in action in the nokogiri extconf. You can even see where clang helped me shake out some bugs, and I think that’s pretty cool.

Heroku’s documentation lists two ways to get full text indexing working with your Heroku application. They talk about using Ferret and Solr for full text indexes. The Ferret option looks OK, but it requires you to rebuild your indexes every time you push. Solr would work, but it requires an EC2 instance or some third party server. Since my budget is precisely $0, using Solr is out of the picture.

But there is a third option. A very secret option. A devious but fun option. You see, Heroku runs PostgreSQL for each rails application database. They’re running a version new enough (Version 8.3) to have full text index support built in. If we’re willing to throw out database agnosticism, we can take advantage of the database’s indexing capability. For this article, I’d like to hop on the Postgres train and show you how to get full text indexes working with Postgres in your rails application. I’ll also show you how to get those indexes on Heroku so we can use them “in the cloud” (Heroku is in the cloud, right?).

For the rest of this article, I’m going to assume you have PostgreSQL version 8.3 or higher installed already and can get your rails application working with Postgres. Installing postgres is outside the scope of this article, but I found these instructions to be very helpful.

Step 1: Go get some coffee

I love it when instructions tell me to go get some coffee because I always do. I have to follow the instructions right?

Step 2: Install Texticle

Texticle is a gem I wrote to help you define your text indexes on a per model basis. To install texticle, we just do the normal gem install:

  $ sudo gem install texticle

The gem is pure ruby and isn’t very long, so I encourage you to peek through the source.

While we’re at it, we should configure rails to load the texticle gem. We need to add it to our envoronment.rb file. Here’s what mine looks like:

RAILS_GEM_VERSION = '2.3.4' unless defined? RAILS_GEM_VERSION

require File.join(File.dirname(__FILE__), 'boot')

Rails::Initializer.run do |config|
  config.time_zone = 'UTC'

  config.gem 'texticle'
end

Texticle also comes with some handy rake tasks (which we’ll talk about later). In order to get those we’ll need update the rails Rakefile:

require(File.join(File.dirname(__FILE__), 'config', 'boot'))

require 'rake'
require 'rake/testtask'
require 'rake/rdoctask'

require 'tasks/rails'

require 'rubygems'

## Our texticle rake tasks
require 'texticle/tasks'

Step 3: Configuring your index

Let’s pretend we have an Article model. The Article model has a “title” field and a “body” field:

class CreateArticles < ActiveRecord::Migration
  def self.up
    create_table :articles do |t|
      t.string :title
      t.text   :body

      t.timestamps
    end
  end

  def self.down
    drop_table :articles
  end
end

To index those two fields, we just create an index block in the model and list the fields we want to index:

class Article < ActiveRecord::Base
  index do
    title
    body
  end
end

Declaring this index automatically defines a “search” method on the model that we can use to search our articles:

>> Article.search('coffee instruction')
=> [#<Article id: 4, title: "coffee", body: "I like getting coffee to be in instructions", created_at: "2009-10-17 21:42:13", updated_at: "2009-10-17 21:42:13">]
>> Article.create(:title => 'kittens', :body => 'kitten poop smells bad, but I still like kittens.')
=> #<Article id: 5, title: "kittens", body: "kitten poop smells bad, but I still like kittens.", created_at: "2009-10-17 21:42:33", updated_at: "2009-10-17 21:42:33">
>> Article.search('kittens')
=> [#<Article id: 5, title: "kittens", body: "kitten poop smells bad, but I still like kittens.", created_at: "2009-10-17 21:42:33", updated_at: "2009-10-17 21:42:33">]
>>

Great! We can search our records. There’s just one catch: we haven’t indexed our data. Doing these types of searches will be slow against large sets of data unless we add an index. Writing these indexes is a PITA, so texticle comes with a handy rake task for generating a migration to create your indexes:

  $ rake texticle:migration
  $ rake db:migrate

After running this, Postgres can use the prebuilt indexes when searching your data.

Just remember: every time you modify columns in your index block, or add new index blocks, you should create a new migration to updated the indexes. If you don’t update the indexes, searches will still work as expected, they just might be kind of slow.

Step 4: Integrating With Heroku

This part is pretty easy. First we update our heroku gem manifest:

  $ echo "texticle" >> .gems
  $ git add .gems
  $ git commit -m'updating gem manifest'
  $ git push origin master

Once your code is up on heroku, just tell heroku to migrate the database:

  $ heroku rake db:migrate

It’s just that easy! Your indexes should be available on the Heroku database server and your application can use them.

Advanced Texticle Usage

Texticle has a few more features I’d like to briefly mention. The first one is search ranking. We can tell Postgres which field has a higher priority. For example, we can tell Postgres to weigh matches in the article’s title higher than matches in the body:

class Article < ActiveRecord::Base
  index do
    title 'A'
    body  'B'
  end
end

The ranks are ‘A’ through ‘D’, and multiple fields can have the same rank.

We can also group indexes. The index we’ve seen so far will search all columns listed. We can add another index so that we only search the “title” field:

class Article < ActiveRecord::Base
  index do
    title 'A'
    body  'B'
  end

  index('title') { title }
end

This gives us a “search_title” method in addition to the “search” method:

>> Article.search_title('kittens')
=> [#<Article id: 5, title: "kittens", body: "kitten poop smells bad, but I still like kittens.", created_at: "2009-10-17 21:42:33", updated_at: "2009-10-17 21:42:33">]
>>

The last thing I want to mention is “rank”. When you perform a search, texticle adds an extra field to your model called “rank”. The rank indicates how well your record matched the search criteria:

>> Article.search('like').map { |x| x.rank }
=> ["0.4", "0.4"]
>> Article.search('coffee').map { |x| x.rank }
=> ["1.4"]
>>

Search results are already returned sorted by rank in descending order, so no need to worry about sorting.

Conclusion

I hope you enjoy tickling text with texticle as much as I do. So far, I’ve been pretty happy with this solution.

Things I like:

• It’s the right price for use with Heroku (namely $0)
• Easy to configure and deploy
• No need to rebuild indexes on pushes
• Postgres can be configured to use different dictionaries, so you aren’t stuck with English

The only drawbacks I’ve found so far are:

• INSERTs and UPDATEs are slower
• It’s database specific

Inserts and updates will be slower, but that comes with the territory of adding database indexes. My data is mostly doing reads, so it doesn’t bother me. Texticle is database specific, but other databases are starting to have full text search support. I think texticle could be extended to support other databases, but I’m quite happy with postgres.

Anyway, thanks for reading. The final step is that you should go get another cup of coffee.

Getting an RFID Reader

First thing to do, is buy an RFID reader. After searching around, I found the touchatag reader. I bought the touchatag starter pack. It’s only $40, USB, and comes with 10 RFID tags. Most importantly, it works well with libnfc (more about that later).

The tags that come with the reader have an adhesive back, so you can stick them to stuff. They also have the unique identifier printed on them so that you can make sure your program output is correct.

Interfacing with the reader

Now that we’ve got the reader, let’s do something with it! I mentioned earlier that the touchatag reader works with libnfc. Libnfc is a C library that knows how to work with NFC devices (nerd talk for “RFID readers”). I’ve written a gem called nfc that wraps up the C library in to something we can use in Ruby.

First thing we need to do is install libnfc. I use macports with OS X. With macports, installing libnfc is quite easy:

    $ sudo port install libnfc

Installing on linux should be just as easy, but you’ll need to consult your package manager. Make sure to install the devel packages too!

After that, simply install the nfc Ruby gem:

    $ sudo gem install nfc

Now that that is out of the way, we can actually read an RFID tag. Here is our code:

require 'rubygems'
require 'nfc'
# Find a tag
NFC.instance.find do |tag|
  # Print out the tag we find
  p tag
end

That’s it! Run the code, then touch a tag to the reader, and boom! We have output. With the tag I’m using, the output looks like this:

$ ruby -I lib test.rb
(NFC) ISO14443A Tag
 ATQA (SENS_RES): 00  44
    UID (NFCID1): 04  D7  62  91  21  25  80
   SAK (SEL_RES): 00

The important part of this output is the UID field. That field is the unique identifier for this tag. The identifier comes back as a list of integers, but they are printed on the tag as hex. We can adjust the program just a little bit to see that list, or to get the same string that’s printed on the tag:

# Find a tag
NFC.instance.find do |tag|
  # Examine the raw numbers
  p tag.uid
  # Get just the UID as a string
  puts tag.to_s
end

The output looks like this:

$ ruby -I lib test.rb
[4, 215, 98, 145, 33, 37, 128]
04D76291212580

That’s pretty much it. Unfortunately, I can’t think of anything fun to do with my tags, but maybe you can! I hooked my tags up to the “say” command that comes with OS X and made each tag say something different.

Non-Blocking NFC interaction

Our previous example blocked until an RFID tag was read. If you run the program without having an RFID tag on the reader, it will just sit there until it can read a tag. Sometimes we might want to tell whether or not there is a tag on the reader right now. In other words, we don’t want our program to block.

Calling find without providing a block will return immediately:

p NFC.instance.find.to_s

You’ll get a return value immediately. The tag returned will either contain a blank uid, or an actual UID. Here is the output run once with a tag sitting on the reader, and once without a tag:

$ ruby -I lib test.rb
"04D76291212580"
$ ruby -I lib test.rb
""

Conclusion

That’s pretty much it. Interacting with the touchatag reader is quite simple and straight forward. Currently the nfc gem supports reading ISO1443A tags (the tags that come with the reader). The reader should be able to read other tag types, but I haven’t had a chance to get other tags to test.

Touchatag provides an official API for their readers. But the API seems difficult and is dependent on a network connection.

Here is a video of me reading some tags.
Here is the code from the video.
Here you can find more photos of the reader.
Finally, here is the source of the NFC gem.

Have fun reading some RFID tags!

Examining the Encoding

If you’ve developed a C extension before, you’re probably familiar with rb_str_new2 and friends. They all basically turn a char * in to a string VALUE. But in Ruby 1.9, what is the encoding of the returned Ruby String? Well, using RubyInline, it’s easy enough to see by calling the “encoding” method. Here is a script that works in Ruby 1.8 and Ruby 1.9:

require 'rubygems'
require 'inline'

class HelloWorld
  inline do |builder|
    builder.c '
      static VALUE test() {
        return rb_str_new2("Hello world");
      }
    '
  end
end

string = HelloWorld.new.test

if string.respond_to? :encoding
  puts string.encoding
else
  puts string
end

In Ruby 1.8, this outputs the string, and in 1.9 we see the encoding. In 1.9, the encoding returned is ASCII-8BIT. Now ASCII-8BIT may be the encoding that you want, but then again, it may not. In Nokogiri, the strings coming from libxml2 are already encoded according to the document declaration. So strings returned must be marked with the appropriate encoding. How can we update the encoding?

Changing the Encoding

In Ruby 1.9, we get a few new functions specifically for dealing with encoding. These functions are defined in <ruby/encoding.h>. We’re going to be dealing with two of them: rb_enc_find_index and rb_enc_associate_index.

The first function, rb_enc_find_index, given a char * will look up the index of your encoding. The function takes a string like “UTF-8″ and returns a magic index number for that encoding.

The second function, rb_enc_associate_index, will associate a string held in a VALUE with the encoding index returned from the first function.

Armed with this knowledge, we can modify our original program to return a string encoded with UTF-8. The only modifications are to include <ruby/encoding.h>, get the index for the desired encoding, then associate the VALUE with the returned index:

require 'rubygems'
require 'inline'

class HelloWorld
  inline do |builder|
    builder.include "<ruby/encoding.h>"

    builder.c '
      static VALUE test() {
        VALUE string = rb_str_new2("Hello World");
        int enc = rb_enc_find_index("UTF-8");
        rb_enc_associate_index(string, enc);
        return string;
      }
    '
  end
end

string = HelloWorld.new.test

if string.respond_to? :encoding
  puts string.encoding
else
  puts string
end

Great! When this is run under Ruby 1.9, the encoding returned is UTF-8. Unfortunately, this example is now specific for Ruby 1.9. Ruby 1.8 does not ship with the correct header files, and definitely does not include the functions for looking up and assigning encoding. This code will just not work under Ruby 1.8. Luckily, this code can be refactored to work under either version of Ruby.

Refactoring for 1.8 Support

Both Ruby 1.8 and 1.9 provide a <ruby.h> header file. The Ruby 1.9 version of that file defines a constant HAVE_RUBY_ENCODING_H that lets us determine whether the proper header file exists. Our final attempt tests for the encoding constant, then defines a macro to wrap rb_str_new2. If the version of Ruby we compile against has encoding support, the macro can add the encoding to the string, otherwise, it just ignores the encoding:

require 'rubygems'
require 'inline'

class HelloWorld
  inline do |builder|

    builder.prefix <<-eoc
#include <ruby.h>

#ifdef HAVE_RUBY_ENCODING_H

#include <ruby/encoding.h>

#define ENCODED_STR_NEW2(str, encoding) \
  ({ \
    VALUE _string = rb_str_new2((const char *)str); \
    int _enc = rb_enc_find_index(encoding); \
    rb_enc_associate_index(_string, _enc); \
    _string; \
  })

#else

#define ENCODED_STR_NEW2(str, encoding) \
  rb_str_new2((const char *)str)

#endif
    eoc

    builder.c '
      static VALUE test() {
        return ENCODED_STR_NEW2("Hello world", "UTF-8");
      }
    '
  end
end

string = HelloWorld.new.test

if string.respond_to? :encoding
  puts string.encoding
else
  puts string
end

In 1.8, the macro just returns the new string. In 1.9, the macro returns the string and additionally sets the encoding. Now if we use this macro wherever we create new strings, we’ll be working well with 1.8 and 1.9!

Final Notes

This example was slightly simplified. Since the encoding index is determined at runtime, there could be problems. If rb_enc_find_index cannot find the requested encoding, it simply returns a -1. The macro should handle that case.

Also, if you’re playing along at home, remember to save the file between running it with 1.8 and 1.9. RubyInline examines the mtime of the ruby file, and will only recompile when the rb file has been written to. That means if you run it with 1.8, then immediately run again with 1.9, it won’t recompile it for 1.9. I suppose I should send in a patch.

One last thing… There may be better ways to do this. I needed to determine the encoding at runtime because XML files declare their encoding scheme. If you parse an XML file that declares it’s encoding as EUC-JP, it would make sense that the strings you pull our are encoded in EUC-JP, right? If you know that you’re always going to be returning UTF-8 strings from your C extensions, it could be a different story. Either way, using macros and checking for constants should make sure your code works with 1.8 or 1.9.