TL;DR: Rack API is poor when you consider streaming response bodies.

ZOMG!!!! HAPPY THURSDAY!!!! Maybe I shouldn't be so excited now. I want to talk about stuff I've been working on in Rails 3.1, and problems I'm encountering today. I want to use this blllurrrggghhh blog post to talk through through the problems I've been having, and to share the pain with others.

Pie is delicious!

One feature that would be useful to add to Rails is having a streaming response body. When Rails processes a response, the entire response is buffered in memory before it can be sent to the user. Some information like Content Length (among other things) is derived, and the response is sent.

Sometimes buffering a response is less than ideal. It would be nice if we could send the head tag along with any css or script includes to the browser as quickly as possible. Then the browser can download external resources while we're still processing data on the server. If this were possible, total response time may remain the same, but the time to first byte would be decreased and the page would load faster as external resource can be downloaded in parallel.

This feature sounds great, but there are many things to think about before it can be implemented. We need to support infinite streams, chunked encoding, prevent header manipulation, ensure database connections, blah, blah blah.

Rack interface

I'm getting ahead of myself. Before we get to our ultimate "pie in the sky" streaming solution, let's take a look at the Rack API. Rack defines an interface for writing web applications. A rack handler must respond to call which takes one parameter, the request environment. call must return a three item list of:

• Response code
• Headers
• Body

The response code should be a number (like 200), the headers are a hash (like { 'X-Omg' => 'hello!' }). The body must respond to each and take a block. The body must yield a string to the block, and the string will be output to the client. Optionally, the body may respond to close, and rack will call close when output is complete.

An Example Rack application

Let's write an example application. Our sample application will simulate an ERb page. We'll add some sleep statements to simulate work happening during the ERb rendering process:

class FooApplication
  class ErbPage
    def to_a
      head = "the head tag"
      sleep(2)
      body = "the body tag"
      sleep(2)
      [head, body]
    end
  end

  def call(env)
    [200, {}, ErbPage.new.to_a]
  end
end

For the purposes of demonstration, we'll be using a fake implementation of rack:

class FakeRack
  def serve(application)
    status, headers, body = application.call({})
    p :status  => status
    p :headers => headers

    body.each do |string|
      p string
    end

    body.close if body.respond_to?(:close)
  end
end

If we feed our application through FakeRack like this:

app  = FooApplication.new
rack = FakeRack.new

rack.serve app

We'll see output from the rack application, and the total program run time is about 4 seconds:

$ time ruby foo.rb                                       
{:status=>200}
{:headers=>{}}
"the head tag"
"the body tag"

real    0m4.008s
user    0m0.003s
sys     0m0.003s

Great! So far, no problem. Why don't we add a middleware to time how long the response takes.

Rack Middleware

Rack Middleware is simply another Rack application. With Rack, we set up a linked list of middleware that eventually point to the real application. We give the head of the linked list to Rack, Rack calls call on the head of the list, and it is the list's responsibility to call call on it's link.

Here, we'll write a Rack middleware to measure how long the "ERb render" takes and add a header indicating the response time.

class ResponseTimer
  def initialize(app)
    @app = app
  end

  def call(env)
    now                        = Time.now
    status, headers, body      = @app.call(env)
    headers['X-Response-Took'] = Time.now - now

    [status, headers, body]
  end
end

When we construct the ResponseTimer, we pass it the real application. Then we pass the response timer instance to rack:

app   = FooApplication.new
timer = ResponseTimer.new app
rack  = FakeRack.new

rack.serve timer

When rack calls call on the response timer, it records the current time, then calls call on the real application. When the real application returns, the response timer then adds a header with the time delta. The output of this program will look like this:

$ time ruby foo.rb 
{:status=>200}
{:headers=>{"X-Response-Took"=>3.999937}}
"the head tag"
"the body tag"

real    0m4.010s
user    0m0.004s
sys     0m0.004s

Speeding up our response time

We've noticed a problem with our Rack application. When a client connects, it takes 4 seconds before they receive any data! It would be nice if we could feed our client the head tag ASAP so they can download external resources.

We know that Rack will call each and (depending on your webserver) immediately send data to the client. Rather than computing values in ERb ahead of time, we'll compute them when Rack asks for them (when each is called).

Let's refactor the ERb page to be lazy about calculating values:

class FooApplication
  class ErbPage
    def each
      head = "the head tag"
      yield head

      sleep(2)

      body = "the body tag"
      yield body

      sleep(2)
    end
  end

  def call(env)
    [200, {}, ErbPage.new]
  end
end

Now no values are calculated until rack calls each on our body. If we run the program, we'll see output from the application more quickly than before.

However, the output is somewhat strange:

$ time ruby foo.rb
{:status=>200}
{:headers=>{"X-Response-Took"=>1.1e-05}}
"the head tag"
"the body tag"

real    0m4.032s
user    0m0.027s
sys     0m0.016s

The time command reports that our response was about 4 seconds. But our response header says that the response took nearly 0 seconds! Why is this?

If we look closely at our timer middleware, we can see it is only timing how long it took for call to return.

We cannot guarantee that any processing happened during the call method.

Let me say that again:

We cannot guarantee that any processing happened during the call method.

We wanted our response timer to time how long the ERb took to render, but really it is just timing how long the call method took.

ZOMG HOW FIX?!?

Iterating over the body

One way to fix is to iterate over the body. If the timer iterates over the body, then we can calculate the real time:

class ResponseTimer
  def initialize(app)
    @app = app
  end

  def call(env)
    now                        = Time.now
    status, headers, body      = @app.call(env)

    newbody = []
    body.each { |str| newbody << str }
    headers['X-Response-Took'] = Time.now - now

    [status, headers, newbody]
  end
end

But this solution is no good! Our response timer now buffers the response, and our client ends up waiting for 4 seconds before they get any data.

We know that Rack calls close on the body after it's done processing the request. Why don't we try hooking on that method?

Introducing a Proxy Object

One way we can hook on to the close method is by wrapping the response body in a proxy object. Then we can intercept calls made on the body and perform any work we need done:

class ResponseTimer
  class TimerProxy
    def initialize(body)
      @now     = Time.now
      @body    = body
    end

    def close
      @body.close if @body.respond_to?(:close)

      $stderr.puts({'X-Response-Took' => (Time.now - @now)})
    end

    def each(&block)
      @body.each(&block)
    end
  end

  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body = @app.call(env)

    [status, headers, TimerProxy.new(body)]
  end
end

Wow! Suddenly our middleware is not so simple. This proxy solution is sub-optimal for a few reasons. We're required to make a new object for every request, and our proxy object will add another stack frame between calls from rack to the response body. Even worse, every middleware that needs to do work after the response is finished must define this proxy object.

This solution does get the job done. If we look at the output from the program, we'll see that the TimerProxy in fact measures ERb processing time correctly:

$ time ruby foo.rb 
{:status=>200}
{:headers=>{}}
"the head tag"
"the body tag"
{"X-Response-Took"=>4.000268}

real    0m4.044s
user    0m0.029s
sys     0m0.015s

Diligent readers will note that the response time is no longer part of the response headers. This is because when the body is flushed, the headers must be flushed too. We no longer have the opportunity to add extra headers when each is called on the body.

Our solution isn't too bad, but it actually isn't complete. The full awkwardness of this API along with a complete solution can actually be felt (and read) in the Rack source itself.

Lady Gaga Solution

Another possible solution is to decorate the body using a module. We can define a module, then simply call extend on the body with the module:

class ResponseTimer
  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body      = @app.call(env)
    body.extend(Module.new {
      now = Time.now

      define_method(:close) do
        super if defined?(super)

        $stderr.puts({'X-Response-Took' => (Time.now - now)})
      end
    })

    [status, headers, body]
  end
end

The body is extended with an anonymous module. During module definition, the time is recorded. We use define_method because it uses a lambda which will keep a reference to the previously calculated time. In the close method, we call super if it's defined, then output our time.

This example also works, but has a few downsides. It is different than previous examples because we are timing only the ERb rendering and not call plus ERb rendering. Using this solution, we're required to create a new module on every request, and also break method caching on every request. Similar to the proxy object solution, we must create a new module and extend for every middleware that must to processing after the response is finished.

ZOMG YOUR EXAMPLE IS CONTRIVED

Yup. But I merely simplified a real world problem. As I mentioned earlier, you can see the awkwardness of this API in rack.

But now that we know about this problem, we can identify middleware that will break streaming responses. For example, Rails defines a middleware that checks connections back in to the connection pool. If our ERb in Rails was streaming, we would lose the database connection during ERb render. The same is true with the query cache in active record. Surely, these cannot be the only middleware that will break when a streaming body is used!

Lifecycle hooks

I think a good solution to this problem would be if Rack provided lifecycle hooks. A Place where we can say "run this when the response is done". We can define something like that today using middleware:

class EndOfLife
  attr_reader :callbacks

  def initialize(app)
    @app       = app
    @callbacks = []
  end

  def call(env)
    status, headers, body = @app.call(env)
    body.extend(Module.new {
      attr_accessor :eol

      def close
        super if defined?(super)
        eol.callbacks.each { |cb| cb.call }
      end
    })
    body.eol = self

    [status, headers, body]
  end
end

app = FooApplication.new
eol = EndOfLife.new app
eol.callbacks << lambda { puts "it finished!" }

rack  = FakeRack.new

rack.serve eol

This keeps us from defining many proxy objects or module extensions during a response. We only define one module extension, and hook any "end of life" hooks on to this instance. The downside is that we cannot guarantee the position of this middleware in the middleware linked list. That means that the "end of life" middleware may not actually execute at the end of the response!

A "real" solution

Rack's interface is simple, and I like that. The simplicity is attractive, but the API seems to fall on it's face when we start talking about streaming web servers. If I remember correctly, Apache 1.0 modules suffered the same problems that Rack is presenting us today. Maybe we should look at Apache 2.0 buckets and filters and design our API using patterns from a project that has already solved this problem.

ZOMG I AM TIRED OF TYPING!!

I'm not happy with any of the solutions I've presented. All of them have downsides that I find unattractive. We can live with the downsides, but life will suck. If any of you dear readers have better solutions for me, I am all ears!

Thanks for listening, and HAVE A GREAT DAY!!!!

<3 <3 <3 <3 <3

Edit: I just noticed that Rack contains a “timer” middleware similar to the one I’ve implemented in this blog post. You can view the broken middleware here.