Category: rails

Easy Markup Validation

Posted by Aaron Patterson – June 12, 2009

I wanted a test helper that would assert that my XHTML was valid XHTML. So I wrote one and called it “markup_validity“. You can use it too, and I will show you how.

First, install the gem:

  $ sudo gem install markup_validity

Then, use it in your tests:

require 'test/unit'
require 'rubygems'
require 'markup_validity'

class ValidHTML < Test::Unit::TestCase
  def test_i_can_has_valid_xhtml
    assert_xhtml_transitional xhtml_document
  end
end

Oh. You use RSpec? It supports that too:

require 'rubygems'
require 'markup_validity'

describe "my XHTML document" do
  it "can has transitional xhtml" do
    xhtml_document.should be_xhtml_transitional
  end
end

Debugging invalid markup can be a pain. MarkupValidity tries to give you helpful errors to make your life easier. Say you have an invalid piece of XHTML like this:

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
  </head>
  <body>
    <p>
      <p>
        Hello
      </p>
    </p>
  </body>
</html>

The error output from MarkupValidity will be this:

.Error on line: 2:
Element 'head': Missing child element(s). Expected is one of ( script, style, meta, link, object, isindex, title, base ).

1: <html xmlns="http://www.w3.org/1999/xhtml">
2:   <head>
3:   </head>
4:   <body>
5:     <p>

Error on line: 6:
Element 'p': This element is not expected. Expected is one of ( a, br, span, bdo, object, applet, img, map, iframe, tt ).

5:     <p>
6:       <p>
7:         Hello
8:       </p>
9:     </p>

MarkupValidity provides a few assertions for test/unit:

  • assert_xhtml_transitional(xhtml) for asserting valid transitional XHTML
  • assert_xhtml_strict(xhtml) for asserting valid strict XHTML
  • assert_schema(schema, xml) for asserting that your xml validates against a schema
  • assert_xhtml which is an alias for assert_xhtml_transitional

The methods provided for RSpec are quite similar:

  • be_xhtml_transitional for asserting valid transitional XHTML
  • be_xhtml_strict for asserting valid strict XHTML
  • be_valid_with_schema(schema) for asserting that your xml validates against a schema
  • be_xhtml which is an alias for be_xhtml_transitional

MarkupValidity even works well with rails. Here is an example rails controller test:

require 'test_helper'
require 'markup_validity'

class AwesomeControllerTest < ActionController::TestCase
  test "valid markup" do
    get :new
    assert_xhtml_transitional @response.body
  end
end

Nokogiri’s Slop Feature

Posted by Aaron Patterson – December 4, 2008

Oops! When I released nokogiri version 1.0.7, I totally forgot to talk about Nokogiri::Slop() feature that was added. Why is it called “slop”? It lets you sloppily explore documents. Basically, it decorates your document with method_missing() that allows you to search your document via method calls.

Given this document:

doc = Nokogiri::Slop(<<-eohtml)
<html>
  <body>
    <p>hello</p>
    <p class="bold">bold hello</p>
  <body>
</html>
eohtml

You may look through the tree like so:

doc.html.body.p('.bold').text # => 'bold hello'

The way this works is that method missing is implemented on every node in the document tree. That method missing method creates an xpath or css query by using the method name and method arguments. This means that a new search is executed for every method call. It’s fun for playing around, but you definitely won’t get the same performance as using one specific CSS search.

My favorite part is that method missing is actually in the slop decorator. When you use the Nokogiri::Slop() method, it adds the decorator to a list that gets mixed in to every node instance at runtime using Module#extend. That lets me have sweet method missing action, without actually putting method missing in my Node class.

Here is a simplified example:

module Decorator
  def method_a
    "method a"
  end

  def method_b
    "method b: #{super}"
  end
end

class Foo
  def method_b
    "inside foo"
  end
end

foo = Foo.new
foo.extend(Decorator)

puts foo.method_a # => 'method a'
puts foo.method_b # => 'method b: inside foo'

foo2 = Foo.new
puts foo2.method_b # => 'inside foo'
puts foo2.method_a # => NoMethodError

Module#extend is used to add functionality to the instance ‘foo’, but not ‘foo2′. Both ‘foo’ and ‘foo2′ are instances of Foo, but using Module#extend, we can conditionally add functionality without monkey patching and keeping a clean separation of concerns. You can even reach previous functionality by calling super.

But wait! There’s more! You can stack up these decorators as much as you want. For example:

module AddAString
  def method
    "Added a string: #{super}"
  end
end

module UpperCaseResults
  def method
    super.upcase
  end
end

class Foo
  def method
    "foo"
  end
end

foo = Foo.new
foo.extend(AddAString)
foo.extend(UpperCaseResults)

puts foo.method # => 'ADDED A STRING: FOO'

Conditional functionality added to methods with no weird “alias method chain” involvement. Awesome!

I love ruby!

Write your Rails view in……. JavaScript?

Posted by Aaron Patterson – May 6, 2008

In my last post about Johnson, I said that next time I would talk about the JavaScript parse tree that Johnson provides. Well, I changed my mind. Sorry.

I want to write about a rails plugin that I added to Johnson. Brohuda Katz wrote an ERb type parser in JavaScript, and added it to the (yet to be released) Johnson distribution. With that in mind, and looking at the new template handlers in edge rails, I was able to throw together a rails plugin that allows me to use JavaScript in my rails view code.

Lets get to the code. Here is my controller:

class JohnsonController < ApplicationController
  def index
    @users = User.find(:all)
  end
end

And my EJS view (the file is named index.html.ejs):

<% for(var user in at.users) { %>
  <%= user.first_name() %><br />
<% } %>

The johnson rails plugin puts controller instance variables in to a special javascript variable called “at”. The “at” variable is actually a proxy to the controller, lazily fetching instance variables from the controller and importing those objects in to javascript land.

Lets take a look at the plugin, its only a few lines:

class EJSHandler < ActionView::TemplateHandler
  class EJSProxy # :nodoc:
    def initialize(controller)
      @controller = controller
    end

    def key?(pooperty)
      @controller.instance_variables.include?("@#{pooperty}")
    end

    def [](pooperty)
      @controller.instance_variable_get("@#{pooperty}")
    end

    def []=(pooperty, value)
      @controller.instance_variable_set("@#{pooperty}", value)
    end
  end

  def initialize(view)
    @view = view
  end

  def render(template)
    ctx = Johnson::Context.new
    ctx.evaluate('Johnson.require("johnson/template");')
    ctx['template'] = template.source
    ctx['controller'] = @view.controller
    ctx['at'] = EJSProxy.new(@view.controller)

    ctx.evaluate('Johnson.templatize(template).call(at)')
  end
end

ActionView::Template.register_template_handler("ejs", EJSHandler)

When the template gets rendered (the render method), I wrap the controller with an EJS proxy, then compile the template into a javascript function, and call that function. The “at” variable is set to the EJSProxy before executing the template, and all property accessing on the “at” variable is passed along to fetching instance variables from the controller.

Server side javascript coding in rails. Weird, eh?

Profiling Database Queries in Rails

Posted by Aaron Patterson – March 13, 2008

Despite the recent Ruby webserver speed contests, most of the slowness at my job results from slow (or too many) database queries.

To help keep database queries down, I added a stats to every page that shows the number of queries vs. cache hits, the number of rows returned, and the amount of data transferred from the database. In this screenshot I’m using the “live” environment, 3 cache hits, 169 misses, 577 rows returned, and 458.9k data transferred. Clicking the box hides it, and clicking “Super Hide!” hides the box and sets a cookie so that the box doesn’t show up again for a while.

Debug Window

To get this working, first I monkey patch the MysqlAdapter to collect database stats:

  ActiveRecord::ConnectionAdapters::MysqlAdapter.module_eval do
    @@stats_queries = @@stats_bytes = @@stats_rows = 0

    def self.get_stats
      { :queries => @@stats_queries,
        :rows => @@stats_rows,
        :bytes => @@stats_bytes }
    end
    def self.reset_stats
      @@stats_queries = @@stats_bytes = @@stats_rows = 0
    end

    def select_with_stats(sql, name)
      bytes = 0
      rows = select_without_stats(sql, name)
      rows.each do |row|
        row.each do |key, value|
          bytes += key.length
          bytes += value.length if value
        end
      end
      @@stats_queries += 1
      @@stats_rows += rows.length
      @@stats_bytes += bytes
      rows
    end
    alias_method_chain :select, :stats
  end

Next I patched the QueryCache to keep track of hits and misses:

  ActiveRecord::ConnectionAdapters::QueryCache.module_eval do
    @@hits = @@misses = 0

    def self.get_stats
      { :hits => @@hits,
        :misses => @@misses }
    end
    def self.reset_stats
      @@hits = @@misses = 0
    end

    def cache_sql_with_stats(sql, &block)
      if @query_cache.has_key?(sql)
        @@hits += 1
      else
        @@misses += 1
      end
      cache_sql_without_stats(sql, &block)
    end
    alias_method_chain :cache_sql, :stats
  end

Then modify ActionController to reset stats for each request:

  ActionController::Base.module_eval do
    def perform_action_with_reset
      ActiveRecord::ConnectionAdapters::MysqlAdapter::reset_stats
      ActiveRecord::ConnectionAdapters::QueryCache::reset_stats
      perform_action_without_reset
    end

    alias_method_chain :perform_action, :reset

    def active_record_runtime(runtime)
      stats = ActiveRecord::ConnectionAdapters::MysqlAdapter::get_stats
      "#{super} #{sprintf("%.1fk", stats[:bytes].to_f / 1024)} queries: #{stats[:queries]}"
    end
  end

Just drop all that inside the after_initialize in your development.rb and you’ll get the nice stats. After that, just create a partial that displays the stats and include the partial at the bottom of your layout. Our partial looks like this:

<% unless %w(production test).include?(RAILS_ENV) -%>
  <h4 id="debug" onclick="$(this).remove()" style="background:pink;text-align:center;position:absolute;top:16px;left:35%;padding:0.5em;border: 2px solid red;">
  <%= RAILS_ENV %>
  <br />
  <% if ActiveRecord::ConnectionAdapters::QueryCache.respond_to?(:get_stats) %>
    <% stats = ActiveRecord::ConnectionAdapters::QueryCache.get_stats %>
    Queries: <%= stats[:hits] %> / <%= stats[:misses] %> /
    <%= number_to_percentage((stats[:hits].to_f / (stats[:hits] + stats[:misses])) * 100, :precision => 0) %>
    |
  <% end %>
  <% if ActiveRecord::ConnectionAdapters::MysqlAdapter.respond_to?(:get_stats) %>
    <% stats = ActiveRecord::ConnectionAdapters::MysqlAdapter.get_stats %>
    Rows: <%= stats[:rows] %> |
    Transfer: <%= sprintf("%.1fk", stats[:bytes].to_f / 1024) %>
  <% end %>
  <p style="margin:0">
    <a style="color:magenta" href="#" onclick="superHide()">super hide!</a>
  </p>
  </h4>
  <script type="text/javascript">
    function superHide() {
      document.cookie = 'debug=hidden; path=/; domain=<%= request.host %>; max-age=14400';
    }
    if(document.cookie.indexOf('debug=hidden') != -1) {
      $('debug').hide();
    }
  </script>
<% end -%>

It’s a little work, but it helps keep my mind on reducing the queries. With enough work, one of these days the speed of the webserver will matter to me. Thanks to Adam Doppelt for the basis of this monkey patch. Any bugs are mine, not his!

Trigger Happy

Posted by Aaron Patterson – March 1, 2007

I’ve just released “Trigger Happy“, a rails plugin that adds support for triggers in your Active Record Migrations. To install the plugin just do this:


script/plugin install svn://rubyforge.org/var/svn/artriggers/trunk/trigger_happy
[/sourcecode]

To add a trigger do this:

add_trigger "ai_people",
  :o n         => 'people',
  :timing     => 'after',
  :event      => 'insert',
  :statement  => 'INSERT INTO log (id, timestamp) VALUES (NEW.id, NOW())'

To drop a trigger do this:

drop_trigger "ai_people"

It only supports mysql for now, but I plan on having other database supported in the future.

Audit Logs with ActiveRecord

Posted by Aaron Patterson – March 1, 2007

I’ve been trying to create an audit log for a few (12) tables, and unfortunately ActiveRecord seems to be falling flat for what I want to do. First I’ll describe whats out there already to do this, then I’ll talk about what I had to do.

There are a couple nice plugins out there for helping you keep track of changes to your records. The first one is acts_as_versioned. acts_as_versioned will copy your record to a version table whenever there is an insert or update, then increase a version number in the original table. It also automatically adds a list of versions to your original model, and lets you revert to any particular version. This is great, and almost exactly what I needed. I didn’t really need the ability to revert to a version, but that just seemed like gravy on top. The only problem with acts_as_versioned comes in when you try to keep track of changes to habtm relationships. But this is where the second plugin comes in to play.

The second plugin is called acts_as_versioned_association. This plugin will help you keep track of changes to your relationships. acts_as_versioned_association is built on top of acts_as_versioned. They way it works is by setting the owner model to acts_as_versioned, then when any associations are updated, it writes a new version of the owner model, then writes the associations to a associations version table. So if you were to have an Article model that has and belongs to many Documents, you would need 5 tables to represent that:

  1. articles
  2. articles_versions
  3. documents
  4. articles_documents
  5. articles_documents_versions

If the association changes, a record is written to articles (to increase the version number), articles_versions, articles_documents, and articles_documents_versions. But what happens if Article has 10 versioned habtm relationships, and just one of those relationships changes? Then a record will get written to every habtm version table for just one change. Thats 12 writes for a change to just one relationship. That will not scale….

Fortunately I don’t care about reverting to a previous version. All I care about is what changed, and when. So my favorite solution for this problem is to add triggers to the tables that may change. That way I only get one extra write when a relationship changes. Just copy the row to another table with a timestamp and an action.

But what about has_many :through?

has_many :through allows you to put a model on top of the join table. Then I could just drop acts_as_versioned on top of that model and be done. I would have used this solution except that I ran in to a bug. has_many :through does not support all of the same array manipulation that habtm does. For example, you can append (<<) to has_many :through and habtm, but the clear method does not work the same way on has_many :through. Also, has_many :through does not set an attribute= method like habtm does.