It’s also worth noting (potentially for future Rubygems patches), that RubyGems doesn’t need whole gemspecs to do it’s general work (for booting apps and such). In fact, on JIT’d rubies, loading specs is quite a lot more expensive than doing some potential alternatives.

Consider for example, a simple data file containing: [name, version, dependencies, load_paths]

Make this datafile in simple data structures (e.g. tuples) and keep parsing cheap. Use a weakref cache and you’ll be reasonably balanced, other than a small amount of GC fragmentation in the early slots (which is a problem now anyway – just dump ObjectSpace and see how much crap gets retained from gemspec loads and the like even after a bunch of GC runs).

Now crunch those indexes into a single file and maybe (if ruby-core adds mmap to stdlib) mmap it, suddenly all but the stat problem is largely solved.

Food for thought.

Folks who work on many different projects often spend more time waiting for “bundle install” than they do waiting for rails to boot.

I blogged about this issue many years ago, and it’s still really difficult to solve. The problem is that stat(2) (and indeed almost every viable discovery method for filesystem paths) actually touches the disk, rather than just looking in the filesystem cache.

This can be avoided / solved a number of ways:

* Run your app in a VM, as then even the stat(2) calls get cached
* Run your app off of a ramdisk (because ramfs is much faster at stat(2) than even Sata-III and a big on-disk-cache)
* Mount your app on an in-process stat-cached filesystem backed fusefs driver (no really, that can work fast, and relieve the need to remember to copy files back over to disk from a ramdisk).

There are some things we can do in RubyGems and Ruby to avoid the mad stat calls, but the best solution without changing the existing semantics will rely on Eric Hodels $LOAD_PATH object proposal in order to be cleanly implemented. Ideally, in order to mainain a reasonable time-space efficiency, we need a lightweight heap datastructure in the ruby stdlib that can represent the filesystem tree that RubyGems has installed, so we can walk that heap instead of smashing the filesystem all the time. This isn’t flawless, but as stated (once you research it) it can be solved quite nicely with the $LOAD_PATH object. The issue with implementing it without, is that the load path is managed by more than just RubyGems (sadly), and this has real practical order limitations.

It’s also worth noting though, that load path objects will be a semantic change that will break some popular libraries out there, so as with so many of these age old problems, we’re in for a fun ride if we want to actually fix the root cause.

As for gemsets, once again, please people, realize that if you’re going to install the same gems again and again and again, you may not actually be saving yourself time.

I recently blogged about my setup here: http://jimeh.me/blog/2011/11/01/my-ruby-development-environment/

Why is rubygems unaware of Gemfile? Shouldn’t that be part of its domain?

Why do we still need “bundle exec” in front of everything? There was a proposal from RVM to manage that in the presence of a Gemfile, but it was rejected. (I think rbenv has a plugin to do that).

I know I’m just ranting, but this whole gems thing shouldn’t be so clumsy…

gem list | cut -f1 -d ” ” | xargs gem uninstall -aIx

… creating clean gemsets for my apps!