I’ve been using RSpec in earnest for the past 6 months now, so I thought it’s time to write a blurrrgh poast comparing RSpec with Minitest. I’ve used Minitest for years, and RSpec only for 6 months, so please keep that in mind when reading!
PLEASE REMEMBER, I don’t care what test framework you use, as long as you are testing your code. This is a post just about my experience with these two frameworks. In other words, this is an opinion piece.
I really believe that all test frameworks are essentially the same. No test framework can do something that’s impossible for another to do. It’s just code. So what differentiates test frameworks? I would say the main difference would be user interface. So I’m really going to compare user interfaces between these frameworks.
Things I like about RSpec.
By far and away, my favorite thing about RSpec is that if there is a failure, it prints how to run just that one test at the bottom of the output. I can easily copy and paste that line and run just the one test. For people who don’t know, here is an example:
describe "something" do it "works" do expect(10).to equal(11) end it "really works" do expect(11).to equal(11) end end
When you run this, the output looks like this:
[aaron@TC tlm.com (master)]$ rspec code/fail_spec.rb F. Failures: 1) something works Failure/Error: expect(10).to eq(11) expected: 11 got: 10 (compared using ==) # ./code/fail_spec.rb:3:in `block (2 levels) in <top (required)>' Finished in 0.00504 seconds (files took 0.20501 seconds to load) 2 examples, 1 failure Failed examples: rspec ./code/fail_spec.rb:2 # something works [aaron@TC tlm.com (master)]$
All you have to do to rerun just the one failing spec is to copy and paste that one line like this:
I don’t have to think or type, I just “copy, paste, run”. I like to think of this as a “learn as you play” feature.
The other thing I like about RSpec is that it ships with a colored output formatter, so if you do
rspec --color, you’ll get colored output. I’ve used Minitest for years without colored output, but I’ve come to really like the colored output from RSpec. It helps me quickly see the most important parts of the test output, the assertion failure, and stack traces (though I highly suspect that the RSpec core team all use black backgrounds in their terminals because sometimes the colors aren’t so nice on my white terminal).
Things I like about Minitest
The main thing that I like about Minitest is that a Minitest test is just a Ruby class. Here is an example test similar to the RSpec sample I showed earlier:
require 'minitest/autorun' class Something < Minitest::Test def test_works assert_equal 11, 10 end def test_really_works assert_equal 11, 11 end end
I like this because I know exactly where the
test_works method is defined. It’s no different than any other class in Ruby, so I don’t have to learn anything new. I like it because I am a heavy user of CTags, so I can easily jump to test methods or classes inside my editor. Another thing that’s nice about using regular Ruby classes is that when I need to refactor my tests, I just use my normal refactoring skills: I extract methods, classes, modules, change to inheritance, etc. I use the same refactoring techniques on my test files as I do on my library files. I think that is where Minitest really shines.
Things I dislike about RSpec
RSpec is a DSL for writing tests. I think this is probably RSpec’s largest weakness. I am a programmer, I can read code, so I don’t particularly care whether or not my test code “reads like English”. I don’t understand the value of the DSL, especially when I have a nearly 4000 line specfile that I’d like to refactor. How do I go about refactoring? I can extract methods:
describe "something" do def helper 11 end it "works" do expect(10).to eq(helper) end it "really works" do expect(11).to eq(helper) end end
But where is the
helper method defined? What’s its visibility? Can I put it in a module? Can I use inheritance? Who can call it? Can I call
super from the extracted method? If so, where does
super go? These are not the types of questions I want to be pondering when I have 3000 test failures and really long spec files to read through. My mind spins off in to thoughts like “I wonder how RSpec works?”, and “I wonder what my cat is doing?”
From what I can gather, calling
describe in RSpec essentially defines a class. But if that’s the case, why not just use Ruby classes? Then I don’t need to guess about method visibility, modules, inheritance, etc. Not to mention, the library code that I’m working with is just using Ruby classes. Why do I need to switch to some other language for testing?
Nesting a describe seems to perform inheritance, but only for certain things like
before blocks. If you run this code:
describe "something" do before do puts "hi!" end it "works" do expect(10).to eq(helper) end it "really works" do expect(11).to eq(helper) end context "another thing" do before do puts "hello!" end it "really really works" do expect(11).to eq(helper) end end def helper 11 end end
You’ll see “hi!” printed 3 times, once for each
it, and “hello!” printed once. For the nested context, I would like to print “hello!” before it prints “hi!”. In a normal inheritance situation, I would change the call to
super. I am not sure how to do what I just described without learning a new thing (I think I would need
let but I don’t know). For now, I’m coping with this by refactoring to classes outside of the specs themselves, then calling the objects from the specs. Essentially “composition” in my tests.
Another thing that bugs me is that if I do try to refactor my RSpec tests, and I change any lines, then the line that I copy and pasted to run just the one test won’t work anymore. I have to figure out the new line. I can use the name of the test by specifying the
-e flag on RSpec, but then I lose the “copy, paste, run” feature that I love so much.
Anyway, I will stop picking on RSpec now. To me, RSpec seems to have some sort of identity crisis and is caught somewhere between “a language for writing tests” and “just Ruby”. That is RSpec’s biggest weakness.
Things I dislike about Minitest
If RSpec does one thing better than Minitest, I would say that the command line UI easily outshines Minitest. Here is one example.
I can run one test at a time like so:
$ ruby code/fail_test.rb -n Something#test_works
This will run the
test_works method on the
Something class. The problem for me is that it’s not clear to do that when you see the failing test output. If we run the whole suite, the output looks like this:
[aaron@TC tlm.com (master)]$ ruby code/fail_test.rb Run options: --seed 5990 # Running: .F Finished in 0.002047s, 977.0396 runs/s, 977.0396 assertions/s. 1) Failure: Something#test_works [code/fail_test.rb:5]: Expected: 11 Actual: 10 2 runs, 2 assertions, 1 failures, 0 errors, 0 skips [aaron@TC tlm.com (master)]$
Looking at this, how would you know how to run just that one test? With RSpec, it’s no problem. I was able to learn how to run just one test the first time I ever used RSpec. I didn’t read any docs, just ran the tests, and the info was right there. Coming from RSpec, you might need to think you could do this:
$ ruby code/fail_test.rb:5
But of course that doesn’t work because you’re just using Ruby on the command line.
Here is a short video of me running the single test:
The disadvantages of this over RSpec’s output is I have to type a few characters and know a commandline interface in advance. You might think “it’s just a few characters”, but every time I have to type something is a potential for a mistake that I’ll have to correct, which takes even more time. I really think the advantages of “copy, paste, run” can’t be understated. (Yes, I know there are tools like autotest for both Minitest and RSpec that will run tests for you, but sometimes I can’t / don’t use them)
The other thing I wish Minitest would do is to ship with a color formatter like RSpec does. I am used to the color output from RSpec, and I don’t really want to go back. It really helps me focus on what’s important about my test failures.
Both of these frameworks have things I like and dislike. Being able to reuse my debugging and refactoring skills in my tests is pretty important to me, especially since I deal with lots of legacy code. This is why I’ll stick with Minitest in my personal projects. However, I am sure I will get better at refactoring and debugging in RSpec tests, and when I get better at it, I will share what I learn.
Now, do I care which one you use? No. As long as you test your code, I am happy. A professional developer should be able to work in either one of these because they essentially do the same thing: test your code.