End-to-end testing for API only Ruby on Rails apps with Cucumber syntax

In an earlier project, we had good test coverage with only unit tests produced following TDD practices. But against all our careful work, one unfortunate day, we broke a core function in production, even though our test suite was fully green. As it turned out, the units under test met some difficulties in working together after leaving the safe and isolated environment we put them in for the sake of testing.

This taught me to value integration/acceptance tests more. Given this event and after watching an interesting talk from Ian Cooper, I built serious doubts against deep isolation for the system under test.

What comes after is only my modest experience with testing API only Rails apps. I don’t claim to be an expert guide nor anything ambitious.

My deepest desire

The current project I work on has tests both in RSpec and in Minitest. They are good tools to perform testing at the unit level; however, as we ascend towards the integration level, I feel them less convenient.

I have a deep desire to write end-to-end tests like this RSpec example:

describe 'User pays via debit card' do
  context 'when the transaction succeeds' do
    it 'creates a proper payment record'
    it 'stores the response of the payment gateway'
    it 'sends a payment done metric'
    it 'registers the payment in XYZ service'
    ...
  end
end

This would serve as documentation and also proves that the happy path works for a feature.

One problem: each it block would run the whole scenario to test one thing, which is not overly performant. A solution could be to merge all the examples into one but then we would lose all the titles of the examples. We have enough examples like it 'works' do followed by pages of meaningless expectations…

And how about testing intermediate states between bigger steps? We will come back to that later.

What? Did I hear it right? Someone yelled Cucumber from the back seat?

Cucumber and the gherkin syntax

As we are gliding towards behaviour testing, it seems to be worth investigating tools like Cucumber.

But first, let’s pimp up our previous payment scenario with a 3D secure 1.0 requirement. It basically adds an extra security layer to online card payments where the user has to prove their identity by typing in a pin code received via SMS. It happens usually on the website of the payment gateway, so we can think of it as an extra intermediate step.

With this in mind, I would write the previous test in Cucumber as follows. More precisely, this would only be the list of the steps, whereas the step implementation would go into a different file… but you get the idea.

Feature: User pays via debit card using 3DS
  Scenario: Successful payment
    When the user requests a new payment
    Then create a pending payment record
    And create a pending payment gateway request record
    And redirect the user to the offsite payment page
    When the offsite payment page redirects the user to our endpoint
    Then the user arrives to the successful page
    When receive the payment callback
    Then update the payment record
    And store the response of the payment gateway
    And send a payment.created metric
    And register the payment in XYZ service
    And ...

Its expressiveness speaks for itself.

I have to confess that we tried to integrate Cucumber into our Rails API only project but we met some unexpected surprises along the way, including mocking, VCR etc, so we put this project on hold until we know how to do them properly.

That is until a few days ago, when I came across the Turnip gem, which proudly advertises that it is a Gherkin extension for RSpec and that basically we can write cucumber features in RSpec. Hm… very interesting stuff.

The Turnip gem

Having a curious nature, I immediately gave it a try and what I found out was even more interesting.

A Turnip scenario is basically a lightweight cover around the RSpec it block, meaning the whole scenario runs in the scope of a single it block, which gives us all the advantage of it.

For example, we can separate the expectation from the exercise when we expect to send a message to a class.

step 'the user requests a new payment' do
  allow(XYZService).to receive(:send_payment_created_event)

  post '/api/payments'
end

step 'register the payment in XYZ service' do
  expect(XYZService).to have_received(:send_payment_created_event)
end

Basic Turnip setup

By default, the request test helpers like post 'url' are not available in the step definitions (feature type tests); fortunately, we can always include the right mixin:

# spec/turnip_helper.rb
Dir[Rails.root.join('spec/acceptance/step_definitions/**/*.rb')].each { |f| require f }

RSpec.configure do |config|
  config.include RSpec::Rails::RequestExampleGroup, type: :feature
end

The feature files are like the ones in Cucumber:

# spec/acceptance/user_pays_with_debit_card.feature

Feature: User pays with debit card
  Background:
    Given a user with an active debit card

  Scenario: Successful payment
    When the user initiates a new payment

The step definitions differ a bit; we put them into a module we include later on:

# spec/acceptance/step_definitions/user_pays_with_debit_card_steps.rb
module UserPaysWithDebitCardSteps
  attr_reader :user

  step 'a user with an active debit card' do
    @user = create(:user)
    create(:debit_card, user: user)
  end

  step 'the user initiates a new payment' do
    request_params = {
      amount: '100'
    }

    headers = auth_headers_for(user)

    post '/api/user/loan_application/payments', request_params, headers
  end
end

RSpec.configure { |c| c.include(UserPaysWithDebitCardSteps) }

Exactly what we needed to continue with our end-to-end testing project.

End-to-end tests vs. Unit tests

A general advice is to avoid abusing end-to-end tests, because they take significantly more time than their unit level associates; the GUI likes to change frequently which usually results in broken tests, etc…

However, we have a peculiar situation in the case of an API only app:

• The API changes way less frequently than the GUI.
• We don’t need to render any HTML, except maybe in the case of emails.
• No HTML response, no need to emulate a browser.
• RSpec request tests (cover for Rails integration tests) add only a small overhead.

Our project has many unit tests and they run quite fast but people have had very limited success in reading the tests and understanding how the application should behave on a higher level, not to mention the business level.
This was the main reason that drag me toward end-to-end testing.

Case study

I built a fully featured, detailed, end-to-end Turnip test (like the one above) for a user who successfully pays back his loan with a debit card applying 3D secure authorisation. It can be red and understood by even business people and it runs in under 400 milliseconds on my development laptop.

I’m very much willing to sacrifice this 400ms to have expressive and maintained documentation and to see that the happy path works for a feature that turns our business into a profitable one.

Closing thoughts

There was a tragic lack of end-to-end tests in our project but finally we can cover our business critical flows with high-level tests. This not only documents the functionality for both business and developers but also gives our team more confidence in deploying new changes into production, which reduces the cycle time for trying out new ideas (also) in production.

I bet business people appreciate this last benefit, even if they have a low opinion about tech stuff like testing.