Upgrading Ruby on Rails from 3.2 to 4.2

At work we’ve recently finished upgrading the Rails version from 3.2 to 4.2 of our legacy application created in 2011.

I collected here the experience and the lessons I learned during the process along with some undocumented changes and solutions for them.

Upgrading Ruby on Rails from 3.2 to 4.2

From 3.2 to 4.0

The work had been started before I joined the company so I decided to continue it with a few improvements.

Simplified workflow I followed

  • Listen to podcasts about how others did it
  • Go through the 4.0 Release notes
  • Go through the official upgrade guide
  • Go through the unofficial Fast Ruby upgrade guide
  • Make all tests green
  • Normal test on staging
  • Stress test on staging for a whole night
  • Deploy to one prod server and route more and more web traffic to it
  • Deploy to the remaining servers, let the queues and background jobs run for a while on Rails 4
  • Restore the web traffic

Organising the work

I defined 3 branches:

  • pre_rails4
  • rails4/master
  • post_rails4

Fortunately we could release half of the required changes before deploying Rails 4 itself. This gave us the very big advantage of releasing changes frequently in small chunks. Which came handy in debugging when something went wrong. We used the pre_rails4 branch for it.

A few changes had to wait until Rails 4 was stable, for these we used the post_rails4 branch.

The rest stayed on the rails4/master branch with frequent rebasing.

Lessons learned

Keeping the dependencies up-to-date

Unfortunately this was highly neglected in the project before. So much that basically there was no upgrade in the previous one year and a half. As a result Rails 4 required a big pile of gems to be upgraded. Many were backward compatible, so we organised them into groups according to their risk and we deployed them in every Monday and Wednesday morning.

Waiting for management to allocate time officially

The first time I did everything by the Book. I worked on the upgrade only when I had a ticket in the actual sprint, which was a rather rare phenomenon despite all my pushing efforts. As a result a four-week task took us more than a year to finish.

Interesting contrast: after the major upgrade I went into a brief period of rebellion against the rules in the Book. Meaning, without any ticket or management approval I upgraded the patch version, fixed all the 10 broken tests, tested on staging and deployed the next Monday. It took 3-4 hours all together. And for my greatest surprise everyone was happy with it. As a consequence we fine-tuned the working process a bit to include cleanup and technical debt into the estimations and now both the engineering and business side is very happy!

Unexpected challenges

These minor changes were not advertised in any upgrade guide and yet, they rewarded me with some quality debugging time.

Rails 4 stopped truncating database string/text fields

More precisely as Moomaka pointed out in a Reddit comment, it was not Rails who truncated the content but MySQL itself. Rails 4 enabled strict mode in MySQL by default.

A broader description about the problem and a hotfix can be found here.

Ruby’s Logger is not extended by Rails 4

As a result the following code won’t set the datetime_format of the MyLogFormatter instance:

# config/application.rb
config.logger.formatter = MyLogFormatter.new
config.logger.datetime_format = '...'

The solution:

# config/application.rb
config.log_formatter = MyLogFormatter.new
config.log_formatter.datetime_format = '...'

Tip: How to validate if Rails overrides the datetime_format method?

Rails.logger.method(:datetime_format).source_location
# with Rails 3 it returns
=> ["...activesupport/lib/active_support/core_ext/logger.rb", 65]
# with Rails 4 it returns
=> ["...ruby/lib/logger.rb", 285]

Database connection leaks in threads

The first stress test on staging provided a lot of ActiveRecord::ConnectionTimeoutError errors and the application stopped working completely.

It turned out that any database command running in a thread checks out a database connection from the pool and neglects to put it back. This is true for both Rails versions. The difference is Rails 3.2 calls the clear_stale_cached_connections! method when checking out a connection and there are no more available connections. Rails 4 doesn’t.

The solution was to call ActiveRecord::Base.clear_active_connections! manually or to use ActiveRecord::Base.with_connection with a block.

Thread.new do
  begin
    User.last
  ensure
    ActiveRecord::Base.clear_active_connections!
  end
end

From 4.0 to 4.2

Leveraging all the experience from the previous upgrade this one went relatively smooth and fast. (Especially in the light of the rebelling spirit)

Was it worth it?

¡Totally!

One can be concerned about the cost of upgrading, but I’m quite confident the cost of not upgrading is even worst:

  • Security / bug issues (What would we tell our insurance company if we got hacked?)
  • Implementing and maintaining architecture / features exist in the next Rails versions
  • An old version is not really a selling point in the hiring process

On of my favourite new features is Active job. A unified API for background jobs. And finally we can send any sort of work to the background easily, especially sending emails.

From 4.2 to 5.0

It is also worth pointing out that Rails 6.0 RC1 has been released for quite some time, meaning Rails 6 is imminent. And therefore as soon as the first stable release is out of Rails 6, they will drop the support for 4.2.

Speaking of which, after dropping the support, there is nothing left except dropping the bass. (I’m sorry, the temptation pulled too strong to resist)

Our future plan is to try a bit different approach:

Instead of using a long-running branch we will follow how GitHub dealt with the problem. We will hook dual boot into our CI pipeline to require Rails 5 compatibility for new changes. And when everything is green, we will flip the switch.

Please let me know if you have met any unexpected challenge during an upgrade.

Comments

  1. Currently doing this same thing now (which makes this the third Rails 3 => 4 upgrade I've had to do). I found your rebellion story interesting and relatable ;) I wasn't aware of the thread issue so thanks for pointing that out

    ReplyDelete
    Replies
    1. I'm happy to hear that more people are upgrading. I think it is important! Regarding rebelling, we tuned our workflow since the first Rails upgrade. Now we include the cleanup / tech debt part in the time estimations. Against all of my expectation this works like charm, and now the engineers are just as happy as business people:D For the thread issue, I discovered it only because I tried to run some stress test, which are really valuable as we can see.

      Delete

Post a Comment