The Impact of Performance Tools on Performance -OR- Use Your Environments!
Using different ‘environments’ in Ruby on Rails development is an established best practice. Rails provides you with 3 default ones: Production, Development, and Test, and at Highgroove we like to add a ‘Staging’ environment as well. Heroku supports multiple environments, and they make life a lot easier when building applications.
In the production environment, some things are cached by default, while in the development environment, things are loaded whenever they are changed which makes working on a project a lot easier because you don’t have to restart the server to see your changes. A staging environment lets you ensure that your staging system and production system stick to their own databases and API services without requiring any server customizations other than setting RAILS_ENV.
This all sounds great, but if you’re not careful early on it can lead to some intersting problems!
For a large project I’m working on, a handful of DelayedJob jobs run to sync about 200,000 files on the filesystem with corresponding records in ActiveRecord, which I would then push up to the production environment on Heroku for the customer to see. These jobs were taking about a week to run each time, and running these jobs in the development environment meant that I had to be careful about the changes I made to models while jobs were running. After a few weeks of this, I’d had enough and started to investigate the slowness.
First, I looked at the usual suspects:
Indexes: Any queries taking longer than 1ms in the query log meant that something probably wasn’t indexed properly, but none of these looked too terrible. I Added a few indexes which sped up some about 40ms queries to under 1ms but didn’t put much of a dent in the overall job time
Unnecessary object creation: Manually tracing though the code, only required objects were being created and I wasn’t setting any globals or other things that the Garbage Collector wouldn’t take care of
Nothing obvious! So out came the perftools gem.
#Gemfile gem 'perftools.rb' # https://github.com/tmm1/perftools.rb
I ran a profile on the slow job and saw that about 40% of the time was being spent in Garbage Collection. This is an unfortunate consequence of using Ruby. Garbage Collection can be disabled around problematic code with GC.disable and GC.enable but when processing huge numbers of files and scanning huge numbers of objects processes can quickly bloom to several Gigabytes of memory which laptops and small virtual machines don’t get too happy about. Additionally, with Garbage Collection disabled, Ruby won’t automatically close unused file handles and (in this case in particular) can run into a “too many files open” error.
Taking another look at the call graph, 55% of the time was being spent in attr_missing which was unusual! They were being called from bullet’s call path. Disabling bullet did the trick: Instead of Ruby using up 100% of the CPU as ‘user’ load, things switched over to postgres using a bit of CPU and more of the process waiting on IO, and the overall run time sped up by 2 orders of magnitude. Awesome!
The lesson here could be that one should be conscious of the tools they are running to automate performance improvements (like bullet) because the data collection that they perform is guaranteed to make things slower.
However, the real lesson is that Rails gives you multiple environments and you should use them: complex things relating to your real data should be done in the production environment. This sync was for production data, and there was no reason for me to be running it in the development environment. My gems were set up properly so that things like bullet and perftools.rb were only being run in the development environment, so running jobs in ‘production’ meant an automatic 1000x+ improvement in performance without the hours I spent figuring this out.
The new (much faster and easier to deal with) workflow is:
RAILS_ENV=production heroku db:pull RAILS_ENV=production rake db_sync RAILS_ENV=production heroku db:push
During the several hours that the sync is running, it doesn’t affect my development and I don’t have to worry about things in development getting pushed up to production. Great success. Now if only I could speed up pushing/pulling a 400MB database to/from Heroku.
Do you have any favorite tools for automating performance enhancements or catching performance problems? Any best practices for working with multiple environments?