New-formula-starburst

Rails rake tasks to sync your remote database to your local development environment

9

So, you’re managing or developing a Ruby on Rails application that you deploy regularly using Capistrano (as any sane Rails developer should), and you want a really easy way to sync up the database on your remote production server with the development environment you have running on your local machine. This is really useful for debugging your app, especially if you have a lot of user-generated content. It’s also just really helpful for design and development if your local environment looks exactly the same as your live environment.

Well, I know it’s not that hard to log in to your remote database, export the whole database, copy it back to your machine, and load it into your development environment. But after a few times doing this, you realize there’s got to be a better way.

Since Capistrano gives us so much flexibility to do pretty much anything on our remote machine, there is an easy way. Here are a few tasks that you can freely copy and paste into your application that will automate all this:

  1. Connect to your production database server
  2. Dump your production MySQL database (sorry, this only works for MySQL right now!)
  3. Download the MySQL dump file to your local machine
  4. Replace the contents of your local development environment with the data and structure from your live production environment

Usage

rake db:production_data_refresh

Yep! That’s all there is to it. It uses the database connection information that you have already set up in your config/database.yml file, so no passwords are ever transmitted. It also uses SSH and SFTP to do all the remote communication and data transfer, so everything is nice and secure.

It is highly recommended that you set up public key authentication so you don’t have to log in with your username and password every time Capistrano wants to connect to your remote host.

Get your copy and paste finger ready … the code is right below the fold

Add these tasks to your lib/tasks/my_app.rake file:

namespace :db do
  desc "Dump the current database to a MySQL file" 
  task :database_dump do
    load 'config/environment.rb'
    abcs = ActiveRecord::Base.configurations
    case abcs[RAILS_ENV]["adapter"]
    when 'mysql'
      ActiveRecord::Base.establish_connection(abcs[RAILS_ENV])
      File.open("db/#{RAILS_ENV}_data.sql", "w+") do |f|
        if abcs[RAILS_ENV]["password"].blank?
          f << `mysqldump -h #{abcs[RAILS_ENV]["host"]} -u #{abcs[RAILS_ENV]["username"]} #{abcs[RAILS_ENV]["database"]}`
        else
          f << `mysqldump -h #{abcs[RAILS_ENV]["host"]} -u #{abcs[RAILS_ENV]["username"]} -p#{abcs[RAILS_ENV]["password"]} #{abcs[RAILS_ENV]["database"]}`
        end
      end
    else
      raise "Task not supported by '#{abcs[RAILS_ENV]['adapter']}'" 
    end
  end

  desc "Refreshes your local development environment to the current production database" 
  task :production_data_refresh do
    `rake remote:exec ACTION=remote_db_runner --trace`
    `rake db:production_data_load --trace`
  end 

  desc "Loads the production data downloaded into db/production_data.sql into your local development database" 
  task :production_data_load do
    load 'config/environment.rb'
    abcs = ActiveRecord::Base.configurations
    case abcs[RAILS_ENV]["adapter"]
    when 'mysql'
      ActiveRecord::Base.establish_connection(abcs[RAILS_ENV])
      if abcs[RAILS_ENV]["password"].blank?
        `mysql -h #{abcs[RAILS_ENV]["host"]} -u #{abcs[RAILS_ENV]["username"]} #{abcs[RAILS_ENV]["database"]} < db/production_data.sql`
      else
        `mysql -h #{abcs[RAILS_ENV]["host"]} -u #{abcs[RAILS_ENV]["username"]} -p#{abcs[RAILS_ENV]["password"]} #{abcs[RAILS_ENV]["database"]} < db/production_data.sql`
      end
    else
      raise "Task not supported by '#{abcs[RAILS_ENV]['adapter']}'" 
    end
  end

end

Now, add these tasks to your config/deploy.rb file:

desc 'Dumps the production database to db/production_data.sql on the remote server'
task :remote_db_dump, :roles => :db, :only => { :primary => true } do
  run "cd #{deploy_to}/#{current_dir} && " +
    "#{rake} RAILS_ENV=#{rails_env} db:database_dump --trace" 
end

desc 'Downloads db/production_data.sql from the remote production environment to your local machine'
task :remote_db_download, :roles => :db, :only => { :primary => true } do  
  execute_on_servers(options) do |servers|
    self.sessions[servers.first].sftp.connect do |tsftp|
      tsftp.get_file "#{deploy_to}/#{current_dir}/db/production_data.sql", "db/production_data.sql" 
    end
  end
end

desc 'Cleans up data dump file'
task :remote_db_cleanup, :roles => :db, :only => { :primary => true } do  
  delete "#{deploy_to}/#{current_dir}/db/production_data.sql" 
end 

desc 'Dumps, downloads and then cleans up the production data dump'
task :remote_db_runner do
  remote_db_dump
  remote_db_download
  remote_db_cleanup
end

Enjoy! Please let me know if anything doesn’t work the way you think it should.

Search the Ruby on Rails API

1

I was just sent the link to RoRAPI. It’s a painfully simple but really useful search engine for the Ruby on Rails API, powered by Google Co-op.

Thanks to Brad Gessler for putting it together.

Generic actions for Rails subclasses

1

Single Table Inheritance in Ruby on Rails is cool, and has made my code oh so beautiful in several cases. I ran across an interesting problem yesterday with inheritance classes, and I thought I’d share my solution.

Imagine you have parent class NinjaTurtle, and four subclasses Leonardo, Donatello, Michaelangelo and Raphael.

class NinjaTurtle < ActiveRecord::Base
end

class Leonardo < NinjaTurtle
  has_many :katana
end

class Donatello < NinjaTurtle
  has_one :bo
end

class Michaelangelo < NinjaTurtle
  has_many :nunchaku
end

class Raphael < NinjaTurtle
  has_many :sai
end

So a typical new or create action in any turtle’s controller would look like a bit like this

@donatello = Donatello.new

What if you want one inherited action called new that will create any type of NinjaTurtle, depending on who’s controller handled the request?

The straightforward way would be to just write an action called new in each of the subclasses controller, but that’s a lot of repetition of basically the same thing.

The solution? Make sure your subclass controller inherits its parent class controller, then you can write a generic new action in the parent controller.

class NinjaTurtleController < ApplicationController
  def new
    @ninja_turtle = params[:controller].camelcase.constantize.new
  end
end

class LeonardoController < NinjaTurtleController ; end
class DonatelloController < NinjaTurtleController ; end
class MichaelangeloController < NinjaTurtleController ; end
class RaphealController < NinjaTurtleController ; end

Now, any request to /leonardo/new would create a new Leonardo object and assign it to @ninja_turtle. Similarly, a request to /donatello/new would create a new Donatello object. The beauty is that there is really only one new method that is flexible enough to create the right type of object depending on the controller that handled the request.

How it works

I discovered yesterday that Ruby’s class names are constants. That means that there is a constant named Leonardo that refers to the Leonardo class.

If we’re using Rails convention, which we should be, the controller name that corresponds to the Leonardo class is leonardo. We can grab the controller name from request parameters.

  params[:controller]                        #=> "leonardo" 
  params[:controller].camelcase              #=> "Leonardo" 

and then we use the Rails built-in method constantize to convert our string into a constant, and invoke the new method on the object that we’re trying to create. It essentially evalutates to Leonardo.new, which is exactly what we wanted!

References:
Programming Ruby
Infovore – Getting a class object in Ruby from a string containing that class’s name

DreamHost Your Own Packages and Gems

29

UPDATE SEP 24, 2007: Almost a year later, this is by far the most popular article on my blog. Thanks to everyone who has contributed their feedback and helped make this guide really work. I have updated the article to reflect some new versions of some software and filled in a few holes that may have tripped some people up.

DreamHost has become a pretty popular choice for many people looking for a reliable Ruby on Rails host. I’ve been with DreamHost for about six months now, and I’d say they’re pretty stellar. This blog is hosted on DreamHost, as well as my Wamily project (while we’re in development and light testing—hopefully we’ll outgrow the shared host soon) and things are running well. They also offer a ridiculous amount of disk space and bandwidth for the price.

One of the best things about DreamHost is that they allow you to manage pretty much every aspect of your environment. You have the ability to log in to the server via SSH and compile and install any of your own packages or Ruby gems. DreamHost does have a centrally controlled version of most of the basic things (including Ruby and Rails), but a lot of times they’re a little slow on the upgrade path when the latest new versions come out.

For that reason, and also for practice when I really have to maintain my own server, I’ve decided to manage all of my own versions of Ruby, Rails, and most of my gems.

Here’s a quick how-to be a control freak on DreamHost after the fold …

Log In Using SSH

I use PuTTY for all of my SSH needs. Fire it up and connect to your domain using SSH. After you enter your password, you’re at your home directory (replace nateclark here with your username). Its worth mentioning that you should log in with the same username that your web application runs as (whatever you set in the DreamHost control panel when creating your domain).

[/home/nateclark]$ 

Create a Directory for Compiled Packages and Gems

To keep things neat, I created a directory called .packages under my home directory for any compiled packages. You could download and compile stuff right in your home directory, but that could quickly get cluttered. The . in front of the directory name makes it a hidden directory, so it won’t be listed in a regular ls command. You have to use ls -a to list all directories including hidden ones.

In my ~/.packages directory, I currently have installed my own versions of Ruby 1.8.6, Trac 0.10, Python 2.4.3, ImageMagick 6.3.0, Subversion 1.3.2 and a few other smaller things.

I also have a ~/.gems directory to store all of my own gems.

Set Up Your Paths

Ok, this is the important part—your path variables tell the shell where to look for executables and gems. We’ll set these up in your ~/.bashrc file, which is executed by bash for non-login shells. For regular login shells, you want to use the same path variables, and ~/.bash_profile sets this up. I’ve chosen to source ~/.bashrc at the end of ~/.bash_profile. For Linux newbies, ~ is a shortcut for your home directory.

My ~/.bash_profile looks like this:
# ~/.bash_profile: executed by bash(1) for login shells.

umask 002
PS1='[\h:$PWD]$ '
alias ll="ls -l" 
EDITOR="/usr/bin/vim" 
. .bashrc
And my .bashrc looks like this:
# ~/.bashrc: executed by bash(1) for non-login shells.

export TZ=EST5EDT # Sets my timezone to Eastern U.S. time
export LD_LIBRARY_PATH="$HOME/.packages/lib" 
export PATH="$HOME/.packages/bin:$HOME/.gems/bin:${PATH}" 
export GEM_HOME=$HOME/.gems
export GEM_PATH="$GEM_HOME:/usr/lib/ruby/gems/1.8" 

NOTE: Some people (including me) have had problems with Dreamhost’s shared gems conflicting with gems that you install locally. To force your environment to use ONLY your local gems and not the Dreamhost managed gems at all, change the last line above to:

export GEM_PATH="$GEM_HOME:/usr/lib/ruby/gems/1.8" 
Now that these paths are set, simply log out and log in again to get them to work. Or, you can just source the file at the prompt:
. ~/.bash_profile
To test if it worked, just echo your path. You should see something like this:
$ echo $PATH
  > /home/nateclark/.packages/bin:/home/nateclark/.gems/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
$ echo $GEM_PATH
  > /home/nateclark/.gems:/usr/lib/ruby/gems/1.8

The paths that point to your directories are listed first, and then the DreamHost shared location is next. Sweet.

Configure Your Rails Environment

In your Rails applications, you’ll also have to tell Rails where to look for your gems. Add this line to the top of your config/environment.rb, file:
ENV['GEM_PATH'] = '/home/nateclark/.gems:/usr/lib/ruby/gems/1.8'

Install Your Packages

Ok, now you’re ready to install your packages into your ~/.packages directory. You can do this just like you would manually compile a package normally, with the only difference that you need to use the --prefix=$HOME/.packages option when you configure. For example, here’s how I installed Ruby 1.8.6:

First, install readline. This is required if you ever want to use script/console on your Dreamhosted rails app.

$ cd ~/.packages
$ wget ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
$ tar zxvf readline-5.2.tar.gz
$ cd readline-5.2
$ ./configure --prefix=$HOME/.packages
$ make
$ make install
Now, download and compile the latest version of Ruby:
$ cd ~/packages
$ wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.6.tar.gz
$ tar zxvf ruby-1.8.6.tar.gz
$ cd ruby-1.8.6
$ ./configure --prefix=$HOME/.packages --with-readline-dir=$HOME/.packages
$ make
$ make install

Occasionally, Dreamhost will kill a process that is using a lot of CPU or memory and would be bogging down the server. A few times, they have killed my make command. If this happens, just run it again until it completes successfully.

Then, make sure you’re actually using the new version:

$ which ruby
  > /home/nateclark/packages/bin/ruby
$ ruby -v 
  > ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-linux]

Update: Some of you had problems with gem. That’s cause I left out the part about installing rubygems. Oops.

$ cd ~/.packages
$ wget http://rubyforge.org/frs/download.php/20989/rubygems-0.9.4.tgz
$ tar zxvf rubygems-0.9.4.tgz
$ cd rubygems-0.9.4
$ ruby setup.rb config --prefix=$HOME/.packages
$ ruby setup.rb setup
$ ruby setup.rb install

Install Your Gems

Installing gems is just as simple as always. Since your $GEM_HOME is set, all your gems will go into the directory that you specified. For example, install your own version of Rails:
$ gem install rails --include-dependencies
And make sure that you’re using the right one:
$ which rails
  > /home/nateclark/.gems/bin/rails

Thats it! Now you can manage your own versions of pretty much any package, library or gem. Of course with that comes the responsibility of keeping everything patched and up to date.

Let me know if I’ve missed anything. Good luck.