Obama '08
New-formula-starburst

DreamHost Your Own Packages and Gems

20
Oct
33
posted 07:05 am in ,
tags / DreamHost, ruby, Wamily, Ruby on Rails

UPDATE SEP 24, 2007: Almost a year later, this is by far the most popular article on my blog. Thanks to everyone who has contributed their feedback and helped make this guide really work. I have updated the article to reflect some new versions of some software and filled in a few holes that may have tripped some people up.

DreamHost has become a pretty popular choice for many people looking for a reliable Ruby on Rails host. I’ve been with DreamHost for about six months now, and I’d say they’re pretty stellar. This blog is hosted on DreamHost, as well as my Wamily project (while we’re in development and light testing—hopefully we’ll outgrow the shared host soon) and things are running well. They also offer a ridiculous amount of disk space and bandwidth for the price.

One of the best things about DreamHost is that they allow you to manage pretty much every aspect of your environment. You have the ability to log in to the server via SSH and compile and install any of your own packages or Ruby gems. DreamHost does have a centrally controlled version of most of the basic things (including Ruby and Rails), but a lot of times they’re a little slow on the upgrade path when the latest new versions come out.

For that reason, and also for practice when I really have to maintain my own server, I’ve decided to manage all of my own versions of Ruby, Rails, and most of my gems.

Here’s a quick how-to be a control freak on DreamHost after the fold …

Log In Using SSH

I use PuTTY for all of my SSH needs. Fire it up and connect to your domain using SSH. After you enter your password, you’re at your home directory (replace nateclark here with your username). Its worth mentioning that you should log in with the same username that your web application runs as (whatever you set in the DreamHost control panel when creating your domain).

[/home/nateclark]$

Create a Directory for Compiled Packages and Gems

To keep things neat, I created a directory called .packages under my home directory for any compiled packages. You could download and compile stuff right in your home directory, but that could quickly get cluttered. The . in front of the directory name makes it a hidden directory, so it won’t be listed in a regular ls command. You have to use ls -a to list all directories including hidden ones.

In my ~/.packages directory, I currently have installed my own versions of Ruby 1.8.6, Trac 0.10, Python 2.4.3, ImageMagick 6.3.0, Subversion 1.3.2 and a few other smaller things.

I also have a ~/.gems directory to store all of my own gems.

Set Up Your Paths

Ok, this is the important part—your path variables tell the shell where to look for executables and gems. We’ll set these up in your ~/.bashrc file, which is executed by bash for non-login shells. For regular login shells, you want to use the same path variables, and ~/.bash_profile sets this up. I’ve chosen to source ~/.bashrc at the end of ~/.bash_profile. For Linux newbies, ~ is a shortcut for your home directory.

My ~/.bash_profile looks like this: 
# ~/.bash_profile: executed by bash(1) for login shells.

umask 002
PS1='[\h:$PWD]$ '
alias ll="ls -l" 
EDITOR="/usr/bin/vim" 
. .bashrc
And my .bashrc looks like this: 
# ~/.bashrc: executed by bash(1) for non-login shells.

export TZ=EST5EDT # Sets my timezone to Eastern U.S. time
export LD_LIBRARY_PATH="$HOME/.packages/lib" 
export PATH="$HOME/.packages/bin:$HOME/.gems/bin:${PATH}" 
export GEM_HOME=$HOME/.gems
export GEM_PATH="$GEM_HOME:/usr/lib/ruby/gems/1.8"

NOTE: Some people (including me) have had problems with Dreamhost’s shared gems conflicting with gems that you install locally. To force your environment to use ONLY your local gems and not the Dreamhost managed gems at all, change the last line above to:

export GEM_PATH="$GEM_HOME:/usr/lib/ruby/gems/1.8"
Now that these paths are set, simply log out and log in again to get them to work. Or, you can just source the file at the prompt: 
. ~/.bash_profile
To test if it worked, just echo your path. You should see something like this: 
$ echo $PATH
  > /home/nateclark/.packages/bin:/home/nateclark/.gems/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
$ echo $GEM_PATH
  > /home/nateclark/.gems:/usr/lib/ruby/gems/1.8

The paths that point to your directories are listed first, and then the DreamHost shared location is next. Sweet.

Configure Your Rails Environment

In your Rails applications, you’ll also have to tell Rails where to look for your gems. Add this line to the top of your config/environment.rb, file: 
ENV['GEM_PATH'] = '/home/nateclark/.gems:/usr/lib/ruby/gems/1.8'

Install Your Packages

Ok, now you’re ready to install your packages into your ~/.packages directory. You can do this just like you would manually compile a package normally, with the only difference that you need to use the --prefix=$HOME/.packages option when you configure. For example, here’s how I installed Ruby 1.8.6:

First, install readline. This is required if you ever want to use script/console on your Dreamhosted rails app.

$ cd ~/.packages
$ wget ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
$ tar zxvf readline-5.2.tar.gz
$ cd readline-5.2
$ ./configure --prefix=$HOME/.packages
$ make
$ make install
Now, download and compile the latest version of Ruby: 
$ cd ~/packages
$ wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.6.tar.gz
$ tar zxvf ruby-1.8.6.tar.gz
$ cd ruby-1.8.6
$ ./configure --prefix=$HOME/.packages --with-readline-dir=$HOME/.packages
$ make
$ make install

Occasionally, Dreamhost will kill a process that is using a lot of CPU or memory and would be bogging down the server. A few times, they have killed my make command. If this happens, just run it again until it completes successfully.

Then, make sure you’re actually using the new version:

$ which ruby
  > /home/nateclark/packages/bin/ruby
$ ruby -v 
  > ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-linux]

Update: Some of you had problems with gem. That’s cause I left out the part about installing rubygems. Oops.

$ cd ~/.packages
$ wget http://rubyforge.org/frs/download.php/20989/rubygems-0.9.4.tgz
$ tar zxvf rubygems-0.9.4.tgz
$ cd rubygems-0.9.4
$ ruby setup.rb config --prefix=$HOME/.packages
$ ruby setup.rb setup
$ ruby setup.rb install

Install Your Gems

Installing gems is just as simple as always. Since your $GEM_HOME is set, all your gems will go into the directory that you specified. For example, install your own version of Rails: 
$ gem install rails --include-dependencies
And make sure that you’re using the right one: 
$ which rails
  > /home/nateclark/.gems/bin/rails

Thats it! Now you can manage your own versions of pretty much any package, library or gem. Of course with that comes the responsibility of keeping everything patched and up to date.

Let me know if I’ve missed anything. Good luck.

Comments

Leave a comment

  • Avatar FatDude said 12 days later:

    Nate, my new resident guru! Thanks for this blog and the pointers via email, both have enabled me to keep what little hair I have remaining.

    Cheers!

  • Avatar Aaron Powell said 19 days later:

    Hi-

    Thanks for this guide. I’m running into difficulties, however. Everything worked exactly as you said until I got to actually installing gems. When I type “gem install rails—include-dependencies”, I get the following:

    ruby: No such file or directory—gem (LoadError)

    Any idea what I’m doing wrong? Thank you.

  • Avatar David Honsvick said about 1 month later:

    I got this error Also

    ruby: No such file or directory—gem (LoadError)

    Anyonme solve this yet?

  • Avatar David Honsvick said about 1 month later:

    Well Seemed to get around it,

    The line : $ cd ~/.packages

    Should have been cd ~/packages

    Thanks for the info, tutorial, its has been great

  • Avatar David Honsvick said about 1 month later:

    The Last Piece

    If you plan you use other libraries such as ruby/amazon

    #!/home//packages/bin/ruby

  • Avatar Eduardo said about 1 month later:

    It will never work.

    1. you need to change the shebang, otherwise your app will keep using the DH ruby

    2. if you change your shebang you will get stuck with this:

    ../config/../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:490:in `require’: no such file to load—fcgi (MissingSourceFile)

  • Avatar David Honsvick said 2 months later:

    I got it working, Just very Slow. Compiled my own ruby / gems just as you see above. Changed the Gems Env in envireoment.rb, changed the shabang in the dispatch files to #!/home/dontyouwishyouknew/path/to/ruby

    Server as unusable until I ended the existing dispatch.fcgi processes and it works. How ever it seems to be about 50% slower than just using thier librarys. Anyone have any thoughts?

  • Avatar Daniel Zhang said 3 months later:

    Great!
    I missed this line in evniroment.rb:
    ENV[‘GEM_PATH’] = ’/home/wamily/.gems:/usr/lib/ruby/gems/1.8’

    Thank you~
    very helpful~
    :)

  • Avatar Brad Gessler said 3 months later:

    I’ve found if you’re getting the ‘ruby: No such file or directory—gem (LoadError)’ error after you run:

    $ ruby setup.rb config—prefix=$HOME/packages

    You need to first go into the rubygems-0.9.0 after you decompress the rubygems-0.9.0 file by entering:

    $ cd rubygems-0.9.0.tgz

    after the ’$ tar zxvf rubygems-0.9.0.tgz’ command. Then run:

    $ ruby setup.rb config—prefix=$HOME/packages

    And finish up the rest of what is in this blog post. That should fix it!

  • Avatar johnjoyce said 3 months later:

    Excellent, just what I was looking for. Now a little help getting RadiantCMS up and runnin..

  • Avatar Chris (1me.com) said 3 months later:

    Hello,

    Can you please break down for me what this line is actually doing? ENV[‘GEM_PATH’]= ’/home/wamily/.gems:/usr/lib/ruby/gems/1.8’

    Here’s my confusion. What is “wamily?” I had a hunch to replace it with my home directory name as the rest of your examples use /home/nateclark.

    Even if wamily works I still would like to know what exactly I’m doing :-D.

    Thanks,

    Chris.

  • Avatar Nate Clark said 4 months later:

    @Chris – thanks for pointing that out. Wamily is another one of my projects … looks like a little copy and paste error. You’re right, it should be 

    /home/nateclark/.gems
    I updated the post.

  • Avatar gfdickinson said 5 months later:

    The addition of the ENV[‘GEM_PATH’] seems like it belongs in an envirnment specific file for production. Can someone describe how I can get this out of the environment.rb file and into a file that will only be used when in the production environment?

  • Avatar Nate Clark said 5 months later:

    @gfdickinson—you can put the ENV[‘GEM_PATH’]= line in config/environments/production.rb and then it will only be set when you load the production environment.

  • Avatar Tom said 7 months later:

    “Then, make sure you’re actually using the new version…” - Well, which ruby returns my local install, but ruby -v still shows the old version. If I run ~/packages/bin/ruby -v, I get the new version number. What am I supposed to do now? Just ignore that? Ruby 1.85 is not bad after all… I just hope the rest is gonna work.

  • Avatar Jorge Calás said 7 months later:

    Thanks very much for this article.

    I could be able to install latest ruby/gems/rails following your instructions. My site start running after installing custom rails but when I tried to make:

    ruby script/console

    I just get errors regarding readline after googling for a while I found the solution by reading this posting http://forum.dreamhosters.com/troubleshooting/58403-Use-my-compiled-Ruby.htm

    What I exactly did was:

    get readline and extract

    cd $HOME/packages  
wget ftp://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz
tar xvzf readline-5.2.tar.gz

    install readline

    cd readline-5.2
./configure --prefix=$HOME/packages
make
make install
cd ..

    re-compile and re-install ruby

    cd ruby-1.8.5-p2
./configure --prefix=$HOME/packages --with-readline-dir=$HOME/packages
make
make install

    After that I was able to run script/console without any problem. It was really straighforward.

  • Avatar Dirceu Jr. said 9 months later:

    In RubyGems 0.9.4 only work for me using: $ ruby setup.rb setup $ ruby setup.rb install—prefix=$HOME/packages[/code]

    ruby setup.rb config --prefix=$HOME/packages
  • Avatar stephen murdoch said 9 months later:

    I really love this tutorial…thanks man it worked

    I had to change a few things though… :)

    _____ First of all, the following line, when downloading ruby using ‘wget’:

    bq.$ wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.5-p12.tar.gz

    needed to be changed to:

    bq.$ wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.5.tar.gz

    It’s subtle but it mattered.

    * 2ndly, you have said, just after the afformentioned line, to type the following:

    bq.$ cd ruby-1.8.5—p12.tar.gz

    Well you can’t ‘cd’ into at ’.tar.gz’, (and as I mentioned, I downloaded ruby1.8.5.tar.gz instead of ruby-1.8.5—p12.tar.gz, as the latter doesn’t exist at the time of writing this) so I changed the line to:

    bq.$ cd ruby-1.8.5

    &&&&&&&&&&&&&&&&&&& And finally, after unpacking rubygems-0.9.2, you should ‘cd’ into the directory before running the ruby serup.rb ** commands

    ______ Still you have REALLY helped me out in a very big way…. Thanks a lot for taking the time to make this kicka$$ tutorial..

  • Avatar stephen murdoch said 9 months later:

    oops, sorry about that last post – I thought I was doing ‘textile’ by inserting bq. before certain lines to make them indented – if you have any problems with my explanation, just remove the ‘bq.’ from my code

  • Avatar Anthony Altemara said about 1 year later:

    Grazie a mille!!!

    I was looking for a way to work around a slow webhost, and first found stuff on freezing plugins.

    This is so much better than freezing plugin versions, or just installing gems local to your railsapp.

    I will link link to this when I get my help section up.

  • Avatar Sam said about 1 year later:

    This post has been very helpful. However, when I run ./dispatch.fcgi I get the error “bad interpreter: No such file or directoryages/bin/ruby”

    I think the problem has to do with the first line of my dispatch.fcgi file I changed the line to:

    #!/home/username/.packages/bin/ruby

    However, after re-reading all of the comments I think my .packages folder may need to be changed just to packages (with no dot).

    When is it correct to use .packages and when is it correct to just use packages?

    Is that the reason I am getting the “bad interpreter:” error?

  • Avatar Mike said about 1 year later:

    Great article and got me started. I couldn’t get the DH Rails to even create a controller, (sometime I’ll go back and figure that out), so I decided to do my own install.

    The directions here were great, but I was getting the infamous

    FastCGI: incomplete headers (0 bytes) received from server

    in my /home/myname/logs/mysite.net/http/error.log when using fcgi. I couldn’t figure out what was wrong until I ran ./dispatch.fcgi on the command line. From there I got the message:

    /home/myname/.packages/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such file to load -- fcgi (MissingSourceFile)

    After looking around, I realized I needed the fcgi.rb file and others.

    So anyhow, I found “http://osdir.com/ml/lang.ruby.ferret.general/2006-04/msg00060.html”, which explains how to install all the above that’s explained above, plus fcgi. I installed fcgi and ruby_fcgi, then did

    gem install rails --include-dependencies"

    I’m not sure if that was necessary but, in any case, then everything worked!

    Note that I use none of DH’s gems. I believe the directions in the main article body should read:

    To force your environment to use ONLY your local gems and not the Dreamhost managed gems at all, change the last line above to:

    export GEM_PATH="$GEM_HOME"

    Anyhow, hope this helps some. I’d still like to figure out why trying to use DH’s installed Rails doesn’t work. Also, I tried plain old CGI with my own Gems and it too wouldn’t work.

  • Avatar EnvyGeeks said about 1 year later:

    Ruby 1.8.6 has been moved to Stable.

    wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.6.tar.gz

  • Avatar Tom said about 1 year later:

    Any idea where rubygems.rb is supposed to end up after all of this? I end up in this state:

    > gem /home/tlianza/packages/bin/gem:8:in `require’: no such file to load—rubygems (LoadError) from /home/tlianza/packages/bin/gem:8

    And if I do a search for rubygems.rb, I end up with:

    ./packages/lib/rubygems.rb ./packages/rubygems-1.0.1/lib/rubygems.rb ./.packages/lib/rubygems.rb

    The ‘gem’ command doesn’t appear to be finding/loading any of them.

  • Avatar sammy said about 1 year later:

    thanks so much for this post, just wondering how you guys got around the problem of dreamhost killing the gem install process when you try to install your own copy of rails.

    I had trouble with it but it didn’t actually seem necessary because all I wanted to do was install my own gems and having my own copy of ruby is sufficient.

  • Avatar allie potter said about 1 year later:

    dear friend, I think you mite want to sell red,pink,blue,green,yellow or white gems and nice ones.

  • Avatar Greg said about 1 year later:

    After having made it through all of these builds, I’m having trouble actually installing gems on DH because of the hideous slowness of gem install. After a few minutes of “Bulk updating Gem source index for: http://gems.rubyforge.org” DH kills my process and my gems never install. Is there any workaround for this?

  • Avatar Adam Farnsworth said about 1 year later:

    Is anyone else getting this error when trying to install the Rails gem:

    /home/username/.packages/bin/gem:23: uninitialized constant Gem::GemRunner (NameError)

    I can’t get around it, any help would be appreciated…

  • Avatar Shados said about 1 year later:

    Ahh, this proved extremely useful to me, especially as it opened my eyes to the possibilites of overriding the systemwide-default software packages with my own local ones :P. Much gratitude to you.

  • Avatar sbeckeriv said about 1 year later:

    For rails projects I like to put all my gems in vendor. This does force me to have a copy of each gem for each rails projects. Kinda like rake freezing rails. It also prevents the soap4r error

    http://dev.rubyonrails.org/ticket/10001

    gem install hpricot -i vendor/gems

    and inside my environment.rb

    ENV[‘GEM_PATH’] = ”#{File.dirname(FILE)}/../vendor/gems” require “rubygems” Gem.clear_paths

    This comment has been flagged for moderator approval. It won't appear on this blog until the author approves it.
  • Avatar Jake said about 1 year later:

    Great post, got me straight to work with the latest Ruby, RoR and Gems.

  • Avatar sagi said over 2 years later:

    I have followed this article for my ROR hosting on Dreamhost, I am getting below an error while installing the gems

    /home/trendi_www/.packages/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:92:in `load_specification’: private method `specification_version=’ called for # (NoMethodError)

  • Avatar rev said over 2 years later:

    Im also getting the NoMethod Error, anyone help?

Trackbacks

Use the following link to trackback from your own site:
/articles/trackback/6