So it seems that running rake tasks on Dreamhost doesn't always work terribly well.

First some background.
I've got Redmine installed and have it set up so that it can process replies to notification emails. After a ton a fiddling, I finally got it working. Overall, it seemed that things worked.

And then one day... I noticed an email reply that wasn't getting processed. Ok. I'd actually noticed the problem earlier, but regardless of it not quite fitting the facts, I had just assumed that my cron job was running hourly instead of every 10 minutes. This time around, however, it had been several hours, so I figured I'd take a look. I checked the cron settings on Dreamhost, and sure enough it was set to 10 minutes. I had email notifications off since the rake task would always return "(in /home/username)".

So the first step was to figure out what was going on. I contacted support as it seemed that initially things were only working if I had email notification on. But then it started failing there too, so it wasn't because of the email address. So I updated the cron task to include STDERR in the email:
rake .... > logfile 2>&1

And viola!
It started showing me:
sh: line 1: 16870 Killed rake ....

So again, I asked DH about it, and they say that I'm going over the CPU or memory limit and tell me to go to:
http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage

Tried everything there, nothing helped.
So I try top. And what do you know... Ruby is sitting there chewing up over 100MB or ram. So whenever the cron job runs, I'm assuming that rake ends up bringing me over the threshold (~200MB) for getting my processes killed by DH's process watcher bot.

So what to do?

DH tells me to sign up for DreamhostPS. And that's exactly what I did... not!
My setup still isn't perfect, but so far it seems to be working a fair bit better.

First, I installed Todd Huss's Dreamhost Rails plugin:
http://gabrito.com/post/ruby-on-rails-dreamhost-plugin
This will help keep your website alive until it's finished the current request if it gets attacked by DH's bot.

Next, I channeled my cron job through a small php script that echos something useful if there actually is an error, otherwise it returns 0, so I no longer have notification emails when things are working as they should. It also adds a timestamp to the log file.

Last but not least, when executing rake, I set the memory limit to 10MB using ulimit, and then run rake through nice.
So the line looks something like:

$output = `ulimit -m 10240; nice -n 19 2>&1 /usr/bin/rake -f $pathToRakefile -q redmine:email:receive_imap RAILS_ENV="production" host=$host port=143 username=$username password=$password folder=INBOX move_on_failure=INBOX.failure move_on_success=INBOX.success tracker=bug priority=normal allow_override=tracker,priority`;


Hope it helps someone.