Xen and the Art of Rails Deployment

Xen and the Art of Rails Deployment

Ezra Zygmuntowicz - EngineYard.com

Founder of Engine Yard.

Deploying Rails has changed a lot over the last few years. Servers change, but the basics remain the same. CGI to Apache/mod_fastcgi, Lighttpd/fcgi, lighttpd/scgi, lightspeed none of them are great.

Mongrel is the way to do rails deployment.

What is Mongrel?

HTTP Server Librbrary written by Zed Shaw.

• Fast HTTP Paarser
• Fast URI Classifier
• Ragel State machine parser with provable correctness

HTTP is a well kown and well tooled protocol, Easier to setup, transparent wire protocol

But Rails isn’t thread safe, so one concurrent request per mongrel.

You can use a bunch of different servers to front your mongrel cluster.

Nginx:

Russian web server focused on performance. Small resource footprint, stands up under the heaviest loads without leaking memory. Killer rewrite and proxy modules. Has an approachable author and growing community.Direct hit to memcache.

Nginx + Mongrel

• his favorite stack
• blazing fast
• flexible and readable config

Gotchas:

• Buffers uploads so no mongrel)upload_progress. Great performance improvement, but no progress bars.
• No connection rate limiting for proxy module.

Dropping mod_rewrite, adding httpscript_module that embeds NekoVM directly in nginx.

Perfect Simple Stack:

• Linux
• Nginx
• Mongrel(mongrel)cluster)
• Monit

Swiptiply: Evented Mongrel

• Hot patches ruby to Removes Ruby’s Thread’s and Socket handling from MOngrel Core replacing them with an EventMachine loop. So Mongrel becomes single threaded and event driven.
• Ruby uses green threads which have a huge context switching overhead.
• Mutex threads are expensive.

Event driven higher throughput because of the less overhead of green thread context switching. I don’t follow all of this, I will have to read up on it.

Swiptiply: Proxy

faster than haproxy. Mongrels connect to the proxy server instead of the other way around. They keep an open, http connection that reduces overhead.

You can start and stop as many mongrels as you want td they get auto configured in the proxy!

The Zen of Xen

Virtualization is the future.

• Old Linux way wsa dedicated boxes running all services in one big hodgepodge on one kernel.
• New school is sharply targeted virtualized linux with each VM running just a single tier or service. This allows you to scale much easier, because all you need to do is move VM’s / allocate resources not redo architecture.
• Don’t think of physical machines, think of the abstraction of the machine.
• If you have a perf issue you can easily find what service is the bottleneck.

Scaling to multiple boxes

• Add a new compute node with Xen installed.
• Pick the services that need more resources and migrate them LIVE to the other machine.
• Much easier to scale.

Description of EngineYard’s setup. Pretty cool. I don’t know much about that infrastructure stuff so I don’t have much of an opinion.

What you end up with is a Fabric of Computing and Storage. So you get cahing across instances, easy deployment, easy scaling up and scaling down.

Most Rails apps are RAM bound way before they are CPU bound.

• Rmagick abnd :include are the worst culprits
• 95% of Rail apps will leak memory at one pint or another
• He’s talking about lack of tooling to monitor memory usage, I think this is where the DTrace stuff will be awesome.

Friend relationships in social networks are a good example of a w

Rails eats DBs

• Indexes are your friends learn when and where to apply indexes.
• Check your logs for the activerecord generated sql, a little optimization can go a long way.

Tips and tricks:

• don’t ever use file system sessions.

script/runner is crazy ineficient. you don’t necesarrily want to store all of rails in your background tasks. Think about doing it in plain ruby with the db library.

Don’t use script/runner to process incoming emails. Run a daemon in a loop that plls the mail server with net/pop3 net/imap. Otherwise you’ll fork a whole Rails process ofr each incoming email this will NEVER work in a production environment.

Rails lives the 80/20 rule, but if you need that las 20% you need to learn to write custom mongrel handlers. Figure out when optimization isn’t premature.

Caching is the key. You can’t get faster than cached static html Memcache is your friend.Don’t store your whole AR object, just store the attributes hash.

Parting Thought: When it comes to performance test and benchmark changes for yourself to be sure. Trust what people say, but verify it to be safe.

Questions

Is the event driven mongrel available?

Yes, see Swiptiply.