Friday 11 January 2013

Bundler is cool and all, but I like YUM better.

Managing Ruby, its different versions and gems is kinda funny thing, funny as in it has many challenges.
First of all, you might want to handle different projects that work with different Ruby versions and each have a different gem set, so RVM is born and you can change these settings with ease, one less problem - although I do have a co-worker that so far failed to install certain gems with it.

Then there's one of the most annoying problems in Ruby, say your project uses gem "a 1.1" and when you try to deploy with "gem install a" you get version 1.2 and this for some reason causes your application to fail and it fails hard! It's one of the hardest errors to pin-point. Now multiply this problem for each gem and it's dependencies you use, it can really hurt.

That's why most Ruby developers (I use this term broadly, including anyone that programs in Ruby in any way) I know use Bundler and it's great, you get a list of all the gems you have on your system and it freezes the versions you know that make your application works and when you try to deploy it simply installs exactly the same gems you have on your devel machine so you don't have any gem difference problem.
But then you get gems with extensions and they don't quite compile, sure you spend some time doing "bundler install" and then "yum/apt-get install [some lib dependency]" and eventually you're there and all is good. Except if you have some nasty gem like ruby-oci8 which can be very annoying to install and not quite  as they say.
If I still got your attention so far, perhaps it's worth noting that my biggest gripe with these solutions is that they don't solve this problem of gem's unspoken dependencies.

Then you use RVM and Bundler together and all your problems are solved and little or no worries on your side. Although I do hear on occasion that those two have a few kinks working like that, but nothing major.

Then there's what's quickly becoming my favorite way of dealing with gems dependencies and so such: YUM!
It started first as an internal rule of the company I work for, the simple version: all software developed within must be packed in a RPM (for RHEL), and all of it's dependencies, including externally developed libraries (such as gems) must also be packed and available in the company's YUM server.
The reasoning was simple, once the software is sold, one must be able to go to the client's server and do "yum install software" and it will just work. (Later I heard this was because one account of failed installation caused the company to loose millions due to some contract provisions I don't really care to understand.)
While I did find it annoying going through these hops at first, I've been growing into this system and I'm grateful that they actually gave my training on software packaging (which is kinda uncommon for companies in Portugal).

YUM allows us to do some pretty fun things, such as, on a brand new server, I can use yum install to deploy my software with just "yum install" and yes, this does solve my problem of unspoken dependencies.

Take for example, an attempt to install my team's testing library on a brand new ruby installation on RHEL 6:


Notice how the first package in the list is NOT a gem, but rather an unspoken dependency for ruby-oci8. Pretty cool uh?
That also means that I have a greater control of all little things that get installed on the system for my app to work and also that I can purge it with considerable less effort. For instance, "yum remove ruby*" will remove ruby, ruby devel, rubygems, ruby libs, irb, all the rubygems and everything that was installed a dependency of any of those.
Sure RVM let's you easily switch between gem sets, but I can easily switch between "everything sets" - of course that is if I don't mind downloading them all again.
Also, with this you can apply patches to gems that are broken and abandoned despite still being popular.

There's a price for that, like everything in that area, it's knowledge, maybe the knowledge of packing is greater than the knowledge needed for Bundler + RVM, but to fix that, there's a tool, it's called "rgem2rpm" (which I use so much that the latest version, 1.4.9, is the release of a pull request I did on Github), with it, you can simply "rgem2pm gemname.gem" and you get a nifty rpm file. You get a few more options to customize it to your needs, and in the case of gems with unspoken dependencies you can pass an easy to make spec template file (you can use the default template as base for it) and add "requires".
When I took the above screenshot I noticed something was amiss, turns out, the testing version of rubygem-headless that we had on repository was not properly packed, so with 5 minutes to make a template file I compiled a new one and voilá:


See how much stuff you truly need to run the gem headless? Well to be fair, the only true dependency is "xorg-x11-server-Xvfb", all others are dependencies of that.

I know what you might be thinking now, for this to work, you need a yum repo with a lot of gems, at least all the gems you use (and their dependencies), well, yes, right now, I believe that we have more ruby gems as rpm then Fedora's Koji and OpenSuse combined (well, combined is probably an overstatement, but I'm confident we have more than each of them, at least).

All of this is packed for RHEL, they would work on Fedora (and probably others) I'm confident, but not by design.
I have in mind setting up a YUM server and compiling all these gems to Fedora, and providing a way for those using Centos and OpenSuse to use it as well, even when though I think there's little public interest in this, it will would make the setup of my devel computers (and co-workers') much, much quicker.

Edit: I almost forgot to mention, for most people, Ruby is a tool for either Rails or Sinatra web development, with a little bit of effort you can pack your web app into an  rpm and deploy on the production machine with YUM install, then have it start restart the web server (and whatever more you need for it to work, such as running database queries and whatnot) and launch your automated tests.
The more I use it, the more I feel that YUM is a very powerful tool that many neglect in favor of other tools that each do a part of what yum does alone.

PS: I do imagine that APT, PACKMAN, PACMAN and other package managers for the many Linux families are just as capable, but since I started making RPM's for RedHat and that's what I have to deal in a daily basis, I'm bias towards YUM.


4 comments:

  1. Great article Arthur, thanks for sharing this experience.

    I've been thinking myself in doing something similar with APT/Ubuntu, including the stuff you said about packaging the application to restart the server and stuff.

    I found rgem2rpm pretty easy, but maybe later you can do another article more like a tutorial. Anyway, congratulations. :)

    ReplyDelete
    Replies
    1. Glad you liked it.

      What kind of tutorial you're thinking of? rgem2rpm usage or general rpm packing?

      Delete
  2. We're deploying our Rails app (http://www.katello.org/) through RPMs and as a guy coming from RVM+bundler workflow to this, I consider it as a big win and agree with this article. Thanks for putting this together.

    A thing however, that one needs to be aware of, is that bundler is your biggest enemy in this workflow: it generates Gemfile.lock, with version numbers bundled there. That means updating some dependency through yum breaks your app, unless you remember to delete the Gemfile.lock. Not cool.

    We solved the problem with using bundler_ext (https://github.com/aeolus-incubator/bundler_ext), that uses the Gemfile as well, but does only require thing of bundler, and leavs the rest for right tool (yum).

    Another thing is, for development, RVM + bundler is not useless. But there is still the problem with external dependencies (gem install nokogiri, then grrr. what other deps do I need?). This project tries to solve the issue (and works pretty good for me) https://github.com/voxik/gem-nice-install. For now, it supports yum based systems, but it's meant to be extensible.

    ReplyDelete
    Replies
    1. That bunlder being the greatest enemy was the first thing I noticed, when they told us we had to use yum we abandoned bundler completely on the first day.

      Nice links, I'll look into them

      Delete