29 Mar 2013
Rubygems, OpenSSL, and you
The last month or two has been a pretty tumultuous time for Rubygems, the Rubygems.org servers, and Bundler itself. This post is my attempt to summarize the history of what happened and make some concrete suggestions for rubyists that care about using Rubygems or Bundler (which is probably most of them). The practical upshot is that you should upgrade your installation of Rubygems to version 2.0.3 or later, and your Bundler gem to at least 1.3.4. If you’re interested to hear why, keep reading.
This year has been a rough for Ruby and Rails developers. The last three months have included almost as many security issues as all of 2012 and 2011 combined. Just a couple of months ago, the Rubygems.org server was compromised and had to be rebuilt from scratch. In the midst of all the security hullabaloo, the Rubygems and Bundler teams have also been working to support all of the changes made in Ruby 2.0 by the time it reached its first release on February 24.
Developers began voicing concerns about security shortly before Bundler 1.3 and Rubygems 2.0 were released in order to support Ruby 2. In addition to Ruby 2 support, Bundler 1.3 began to verify the SSL certificate provided by rubygems.org, which 1.2 did not do. It also deprecated the default :rubygems source from version 1.0 in an attempt to increase visibility into the connection method Bundler uses, and to encourage HTTPS instead of HTTP without breaking backwards compatibility.
Unfortunately, the initial 1.3 release was not fully compatible with installations of Ruby compiled without OpenSSL. Even worse, it was unable to verify the rubygems.org SSL certificate on machines with older OpenSSL CA certificates, which includes many “stable” OS releases used on production servers. I wrote error-handling code that should have at very least explained what the problem was, but one if my assumptions about exceptions was wrong, and so the explanatory message was not even displayed. Resolving these issues took time and cooperation from helpful users, and that left a lot of people in the frustrating position of seeing unexplained OpenSSL certificate verification errors every time they tried to run bundle install.
Compounding these issues, a Rubygems bugfix release around the same time included a bug in certificate validation, causing OpenSSL exceptions similar to those seen in Bundler. The ultimate solution that both Rubygems and Bundler settled on is to include the well-known CA certificates needed to verify SSL connections to rubygems.org. Those built-in certificates are used as a fallback on machines with older CA certificates installed, preventing the OpenSSL errors that people were seeing.
Finally, Rubygems 2.0.2 included, with the best intentions, a change to the code that connected to gem servers. When a server was configured to use HTTP, Rubygems would first attempt to connect via HTTPS, and only use HTTP if that failed. That change resulted in additional intermittent failures in Bundler, but only for those running the very latest Rubygems. Confusing things even more, the certificate for api.rubygems.org actually expired while all this was going on. That caused a rash of legitimate OpenSSL certificate validation exceptions, but they were mostly drowned out by the noise of everything else that was going on simultaneously.
Eventually, api.rubygems.org got a valid certificate, Bundler 1.3.4 was released with improved HTTPS support and built-in CA certificates, and Rubygems 2.0.3 was released with built-in CA certificates but without automatic HTTPS. Things should now be functional whether your Ruby was built against OpenSSL or not. Additionally, connections to rubygems.org should not cause certificate validation exceptions, even if your system has older CA certificates.
Making sweeping changes to fundamental Ruby infrastructure is hard, especially when we need to keep things compatible for users who haven’t been able to upgrade other parts of their stack yet. It’s easy to make helpful changes without realizing the potential problems, especially when prereleases and release candidates aren’t tested by very many people. If you care about this stuff, we’d love to have your help. Talk to us in #bundler and #rubygems on irc.freenode.net, or read the Bundler contributing guide for a tour of the helpful things that anyone can do.
Special thanks to Josh Kalderimis for inspiring me to write this post with his confusion about what the hell was going on, anyway.