15 May 2013
Gem Activation and You, Part 2: Bundler and Bin Stubs
Most of this article will expect some basic familiar with Bundler, and Gem Specifications, Activation, and Dependencies.
There’s been a bit of discussion in the past about Bundler, when it should be used, when it should not be used. Yehuda Katz has written extensive commentary on the topic, and you should read that, but this will discuss how Bundler works and what it’s good at solving for you.
I’m going to boil it down, and will elaborate implicitly later:
- If you have a standalone executable for others to use, Bundler can make dependency problems less obvious by hiding dependencies which are valid according to your gemspec, but broken. You should at least ensure it’s operating without bundler before releasing it.
- If you have a project directory, use Bundler extensively, and set everything
you use in the
- Consequently, if you’re developing a gem, that is your project directory.
Just don’t check in the
Gemfile.lock, but still use
bundle execlike it’s going out of style. Routinely
bundle updateor set arbitrary hard dependencies in your
Gemfileto ensure your gem plays well with others.
- Consequently, if you’re developing a gem, that is your project directory. Just don’t check in the
Why all this?
Because even though we’ve been using Semantic Versioning long before Tom Preston-Werner wrote his treatise on the subject, you still have to play ball with a lot of people. A lot of people don’t use Semantic Versioning.
Ivory Towers are for people who never get dirty; ignore the real world at your own peril. Bundler is a tool for assisting you with dealing with the real world. Just like you have things like the CGI specification, and HTTP, Rails is there to assist by putting XSS and CSRF protection – things you need for modern web programming. RubyGems is the basics, and Bundler is the cherry on the top to assist with real world application problems.
That said, using bundler liberally can hide certain classes of problems, or empower you to discover them.
How does Bundler work?
Let’s start with a quick note on what Gem Requirements are first. So, a Gem
Requirement is a specification of a version, such as
>= 0, which always means
the latest version, or
~> 1.2.3, which means anything
>= 1.2.3 but also
<= 1.3.0. Gem Requirements have a
few operators which have basic code mappings.
You should read them.
How Bundler works, in a nutshell: For a given
Gemfile, Bundler will use the
latest version of everything that fits the default Gem Requirement (the default
>= 0), and given any conflicts, slowly reduces the value of
each Gem’s version until it violates the
Gemfile's Requirement or the
Specification’s Requirements. Presuming it’s able to solve the formula, it
spits out a
Gemfile.lock which contains what conclusion it came to. If not,
it tells you where the conflict lies.
Let’s see this in action
As mentioned in the previous article, both the
gems do not play nicely together on a dependency level. However, if you’re
willing to accept
10.18.2 instead of the latest hotness,
can use it with
Here’s an example Gemfile to play with:
gem 'chef' gem 'vagrant', '= 1.0.7'
Put that in a directory and run
bundle. You should see something like this:
Using archive-tar-minitar (0.5.2) Using bunny (0.7.9) Using erubis (2.7.0) Using highline (1.6.18) Using json (1.5.4) Using mixlib-log (1.6.0) Using mixlib-authentication (1.3.0) Using mixlib-cli (1.3.0) Using mixlib-config (1.1.2) Using mixlib-shellout (1.1.0) Using moneta (0.6.0) Using net-ssh (2.2.2) Using net-ssh-gateway (1.1.0) Using net-ssh-multi (1.1) Using ipaddress (0.8.0) Using systemu (2.5.2) Using yajl-ruby (1.1.0) Using ohai (6.16.0) Using mime-types (1.23) Using rest-client (1.6.7) Using polyglot (0.3.3) Using treetop (1.4.12) Using uuidtools (2.1.4) Using chef (10.18.2) Using ffi (1.8.1) Using childprocess (0.3.9) Using i18n (0.6.4) Using log4r (1.1.10) Using net-scp (1.0.4) Using vagrant (1.0.7) Using bundler (1.3.5) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
Notice how we’ve specified the latest version of
chef, but we got
instead? This is because of the
net-ssh dependencies they share –
depends on a version of
vagrant is ok with, so Bundler, to
solve the formula, rolls our chef back.
Change your Gemfile to look like this:
gem 'chef', '~> 11.0' gem 'vagrant', '= 1.0.7'
chef to have a minimum version of
11.0 but not as high as
bundle update. You will see this:
Resolving dependencies... Bundler could not find compatible versions for gem "net-ssh": In Gemfile: chef (~> 11.0) ruby depends on net-ssh (~> 2.6) ruby vagrant (= 1.0.7) ruby depends on net-ssh (2.2.2)
Voila! We have a constraint violation on
chef depends on
or better, and
vagrant just isn’t going to let that happen. If you read the
first article, you’ll notice this is the same constraint violation we saw
Bin Stubs, or how those command-line tools get run.
Now that we understand how Bundler works, let’s have fun with tools like
gist. These are tools you commonly would run outside of a
bundled environment, but still have consequences within the RubyGems system.
This is because they correspond to activated gems, and what gems are
activated largely depends on what gets installed. The scripts you actually run
are called “bin stubs”, or little scripts that look a lot like this (this one’s for
 erikh@speyside ~/tmp% cat `which rake` #!/usr/bin/env ruby # # This file was generated by RubyGems. # # The application 'rake' is installed as part of a gem, and # this file is here to facilitate running it. # require 'rubygems' version = ">= 0" if ARGV.first str = ARGV.first str = str.dup.force_encoding("BINARY") if str.respond_to? :force_encoding if str =~ /\A_(.*)_\z/ version = $1 ARGV.shift end end gem 'rake', version load Gem.bin_path('rake', 'rake', version)
gem call again! If you notice, it’s parsing the version out from
_version_, and activating that version or the latest version if omitted
version will be nil). This is just like the example from the first
This means if you have rake
0.9.6 and rake
10.0.0, by default,
10.0.0 will be run. However, if you do this:
rake _0.9.6_ my_target
0.9.6 will be run instead. The point is, the script is there to facilitate
this, and gem activation in general. The importance of these notions will be
important for our next part…
bundle exec is really really really really important for your bundled projects
bundle gem foo – this will create a project skeleton for a gem called
foo. It will generate in the
foo directory a few files, including a
Rakefile, and a
Let’s add something to that
Rakefile. How about this at the end?
require 'json' p JSON::VERSION
And this to the
foo.gemspec in the right spot:
spec.add_dependency 'json', '= 1.5.4'
gem install json to get the latest version, then
If we type the command to get the list of tasks,
rake -T, we should see something like
"1.7.7" rake build # Build foo-0.0.1.gem into the pkg directory. rake install # Build and install foo-0.0.1.gem into system gems. rake release # Create tag v0.0.1 and build and push foo-0.0.1.gem to Rubygems
What? We just told bundler to use 1.5.4! Bundler never got considered here. The
bundle exec was created to ensure that all activations happen under the
watchful eye of bundler.
bundle exec rake -T and see how this changes:
"1.5.6" rake build # Build foo-0.0.1.gem into the pkg directory. rake install # Build and install foo-0.0.1.gem into system gems. rake release # Create tag v0.0.1 and build and push foo-0.0.1.gem to Rubygems
Now, if there were conflicting gems on your machine that you would require, or
just want to make sure you have the right version, running without
exec ensures that’s possible. This is a great thing for one-off commandline
tools, but not so great for applications, or projects in general. If you
develop commandline tools, you should test with and without bundler to ensure
the behavior in the presence of other dependencies is desired.
RubyGems 2.0 can use Gemfiles
Bundler can solve a whole host of constraint problems, but RubyGems 2.0 now
considers Gemfiles as well; this actually made the above example a lot harder
to do that it has been before as
bundle exec is not nearly as necessary
anymore. Still, to be on the safe side, you should use it for now.
Bundler, Bin Stubs and RubyGems all work together to create a smart system at the cost of a little cognitive dissonance – the expectation that there should be one source of truth is honored, but it is evaluated amongst many truths in relationship to its own requirements. When you don’t care, it’s great. When you do, you have this article to help you figure out what to do. :)
Stay tuned for Part 3, where we discuss packaging RubyGems with other packaging systems.
Til next time,
Erik Hollensbe at 03:03