scribble

Holistic Engineering

E-Mail GitHub Twitter

15 May 2013
Gem Activation and You, Part 2: Bundler and Bin Stubs

Most of this article will expect some basic familiar with Bundler, and Gem Specifications, Activation, and Dependencies.

There’s been a bit of discussion in the past about Bundler, when it should be used, when it should not be used. Yehuda Katz has written extensive commentary on the topic, and you should read that, but this will discuss how Bundler works and what it’s good at solving for you.

I’m going to boil it down, and will elaborate implicitly later:

  • If you have a standalone executable for others to use, Bundler can make dependency problems less obvious by hiding dependencies which are valid according to your gemspec, but broken. You should at least ensure it’s operating without bundler before releasing it.
  • If you have a project directory, use Bundler extensively, and set everything you use in the Gemfile. Use bundle exec extensively.
    • Consequently, if you’re developing a gem, that is your project directory. Just don’t check in the Gemfile.lock, but still use bundle exec like it’s going out of style. Routinely bundle update or set arbitrary hard dependencies in your Gemfile to ensure your gem plays well with others.

Why all this?

Because even though we’ve been using Semantic Versioning long before Tom Preston-Werner wrote his treatise on the subject, you still have to play ball with a lot of people. A lot of people don’t use Semantic Versioning.

Ivory Towers are for people who never get dirty; ignore the real world at your own peril. Bundler is a tool for assisting you with dealing with the real world. Just like you have things like the CGI specification, and HTTP, Rails is there to assist by putting XSS and CSRF protection – things you need for modern web programming. RubyGems is the basics, and Bundler is the cherry on the top to assist with real world application problems.

That said, using bundler liberally can hide certain classes of problems, or empower you to discover them.

How does Bundler work?

Let’s start with a quick note on what Gem Requirements are first. So, a Gem Requirement is a specification of a version, such as >= 0, which always means the latest version, or ~> 1.2.3, which means anything >= 1.2.3 but also anything <= 1.3.0. Gem Requirements have a few operators which have basic code mappings. You should read them.

How Bundler works, in a nutshell: For a given Gemfile, Bundler will use the latest version of everything that fits the default Gem Requirement (the default requirement being >= 0), and given any conflicts, slowly reduces the value of each Gem’s version until it violates the Gemfile's Requirement or the Specification’s Requirements. Presuming it’s able to solve the formula, it spits out a Gemfile.lock which contains what conclusion it came to. If not, it tells you where the conflict lies.

Let’s see this in action

As mentioned in the previous article, both the chef and vagrant 1.0.x gems do not play nicely together on a dependency level. However, if you’re willing to accept chef 10.18.2 instead of the latest hotness, 11.4.4, you can use it with vagrant 1.0.

Here’s an example Gemfile to play with:

gem 'chef'
gem 'vagrant', '= 1.0.7'

Put that in a directory and run bundle. You should see something like this:

Using archive-tar-minitar (0.5.2) 
Using bunny (0.7.9) 
Using erubis (2.7.0) 
Using highline (1.6.18) 
Using json (1.5.4) 
Using mixlib-log (1.6.0) 
Using mixlib-authentication (1.3.0) 
Using mixlib-cli (1.3.0) 
Using mixlib-config (1.1.2) 
Using mixlib-shellout (1.1.0) 
Using moneta (0.6.0) 
Using net-ssh (2.2.2) 
Using net-ssh-gateway (1.1.0) 
Using net-ssh-multi (1.1) 
Using ipaddress (0.8.0) 
Using systemu (2.5.2) 
Using yajl-ruby (1.1.0) 
Using ohai (6.16.0) 
Using mime-types (1.23) 
Using rest-client (1.6.7) 
Using polyglot (0.3.3) 
Using treetop (1.4.12) 
Using uuidtools (2.1.4) 
Using chef (10.18.2) 
Using ffi (1.8.1) 
Using childprocess (0.3.9) 
Using i18n (0.6.4) 
Using log4r (1.1.10) 
Using net-scp (1.0.4) 
Using vagrant (1.0.7) 
Using bundler (1.3.5) 
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.

Notice how we’ve specified the latest version of chef, but we got 10.18.2 instead? This is because of the net-ssh dependencies they share – 10.18.2 depends on a version of net-ssh that vagrant is ok with, so Bundler, to solve the formula, rolls our chef back.

Change your Gemfile to look like this:

gem 'chef', '~> 11.0'
gem 'vagrant', '= 1.0.7'

This sets chef to have a minimum version of 11.0 but not as high as 12.0. Run bundle update. You will see this:

Resolving dependencies...
Bundler could not find compatible versions for gem "net-ssh":
  In Gemfile:
    chef (~> 11.0) ruby depends on
      net-ssh (~> 2.6) ruby

    vagrant (= 1.0.7) ruby depends on
      net-ssh (2.2.2)

Voila! We have a constraint violation on net-sshchef depends on 2.6 or better, and vagrant just isn’t going to let that happen. If you read the first article, you’ll notice this is the same constraint violation we saw before.

Bin Stubs, or how those command-line tools get run.

Now that we understand how Bundler works, let’s have fun with tools like rake or thor or gist. These are tools you commonly would run outside of a bundled environment, but still have consequences within the RubyGems system.

This is because they correspond to activated gems, and what gems are activated largely depends on what gets installed. The scripts you actually run are called “bin stubs”, or little scripts that look a lot like this (this one’s for rake):

[15] erikh@speyside ~/tmp% cat `which rake`
#!/usr/bin/env ruby
#
# This file was generated by RubyGems.
#
# The application 'rake' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

version = ">= 0"

if ARGV.first
  str = ARGV.first
  str = str.dup.force_encoding("BINARY") if str.respond_to? :force_encoding
  if str =~ /\A_(.*)_\z/
    version = $1
    ARGV.shift
  end
end

gem 'rake', version
load Gem.bin_path('rake', 'rake', version)

There’s that gem call again! If you notice, it’s parsing the version out from _version_, and activating that version or the latest version if omitted (because version will be nil). This is just like the example from the first article.

This means if you have rake 0.9.6 and rake 10.0.0, by default, rake 10.0.0 will be run. However, if you do this:

rake _0.9.6_ my_target

0.9.6 will be run instead. The point is, the script is there to facilitate this, and gem activation in general. The importance of these notions will be important for our next part…

Why bundle exec is really really really really important for your bundled projects

Type bundle gem foo – this will create a project skeleton for a gem called foo. It will generate in the foo directory a few files, including a Gemfile, a Rakefile, and a foo.gemspec.

Let’s add something to that Rakefile. How about this at the end?

require 'json'
p JSON::VERSION

And this to the foo.gemspec in the right spot:

spec.add_dependency 'json', '= 1.5.4'

Then gem install json to get the latest version, then bundle install.

If we type the command to get the list of tasks, rake -T, we should see something like this:

"1.7.7"
rake build    # Build foo-0.0.1.gem into the pkg directory.
rake install  # Build and install foo-0.0.1.gem into system gems.
rake release  # Create tag v0.0.1 and build and push foo-0.0.1.gem to Rubygems

What? We just told bundler to use 1.5.4! Bundler never got considered here. The tool, bundle exec was created to ensure that all activations happen under the watchful eye of bundler.

Type bundle exec rake -T and see how this changes:

"1.5.6"
rake build    # Build foo-0.0.1.gem into the pkg directory.
rake install  # Build and install foo-0.0.1.gem into system gems.
rake release  # Create tag v0.0.1 and build and push foo-0.0.1.gem to Rubygems

Now, if there were conflicting gems on your machine that you would require, or just want to make sure you have the right version, running without bundle exec ensures that’s possible. This is a great thing for one-off commandline tools, but not so great for applications, or projects in general. If you develop commandline tools, you should test with and without bundler to ensure the behavior in the presence of other dependencies is desired.

RubyGems 2.0 can use Gemfiles

Bundler can solve a whole host of constraint problems, but RubyGems 2.0 now considers Gemfiles as well; this actually made the above example a lot harder to do that it has been before as bundle exec is not nearly as necessary anymore. Still, to be on the safe side, you should use it for now.

Conclusion

Bundler, Bin Stubs and RubyGems all work together to create a smart system at the cost of a little cognitive dissonance – the expectation that there should be one source of truth is honored, but it is evaluated amongst many truths in relationship to its own requirements. When you don’t care, it’s great. When you do, you have this article to help you figure out what to do. :)

Stay tuned for Part 3, where we discuss packaging RubyGems with other packaging systems.


Til next time,
Erik Hollensbe at 03:03

scribble

E-Mail GitHub Twitter