Holistic Engineering

E-Mail GitHub Twitter

16 Mar 2013
The Chef Resource Run Queue

I always cringe a little when I hear the phrase “chef scripts”, largely because it’s rather incorrect and the source of much confusion from even advanced chef users. This is an especially hard notion to defuse with consumers of chef-solo because of its very non-dynamic nature. Chef’s recipes are a way of programming a queue of actions to be run, and why this matters, I hope to make apparent over the next few pages.

The Chef term for the compiled queue is the “Resource Collection”, and it ends up being processed as linear queue with some occasional state machine tomfoolery, as we’ll see in a minute here. It’s very similar to run queues in unix kernels, especially how they relate to syscalls, with the main difference that chef is both the source of the call (the compile phase), and the executor of the request (the converge phase).

And just to be really clear this is how it always has been in chef, and probably always will be, so this topic applies for those of you still stuck on chef 0.6, all the way up to those doing the latest hotness on Chef 11.

The compile and converge phase…

A very simple chef recipe:

execute "echo he's a bad mutha..."

execute "echo shut yo mouth!" do
  only_if { ::File.exist?("/proc/mouth") }

execute "echo Just talkin' bout chef"

execute "echo we can dig it." do
  only_if { ::File.exist?("/proc/mouth") }

During the compile phase, these resources (Chef::Resource::Execute) will get arranged in a queue in order of appearance, with their default action, :run. When chef is finished compiling all the recipes (order determined by the node’s run_list), convergence happens.

During convergence, each queue item is iterated through and has it’s provider (in this case, the rather unsurprising Chef::Provider::Execute) applied to it with the action, after primitive predicates are checked – things like not_if and only_if, as we used up there. Presuming the predicates yield a true result and /proc/mouth exists, we will see four echo statements executed, and their output in the chef log.

So, more illustratively, here’s how this recipe turns into four echo statements:

  • recipe.rb is evaluated by chef-client or chef-solo
  • when each execute statement is encountered:
    • a Chef::Resource::Execute object is created.
    • this object has defaults applied to them such as the :run action and Chef::Provider::Execute provider
    • the body of the statement (the bits between do/end) are applied to the resource via some ruby evaluation magic called instance_eval
    • This object is then added to the end of the ResourceCollection’s queue.
  • after all recipes are evaluated, the compile phase has ended, and the convergence phase begins.
    • chef goes through the ResourceCollection and evaluates each resource in the queue, shifting it off as it encounters it.
    • chef determines if it can apply the provider to the resource by checking the action, and built-in predicates like not_if and only_if.
    • presuming it can apply it, it executes the action’s method in the provider, and the provider communicates back by altering the resource’s state if it did anything. This method is called updated_by_last_action and you’ll want to use it in LWRPs if you’re a good citizen.
    • at this point, any notifications or subscriptions are processed if the resource was told by the provider anything was changed.

The important parts

  • After the compile phase, the recipe no longer matters. It’s not even consulted and actually doesn’t even have to exist any more.
  • This is not a script in the traditional sense – the execution is not top to bottom, it’s two phase, and the second phase is largely responsible for what is executed, and the order it’s executed in.

Lisp vs. C macros, a digression.

You may already be familiar with how C and Lisp macros work and how they are different from each other. I’ll relate these to recipe compilation in a second, but first an explanation is needed.

Before any compiler runs, C executes cpp, or the C pre-processor, (or a derivation thereof depending on the compiler suite) to process macros. It then takes the pre-processed output and compiles that instead.


#include <stdio.h>

#define PRINT_COOL_STUFF(x) printf("not %s again!", (x))

int main(int argc, char **argv)

After running through the C pre-processor:

/* HUUUUUUGE block of shit that stdio.h put here that doesn't matter */
int main(int argc, char **argv)
  printf("not %s again!", ("meatloaf"));

Aside, not kidding about that huge block of shit – stdio.h includes other stuff, has its own macros, sometimes even printf is a macro!

The important point though is that C always deals with the result of the C pre-processor, and no C is compiled until the C pre-processor is done. The pre-processor language is its own, non-C thing and has its own quirks and ultimately ends up being a text replacement, the result of which is compiled.

Lisp macros are different. Lisp, the language, is more or less the syntax tree – while C will be reduced to a parsed form and then manipulated, there is no parsed form of Lisp in the same sense, because you’re typing it into an editor.

Lisp macros are the manipulation of the syntax tree itself, not a text file. Lisp macros are also just lisp with some special additional syntax. They also happen at compile time, but have very different implications.

Example (forgive me for form here, it’s been a while since I seriously lisped):

(defmacro print_cool_stuff (x) 
  (let y (concat x "is cool!"))
  '(print ,y))
(print_cool_stuff "chef")

The result is not very surprising, but how it gets there is a lot different:

(print "chef is cool!")

At compile time, it actually executed the lisp expression:

(let y (concat "chef" "is cool!"))

and yielded another lisp form with that substituted, and that is executed. No variables being passed around, no text files being edited. This is what the compiler gets to see, not us.

If we had run (print_cool_stuff (concat "three" "cool" "things")), the compiler would see this:

(print "three cool things is cool!")

That’s because this happened: (let y (concat (concat "three" "cool" "things") "is cool!"))

So, when you see a sweaty CS dork raving about lisp and how it’s the best language ever designed, this is usually what they’re excited about. Lisp macros really don’t exist in many other systems, because they almost impossible to do without bringing in the code-as-syntax-tree feature of lisp. Ruby just happens to emulate them pretty well due to some properties it has as a language.

Recipes are a DSL for adding Resources to a Queue

Just like with the lisp macro above, your recipe is compiled, and the result put into the queue. Convergence could be rewritten as “evaluating and re-compiling the queue” and not be that far from the truth, but we’ll discuss that in a minute.

Here’s an example of an abuse of this feature in recipes:

(0..10).each do |x|
  if x % 2 == 0
    execute "echo #{x}"

What will end up existing in the queue is:

  • echo 0
  • echo 2
  • echo 4

… and so on up to 10. The odd execute resources were never created, and thus, never added.

This gets less obvious when you use a node attribute:

(0..10).each do |x|
  if x % 2 == node["echo_modulo"]
    execute "echo #{x}"

What gets added here? It depends entirely on the value of node["echo_modulo"] at this point. While this example may seem trivial, consider something that uses a case over node["platform"] and how that might affect the queue.

Chef’s dirty secret: Convergence is also Compilation

Notifications and Subscriptions are great examples of a secondary compilation step that happens during convergence.

This recipe:

execute "echo foo" do
  action :nothing

execute "echo bar" do
  action :nothing
  subscribes :run, "execute[echo foo]", :delayed
  notifies :run, "execute[echo quux]", :immediately

execute "echo quux" do
  action :nothing
  notifies :run, "execute[echo baz]", :immediately

execute "echo baz" do
  notifies :run, "execute[echo foo]", :immediately

Looks like this when run:


But after compile, the queue looks like this and is evaluated in this order:


What happened here? Two things:

  • An item in the queue is a constructed object with two parts:
    • The resource
    • The action
  • notifies and subscribes modify the queue, and the position is dependent on the third argument.
    • :delayed adds to the end of the queue, so it is the last thing executed.
    • :immediately adds to the head of the queue, so it the next thing executed.

The first bit there is really important – if if were not the case, the queue would have no idea that echo baz had already run, and would run it again as soon as it was notified to do so from echo bar.

Additionally, we learn here that not only is the ResourceCollection a queue of things to act upon, but a registry of resources (and their states) that can be referred to later, at convergence time, with the result being more compilation of the queue.

Ok, so what can I do with all this?

Anything you want, really. The dynamic nature of recipes is what makes them more powerful than puppet manifests or ansible playbooks; otherwise, they are not that much different.

I have a silly project I banged out over an afternoon that takes it to its logical absurd extreme: Tyler Perry’s Chefception. It’s a REST service that stores JSON blobs in a sqlite database, then uses a provider to evaluate those into resources and run them as a part of the run queue. The idea being that you could do ad hoc resource management with curl, or your favorite HTTP library, without having to care too much about cookbooks. Of course, this is probably a pretty horrible idea and shouldn’t be used by anyone for anything, but it was fun to write.

There’s a lot you can do, but just remember that a recipe and script are two very different things!

Til next time,
Erik Hollensbe at 07:21


E-Mail GitHub Twitter