I always cringe a little when I hear the phrase “chef scripts”, largely because
it’s rather incorrect and the source of much confusion from even advanced chef
users. This is an especially hard notion to defuse with consumers of
chef-solo because of its very non-dynamic nature. Chef’s recipes are a way of
programming a queue of actions to be run, and why this matters, I hope to make
apparent over the next few pages.
The Chef term for the compiled queue is the “Resource Collection”, and it ends up being processed as linear queue with some occasional state machine tomfoolery, as we’ll see in a minute here. It’s very similar to run queues in unix kernels, especially how they relate to syscalls, with the main difference that chef is both the source of the call (the compile phase), and the executor of the request (the converge phase).
And just to be really clear this is how it always has been in chef, and probably always will be, so this topic applies for those of you still stuck on chef 0.6, all the way up to those doing the latest hotness on Chef 11.
The compile and converge phase…
A very simple chef recipe:
1 2 3 4 5 6 7 8 9 10 11
During the compile phase, these resources (Chef::Resource::Execute) will get
arranged in a queue in order of appearance, with their default action,
When chef is finished compiling all the recipes (order determined by the node’s
run_list), convergence happens.
During convergence, each queue item is iterated through and has it’s provider
(in this case, the rather unsurprising Chef::Provider::Execute) applied to it
with the action, after primitive predicates are checked — things like
only_if, as we used up there. Presuming the predicates yield a true
/proc/mouth exists, we will see four
echo statements executed,
and their output in the chef log.
So, more illustratively, here’s how this recipe turns into four echo statements:
recipe.rbis evaluated by
- when each
executestatement is encountered:
Chef::Resource::Executeobject is created.
- this object has defaults applied to them such as the
- the body of the statement (the bits between do/end) are applied to the
resource via some ruby evaluation magic called
- this object has defaults applied to them such as the
- This object is then added to the end of the ResourceCollection’s queue.
- after all recipes are evaluated, the compile phase has ended, and the
convergence phase begins.
- chef goes through the ResourceCollection and evaluates each resource in
the queue, shifting it off as it encounters it.
- chef determines if it can apply the provider to the resource by checking
action, and built-in predicates like
- presuming it can apply it, it executes the
action’s method in the provider, and the provider communicates back by altering the resource’s state if it did anything. This method is called
updated_by_last_actionand you’ll want to use it in LWRPs if you’re a good citizen.
- at this point, any notifications or subscriptions are processed if the resource was told by the provider anything was changed.
- chef determines if it can apply the provider to the resource by checking the
- chef goes through the ResourceCollection and evaluates each resource in the queue, shifting it off as it encounters it.
The important parts
- After the compile phase, the recipe no longer matters. It’s not even consulted and actually doesn’t even have to exist any more.
- This is not a script in the traditional sense — the execution is not top to bottom, it’s two phase, and the second phase is largely responsible for what is executed, and the order it’s executed in.
Lisp vs. C macros, a digression.
You may already be familiar with how C and Lisp macros work and how they are different from each other. I’ll relate these to recipe compilation in a second, but first an explanation is needed.
Before any compiler runs, C executes
cpp, or the C pre-processor, (or a
derivation thereof depending on the compiler suite) to process macros. It then
takes the pre-processed output and compiles that instead.
1 2 3 4 5 6 7 8 9
After running through the C pre-processor:
1 2 3 4 5 6
Aside, not kidding about that huge block of shit — stdio.h includes other
stuff, has its own macros, sometimes even
printf is a macro!
The important point though is that C always deals with the result of the C pre-processor, and no C is compiled until the C pre-processor is done. The pre-processor language is its own, non-C thing and has its own quirks and ultimately ends up being a text replacement, the result of which is compiled.
Lisp macros are different. Lisp, the language, is more or less the syntax tree — while C will be reduced to a parsed form and then manipulated, there is no parsed form of Lisp in the same sense, because you’re typing it into an editor.
Lisp macros are the manipulation of the syntax tree itself, not a text file. Lisp macros are also just lisp with some special additional syntax. They also happen at compile time, but have very different implications.
Example (forgive me for form here, it’s been a while since I seriously lisped):
1 2 3 4
The result is not very surprising, but how it gets there is a lot different:
At compile time, it actually executed the lisp expression:
(let y (concat "chef" "is cool!"))
and yielded another lisp form with that substituted, and that is executed. No variables being passed around, no text files being edited. This is what the compiler gets to see, not us.
If we had run
(print_cool_stuff (concat "three" "cool" "things")), the
compiler would see this:
(print "three cool things is cool!")
That’s because this happened:
(let y (concat (concat "three" "cool" "things")
So, when you see a sweaty CS dork raving about lisp and how it’s the best language ever designed, this is usually what they’re excited about. Lisp macros really don’t exist in many other systems, because they almost impossible to do without bringing in the code-as-syntax-tree feature of lisp. Ruby just happens to emulate them pretty well due to some properties it has as a language.
Recipes are a DSL for adding Resources to a Queue
Just like with the lisp macro above, your recipe is compiled, and the result put into the queue. Convergence could be rewritten as “evaluating and re-compiling the queue” and not be that far from the truth, but we’ll discuss that in a minute.
Here’s an example of an abuse of this feature in recipes:
1 2 3 4 5
What will end up existing in the queue is:
- echo 0
- echo 2
- echo 4
… and so on up to 10. The odd
execute resources were never created, and
thus, never added.
This gets less obvious when you use a node attribute:
1 2 3 4 5
What gets added here? It depends entirely on the value of
at this point. While this example may seem trivial, consider something that
node["platform"] and how that might affect the queue.
Chef’s dirty secret: Convergence is also Compilation
Notifications and Subscriptions are great examples of a secondary compilation step that happens during convergence.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Looks like this when run:
1 2 3 4
But after compile, the queue looks like this and is evaluated in this order:
1 2 3 4
What happened here? Two things:
- An item in the queue is a constructed object with two parts:
- The resource
- The action
subscribesmodify the queue, and the position is dependent on the third argument.
:delayedadds to the end of the queue, so it is the last thing executed.
:immediatelyadds to the head of the queue, so it the next thing executed.
The first bit there is really important — if if were not the case, the queue
would have no idea that
echo baz had already run, and would run it again as
soon as it was notified to do so from
Additionally, we learn here that not only is the ResourceCollection a queue of things to act upon, but a registry of resources (and their states) that can be referred to later, at convergence time, with the result being more compilation of the queue.
Ok, so what can I do with all this?
Anything you want, really. The dynamic nature of recipes is what makes them more powerful than puppet manifests or ansible playbooks; otherwise, they are not that much different.
I have a silly project I banged out over an afternoon that takes it to its logical absurd extreme: Tyler Perry’s Chefception. It’s a REST service that stores JSON blobs in a sqlite database, then uses a provider to evaluate those into resources and run them as a part of the run queue. The idea being that you could do ad hoc resource management with curl, or your favorite HTTP library, without having to care too much about cookbooks. Of course, this is probably a pretty horrible idea and shouldn’t be used by anyone for anything, but it was fun to write.
There’s a lot you can do, but just remember that a recipe and script are two very different things!