Atom Feed: Whole Site

An attempt at a Ruby 1.9p1 porting guide

2008-11-27T11:28:02Z

Ruby 1.9p1 has been out for a few weeks, and bringing some of the libraries I maintain into compliance with it has been the focus of my free time recently.

I have collected a few things here which may help others in my position save a bit of time debugging and bug reporting. This does not cover the changes that were made between 1.8.6 and 1.8.7, many of which were "staging changes" for 1.9.

This is also a work in progress, and will likely change at least once or twice over the next month; I will note here what has changed.

Module/Class Changes

Low-hanging fruit:

The 'parsedate' stdlib package no longer exists; this functionality has been rolled into the date/time packages.
UnboundMethod now pulls some functionality out of Method, and as a result rebinding or calling methods via Object#method may yield unexpected results. Read the docs.

Date/DateTime/Time .parse changes

These now try a more international DD/MM/YYYY format first when parsing than the previously U.S.-centric MM/DD/YYYY format. As a result, you may have issues like this:

    1 # old
    2 d = Date.parse("10/11/2008")
    3 d.strftime("%Y-%m-%d")
    4 # "2008-10-11"
    5 d = DateTime.parse("10/11/2008 10:01:02")
    6 d.strftime("%m/%d/%Y %H:%M:%S")
    7 # "10/11/2008 10:01:02"
    8 
    9 # new
   10 d = Date.parse("10/11/2008")
   11 d.strftime("%Y-%m-%d")
   12 # "2008-11-10"
   13 d = DateTime.parse("10/11/2008 10:01:02")
   14 d.strftime("%m/%d/%Y %H:%M:%S")
   15 # "11/10/2008 10:01:02"

delegate.rb changes

The bad news is that a great deal of code that uses this stdlib addition just had the rug pulled out from under it. The good news is that all the changes are intended to be for the better, and you can reap a great deal more flexibility from them. The good news is, deep copies are no longer required to properly clone the managed object over a #clone or #dup.

Marshal and DelegateClass play together very differently than they used to; if you were using the old Marshal.load(Marshal.dump(self)) trick to do deep copies on delegates, you will run into problems. This is because 1.9 Marshal now has dump/load hooks, and the new delegate.rb takes advantage of them, and they operate differently than you'd think. Good for us that they're easy to overload. Look into Marshal, or Marshal.dump for how to perform this.

Additionally, the new 1.9 delegate.rb includes two methods on each class of delegate (DelegateClass, Delegator, etc) named __getobj__ and __setobj__, which act as accessors for the underlying object. So, this idiom no longer works:

    1 class Foo < DelegateClass(Array)
    2     def initialize
    3         @arr = [1,2,3]
    4         super(@arr)
    5     end
    6 
    7     def reset
    8         @arr.replace([])
    9     end
   10 end
   11 
   12 f = Foo.new
   13 f2 = f.dup
   14 f2.reset
   15 f2[0] # 1? wtf?

This is because now, the value @arr holds is #dup'd and assigned to an underlying instance var (@delegate_dc_obj in this case) controlled by these new methods. When #dup/#clone are called, they are treated as separate objects (as it should be). However, dispatch is only implicitly given to the one created during the super() call, so while we've cleared out @arr, methods on this class and the ones in Array will be working with the different data in @delegate_dc_obj. Quick thinkers may notice the common thread between the deep copy problem and this one.

To fix this issue with minimal impact, have the object accessors produce the object we want, like so:

    1 class Foo < DelegateClass(Array)
    2     def initialize
    3         @arr = [1,2,3]
    4         super(@arr)
    5     end
    6 
    7     def reset
    8         @arr.replace([])
    9     end
   10 
   11     if RUBY_VERSION =~ /^1\.9/
   12         def __getobj__
   13             @arr
   14         end
   15 
   16         def __setobj__(obj)
   17             @delegate_dc_obj = @arr = obj
   18         end
   19     end
   20 end

You may notice the assignment to @delegate_dc_obj in __setobj__, which is a dispatcher bug. Here is the ticket. As of this writing it has been closed, but the bug still exists.

For Ruby/DBI's DBI::Row (which unsurprisingly has both of these issues), this is working and well-exercised in the test suite, passing on both 1.8.7 and 1.9:

    1 if RUBY_VERSION =~ /^1\.9/
    2     def __getobj__
    3         @arr
    4     end
    5 
    6     def __setobj__(obj)
    7         @delegate_dc_obj = @arr = obj
    8     end
    9 else
   10     def clone
   11         Marshal.load(Marshal.dump(self))
   12     end
   13 
   14     alias dup clone
   15 end

Syntax/Core changes

~~Method prototypes~~

This was poorly researched and I am sorry if I have mislead anyone.

Explicitly prototyped blocks are now required in the call. For example:

    1 # explicit block...
    2 def foo(arg1, arg2, &block)
    3     p [arg1, arg2]
    4     yield arg1, arg2
    5 end
    6 
    7 foo(1, 2) # wrong
    8 foo(1, 2) { } # right
    9 
   10 # implicit block...
   11 def bar(arg1, arg2)
   12     p [arg1, arg2]
   13     yield arg1, arg2 if block_given?
   14 end
   15 
   16 # both of these work
   17 bar(1, 2)
   18 bar(1, 2) { }

Even if you don't do this in your own code, be aware that many oft-used packages (like Rake) make heavy use of this idiom and will require code changes on your behalf.

Testing changes

One of the largest changes I've been faced with is that Test::Unit no longer exists, superceded by MiniTest. MiniTest has some compatibility layers that will likely work with most test suites, however MiniTest is as it is named and as a result, does not include all the functionality the packages it's compatible with contains. If you have an unorthodox or esoteric test suite, be prepared for a little hell.

The good news is that Test::Unit, like RSpec, is now maintained separately from Ruby, and available as a gem. Here are a few test suites that you may want to look at to get your 1.8 test suite working again:

xUnit: test-unit 2.0. The developer is extremely responsive (my bug reports have been fixed within minutes), so don't hesitate because it's a gem.
BDD: RSpec. The old standard. I'm not a BDD guy but you guys win this round in the "my test suite still works" department.
BDD: (Github) Bacon - gem install bacon. Lightweight RSpec clone, plenty I know love this.

Just to repeat myself, most people will not need to sway from the Ruby stdlib to get Test::More functionality.

HTH! If you have corrections, additions, or just want to flame me, email me here.

Alistair Cockburn: Reading for Management Types

2008-10-11T07:28:34Z

This article is new to me... A co-worker gave it to me, recommending I use it as a basis to weigh against certain process changes I've been proposing at work, as a lot of that is part of my job these days.

I find it interesting that I am certainly not the first (and definitely not the last) person to read this article for the first time, but so, so many of these things are fairly obvious notions (Mr. Cockburn doesn't deny they are) and are completely absent of consideration or even acknowledgement in many software houses.

Really, it boils down to (for me):

First and Foremost, People are People
The people you work with are typically smart and have formed their own opinions about the current, and proposed development processes.
People are unique.
Simple processes cater more to people; even if the complex process is bulletproof, if no one likes or can remember all of it, it's futile
People will find a way to get out of anything they don't like, even if only temporarily.
Enough dissonance will result in a pointless process that results in wasting developer time without any tangible result.

Which can be summarized as, "if they don't like it, it ain't gonna happen". It's more than that, with case studies and lots more detail, and the more valuable part of the article is how to shape your processes so that even if people don't like it, it doesn't impact them so heavily they can't deal with it.

Two of the agendas I've been trying to get kickstarted are TDD and releasing our web backend as a product (as opposed to just checking in whenever, wherever we feel like... which as much as you might cringe at the concept, it is how the majority, not the minority of small shops operate) and while I'm fairly certain the other senior agrees with me I think he's more worried about spending a lot of time (money) for a fruitless result.

So far the people I have on the wagon seem to be excited about it (we started converting legacy tests to something more TDD-able, static database between suites and whatnot) and the other ones simply haven't been affected by it; this is somethign I have to address in the near future, and will be the ultimate test.

My Problem With Republicans and Democrats

2008-10-10T06:41:52Z

These links sum it up much better than I could:

Schema management: keep your framework out of this

2008-10-08T07:23:42Z

Fixtures (database fixtures, specifically) and migrations (database versioning) never cease to amaze me as a powerful tool to empower testing and enhance the quality of any application. After all, why do you treat your source code as some sacred, version controlled scion, but the database gets the rough equivalent of cp $i $i.old version control. Really folks, how many applications do you know of where the schema and the manipulations to the database aren't as important, if not more important, than the code used against them?

However, almost all of our options today are housed in frameworks which are mostly whole-hog; you must accept the rest of the framework (or at least, do a good job of faking it) to get these powerful features, or you need to write them yourself. This is the second time I've been put to task implementing these for a company with a custom framework and I'd rather avoid it in the future; if you're saying to yourself, "they should just use the framework", go to business school. You are not an engineer.

Fixtures are more important to your test suite than your web app, and while there are basic fixtures in your various SUnit clones out there, the concept of tying them to a database is solely in the realm of web frameworks, which do provide some test additions, but ultimately add fixtures and a bunch of other crap you may or may not use; there are plenty of ways to test your web application without having to use any of those methods. I must dare to say that they are no less effective... and often, are much more, especially when it comes to frameworks that mock the webserver integration bits. This posting wouldn't be the first time someone's complained that a mocking of something as critical as mod_perl or FCGI hasn't failed them horribly.

More importantly, fixtures provide you with a baseline set of data. Someone that uses fixtures effectively will re-init the database, perform that ugly mess of joins that ultimately result in an update, and examine the result. If they get it wrong, they do it again. They may never need or want the framework to do this, but they do need the fixtures. I recall one time one of our tech-savvy content team members dropped an image relation table from the database our test suite ran against, and we were down (development-wise, at least) a whole day getting the DBA involved to get our data back. I'd be surprised if I was the only person this has happened to.

Migrations provide us with powerful way of versioning the database; through the use of DDL and DML, we can create efficient and expressive manipulations of the database which create a record and rationale for these changes. If you've ever jaunted across the office to ask Joe Hacker WTF he was thinking when he removed a constraint from the session table, you know exactly what I'm talking about.

Migrations not only serve the development team but the other unsung parties responsible for the database; the DBA, the sysadmin. They can stage these changes before they ever reach production and examine the result without the developer-issued rose-colored glasses, something I think many developers under-appreciate. Additionally, a standalone migration system is the staple of pretty much every DBA I've worked with in the past, and it's almost always homegrown. Why not make this easier on them?

So, Perl users, and Ruby users, why the hell haven't you done this yet? My hands are tied until I can talk it over with my employer, but simple, standalone features for fixture import/export and migration schemes; this isn't rocket science. I don't want your framework (at least, right now). Give me something that I can use.

Using nanoc

2008-10-08T05:59:45Z

We're now using nanoc to replace the wordpress installation. It's not blog software, it's a CMS that precompiles your site on a local machine for you to upload. It's a well-designed system and highly extensible, and I'm not going to attempt to top Denis Defreyne's excellent work in creating a manual, so go read it if you want the basics. However, here are some examples of code I've used to generate this site.

Real content management

nanoc is more than a frontend to a template system, it understands the needs of content maintainers. I need to know about the structure of my content, I need to be able to enable and disable content, I need to be able to create content with as little repeated work as possible. nanoc understands this.

nanoc fills a number of instance variables with data about the content as a whole, and the content being built; this data can then be used to build the document structure of your choice. For instance, pages have parent and child information, allowing you to easily create navigation that's contextual. Here's how I gather the pages for the "siblings" section on the left:

    1 @page.parent ?
    2     @page.parent.children.reject { |page| page.path == @page.path } :
    3     []

Here's the ERB used; this is a part of the main layout for the site, all pages use it. You can see how the siblings section is only generated if needed, as well as the optional parent link.

Apologies that this is not highlighted, I haven't taught the code highlighter embedded ruby yet.

<% 
    siblings = @page.parent ? @page.parent.children.reject { |page| page.path == @page.path } : []
    siblings.reject! { |page| page.filename == "atom" }
    if siblings and siblings.length > 0
%>
    <a class="leftnav-header">Siblings</a>
    <div>
        <%
            if @page.parent
        %>
            <a class="leftnav-link" href="<%= @page.parent.path %>">Parent: <%= @page.parent.title %></a>
        <% end %>
        <% siblings.each do |page| %>
            <a href="<%= page.path %>" class="leftnav-link"><%= page.title %></a>
        <% end %>
    </div>
<% end %>

This produces the document structure the JQuery Accordion UI Extension needs to produce the fancy menus on the left. nanoc provides all the primitives and allows me to relate the content as I see fit; how many "real" CMS's do you know that can do that, especially with that little custom code?

I have other tidbits in lib/default.rb which (like all other things in lib/) is loaded before compiling starts.

Custom processing filters

I wanted to write a filter that highlights code, since I plan to include a lot of it here. This was always a pain in the ass in other systems, integration and performance being the big issues. It's trivial in nanoc, since integration can actually mean running a command-line program without the security drawbacks, and performance is never an issue since all content is pre-compiled. It's written as a post-processing filter and lets me use xml-style tags to indicate where the code is, like this:

    1 <highlight lang="ruby" class="code">
    2 def foo(bar)
    3     do_something_with(bar)
    4 end
    5 </highlight>

The filter strips the "lang" attribute and uses it for the processor. The rest of the attributes are merged with the output, allowing custom styling and other HTML niceties. This produces:

    1 def foo(bar)
    2     do_something_with(bar)
    3 end

What really impressed me was how simple this was to write; in fact, I spent over 4 hours putting this together, due to issues with various HTML parsers that I couldn't resolve. nanoc's handles this so simply that it didn't really cross my mind how little effort it was to get nanoc integrated into the picture until just now.

Here's the basics of the filter class:

    1 module Nanoc::Filters
    2     class Highlight < Nanoc::Filter
    3 
    4         identifiers :highlight
    5 
    6         def run(content)
    7             return content.strip
    8         end
    9     end
   10 end

This code obviously just removes extra whitespace (the real thing is here) but you can see how easy it is to get started. This is what I had to do in my page definition (a YAML file) to enable it:

filters_post:
    - highlight

These definitions have hierarchy so I can do it in my defaults file as well, allowing this filter to run on all or some of my content.

Content is content; Display it how you like

One of the things I wanted to do while writing this article was produce the code sources as separate files; highlighted with the above-described highlight filter with a link to the raw source. Producing this wasn't as hard as it could have been, that's for sure!

Let's establish what needs to be done:

Code needs to be barfed into a file.
Needs to be compiled twice: once with the highlighting, once to a plain text file.

nanoc has a great feature called page representations which we'll use for this task. The same page can be represented any number of times, with different output parameters (filters, filenames, etc).

First, create your page and edit the YAML file to look something like this (this is for highlight.rb):

title: highlight.rb
reps:
    raw:
        skip_output: false
        filename: "highlight"
        extension: "rb"

As mentioned before, nanoc allows defaults that get merged in with this definition. Here is my defaults file:

filters_pre: 
    - erb
skip_output: false
filters_post: 
    - highlight
reps:
    raw:
        skip_output: true
        filters_pre:
            - erb
        filters_post: []
        layout: none
site_name: "Erik's Tub o' Bullshit"

reps starts a representation structure which contains alternative representations. default is the page that'll be generated automatically, and does not necessarily need to be specified: it'll exist unless skip_output is set to true, which is the default for the raw representation because we don't want to represent *everything* that way.

Just like the list of pages to be compiled and the page currently being compiled, we can get at the different representations of the current page being compiled and the representation that is currently being compiled. In this case, I have highlight set up as a post-filter for default representations while it does not exist for raw representations. But now we have a new problem, how do I generate those highlight tags when I need them, but not when I don't?

Yes, there are easier ways to do this: a separate 'highlight' filter that just ran highlight over everything would work, certainly, and maybe sooner or later I'll do that. Until then, I came up with this solution: generate a new layout which builds the tags for us if they're needed, based on the page representation. Here's what the layout looks like (which requires erb, but nothing else):

<% if @page_rep.name == :default %>
<% if @close %>
</highlight>
<% else %>
<p><a href="<%= @page.reps(:raw).path %>">Click here to download</a></p>
<% if @lang %>
<highlight lang="<%= @lang %>" class="code widecode">
<% else %>
<highlight class="code widecode">
<% end %>
<% end %>
<% end %>

Then our actual content looks like this:

<%= render 'code', :lang => "ruby" %>
module Nanoc::Filters
...
end
<%= render 'code', :close => true %>

If you've missed the other links to it, the result looks like this. The render method takes a layout and a splatted hash of keys/values that are interned into instance vars available to the ERB template loaded for the layout. Our layout then generates the highlight tags based on the representation and the vars provided to it.

Overall...

nanoc still has a few rough edges, but for the most part they're able to be hurdled with little effort. There are numerous features to make content production easier for the less technical as well, but you should read about those yourself.

I highly recommend this to anyone building a new website that wants all the features of a CMS but doesn't need the overkill that is often plone or slash, and doesn't want to resort to using a blogging tool for it either.