scribble

Holistic Engineering

E-Mail GitHub Twitter

31 Aug 2012
Your Ops Team Is Probably Important

I’m not exactly the voice of reason most of the time, and I’m not a big fan of blogs that pride opinions over technical content. That said, I see quite a bit of bickering, from developers and operations folk alike, about the concept of “NoOps” and its good and bad qualities. As someone who did full-time web development for over a decade and is now just over 2 years of a full transition into operations, I see a lot of misguided accusations over the expectations that both devs and ops teams put on each other respectively. It’s my hope to bridge those perceptions in this article, based on things I’ve discovered by contrast myself, and following the changes that have occurred in recent years.

It’s ok to not have an Ops Team.

If you can get by without an ops team – good for you. It’s not a requirement for a lot of modern applications, especially at small scale, and the service offerings out there are maturing at a breakneck pace; Heroku, Google App Engine, Engine Yard, Joyent are all offering compelling alternatives to having devoted operations personnel. To put it another way, if you all work at home, is there really a need for an office janitor? No.

It’s important to understand, though, what an operations team offers you. My experience in development, especially for the web, is that you have a product team which is looking at the road ahead and the development team is typically concerned with the here and now, save the occasional large refactor or requirement thrown downstream that requires a long-term vision of how the application(s) will be structured.

Your ops team is constantly bombarded by the present via pages, service requests from other staff, and so forth. But between those pockets of (let’s be honest here) interruptions your ops team should be fully functioning as a lighthouse for the shore of the future, planning ahead and actively seeking out ways to keep your environment reliable and convenient to work in.

“NoOps” services can offer a great deal of this these days, from taking the thinking out of capacity planning and disaster recovery, to automating the provisioning of servers and so forth. It’ll be exciting to see where they take things in the future. That said, it’s not a complete solution, and removes the human element from the equation.

Your ops team provides a greater sum than answering pages and configuring servers.

Your ops people are as much of a human resource as they are technical ones. Remember, you the developer are their customers, and the services they build to make your lives easier are their product. Keeping you happy is as much a part of their job as making sure the load balancer is distributing traffic evenly.

What frequently baffles me in this new era of ops is how many developers aren’t exploiting this resource. With rare exception, here are a few technical things your ops person will probably be able to help you with:

  • Understanding the penalties and advantages of applications that fork
  • Understanding why that query that takes 0.1ms to run and returns 800,000 rows is still slow
  • Telling you that no, that extremely complicated thing you did that eats tons of resources won’t actually be a problem.

These are all things that you are very unlikely to know as a developer. I’m not saying this as hyperbole, I’m saying this from experience on both ends of this spectrum. They’re just not problems that most developers consider (or in the last case, problems that developers think is a problem but actually isn’t) and your operations team lives in that world, around your systems, data, and network, so why not utilize them?

The reality is, what everyone sees as “Ops” is changing. Rapidly.

To further this point I think it’s important to understand that both the DevOps movement, and NoOps too (even though a lot of operations people are unwilling to admit it) is changing how we do operations, moving it forward every day.

I started as a wet-behind-the-ears 19 year old as a web developer at a famous bookstore in Portland, OR. I’m 34 now. In that time I’ve seen the landscape of operations attract a lot of archetypes, and the systems have changed too. For example, I was asked to not run “bash” on the mail server at said bookstore because it used too much ram, and to use “csh” or “sh” instead. I was introduced to this thing called “ssh”, and I had been using “telnet” for years, and it took a while to understand why I had to use something different. Nowadays people get pissed when faced with a traditional bourne shell, and nobody – nobody – would question the use of ssh. I worked with CCNAs and System Administrators who, by all accounts, could easily do my job in a heartbeat if it was at all interesting to them.

Old man rambling aside, at some point that changed. I started walking into shops where the “system administrator” was more skilled with a crimper than a shell prompt, and where I genuinely felt if I didn’t tell this person what the hell to do we were genuinely screwed. I know I’m not alone in this feeling. Having to have the “why do we have to upgrade $x again, it’s going to take me a week to make packages of this shit / the OS vendor doesn’t support it, fuck off” argument, or getting the rare opportunity to see how something is set up to see, if they didn’t just paste it straight from Google, a complex sea of shell scripts that made the most criticized contributors to DailyWTF seem tame by comparison. I specifically remember a qmail configuration done so poorly and haphazard that nobody in the company – even the sysadmin that set it up – had the cajones to reboot the machine. It wasn’t an uptime thing, they genuinely didn’t think they could bring it back up.

I hated these people. I still hate them, and I’m guessing if you’re a developer with any long-term experience, you’ve probably run into these people and despise them too. Control freaks, “BOFHs” and generally compensating for incomptence through laziness and hand-waving.

The thing that DevOps is preaching is the antithesis of this – supporting developers to realize their needs on the systems, and architecting beautiful systems, networks, and automation to deliver on promises to the company and the engineering teams. NoOps, in my opinion, is the guerilla warfare version of DevOps, namely “I never want to work with a shitbird like that again, so I’m going to write software to render them extinct.”

Both camps have been largely successful. I’ve recently started interviewing again, and the bar 3, 4 years ago was at my feet in comparison to the questions I’m being asked now. I can’t see people that held that attitude, and level of technical prowess finding work anywhere in this modern climate. What’s funny is that I firmly believe that those folks at the start of my career were really just what we’re getting back to now, after a long september of shitbirds ruining the party for everyone else. Now that these types aren’t needed anymore, the real fun can happen for everyone.

Perhaps we should just can this whole argument and call it NoShitBirds, and instead of focusing on roles, worry about what the non-shitbirds are adding to the company.


Til next time,
Erik Hollensbe at 04:03

scribble

E-Mail GitHub Twitter