Devops - not a solution for everyone

A relatively new term "Devops" is starting to become more widespread. As with most new terms it means different things to different people (a bit like "the cloud" or "grid computing" does even now).

Without getting into too much detail, it's about developers (who write the code) and operations (who look after the servers and responsible for the customer experience) working together, smartly, using practices adopted from "agile" methodology. Stephen Nelson-Smith has written a good article about Devops. Damon Edwards has also written an excellent What is DevOps? post.

A stereotypical release cycle consists of developers working hard on some new features, probably being hounded by managers and last minute changes. Finally they finish - it just needs to be released and they can sit back and have a well earned beer. Sadly, that beer may have to wait...

Operations are asked (or told) "we need to do a release today". That's a big problem as they are busy doing a large migration and the release will have to be done tomorrow. Is there a change request? "Bah, paperwork" thinks the developer. "Cowboys" thinks the sysadmin. The developer writes a change control request that says "put new code" on and after a bit of an argument about the lack of detail it emerges that some extra work needs to be done by the sysadmin, some new firewall rules need creating, extra software installed.

Sadly, this "lob it over the wall" and "us/them" mentality occurs far too frequently - and not just between developers and operations.

One concern with devops is that it sets the bar too high for many organisations to achieve with their current skill set. Expecting people to be able to cross the dev/ops boundary can be expecting a lot. Fully automated deployments with rollback capability, test suites and staging environments are fantastic, however it takes time which many organisations can't currently find.

Let's start with something smaller. Talk to each other, both teams work for the same company and towards a common goal (to make that company money). Break it down a bit more and you may need to trade off operations desire for security with developers desire for new functionality. Complaining in private doesn't help - normally after talking to each other some compromise or alternative can be found. Developers will think of things that operations won't know about and vice versa - use the whole skillset you have available. Don't send snotty emails, go and see the people at their desk - or pick up the phone, the human voice carries a huge amount of information which is lost when email or instant messaging is used.

If a release is coming up, developers should be informing ops as soon as they know - tell them what new cool features are in it - make them want to release it too. This way ops get a heads up and can plan for it. If you need a new package installing on the server, perhaps get ops to do that - that way when the production platform is altered they should have notes of what to setup and how. They may write a wiki page with the information you need - saving you time and ensuring consistency (which matters to them).

Similarly operations should be trying to help developers. Explain why change requests are important, but keep it a lightweight process. Change control should be about getting approval, documenting changes (in case of mistakes) and following procedures which are as error free as possible and allow for recovery. Find out how you can help them - perhaps they need some help troubleshooting a problem, or they'd like to try some new technology out but need it setting up.

Some possible ways to start:

  • set up staging environments if you don't have them already
    • operations reduce the risk to production from untested code
    • developers can make more invasive changes knowing that they are tested safely
  • automate deployment
    • lowers the risk of missed steps
    • deployment scripts can be made bulletproof and perform more testing over time
    • reduces time spent by everyone on a deployment
  • remove tedious repetition
    • by automating
    • or other means such as moving configuration into a configuration file rather than hardcoded during the build

In short, better communication and working together is the first step of devops. Later the merging of developer "agile" practices and operations can occur. It really is possible to work better such that "everyone wins".