This article is part of a series that I am writing on how to use the Visible Ops Handbook to stop firefighting and start leading your business. In this article we will look at how to create a repeatable build library that makes it “cheaper to rebuild than to repair” (Visible Ops page 24).
By making infrastructure easier to rebuild than to repair, we will create the data center equivalent of fuses… (Visible Ops page 47)
Eliminate Snowflakes
In step 2, we identified the “fragile artifacts” that seem to break each time you touch them. These are the devices that you want to focus on first. In my case, our fragile artifacts were servers that met the definition of “snowflakes”, they were all uniquely configured. The backups only included a common set of files used in the initial configuration, not the unique one-off site specific configuration. That meant that if we had to restore from backup, we had to somehow recreate the one-off configuration. Obviously this was inefficient and the restoration process was longer and more painful than it needed to be.
“…success in this phase hinges on our ability to control production changes to that fuse, lest the new fuse have radically different behavior than the fuse it replaces” (Visible Ops page 47)
Golden Build
The software vendor bought into what we were attempting to do and delivered just what we needed. We worked with them to create a golden build (since followed by a platinum build) version of the software. A critical requirement was to make sure all site specific files are included in the backup and restore procedure. Any new feature has to meet this requirement and the restoration process tested before we put it into production. We can now restore these servers in less than 30 minutes, where it previously took many times that depending on the site and the skill level of the person performing the restoration.
Benefits
- This step has probably been the most important to reducing our MTTR and unplanned work while simultaneously increasing our change success.
- We no longer have snowflakes so we don’t need special knowledge to support the services.
- Engineers have more time to work on projects that increase revenue or reduce expenses
- Happy customers!
Congratulations! At this point in the process you’ve stopped firefighting and are probably a bit disoriented by the reduced adrenaline levels and the phone missing from your ear. While this is an actual problem for some people, most of us will happily learn to take up a hobby.
How about you? Are you ready to start down this path? Has this series given you ideas on how you can use Visible Ops in your organization?
Related Posts
- 4 Simple Steps to Stop Fighting Fires and Start Leading Your Business
- 4 Steps to Stop Fighting Fires and Start Leading: Introduction
- 4 Steps to Stop Fighting Fires: Step 1 Stabilize the Patient
- 4 Steps to Stop Fighting Fires: Step 2 Identify Fragile Artifacts
- 4 Steps to Stop Fighting Fires: Step 4 Continuous Improvement
Subscribe (free) via RSS
LinkedIn
email
