4 Steps to Stop Fighting Fires: Step 1 Stabilize The Patient

by becki on June 13, 2009

So far, we’ve discussed the goals of this project and provided an introduction to the 4 steps outlined in the Visible Ops Handbook. The main goal is to migrate from constantly responding to crisis situations and move toward driving your organization. Throughout this series I will share examples from my own situation, and hope to spark some thoughts on how you can improve your own processes.

Step 1 Goal: Stabilize the Patient

Today we’re going to focus on Step 1, “Stabilize the Patient and Modify First Response”.  This is where you analyze the cause of your unplanned work. Unplanned work is the most expensive activity in your organization. It does not generate revenue, it does not improve efficiencies or reduce cost, and it takes time away from planned activities that do help your bottom line. At this point your goal should be to reduce unplanned work to 25% or less, while keeping in mind that high performing organizations spend 5% or less of their time on unplanned work. Baby steps my friend.

“What is the precise definition of unplanned work? It is any activity in the IT organization that cannot be mapped to an authorized project, procedure or change request. Any service interruption, failed change, emergency change, or patch or security incident creates unplanned work. ” (Gene Kim, Tripwire http://www.tripwire.com/resources/management-perspectives/article.cfm?aid=4)

Where to Begin

In this step we stop shooting ourselves in the foot. “If 80% of our injuries are self-inflicted, that means we are causing 80% of unplanned work” (Visible Ops Handbook p. 27). Think about that for a minute. 4 days out of 5 your organization is fire fighting. That’s crazy. I’ve been there, and it is no fun spending the day with a phone in each ear and a line of people at your door.

If you don’t know how much time you are spending on unplanned work or the root cause of your service interruptions, your first task is to begin measuring these. Here is how I did this:

  • Created an activity sheet using Excel. Everyone mapped their time to a project or planned support activity. Everything else was mapped to unplanned work. Activity sheets were turned in weekly and analyzed monthly.
  • Service interruptions require a root cause analysis report. Root causes are tracked on a spreadsheet and reviewed monthly.

Once you have data, you can evaluate how to improve your processes. What you are likely to learn is that you have some systems that break every time you touch them. Visible Ops refers to these as “fragile artifacts.” Changing these systems is highly risky, so you must control and audit changes on these systems.

Change Management

No one that I know enjoys change management. Just hearing those words causes eyes to roll and people to grumble. How on Earth do you introduce change management policies and processes without a huge fight? Here’s how I did it:

  • Bought a copy of the Visible Ops Handbook for each member of the department and reviewed it as a group
  • Reviewed outage data, especially root cause
  • Created an emotional connection by discussing the pain of working on service interruptions caused by a failure to follow policy and / or procedure
  • Discussed how we could improve our change management policy and procedures
  • Broke out into focus groups & those groups presented recommendations to the whole department
  • The department refined the proposed recommendations until we ended up with final policies and procedures
  • Policies and procedures are modified as new lessons are learned

Everyone was involved in the entire process and had a hand in creating our change management policy. This is what worked for me and my department, your experience may differ, but it has worked beyond my wildest expectations.

Next Steps

  • Audit the changes (you’ll want to automate this)
  • Enforce the policies
  • Continually work on changing the culture by reinforcing the relationship between changes and service interruptions.
  • Reinforce the effect service interruptions have on customers (internal and external), coworkers, the department, and performance metrics
  • Review the metrics with your people
  • Celebrate successes

Is This Difficult

This is not difficult from a skill point of view, but it does require consistency and patience. Find something to keep you motivated over the long haul. In my case the metrics kept me motivated. I can tell you that you will see immediate improvement, but you are not likely to get to your end goal in a year. It takes time to change a culture, and you cannot move from an average performing organization to a high performing one without changing the culture.

Do you think this information will help you “stabilize your patient” and get a handle on your operation? Do you believe you can get off that treadmill and go home at night and spend time with your family? Have you done something similar in your organization that you’d like to share?

I encourage you to come back next week to learn about Step 2, “Catch and Release and Find Fragile Artifacts.”

Additional Resources

Related Posts


Leave a Comment

{ 1 trackback }

Previous post:

Next post: