Part 1: Learning Culture – Blameless Postmortems

I believe in fostering a culture rooted in transparency and safety. Building and maintaining a psychologically safe environment requires a steadfast commitment to learning. To nurture this growth mindset, we need team practices that promote continuous improvement. In this post, I’ll explore one of two practices that are part of a kaizen culture: the blameless postmortem. I’ll share a story, highlight blameless postmortem’s multi-faceted benefits, and provide actionable steps and resources to help you implement or enhance your existing Root Cause Analysis (RCA) process.

Photo by Nicole Baster on Unsplash

STORY: 2 Days before Christmas

V, a dev lead was doing some maintenance, cleaning up what he thought was a development environment datastore just before Christmas. He removed a database, which took down two production applications. V notified the team immediately via Slack, and 8 of us jumped into a video session to help. We quickly communicated to our internal and external customers, divided and conquered, and started the recovery process.  We didn’t stop there, though. While waiting for the backup to run, we spent 15 minutes or so sharing our own “moments” with him, and laughing together. During our blameless postmortem the next day, we uncovered some holes in our processes and agreed to “buddy system” maintenance. 

From inception, part of our small, global remote team working agreement included Etsy-style blameless postmortems when something went sideways.  That incident and our laughter in the process brought our team closer.  

BENEFITS

For your team.  Done well, with facilitation that uncovers stories and the “how,” trust gets strengthened and reinforced on the team. You are more likely to grow into a tight-knit, high-performing team. Through respectful discussions and insights gleaned, you nurture skills and knowledge on the team.  This levels up earlier career staff.  It encourages a growth mindset.  This practice is one in which folks can step up and contribute in multiple ways, whether starting the process, coordinating it, contributing to the discussion, or facilitating the session.  If you are also into growing “T-shaped” people, then blameless postmortems are yet another means to support that. 

For leadership.  Contexts vary, but hopefully, your team has a foundation of solid leadership and wants you to be autonomous and self-directed.  This practice also demonstrates team competence to your leadership. It fortifies their trust in you, i.e.,  “Don’t worry, we got this covered. Here is what we learned and the actions we have taken (are taking/with timeline), i.e., “What we will modify to reduce the likelihood that this happens again.”  

Resources.  You don’t want to go without once you have experienced the electricity of group learning that gets sparked in a blameless postmortem.  It feels weird NOT to do this when there’s an incident. This practice might seem like an onerous investment of time, but it does not have to be.  It is worth the effort. You get back what you put in.  

Here is how it worked efficiently on our remote-first team.  Each incident got a slack channel using a naming convention like  #12-23-2020-rca-ddedown.  Someone would volunteer to start the process asynchronously in the channel, and someone agreed to be the facilitator and would set a time to discuss 30+ minutes. These were the guidelines & required reading we used to stay true to the spirit of the practice:  Debriefing Facilitation Guide and Blameless Postmortems.

Here is the agenda based on the above articles and GitHub guide:

A. Event Recap, detailed account of

  1. What actions we took at what time.  Just the facts. Someone starts this effort before the group session.
  2. What effects we observed
  3. Understanding of timeline events as they occurred

B. Discussion

  1. The 5+ whys with
  2. What expectations we had
  3. Assumptions we made

C. Major Takeaways, Recommendations

PARTING THOUGHTS

Blameless postmortems are a vehicle for learning. A team with a solid leadership foundation—clear expectations and priorities, decent tools, and encouragement or incentivization to be autonomous and self-directed—can learn, laugh together, and enjoy the ride.   

The team in the story has disbanded, but when I catch up with former teammates, we still talk about that incident in appreciation of our teaminess.  Thanks to my SABA, ATL, and Pensacola mates for many spirited discussions. “Nizz” still loves you. 

Lastly, for today’s musical accompaniment, a song about a car: Fire it Up by Modest Mouse [YouTube]

Related posts:

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.