Why Lessons Learned Rarely Change the Next Project

Every project ends with a list. What went well, what went wrong, what we would do differently. The list is filed. The next project starts, and more often than teams admit, it makes the same mistakes again.

The problem is rarely capture. It is the distance between a lesson identified and a lesson applied. One of the most documented lessons learned programs in the world, NASA's, has measured that distance. Its findings come first. A small loop that closes the distance comes after.

Identified is not the same as learned

A lesson is identified when someone writes it down. It is learned when the organisation behaves differently because of it. Between the two sits a lot of work that most lessons learned processes never assign to anyone.

Research on project organisations names the pattern directly. In the International Journal of Project Management, Duffield and Whitty found that lessons learned processes rarely happen, and when they do, they focus on "lessons identification rather than organisational learning." The output is a document. The intended outcome is a change in how the next team works. Those are different things, and only one of them is easy to produce.

What NASA's own audits found

NASA takes lessons seriously. It runs the Lessons Learned Information System (LLIS), and its procedural requirement NPR 7120.6, the Lessons Learned Process, set out how lessons are collected, validated, assessed and codified. That makes its audits a useful test case. If lessons stall there, the cause is not a lack of intent.

In 2001, the U.S. General Accounting Office (now the Government Accountability Office) surveyed NASA program and project managers. Of the 115 who responded:

• 27% had not been aware of LLIS before the survey.
• 17.5% had applied a lesson they found through LLIS.
• 58% said current systems did not let them retrieve the right lessons at the right time.

GAO concluded that "lessons are not routinely identified, collected, or shared." Learning did happen, but mostly through reviews and conversations with colleagues, not through the system built for it.

In 2012, NASA's Office of Inspector General returned to the question. It surveyed 28 science and space flight projects launched between 2005 and 2011. Sixteen project managers said they used LLIS at all, and only six of them found it useful. Twelve had contributed lessons. Only ten knew of the requirement to contribute. Managers described the system as outdated and hard to search, and many saw their own projects as too unique for past lessons to apply. The OIG also found that only two NASA centers fully complied with NPR 7120.6 in how they ran their own process.

The pattern is consistent across eleven years. Lessons were stored. Few were retrieved. Fewer still changed the work.

Why lessons stall

Read the audits closely and three causes keep returning. They apply well beyond spaceflight.

• Lessons are written as stories, not instructions. A paragraph about what happened is hard to act on. It does not say which document, step or decision should now be different.
• Lessons live apart from the work. A separate database asks busy people to go looking. Most will not, especially under schedule pressure. GAO found that managers learned more from colleagues and reviews than from the system.
• Nobody checks. Capture is counted. Application is not. When the only measure is how many lessons were recorded, teams get very good at recording.

None of these is a technology problem alone. Each is a missing link: between the lesson and the thing it should change, and between that change and the evidence that it happened.

Pause and Learn: learning while the project is still running

NASA Goddard Space Flight Center answered part of this with a practice called Pause and Learn. Instead of waiting for a closeout review, a project team stops at meaningful points in the life cycle and reflects together.

As Edward Rogers and Barbara Fillip describe it, a standard session brings 10 to 20 people from one project team together for about 90 minutes. It covers four or five topics chosen in advance and works through three questions: what went well, what did not, and what we would do differently. Ground rules are read at every session. The discussion is captured as a knowledge map, reviewed with the project manager, and shared in later workshops. Sessions are planned in the project's lessons learned plan, not called only when something breaks.

Pause and Learn gets two things right. It captures lessons while memory is fresh. And it keeps the conversation inside the team that will act on it. What it does not do on its own is guarantee that the next project picks the lesson up. That needs one more step.

A lessons learned process that closes the loop

The international standard for knowledge management systems, ISO 30401, sets requirements for establishing, implementing, maintaining, reviewing and improving a knowledge management system. The NIST Baldrige Excellence Framework asks, in item 4.2, how you build and manage organisational knowledge, how you share best practices, and how you use your knowledge to embed learning in the way your organisation operates. Both point to the same idea: knowledge has a lifecycle, and capture is only its first stage.

In practice, a working loop has three steps.

1. Capture in four fields. For each lesson, write what you expected, what actually happened, why the gap occurred, and what you will change. The last field is the one that matters. If a team cannot name a change, it has an observation, not a lesson.

2. Link the lesson to its target. Every lesson should point to the procedure, checklist, template or decision it changes. Update that document, and record the link. Now the lesson lives where the next team will work, not in a folder they have to remember to open.

3. Check that the next project applied it. When a later project reaches the step the lesson affects, confirm that the updated procedure was used. If it was, the lesson is applied. If it was not, find out why. Perhaps the change was unclear, or the step was skipped. That finding is itself a lesson, and it goes back to step one.

Our rule at Roadyom is proof before claim. For lessons, that means nothing counts as learned until a later piece of work shows it. So measure the loop by its last step. A useful indicator is simple: of the lessons captured last year, how many are linked to a procedure, and how many of those were applied by a later project? That number is harder to grow than a count of lessons recorded. It is also the only one that tells you whether you are learning.

A lesson that names no change is an observation. A change that no later step uses is a draft.

How Prism runs this loop

Prism keeps an organisation's decisions and knowledge as steps in the work itself. A team can open a Pause and Learn record at any step, capture what was expected, what happened, why, and what changes, and link each lesson to the procedure it should update. Prism counts a lesson as infused only when it is linked to a procedure and a later step has applied it. Until then, it stays visible as open. People decide what to change; Prism keeps the record of whether the change held.

Sources

1. Survey of NASA’s Lessons Learned Process (GAO-01-1015R). U.S. General Accounting Office (now Government Accountability Office), 2001.
2. Review of NASA’s Lessons Learned Information System (IG-12-012). NASA Office of Inspector General, 2012.
3. Mapping Lessons Learned to Improve Contextual Learning at NASA. NASA Goddard Space Flight Center (E. W. Rogers, B. Fillip), NASA Technical Reports Server, 2016.
4. Developing a systemic lessons learned knowledge model for organisational learning through projects. International Journal of Project Management 33(2) (S. Duffield, S. J. Whitty), 2015.
5. ISO 30401:2018 Knowledge management systems — Requirements. International Organization for Standardization, 2018.
6. Baldrige Excellence Builder 2021–2022 (Item 4.2, Information and Knowledge Management). NIST Baldrige Performance Excellence Program, 2021.

Comments

Your nameYour comment

Comments appear after our team reads them. Please do not include patients' or colleagues' personal details.

Post comment