Compliance – Tailoring Safer Systems

An audit is a snapshot. It checks, on a fixed date, whether the documentation meets requirements and whether the procedures are described as they need to be described. What it does not check is whether the organisation sees its own weak signals and learns from near misses. The distinction sounds academic but is operationally consequential. It decides whether an organisation builds its safety on the audit date or in between.

There is no need for polemic against audits. Audits do what they should. The serious question is what they structurally cannot do. And what needs to stand beside them so that what happens between audits does not become a blind spot for the organisation.

What audits actually measure

Audits measure conformity within a defined observation window. They check whether documentation at the time of inspection shows what it must show, and whether procedures are described as they must be described. This is a legitimate and non-trivial task. It has its place in the trust landscape that complex societies require. No one would board an airline aircraft without certification, no hospital operates without accreditation, no industrial operator runs without a regulatory framework. Audits produce this trust through a social procedure whose function Michael Power described precisely back in the late nineties in The Audit Society: they are “rituals of verification”, not measuring instruments for the property they claim to test. They produce a legible picture of order, and this picture is compatible with insurance, law and corporate reporting.

Power’s point is not that audits are useless. It is that what they produce is not identical to what they appear to produce. A passed audit says that what was documented at the time of inspection met the requirements. It says nothing about whether the organisation sees the weak signals of its own practice, whether it learns from near misses, whether its adaptive capacity holds under real pressure. These properties are not documentable in the form an audit requires for its findings. They are processes, not states, and an audit is built to test states.

Anyone who does not draw this distinction builds a conception of safety in which compliance and safety are the same thing. Before the Texas City explosion in 2005, BP had a “lost time injury” rate below the industry average, and an audit result confirming that number. What was deteriorating simultaneously in the same plant was process safety: a domain not captured by the prevailing KPIs. Andrew Hopkins described this in Failure to Learn with a clarity that still hurts: the organisation managed what it could measure, and screened out what evaded measurement. The audit confirmed the measurement.

Audits check whether what is supposed to be documented is documented. They do not check whether the organisation sees what it should be seeing.

Audit preparation becomes a permanent task

From this structural property follows a second, which becomes visible in many organisations as soon as you look at how effort is distributed across the year. Audit preparation has become a permanent task, with its own resources, its own roles, its own quarterly rhythms. An internal compliance function that works all year toward a smooth external inspection. Pre-audits, mock-audits, action lists, “gap analyses” meant to anticipate the result. This isn’t dumb bureaucracy, but the understandable response to an audit regime that has grown denser, more formalised and more consequential over the past twenty years.

The difficulty in this rationality arises where it displaces attention. Attention is finite, line time is finite, and what pays off in an audit competes for these exact resources with what does not appear in an audit. Erik Hollnagel describes this effect in Safety-II in Practice as a systematic reinforcement of Work-as-Imagined: the denser the specification, the more energy the organisation puts into maintaining the world of specification. The attention it spends there is missing from the observation of Work-as-Done. The gap between the two grows widest precisely where the most is documented, because the documentation creates its own reality that needs maintenance.

The displacement happens without bad intent. It follows the mechanics of prioritisation under time pressure: an operational signal that isn’t audit-relevant gets moved to the back, and “to the back” means, on the annual rhythm, until the next quarter, when the same logic will apply again. At the moment of prioritisation, the shift is materially correct. Over time, it is a pattern: what doesn’t fit the audit form rarely comes back on the table.

Concretely: in November, a shift lead reports an anomaly at a measurement point that doesn’t appear on any audit checklist. Her supervisor files it as “forward to Q1”. The Q4 audit runs cleanly. In Q1, preparation for the next audit cycle is already underway, the anomaly sits on a list nobody opens any more. In April, a different shift at the same measurement point has a near miss that is connected to the original report. Nobody sees the connection any more.

To see this pattern, you have to step outside the audit logic. From within its sorting, every individual shift looks like clean work.

What needs to stand beside it

Abolishing audits is neither possible nor sensible. They have their function, they are built into the logics of insurance and regulation, they produce the trust a division-of-labour economy requires. What they cannot deliver has to be delivered alongside. Alongside, not instead.

That isn’t an elegant answer. It demands a second layer that belongs in normal operations and works on a different logic from the audit layer. This second layer has a name in the resilience and HOP literature: Operational Learning. It is not the collection of Lessons Learned from incident reports. It is the ongoing reconciliation of the picture of work with what actually happens in operations, before it becomes an event.

In Pre-Accident Investigations, Todd Conklin develops two tools designed exactly for this second layer. The first are Learning Teams: small, time-limited groups of operating personnel and a facilitator, who sit down for one to two hours after a near miss or a routine task. Their job is not to find a solution. It is to reconstruct what was actually done and compare it with what should have been done. The output is an observation, not an action item. Precisely this refusal of the action-item format is the condition under which the observation becomes sharp. Whoever starts looking for solutions immediately stops seeing.

The second are Pre-Job Briefs. These are not the formal safety briefings everyone knows, but short structured conversations at the beginning of a non-routine task: what could go wrong. What trigger means we abort. Who has authority in which situation. The output is a shared mental model, not a list. A well-run Pre-Job Brief practice is hard to show in an audit, because it leaves no paper trail. It is effective in daily safety because it brings what can go wrong into the conversation before the doing.

Both tools share a principle. They are oriented toward seeing, not toward steering. The audit layer steers what is visibly documented. The learning layer makes visible what doesn’t reach the steering. Steven Shorrock and Claire Williams, in their work on human factors in practice, call this the “professional curiosity” of a learning organisation: the willingness to keep reconciling one’s own picture of work with the actual work. This is not a one-off project. It is an ongoing practice.

The distinction from what many organisations report as “Lessons Learned” is important. Lessons Learned are an output format: what we take from a closed incident, formulated as an action item or insight, filable in a system, citable in the next audit. Operational Learning in Conklin’s sense is not an output but an ongoing mode in which the organisation continuously refines its picture of work. One closes something. The other holds something open.

One example shows the difference most sharply. A near miss in a control room: an operator categorises an alarm differently from how the designers intended. In the Lessons-Learned format, this becomes an action item. “Refine alarm labelling, retrain operator.” Done, ticked off, filable in the system. In the Operational-Learning format, it becomes an observation. “Operator under load X reads the alarm in the context of other signals differently from how the designer assumed. The pattern recurs under conditions Y. What we don’t know is which contextual cues drive the interpretation.” A question instead of a result. Rather than closing the investigation, it keeps it open.

Whoever forces Operational Learning into the Lessons-Learned format has just lost the concept.

Why this is harder than it sounds

The audit logic and the learning logic compete for the same resource: the attention of the line. The audit logic almost always wins this competition because its consequences are short-term and visible. A failed audit pulls inquiries, reports, justifications upward. A skipped Learning Team session pulls nothing. It simply doesn’t happen, and nobody notices, until an event happens whose connection with the omitted learning work can no longer be cleanly shown.

On top of that: Operational Learning has no pretty KPI. An organisation can count the number of Learning Teams conducted, but the moment it does, it starts fulfilling the format rather than using it. Charles Goodhart described this effect in 1975 for economic regulation: once a metric becomes the target of control, it loses the property that made it a good metric. This makes the learning layer awkward in upward reporting, and awkwardness is a scarce property in organisations under efficiency pressure.

Whoever wants to build this second layer accepts that it isn’t reportable in the same language as the audit layer. It requires a leadership willing to release time and protected spaces without immediately demanding an impact measurement. It requires a line that recognises observation as a legitimate activity in its own right, not merely as a means to an action-item end. This is the more demanding form of safety work, and it is precisely the kind that happens between the audits, or doesn’t.

Where the audit belongs

Audits will stay because they are functional. What is missing beside them is the second layer: an ongoing learning practice that doesn’t replace the audit but takes on the safety work the audit structurally cannot do. Only with this second layer does the audit become what it should be: a confirmation of the state the organisation knows. Not the principal source of its safety knowledge.

Sources

Michael Power – The Audit Society: Rituals of Verification, Oxford University Press 1997
Todd Conklin – Pre-Accident Investigations: An Introduction to Organizational Safety, Ashgate 2012
Erik Hollnagel – Safety-II in Practice: Developing the Resilience Potentials, Routledge 2018
Andrew Hopkins – Failure to Learn: The BP Texas City Refinery Disaster, CCH Australia 2008
Steven Shorrock & Claire Williams – Human Factors and Ergonomics in Practice, CRC Press 2017

There is a lot of talk about measuring safety. That is something which is easier said than done. This article shares some reflections.

Measuring what?

Before starting to measure, one needs to know what one is measuring. How you define safety will determine what you measure and how you measure. Let us illustrate the problem with three quite common views on safety. As you will see, none of them covers the subject entirely and all have advantages and disadvantages.

Safety as compliance

A very basic way of thinking: safety is following the safety rules. Being compliant with these rules is being safe. This corresponds to the almost automatic reaction that many people have after an accident: if only they had followed the rules, this would not have happened. Many investigations therefore focus on breaches of protocol and deviations. Also, in ‘normal’ situations there is emphasis on compliance. Wear the mandatory safety gear. Hold the railing. Striving for compliance also appeals to the human tendency towards conformity. We are social creatures, after all.

Safety rules are important. They are a basic form of how we teach safety: “Don’t touch the stove, it’s hot!” “Watch left, right, left before crossing the street.” These things we teach our kids, our workers, etc. Safety as compliance works reasonably well in rather simple, ordered and predictable systems. In these situations, you have a reasonable chance to foresee what can happen and conceive actions to deal with variations. If you are on known territory, you can deal with the things that happen by applying prescribed routines. Following ‘best practice’ means acting safely, while acting outside of these scripts is regarded as unsafe.

Safety rules are not perfect, however. We live and work in a world with a lot of variability and we have a limited amount of foresight. This means that we cannot write rules for every eventuality. If we could, the rules would be impossible to handle because of their sheer volume. Besides, rules depend on context. In London it is smarter to look right, left, right before crossing, while this is not the best strategy for Zürich.

Rules are compromises and may sometimes not be enough to keep you safe. Even if you follow all the traffic rules, you can have an accident. For example, when others do not follow the rules. In some situations, following rules is even the unsafe option. One (in)famous example is the Piper Alpha disaster where the people that followed the emergency procedures died while the ones who ignored the procedures and just jumped overboard survived.

Safety as an absence of accidents

Go out on the street and ask a hundred randomly chosen people, “What is safety?” Chances are that many will answer something in the line of “Not having any accidents”. Thinking this way makes intuitive sense to most people. It feels right because in our minds safety and accidents are very much linked. When we do not have any accidents, we have been safe. Or have we? Actually, not necessarily. That nothing has happened does not mean that things are safe. In many cases it only means that nothing has happened yet. Although it can very well be that nothing happens ever.

A simple test is to reverse the definition and see whether it still works. Is “the absence of accidents is safety” true? Absence of accidents can be achieved by other ways. Randomness or luck are possible factors. Your definition of accident is another. Whether people choose to report accidents yet another. However, accidents do give an indication about safety, or rather unsafety. An accident can be regarded as a manifestation of risk, bringing us to the next definition.

Safety as acceptable risk

Whatever you do, there is some risk involved. We cannot avoid this. We even want some risk, but not too much. We need to compromise between various goals (financial, safety, production, quality, etc.), between uncertainty and control. We have only limited resources (money, time, expertise, etc.). Therefore, we must make trade-offs and search for balance.

This view of safety appeals to rational creatures. It suggests deliberation and decision based on ‘facts’. We will always face risks; we just have to make sure that they are acceptably low. The question is therefore what the right level of risk is. We should obviously try to put as much ‘distance’ as possible between ourselves and the hazard and the possible negative futures the hazard could lead to. But we do not want too much distance either. It has to be practicable and affordable. Besides, some hazards we actually do desire. Just think of drinking coffee. We want our coffee hot, but we do not want to burn ourselves. Therefore, we tend to sip our coffee carefully at first, or maybe blow a bit on it, instead of gulping it down at once.

The view of safety-as-acceptable-risk is useful, but there are also some drawbacks. One is its reliance on knowledge, another is how it can lead to quantitative approaches to risk that look more objective than they are, that it may lead to a static view of safety, and the problem of monitoring the risk level. Then there is of course the problem of who decides what is ‘acceptable’ and based on what. Who determines what is included in the assessment and what factors weigh in (and how much)? Who is allowed to participate in the process and how can they participate in the process? What language is used during the process and in the communication of the results?

One example of the latter is how consequences are selected and expressed. Certain risk assessments focus on fatalities, but those are often not the only bodily consequences. So, what to do with injuries? Should one choose a number of severe injuries that equals a fatality? Or should we, as one often sees, translate fatalities and injuries into monetary units? Is that really a good, and fair measure? Can you put a number on a human life? And if so, what number? Sure, you can estimate one person’s economic contribution to society and his/her family, but a person is so much more than his/her economic contribution.

Challenges

The above views of safety all bring their own ways of measuring safety. Regard safety as compliance and you may be tracking citations from the inspectorate, or observations of unsafe acts (e.g. not wearing protective equipment). If safety is seen as the absence of accidents, you will naturally follow up on accident and injury reports. Those who adopted a risk view of safety may have some kind of a risk register, present the most important risks in a risk matrix or heat map and follow up on actions to control the risks.

How you define safety will influence your choice of things you measure – and vice versa. What you measure may very well become your definition of safety, consciously or not. If corporate policy, an ISO standard or the regulator requires you to record accidents and near misses as part of your monitoring, it will become very natural to talk about these metrics when someone asks about “How are we doing at safety?”

Another challenge is that management dashboards and scorecards allow only limited space for the presentation of how things are going. Managers are busy people and they would very much like to get clear, concise, unambiguous and short answers. However, safety is a complex phenomenon. Therefore, we need a variety of measures to give a reasonable description. No one view captures everything. Every view shows some elements of safety, but never the full picture. A good answer thus needs rich information and nuances. Here is a tension between space and attention available and what is needed to give a high-quality answer.

Dumbing it down into an easy measure, no matter how intuitive, will not do justice to the subject. A fatality/injury-based metric only captures a tiny part of a very complex phenomenon. It would be like describing a river exclusively by its temperature – which, by the way, rather depends on its surroundings, location and season than on ‘itself’, just as injury rates may correlate stronger with the context than with safety efforts initiated by the organisation. A trade-off between thoroughness and efficiency is inevitable and carefully addressing this in the management system is essential.

This article is an adapted and abbreviated chapter from the book If You Can’t Measure It… Maybe You Shouldn’t. Reflections on Measuring Safety, Indicators, and Goals.

Tag: Compliance

What Happens Between Audits