Checking Instead of Writing: Generative AI and the Check That Is First to Go Under Pressure.

Anyone who introduces an AI assistant into an organisation usually ships a safety promise along with it: the AI produces the draft, a human checks it and remains responsible. In guidelines this is called “human in the loop”. That sounds like a sensible division of labour: the machine delivers a first version in seconds, the human contributes judgement and expertise, and the result is better and arrives sooner. Or so the calculation goes.

Lisanne Bainbridge would have balked at that calculation. Last week this magazine looked at her 1983 paper “Ironies of Automation”. Her finding: we automate whatever can be automated, and the human gets the rest. That rest consists of monitoring and taking over, and humans are poorly equipped for both, precisely because the automation relieves them of the everyday work. Bainbridge was writing about control rooms and cockpits. With generative AI, the same division of labour moves into the office, and the ironies come with it.

From Writing to Checking

A research group led by Auste Simkute (University of Edinburgh and Microsoft Research) compared user studies of generative AI with three decades of human factors research on automation. The group identifies four reasons why the expected savings in time and effort often turn out smaller or fail to materialise at all. The comparison is sometimes with working without AI and sometimes with what the tools promise; the paper does not define productivity precisely. The four reasons: users’ role shifts from producing to evaluating, workflows are restructured unhelpfully, work gets interrupted, and automation makes easy tasks easier and hard tasks harder. The last is almost word for word in Bainbridge; the first corresponds to her shift from controlling to monitoring.

What this change of role looks like in everyday work is the subject of a survey by a second group, from Microsoft Research and Carnegie Mellon University. In total, 319 knowledge workers described 936 situations in which they used generative AI at work. In their accounts, critical thinking shifts towards verifying information, integrating the AI’s responses into their own work and overseeing the task. Those who trust the AI more report checking less critically. Those who feel confident in their own ability at the task check more. These are self-reports, not measurements. Still, they fit what Bainbridge took from vigilance research: anyone monitoring a system that almost never fails will eventually stop looking.

Checking is also harder than the word suggests. A generated text looks finished. It is fluently written, well structured and supplied with sources. Whether a source exists and says what is attributed to it cannot be seen from the text. The only way to find out is to open it. If the machine writes twenty pages in a minute that would have taken me two days, and checking them takes a few hours, that is still a large gain. On paper, the check is usually planned, often even mandatory. In practice, it is the part of the work that gives way most easily under pressure. Anyone who cuts a check short does not notice at first, and nobody else does either, because the text looks finished. The time gained, on the other hand, is visible at once. Erik Hollnagel calls this the efficiency-thoroughness trade-off: under pressure, people trade thoroughness for efficiency, and most of the time it works out. That is exactly why it becomes a habit.

The check is almost always planned. Under pressure it is the first thing to go, because nobody notices it is missing.

237 Pages, with Footnotes

In July 2025, Australia’s Department of Employment and Workplace Relations published a 237-page report by Deloitte Australia. It reviewed an IT system that automatically imposes penalties on jobseekers in the welfare system. Christopher Rudge of the University of Sydney found up to twenty errors in it, including a fabricated quote from a federal court judgment and references to academic work that does not exist. Deloitte confirmed that some footnotes and references were incorrect and repaid the final instalment of its fee. The contract had been worth around 440,000 Australian dollars. The revised version now states that a language model (Azure OpenAI GPT-4o) was used in preparing it. According to the department, the substance and recommendations of the report were unchanged.

That a review of an automated system should itself trip over automation has a certain irony. More revealing is the type of error. Whether a quote is accurate and a source exists is the most mechanical part of any quality check. How the review at Deloitte was organised is not public, and I do not want to speculate about who missed what. To judge whether a check holds up, individual diligence is the wrong place to look. What matters is whether the check is designed to find errors of this kind, and whether it stands a chance against deadlines and cost pressure in daily practice.

Deskilling, Now Measured

Bainbridge’s second irony concerns skill. Anyone who only watches loses the abilities an emergency calls for. In 2025, one of the first measurements of this from clinical practice was published. It comes from colonoscopy, not from generative AI: there, AI systems highlight suspicious areas in the image during the examination.

A team led by Krzysztof Budzyń examined how 19 experienced endoscopists at four centres in Poland performed without AI after they had been using it routinely for some time. Each of them had performed more than 2,000 colonoscopies. Their adenoma detection rate (adenomas are precursors of bowel cancer) in examinations without AI fell from 28.4 to 22.4 per cent. It is an observational study, and the results need confirming. By their own account in an interview, the researchers had assumed that experienced practitioners would be less affected.

This study does not show whether the same happens with writing and analysis. The connection to the Microsoft and Carnegie Mellon survey is uncomfortable all the same. There, how thoroughly people checked depended on their own skill: those who felt confident in a task checked more. That skill comes from doing the work, and that is precisely what the tool takes over. Anyone who spends years checking AI drafts rather than writing their own will gradually lose the judgement the checking relies on.

The Dangerous Middle

Two approaches are internally consistent. In one, people do the work themselves and use the tool where they have mastered the subject and can genuinely do the checking. In the other, the organisation lets the system do the work and openly accepts the residual risk, for example for internal summaries where an error costs little. The middle is where the danger lies: human review counts as a safety measure and is written into policy and process, but in daily practice it gives way under pressure.

In 2019, technology researcher Madeleine Clare Elish coined a term for what happens next: the “moral crumple zone”. She analysed cases including Three Mile Island, the loss of AF 447 and accidents involving self-driving cars. In highly automated systems, she found, responsibility is often attributed to a human who had only limited control over how the system behaved. The crumple zone in a car protects the person at the wheel. The moral crumple zone protects the technical system, at the expense of the human closest to it. In Tempe, that was the safety driver. In knowledge work, it is the person whose name is on the document.

Legislators know the problem and still rely on the human as the overseer. Article 14 of the EU’s AI Act requires that high-risk systems can be effectively overseen by natural persons. Those overseeing them are to “remain aware of the possible tendency of automatically relying or over-relying on the output”. The law names the problem and answers it with a call for vigilance. Bainbridge cites vigilance research showing that this is exactly where vigilance runs out: even highly motivated people cannot keep their attention for more than about half an hour on a display where hardly anything happens. Article 14 does not apply to most office writing assistants anyway, since they are not classified as high-risk systems. And in summer 2026 the EU postponed the obligations for most high-risk applications to December 2027. In Switzerland, the Federal Council intends to ratify the Council of Europe’s AI convention and make targeted changes to existing law. The consultation on it had been announced for the end of 2026 and was postponed in late October 2026 to spring 2027.

What a Check Needs

The first step is a decision that is rarely taken: for which documents and analyses does the organisation accept that errors from the AI will slip through? That question belongs where decisions about deploying the tools are made, and the answer belongs in writing. Where the residual risk is too high, a check worthy of the name is needed.

Such a check has a defined scope and a time budget. “Open every source and compare it with the claim” is a checking task. “Have another look over it” is not. Writing it into the rules is not enough, though, because this is the familiar gap between work-as-imagined and work-as-done. Whether checking happens shows in the work as it is actually done and in the incentives: if the time saved is counted as more output per person, if fast delivery counts as performance, and if a caught error stays invisible, the pressure lands on the one step where time is easiest to claw back. Unless that pressure is taken off, the check has in effect been dropped, even if it is still written into the process. Bainbridge’s solutions rely on displays that make deviations visible rather than hoping someone will notice them. Applied here, that would mean tools that show which claim rests on which source, and where the model found nothing.

Bainbridge’s point about training transfers most easily. It is the most successful automated systems, the ones that rarely need the human, that may need the greatest investment in operator training. For knowledge work, this means that anyone expected to check AI drafts has to keep doing the work themselves regularly. That costs part of the time the tool was meant to save, and that calculation belongs openly on the table before the tool is introduced.

“Someone will look over it anyway” sounds like a safety measure. As long as the system puts that very check under pressure, the phrase only settles who is responsible afterwards.

Sources

• Lisanne Bainbridge – Ironies of Automation, Automatica 19 (6), 1983, pp. 775–779
• Auste Simkute, Lev Tankelevitch, Viktor Kewenig, Ava Elizabeth Scott, Abigail Sellen & Sean Rintel – Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction, International Journal of Human–Computer Interaction 41 (5), 2025, pp. 2898–2919
• Hao-Ping Lee, Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks & Nicholas Wilson – The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers, CHI ’25, ACM 2025
• Krzysztof Budzyń et al. – Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study, The Lancet Gastroenterology & Hepatology 10 (10), 2025, pp. 896–903
• Marilynn Larkin – Is AI Use Causing Endoscopists to Lose Their Skills?, GI & Hepatology News (MDedge), 18 September 2025 (interview with the lead authors)
• Madeleine Clare Elish – Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction, Engaging Science, Technology, and Society 5, 2019, pp. 40–60
• Erik Hollnagel – The ETTO Principle: Efficiency-Thoroughness Trade-Off. Why Things That Go Right Sometimes Go Wrong, Ashgate 2009
• Associated Press – Deloitte to partially refund Australian government for report with apparent AI-generated errors, 7 October 2025
• European Union – Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Art. 14; amended by Regulation (EU) 2026/1744
• Schweizerischer Bundesrat – KI-Regulierung: Bundesrat will Konvention des Europarates ratifizieren, press release, 12 February 2025
• Schweizerischer Bundesrat – Bundesrat nimmt Zwischenstand der KI-Regulierung zur Kenntnis, press release, 28 October 2026