Someone posted about a conference where the AI use by academics was, in their words, absurd. The complaint lands. But "people are using AI badly" is not a useful finding on its own, because it doesn't tell you what using it well looks like.
I use AI for a large share of my engineering work. Not for autocomplete. For reviewing whole systems, for reading unfamiliar framework code, for finding the bug I can't see. It works. It also produces confident, well-argued, completely wrong output on a regular basis, and the difference between those two outcomes is almost never the prompt.
It's whether anybody checked.
The rule
A model's finding is a lead, not a fact.
That sounds like a platitude until you've watched what happens when you skip it. Three cases from my own projects, all inside the last few months.
Case one: two models, opposite answers, same question
I was reviewing a payment-splitting module before it went near production. Money movement, so I ran the review as multiple independent passes rather than one, each with its own focus, none of them seeing the others' output.
Two of those passes reached opposite conclusions on the same question: whether a credit memo's net amount cancelled out an over-refund of tax. One said the bug was real. One said it was defused.
Both arguments read as competent. Both cited specific behaviour. There was no way to pick a winner from the text, because the disagreement wasn't about logic. It was about an assumption neither of them had stated. One had silently assumed a line item with no tax on it.
The only thing that settled it was opening the framework's own source and reading the method that registers a refund against a memo item. The field in question is tax-exclusive. That one line of vendor code decided it. The bug was real.
What I want to draw out: neither model was being careless. The failure mode is subtler than hallucination. It's an unstated premise carried confidently through an otherwise sound chain of reasoning, and the only thing that dislodges it is ground truth.
Case two: the green test suite that certified a worse bug
Different project, a personal finance tool that parses transaction alert emails.
A review pass flagged a minor cosmetic issue: HTML entities like   sitting next to an amount were confusing the parser and pushing clean transactions into manual review. Genuinely minor. The fix is obvious: decode numeric entities before parsing.
So I decoded them. Roughly:
text.replace(/&#(\d+);/g, (_, code) =>
String.fromCodePoint(Number(code))
)Then I ran everything. Type check clean. Lint clean. 490 tests passing. Build fine.
String.fromCodePoint throws a RangeError on any value above 0x10FFFF. An email containing �, a 91-byte message, throws out of the ingest function. And because the message stays in the mailbox, it throws again on every retry, forever. The feed stops. Not degrades. Stops.
A batch that should have taken three real transactions took one.
Every deterministic gate I own was green over that. They were green because they test the code I wrote against the inputs I imagined, and I had not imagined an integer larger than Unicode. The fix for a cosmetic annoyance had created something far worse than the annoyance.
It was caught by accident. A second review pass happened to still be running against the tree I'd just changed, and it read the new code.
The generalisation I took from it: a fix is a change that has not been reviewed yet. Re-running the test suite after a fix proves nothing about the class of defect that the test suite couldn't see in the first place. If a review found a problem in some area, the fix to that area needs the same review again, fresh, with no memory of the previous round.
Case three: the brief was the only variable
This one is about how you ask, and it's the cleanest experiment I've accidentally run.
One review pass, one codebase, two deliberate trade-offs in the code. Both were decisions I'd made on purpose, with reasons. In the instructions I gave the reviewer, I explained the reasoning behind the first one and, through nothing but haste, left the second unexplained.
It skipped the first. Correctly. It had the rationale, it weighed it, it moved on.
It reported the second as a critical defect.
Same pass, same code, same model. The only difference was whether the why was in the brief. And if I'd acted on that critical without checking, I would have "fixed" a deliberate decision and reintroduced the thing it was there to prevent.
This is the part people miss when they complain about AI output quality. A reviewer with no context reports every deviation from the obvious as a defect, because from where it's standing that's exactly what a deviation looks like. Context is not a nicety in the prompt. It's the difference between a finding and a false alarm.
What separates augmentation from laziness
Not effort. Not prompt engineering. Not model choice, though that matters at the margins.
It's whether there's a verification step that doesn't run through the same mind that produced the output.
Three things I actually do now:
Re-read the cited code before acting on any finding. Every time. file:line or it didn't happen. Roughly a third of what comes back doesn't survive this, and the ones that don't survive are often the most persuasively written.
Go to the source for framework behaviour. Not the docs, not the model's summary, not my own memory of how the ORM works. The vendor file. This resolved case one and would have caught the tax-exclusive assumption before either pass ran.
Never let the author certify the work. If the same context wrote the fix, it isn't in a position to judge the fix. A fresh pass with no memory of the conversation finds things the original never will, precisely because it doesn't share the assumptions.
The uncomfortable bit
None of this is about AI, which is why the academics-at-a-conference complaint slightly misses.
A junior developer's pull request is a lead, not a fact. A vendor's changelog is a lead. A green test suite is a lead: case two is exactly that, five separate green gates over a defect that killed a production feed. My own confident recollection of how a framework behaves is a lead, and it's wrong often enough that I've stopped trusting it.
AI just makes leads cheap. It generates plausible-sounding claims at a rate no human process ever could, which means the verification step is now the bottleneck, and skipping it produces failure faster than it used to.
The people using it badly aren't using it too much. They're verifying too little, and the volume makes that visible in a way it never was before.
Do the checking. It's the whole job now.
Source / further reading: https://lobste.rs/s/qn7jtv