Agentic code quality: tests, constraints and what still needs a human

Quality moves into the constraints

Addy Osmani put the idea plainly in Agentic Code Quality, republished on O'Reilly Radar on 15 October: “Software quality now depends on the constraints you set around your agents.” He describes a layered set of quality gates: unit, property and acceptance tests, mutation testing, complexity limits, security policies and deployment rules. Some shape the work before the agent starts, some give feedback while it works, and some decide whether its output can reach production at all.

His summary of the model is a useful rule for any team: “An agent can propose anything. Your constraints decide whether a proposal is safe enough, correct, scoped, and useful, for you and your team to ship.” He also points out that the same things that trip up human developers trip up agents: brittle environments, nondeterministic builds, missing permissions and weak tests. The goal, he writes, is “one where an agent can do real work, get feedback it can trust, and fail without doing much damage.”

The problem: agents write valid but weak tests

If tests are the main constraint, their quality matters more than ever. Several October 2026 papers suggest that is exactly where agents are weakest.

TestSuiteBench, from researchers at ETH Zurich and LogicStar.ai, asked coding agents to write complete test suites for 165 real open-source command-line applications from their specifications alone. Each application came with about 74 planted bugs (mutants), each breaking a single requirement. The generated tests were mostly valid, with 88 percent of them passing on the real implementation, but the suites caught too few of the planted bugs: mutation scores stayed below 60 percent for every agent. The authors conclude: “These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.”

TestPrism looked at the same problem from another angle. Its tests had to accept every valid implementation of a task and reject every invalid one, not just pass against a single reference solution. Across fourteen coding agent setups, the best score was 28.00 percent, compared with 59.67 percent when judged against a single reference. The largest group of failures (50.0 percent of the analysed cases) were faulty checks, where a test exercised the right behaviour but judged it wrongly, followed by coverage gaps (34.6 percent), where tests missed the input combinations that expose a bug.

A third paper, TestJack, audited coding benchmarks themselves. Across six frontier model backends and five benchmarks, it found that about 34.4 percent of agent attempts currently judged correct actually violate the task requirements, lowering the overall resolution rate from 50.6 percent to 33.2 percent. Passing the tests and doing the job are not the same thing.

Passing is not the same as quality

Correctness is also only one part of quality. A study of 65,663 programs from 73 models, Beyond Functional Correctness, ran generated Python code through Pylint, Radon and SonarQube and found systematic differences between models that pass rates do not show, with maintainability as the clearest source of variation.

Coverage has the same limit. In a 13 October Harness post, Rohan Bhaumik and Mrinalini Sugosh give the example of a new timeout path: the existing tests all pass, but none simulate a timeout, so a bug in the new branch slips through. As they write, “It's important to note that code coverage is not the same as quality.” Their conclusion: “Code quality has to keep pace with code generation.”

Kousigan A, writing about turning a definition of done into gates agents cannot skip, makes the same point in one line: “Green CI means the checks you wrote passed, not that the requirement is met.”

Constraints that actually work

Architecture is another constraint tests miss. On Sonar's The Next Commit podcast, Matt Erman, a senior software engineer at Blackboard and author of Grokking Software Architecture, argued that agents are risky because they know every pattern and will mix them together if left unconstrained. His answer is architectural fitness functions, checks that fail the build when code breaks structural rules, plus small phases with a clean commit after each one. “You don't optimize the code, you optimize the feedback loop,” he said. And on trust: “You don't trust the AI agent blindly, but you trust the verification system you build around the agent.”

GitLab's AI-Native Engineering Playbook, introduced in a 20 October GitLab blog post, recommends a layered verification system: objective checks such as formatting, types and dependency policy close to authoring, small changes, expected behaviour defined up front, an independent reviewer, review depth matched to risk, and a record of every agentic review decision.

What still needs a human

None of this removes people. It changes where their attention goes. Osmani argues that human attention is scarce and should go to the problems that need judgment, such as taste, intent and architecture. GitLab adds a warning about rubber-stamping: “A checkpoint nobody actually exercises is worse than none, because it creates confidence where the system needs scrutiny.”

In practice, a person should still own the acceptance criteria (what done actually means), review changes to sensitive areas such as payments, authentication and customer data, judge whether new tests really prove the behaviour, approve anything that cannot be undone, and decide when an agent has earned more autonomy based on evidence.

A practical checklist for agent-written code

• Write the acceptance criteria before the agent starts, so there is something concrete to check.
• Make lint, type checks, the build and a stable test suite required checks that block the merge.
• Do not let the same agent write the code and be the only judge of its tests: add mutation testing, a second independent agent or a human review of the tests themselves.
• Track coverage on changed lines, not just one repository-wide percentage.
• Encode architecture rules as automated checks so violations fail the build.
• Keep changes small and commit after each verified phase so regressions are easy to trace.
• Require a named human approval for payments, authentication, data handling and deployments.
• Turn every bug that slips through into a regression test.

Where CoreCodify fits

Whether a website or app is written by people, agents or both, what clients get is the result: a product that works, stays fast and is easy to change next year. Agentic tools let small teams ship far more, but only if the tests, gates and human checks around them are designed with the same care as the code.

At CoreCodify we design and build websites, mobile apps, UI/UX and AI solutions, and we test and review the work before it goes live. If you want a website or app built with proper quality checks, or help adding AI features to your product the safe way, hire me for custom work at usamify.com.