Three weeks of agentic refactoring: could we trust the result?

That one was on us.

We use guardrails because coding agents make mistakes. But what happens when the verification itself is wrong? An architecture check doesn't establish which implementation a test process actually loaded. For much of this refactoring, generating code was easier than deciding what should change and establishing whether the result was correct.

I've been developing DATAMIMIC for years. Our DataSourceRegistry, for example, had grown to around 1,000 lines. Runtime contexts, XML statements, script evaluation, file formats, database access. Too much in one place.

We wanted to align Community Edition's architecture with Enterprise Edition, with coding agents doing much of the refactoring.

Agents can change code faster than I can review it. But they don't necessarily understand why we made certain decisions years ago.

With multiple agents working in separate sessions, each with its own context, small local decisions can slowly change the architecture without anyone agreeing to that change.

So we defined the target architecture first. Components, responsibilities, allowed dependencies and public interfaces. A versioned contract in Git, something the agents and I could work against. ArchKeel checks the actual code against that contract.

36 selected Git checkpoints, from 21 September to 12 October 2026. Each tile represents a Python file; rename identities are inferred by Git.

I also develop ArchKeel, so this wasn't an independent evaluation. We'd tested it and run some smaller experiments. I thought its Python support was ready for a larger codebase.

It wasn't. DATAMIMIC exposed weaknesses and edge cases we hadn't seen. Typed dictionaries, callback types, inherited methods. And findings that weren't clear enough to make a decision.

Sometimes we couldn't immediately tell whether the problem was in DATAMIMIC, the architecture contract or ArchKeel.

We were changing DATAMIMIC, our architecture contract and ArchKeel at the same time. I had underestimated how much work verifying ArchKeel would take.

On September 25, an agent working on locale error handling introduced four imports from an internal module:

from datamimic_ce.errors.factory import invalid_locale_error

The import worked. But we'd explicitly decided that services should use the public error interface:

from datamimic_ce.errors import invalid_locale_error

ArchKeel reported four new INTERFACES-ONLY violations. We fixed the imports and ran the check again. All four violations disappeared. No changes to the contract or baseline.

Original agent commit · Correction

It caught the change after the commit. The implementing agent had checked focused tests, lint and types; the architecture check ran later during integration. That left a gap in our agent workflow: this check belonged before the agent handed over its commit.

Four imports aren't a huge issue. But one reasonable local decision had crossed an agreed boundary.

We also had cases where the rule itself needed discussion. The SQL dispatch path, for example, exposed a conflict between our blanket cast ban and the existing dynamic-client contract. We checked the callers and documented a narrow exception to preserve that contract.

We still have to challenge the architecture. ArchKeel should check whether the implementation matches what we've agreed on.

DATAMIMIC is a test-data engine. Customers have existing XML models, and we didn't want the refactoring to silently change their behavior.

We had 931 XML inputs to account for. At one checkpoint we'd reviewed 180. Another 751 were still open.

We linked inputs to existing tests and recorded results, then added focused checks where evidence was missing.

And we had to distinguish between successful executions, expected rejections, known failures, diagnostics and missing local coverage. Some grouped negative tests timed out, so we split them into smaller checks.

Eventually all 931 inputs had an evidence classification. Of these, 874 had paired evidence from both revisions, including 12 with the same pre-existing error. Another 8 were covered by a passing parent run with readback, 1 had an approved behavior change, and 48 were classified as fixture or legacy debt, or had only module-level CI evidence. The technical report preserves the full breakdown.

That doesn't mean all 931 were successfully replayed. I didn't want to turn different kinds of evidence into the same green PASS.

That distinction mattered even more after the module-loading problem from the beginning. We'd challenged the agents' output. We hadn't challenged our own verification setup enough.

On October 12, we merged the refactoring into development. ArchKeel 2.0.0 reported zero declared architecture violations across 488 observed Python files: 471 core files and 17 under resources/.

The final CE component structure at the accepted revision.

The final PR CI passed, including architecture checks, product tests, static analysis, build checks and seeded replay. Some branch-conditioned E2E and release jobs were skipped.

We still had 286 architecture UNKNOWN records: unresolved cases such as inherited methods, property bindings and type aliases. These are separate from the 931 XML inputs.

Those were assessed through separate agent review tasks, but within the same model/provider ecosystem. That's another review pass, not independent external verification.

We also changed our acceptance criteria during the project. Originally the gate required zero material UNKNOWN positions. The final gate accepted zero declared violations with documented assessment of the remaining unknowns. I accepted that because unresolved analyzer cases needed explicit review; forcing the counter to zero could mean changing valid Python just to satisfy the checker. The assessments still record remaining code obligations, and the unknowns remain unknown.

We didn't meet the original gate. We met the revised one.

The codebase didn't get smaller either. Core Python files increased from 437 to 471, and lines of code went up around 3.8%. Some large modules still need work.

So was it successful?

I think responsibilities and boundaries are clearer now. And we have a concrete example where ArchKeel caught an unintended change against the declared architecture.

But that doesn't prove every architecture decision is correct or every existing behavior has been preserved.

What I'd keep is the shared view of the contract and its findings. The saved results identify the source, target and analyzer, so we could inspect the checks themselves rather than rely only on an agent's summary. On October 8, ArchKeel 1.0.0 reported 91 declared violations; on October 12, ArchKeel 2.0.0 reported zero. The checker and target changed in between, so that isn't a clean before-and-after. And zero didn't settle acceptance: the 286 remaining unknowns still needed explicit assessment.

Did the agents make us faster? I don't know.

We spent three weeks on this, but part of that time went into ArchKeel development and fixing the verification. We didn't have a control run or a complete cost ledger, and we didn't track those work streams separately enough to give a defensible percentage. I can't separate their effects well enough to claim a productivity gain.

Next is DATAMIMIC EE Core. We'll start with published ArchKeel 2.0.0, run the architecture check before an agent hands over its commit, and track refactoring work separately from checker development. I want to see how much of what we've learned transfers.

I'd like to spend more time challenging the architecture and less time debugging the checker.

The technical report has the evidence, including the invalidated results, remaining limitations and acceptance changes. As of October 12, the refactoring is merged, but isn't part of a published DATAMIMIC CE release yet.

Where have your own guardrails failed?