How to build production-ready software assisted with AI tools

How to properly use AI tools (and the harness around them) to ship production-ready software instead of plausible-looking code.

Introduction

I've hit this more than once: the AI generates a 15-file change in two minutes, it looks promising, the tests pass, and a few days later I find out it quietly deleted the one test that would have caught a bug and padded the codebase with boilerplate nobody asked for. "Vibe coding" makes fast output easy. Production-ready is a different problem.

Over the last year, I wrote twice about how I actually code with AI: first the Cursor and Claude Code Max setup, then a real pull request done plan-first and test-driven. Both worked at the time, but reading them now, the weakness is obvious: I was enforcing all the discipline by hand. Remember to plan before writing. Remember to ask for the failing test first. Remember to run the program, not just trust a green unit test. I was the senior engineer, and the AI was the junior. I was also the only thing standing between the junior and production.

This post is about the next step: moving that discipline out of my head and into the machinery around the model. The framework I use for it is Pilot Shell, created by Max Ritter, and I run it every day on both Claude Code and Codex. Back in that August 2025 post, I mentioned an evening workshop in Munich with Max; he turned a lot of the ideas from that conversation into Pilot Shell, so this is very much a follow-up.

But before the tool, the diagnosis. I got it wrong for a long time.

The real problem isn't the model

Left alone, the model doesn't write bad code. It writes plausible code, very quickly, and plausible is exactly the trap. My recurring failure modes, after a year of daily AI-assisted work:

• You ask for one thing, the diff touches eight files.
• The test gets skipped, or written so it passes against the buggy code.
• "Tests pass" is reported as "it works", when the program was never actually run.
• Yesterday's design decision is gone, so today it gets re-derived differently.
• When a test is in the way, the test gets deleted instead of the code being fixed.

For a long time I treated these as prompting problems. Write better instructions, add a CLAUDE.md, and remind it more firmly. It never held: a prompt is a one-time suggestion, not a standing rule, and nothing about it survives a context reset.

Look at that list again, though, and none of these are really model failures. They're failures of what's around the model:

A better model next year doesn't fix this list. It makes the code arrive more quickly.

From prompt engineering to harness engineering

The industry has been converging on the same conclusion, and you can read it in how the vocabulary has shifted. First everything was prompt engineering: what you tell the model. Then came context engineering: what the model sees, the realization that the content of the context window, not the cleverness of the instruction, dominates output quality. The current frontier is scaffold engineering, or harness engineering: what runs around the model, the deterministic machinery of checks, gates, workflows, and state that the model itself cannot skip, forget, or argue with.

flowchart LR
    PE["Prompt engineering<br/>what you tell the model"] --> CE["Context engineering<br/>what the model sees"] --> SE["Scaffold engineering<br/>what runs around the model"]

Here's the part that took me a year of daily use to internalize: these are not three problems. They're one problem, and solving any one of them alone buys you almost nothing.

• Perfect prompts with no scaffold: the model follows your standards for one session, then drifts. I've watched a careful TDD instruction dissolve on the very next context reset.
• Perfect context with no scaffold: the model knows exactly how your codebase works and still declares itself done without running anything, because nothing stops it.
• A scaffold around bad context: gates firing on work that was misguided from the start, enforcement wrapped around confident guessing.

The teams getting production-ready output from coding agents are the ones running all three as a single system. And that raises an uncomfortable question for everyone else: who is going to build and maintain that system for you? It's a real engineering project, one whose ground shifts every month as models, CLIs, and best practices move.

Pilot Shell: the harness as a product

That's what Pilot Shell actually is, and why I stopped thinking of it as "a config for Claude Code". It's the full harness (prompt engineering, context engineering, and scaffold engineering) shipped as a single solution on top of the extension points that Claude Code and Codex CLI already provide: same model, no fork, one install. Everything runs locally, which, for client work, is often the property that gets it approved at all.

flowchart TB
    subgraph P[Pilot Shell — one install]
        direction LR
        R[Standards & rules] ~~~ T[Context tools] ~~~ M[Memory] ~~~ H[Enforcement hooks] ~~~ W[Workflows]
    end
    P --> A[Claude Code / Codex CLI]

The mapping to the three disciplines is direct: the standards are prompt engineering matured, the context tools and memory are context engineering, and the hooks and workflows are the scaffold.

Here's the harness the way it shows up in my day, high-level on purpose. Each layer deserves its own detailed post, and I'll get to those.

Standards it can't forget: prompt engineering, grown up

The mature form of prompt engineering isn't a better prompt; it's standing standards that load into every session and get treated as law, not as a suggestion from forty messages ago. Pilot ships them curated: parsimonious testing (behavioral tests, no coverage padding, no ten-redundant-tests-per-class), real verification ("tests pass" is never accepted as "it works"), the lineage test (every changed line must trace back to what I asked for), systematic debugging (root cause before fix), hard git safety.

The part that makes this mine rather than generic: I run /setup-rules once on a repo and it reads the codebase and writes project-specific rules on top of the built-in ones. When a project conflicts with a default (say, it wants strict TDD on every change), a small override file changes only that section, not the whole rulebook.

Context instead of confidence: context engineering, both directions

Context engineering cuts in two directions, and Pilot works both.

Getting the right things in: most bad AI output comes from the model not knowing the codebase and guessing confidently, not from a reasoning failure. Pilot replaces guessing with lookups: a code knowledge graph for structural questions (who calls this, what breaks if I change it), hybrid semantic search for intent questions ("how does authentication work here"), live library documentation instead of trained-in memories of an old API version, and real browser automation so a frontend change is verified against the actual page.

Keeping the wrong things out: the context window is a budget, and most setups squander it. Pilot loads language standards only when the matching file type is in play, compresses dev-tool output before the model reads it, and returns search results as matching chunks instead of whole files. A lean window is cheaper, and the model reasons better when it isn't wading through output it never needed.

Memory: context engineering across time, and across the team

The sharpest limitation of a bare agent is that it's stateless: every session starts from zero. Pilot treats memory as context engineering across time. Meaningful events (decisions, bug fixes, gotchas, trade-offs) get captured as I work, summarized, and stored locally. Reopen the project and the relevant history is there at session start: what I decided, what bit me, what we chose not to do and why. And because this memory lives outside the context window, it survives both compaction and session boundaries.

The newer part is that this stopped being personal. Team memories travel through the repo itself: captured decisions are committed like any other file, reviewed like any other diff, and load automatically at every contributor's session start, on Claude Code and Codex both. The new teammate's agent knows on day one why the payment retry works the way it does. No cloud service, no sync infrastructure; git carries the "why" just as it carries the code.

Enforcement that doesn't get tired: the scaffold itself

Standards alone are still suggestions. The scaffold is what makes them non-negotiable: small deterministic checks firing on lifecycle events (every edit, every tool call, every attempt to stop), whether or not anyone remembers them.

Every file edit is linted, formatted, and type-checked the moment it lands. A TDD check notices when production code is written without a failing test in place. And my favorite: the agent cannot declare a workflow finished while verification is still red. It gets blocked at the stop and sent back until the work is verified. Done means verified, not "the model feels finished".

Two properties matter more than any individual check. The enforcement is deterministic: string comparisons and exit codes, not an LLM judging another LLM, so it can't be argued with. And it's invisible while you work: none of it depends on my attention or my discipline on a given day. The scaffold also keeps long sessions survivable. When the context window fills and compacts, the active plan and task state are snapshotted and re-injected, so the model never loses the thread mid-feature.

Workflows matched to the shape of the work: the scaffold's rails

On top sit the workflows I actually type. There are four, and the routing question is never "how big is this?" It's what is the work measured against?

/spec is the one I live in for planned work: it explores the codebase, asks its clarifying questions, writes a plan with a definition of done per task, has the plan independently reviewed for risky assumptions, and then waits for my approval before a single line is written. After approval, it implements test-first, and the verify phase runs the full suite plus a real end-to-end pass before an independent review reads the final diff against the plan.

flowchart LR
    E[Explore] --> P[Plan] --> A[Approve] --> I[Implement] --> V[Verify] --> D[Done]
    V -- issues found --> I

/build is the newest and the most interesting, because it inverts the contract. Some work shouldn't be planned up front. The approach is better discovered while building: a landing page that should feel alive, a port that should come out better than the original. So /build locks the definition of done instead of the path: you name the end state, it freezes acceptance criteria before any building starts, then works in rounds (build, judge every criterion pass/fail, turn failures into the next round's tasks) until the criteria actually pass. No drifting scores, no quietly lowered bar, a hard ceiling on rounds, honest reporting of anything that wouldn't close. It runs autonomously from goal to hand-back precisely because it was fixed before the loop began.

flowchart LR
    G[Goal] --> C[Freeze criteria] --> B[Build] --> J[Judge] --> V[Verify] --> H[Hand back]
    J -- failed criteria --> B

/fix is the same discipline in a quick lane: reproduce the bug, write the failing test, fix at the root cause, then run the actual program with the original input to confirm the symptom is gone. And it knows its limits: a bug that needs architectural change gets handed to /spec instead of quietly growing into a rewrite.

flowchart LR
    I[Reproduce] --> R[Failing test] --> F[Fix root cause] --> V[Verify live] --> D[Done]
    I -- too big --> S["/spec"]

When I want the work isolated, any of the three runs in its own git worktree and merges back only after approval. My branch stays clean, and several runs can even proceed in parallel.

For teams and enterprises

Here's the argument I now make to enterprise and team clients, because it's where this stops being a personal productivity story.

Every serious engineering organization adopting Claude Code or Codex CLI faces the same build-or-buy moment, whether they've named it or not. The agent arrives as a brilliant, unharnessed capability, and someone internal starts accreting the harness around it: a CLAUDE.md here, a hook script there, a homegrown planning convention, a memory hack. Six months later, every team has a different half-finished scaffold. None of it is maintained, and the ground has already shifted beneath it (new models, new CLI capabilities, new best practices). Harness engineering is a moving target, and chasing it is a full-time platform job that almost no team actually budgets for.

Pilot Shell is the answer to that build-or-buy question: the current best practices for running these agents (the standards, the enforcement, the context machinery, the workflows) as one maintained baseline that tracks the moving target for you. When the model generation changes or the CLI grows a new capability, the harness updates; your team doesn't rebuild it.

And critically, a shared baseline is not a straitjacket. The flexibility is designed in, because no two organizations have the same stack:

• Your languages and frameworks: per-language standards activate by file type, and /setup-rules generates rules from your codebase's actual conventions, whatever the stack.
• Your industry and environment: everything runs locally, nothing leaves the machine, which is what makes it usable in regulated and client-confidential settings in the first place. It works in any existing project, with no scaffolding-the-repo or restructuring required.
• Your practices: any built-in rule can be overridden per project, a section at a time, not fork-the-rulebook. Custom rules, skills, and extensions sit alongside the shipped ones and survive updates.
• Your team: skills, rules, and extensions sync across machines and teams through a connected repo, so the whole organization runs one baseline instead of five diverging personal setups, and improves it in one place.

That's the enterprise pitch in one sentence: use Claude Code and Codex CLI with the best current harness on day one, keep it current without staffing, and keep your own languages, frameworks, and standards.

What it costs to run

Here's the objection I'd have raised a year ago: a harness this thorough (reviews, verification, browser runs) surely burns tokens, and tokens are money.

The opposite turned out to be true, because context engineering's "keep the wrong things out" direction also cuts cost directly:

• Tool output is compressed before it enters the context — 60–90% of typical dev-tool noise gone before the model ever reads it.
• Search returns answers, not files — matching chunks instead of whole files, orders of magnitude less context than the grep-and-read loop, for a better answer.
• Only the relevant standards load — a Python session never pays for the TypeScript rulebook.
• The right model does the right job — model switching runs the expensive reasoning where it changes the outcome: the strong model plans, the efficient model implements and verifies, automatically per workflow phase.

In practice, the context these measures save more than covers the tokens the added verification spends. And you don't have to take that on faith, because spend is visible. That brings me to the last piece.

The Console

Everything above runs headless, but it's all observable in one place: the Console, a local dashboard showing my sessions, the memory the system has captured, every plan and build with its live status, and token usage with actual cost trends over time. It's also where the knobs live: how much autonomy each workflow gets, which reviews run, whether plans need my approval, what gets shared with the team.

flowchart TB
    subgraph C[The Console — local dashboard]
        direction LR
        S[Sessions] ~~~ M[Memory] ~~~ P[Plans & Buildouts] ~~~ U[Usage & costs] ~~~ SH[Sharing] ~~~ ST[Settings]
    end

Two things about it matter to me. It's local, like everything else, so nothing about my sessions or my client's code leaves the machine. And it changes the relationship with the "black box": when the agent is working through a plan, I can watch the plan file move through its states, annotate it mid-run to steer without stopping, and see exactly what a session costs.

Where the human stays

The harness runs the process. It does not run the project, and that split is the design, not a limitation of it.

What gets built is mine. The agent can find an approach, but it has no taste: which feature is worth its complexity, what good looks like for these users, where the bar sits. That judgment enters through the plan I approve, the acceptance criteria I set, and the questions I answer before a build starts. An agent can pass every mechanical gate while solving the wrong problem. Deciding what the right problem is was never its call.

Control is mine. Nothing irreversible happens without me: no plan gets implemented before I approve it, nothing merges before I've read the diff, no git write runs without my say-so. Every gate sits before the irreversible step, not after it.

And responsibility is mine. I ship the code, so I answer for it, to the client, in the incident review, in production. "The AI wrote it" is not an answer anyone accepts. What the harness provides is the ability to carry that responsibility honestly: verified work, a readable diff, a record of what was decided and why. It gives me the evidence. It never takes the accountability.

Conclusion

Go back to that deleted test I opened with. No better prompt was ever going to catch it, and no better model will either, because it was never an intelligence problem. Every team runs the same models; the difference in what ships comes from the harness around them. Prompt engineering, context engineering, scaffold engineering: one problem, and it has to be solved as one system.

For any team whose AI output is fast but not shippable, the fix is to decide who builds and maintains your harness, not to write better instructions. Pilot Shell is that harness as a product: installed in an afternoon, running entirely on your machines, kept current without you chasing the moving target, and flexible enough to stay with your stack and your standards. The result is work that holds up in code review and in production, not more impressive demos.

This was deliberately the high-level tour. There's real machinery under every layer here, and I want to turn it into a technical series next. So I'll leave it as an open question: what would you want to read about first? Write me.

Happy shipping!

This post was written with AI assistance and verified plus enhanced by a human.