Harness Engineering: How a Small Team Builds and Operates Twinbook

Twinbook Trust & Engineering series · ~10 min read · engineering culture

There's an idea gaining traction in AI labs called harness engineering. The short version: a powerful model on its own doesn't get much done. What turns a model into a system that ships real work is the harness around it — the tools it can call, the context it's given, the memory it can rely on, the checks that catch its mistakes, and the loop that lets it act, observe, and correct. The model is the engine; the harness is the car.

We've taken that idea and pointed it inward. Twinbook isn't built by a large engineering org. It's built and operated by a small team that uses AI agents — inside a deliberately engineered harness — to do the operational heavy lifting: ingesting content at scale, running deployments, testing the system, and keeping a durable, written memory of how everything works. This post is about that harness, because it's a big part of how we move quickly without getting sloppy about the things (security, correctness, learners' data) where sloppiness is unacceptable.

The core insight: write the harness down in Markdown

The most boring and most important component of our harness is plain Markdown files.

Over time we've built up a layered, version-controlled body of operational knowledge that both humans and agents read:

Project rules — a top-level set of non-negotiables ("fix root causes, don't add fallbacks," "this field must come from the UI, never inferred," "never remove this auth middleware"). These act as guardrails that every agent and every change is held to.
Runbooks — step-by-step playbooks for repeatable, high-stakes operations (e.g., ingesting a full textbook into a clean public space: how to split it, how to detect chapters, how to deploy the three services involved, and the specific failure modes to watch for).
Specs — design documents for substantial features, written before implementation, including the hard parts, the trade-offs, and what "done" looks like.
A persistent memory index — a curated record of what we learned the hard way: the incident where a green CI run silently didn't deploy; the time a config drift made a fix invisible in production; the exact reason a deployment looked healthy but was serving a week-old revision.

This matters because context is the scarcest resource in agent work. A capable agent with no memory re-derives (and re-breaks) the same things every session. By writing the harness down, every operation starts from accumulated experience instead of a blank page. The runbook is the institutional knowledge — and it doesn't leave when a tab closes.

Agents do the toil; specs and checks keep them honest

We use agents for the operations that are repetitive, scriptable, and verifiable:

Content ingestion at scale. Bringing a large textbook into Twinbook is real work — splitting a thousand-page PDF into chapter-sized parts, driving each through the pipeline, and verifying the result. We've run these ingestions with orchestrator scripts and agents working from a runbook, handling dozens of source files in a session. The runbook encodes every lesson from the previous ingest, so each one is cleaner than the last.
Deployments. Our deploys run through CI, but the verification — did the new revision actually take traffic? did the route mount correctly? did the secret resolve? — follows a written ritual, because we've been burned by "CI is green" not meaning "the fix is live."
Security testing. Our pre-launch security verification was run as a structured, autonomous pass against every reachable surface, with non-destructive probes and a written report of what passed, what failed, and what got fixed (the basis for our Trust & Security pages).

But — and this is the part that matters for anyone trusting us with their data — the harness is built around discipline, not autonomy for its own sake:

Substantial work starts from a spec that a human reviews.
Changes are verified against explicit success criteria, not declared done because a script exited zero.
High-stakes surfaces (auth, data deletion, anything touching learner data) are governed by hard rules and human review, not left to an agent's judgment.
We treat agent output the same way we treat any other input: as a draft to be checked, not as ground truth. (That's the same principle behind how we treat AI output for learners — see Our Approach to AI Safety in Education.)

The point of the harness isn't to remove humans. It's to remove toil so the humans can spend their attention on judgment, security, and correctness.

A verification culture, written down

If there's one theme in our operational memory, it's "green doesn't mean done." Some of the most valuable entries in our harness are scar tissue:

A CI pipeline that passed while the service quietly kept serving an old revision (a missing dependency). Lesson written down: verify the new revision took traffic, not just that CI was green.
A route that worked locally but 404'd in production because a mount was excluded from a sync step. Lesson written down: diff the route mounts after every sync.
A "login works but data fails" symptom that was actually database connection exhaustion masquerading as an auth error. Lesson written down: check the real root cause before trusting the surface error.

Each of these became a permanent checklist item. The harness gets more reliable over time precisely because failures are captured as durable, reusable knowledge instead of being re-learned.

Why this is the right model for a learning company

There's a nice symmetry here. Twinbook's product is about memory — capturing what you've learned so you don't have to re-learn it. Our engineering practice is the same idea applied to operations: capture what the team has learned, in writing, so we don't re-learn it the hard way.

For a small team, harness engineering is leverage. It lets us:

ingest large amounts of high-quality content reliably,
ship and verify changes quickly without skipping the checks that protect users, and
keep a single, durable source of truth for how the system works and why.

And it does this without compromising the things we refuse to compromise on. The guardrails, the specs, the verification rituals, the "fix the root cause" rule — those aren't friction we tolerate despite using agents. They're the harness that makes using agents safe.

What we're still figuring out

Harness engineering is young, and we're learning as we go: how much to encode in rules versus leave to judgment, how to keep written memory from going stale, where automation genuinely helps versus where it adds risk. We'll keep writing about it. But the core bet — that the harness, not the model, is where the real engineering lives — has held up well enough that it now shapes how we build everything.