Our Approach to AI Safety in Education

Twinbook Trust & Engineering series · ~9 min read

An AI study tool that confidently teaches you the wrong thing is worse than no tool at all — especially in a field like medicine, where a misremembered fact has consequences. So when we talk about "AI safety" at Twinbook, we don't mean it abstractly. We mean a set of concrete engineering decisions about how the model is used, what it's allowed to influence, and how its output is checked before it reaches a learner.

This post walks through those decisions.

The risks we're actually defending against

We organize our thinking around the OWASP Top 10 for LLM Applications — the community's reference list of how language-model features go wrong. For a product like Twinbook, the ones that matter most are:

  • Prompt injection — a malicious or accidental instruction hidden inside an uploaded document ("ignore your instructions and...") trying to hijack the assistant.
  • Insecure output handling — model output rendered or executed unsafely (e.g., HTML/script injection).
  • Overreliance / hallucination — the model stating something false with confidence, and a learner believing it.

Below is how we address each.

Prompt injection: documents are data, not commands

The core defense is a boundary: content retrieved from your uploads is treated as material to reason over, never as instructions to obey. The assistant's behavior and its access to tools are controlled by Twinbook's own system design, not by whatever text happens to be on page 12 of a PDF. An instruction embedded in a document doesn't get promoted to a command.

We reinforce this with retrieval grounding. The Space Assistant doesn't free-associate over an entire corpus; it answers from a retrieval layer scoped to your materials and cites the sources it used. Grounding-plus-citation does two things at once: it improves factual accuracy, and it shrinks the blast radius of any injected instruction, because the answer is anchored to retrieved passages a user can check.

Insecure output handling: safe rendering by default

Anywhere model output becomes a rendered artifact, we sanitize it:

  • Generated and exported HTML is sanitized against an allowlist — no <script>, no <iframe>, no event handlers.
  • Our PDF / notebook export runs a hardened headless browser with JavaScript disabled and network requests restricted to what's needed. Even if generated content tried to do something clever, the rendering environment won't execute it.

The principle: never assume model output is safe to render. Treat it like any other untrusted input.

Overreliance and hallucination: output is a draft to be checked

We design the product to communicate, in both UI and behavior, that AI output is a study aid, not an oracle.

  • Grounding + citations (above) keep answers tied to source material the learner can verify.
  • We validate structured outputs (questions, flashcards, structured summaries) against expected schemas, so malformed or off-task generations are caught rather than shown.
  • We run an evaluation process over AI outputs — checking that they're grounded in the source and on-task — before relying on them. (There's a dedicated post on this: How We Evaluate AI Outputs Before They Reach Learners.)
  • Where appropriate, the product frames outputs as drafts/aids and encourages checking against primary sources and instruction.

The educational-use boundary

This is a safety control too. Twinbook supports learning, study, and review. It is not a clinical decision-support system, not a diagnostic tool, and not a source of authoritative professional advice. That's not boilerplate — it shapes what we build and what we tell users. For our medical audience specifically, AI output must never be used for patient care, and identifiable patient data shouldn't be uploaded at all without the right institutional agreements (see our HIPAA & PHI policy).

Cost and abuse safety

There's a quieter dimension of AI safety: making sure the AI pipeline can't be driven by someone it shouldn't be. Our internal AI services don't accept anonymous requests — they're reached only through authenticated, attributable internal authorization, and AI usage is metered per account. Together these prevent an outsider from running model work without a valid identity — which protects both our costs and the integrity of the system learners rely on.

What we don't do

  • We don't train or fine-tune models on your content (see How Twinbook Protects Student Learning Data).
  • We don't let document contents control system behavior.
  • We don't render model output without sanitizing it.
  • We don't present AI output as authoritative in domains where being wrong has real cost.

Where we're still improving

AI safety is not a finished checkbox. We're continuing to expand our evaluation coverage, tighten content-rendering policies, and improve how clearly the product communicates uncertainty. We'll keep writing about it here as it evolves — and we'd rather tell you what we're still working on than imply the problem is solved.

More: Trust & Security · How Twinbook Safely Processes Dense Learning Content · How Twinbook Protects Student Learning Data.