An AI coach that can't hallucinate

Scoutr.gg is a real-time coach for League of Legends: it watches your minimap, tracks the enemy jungler, and talks to you while you play. The single hardest constraint in building it had nothing to do with computer vision or latency. It was this: a coach that confidently says something false is worse than no coach at all.

If the app tells you "jungler top side, you're safe to push" and the jungler is in the river brush next to you, you don't just lose the exchange — you lose trust in every future call-out. And large language models, left unconstrained, will absolutely do this to you. They don't know when Baron spawns. They'll happily invent it.

So the architecture question became: how do you put an LLM in the loop of a product whose entire value is being right about a game state that changes every second?

The split

The answer we landed on is a hard split into two halves that never blur:

• A deterministic core that runs on your machine. Everything that must be true is computed, not generated: minimap tracking (classic computer vision — HSV masks and template scoring against champion icons, no ML), gank detection, gold differentials, wave-state classification, objective timers, and even the timing rules for when the voice is allowed to speak. This half is plain code. It cannot hallucinate, because it cannot generate — the worst it can do is be wrong the way a thermometer is wrong.

• An AI brain that runs in the cloud. Strategy is genuinely fuzzy — "you're 40 gold up but your wave is crashing and their jungler was last seen bot" doesn't reduce to an if statement. The AI half reads the deterministic half's output and makes judgment calls: what matters right now, what to say about it, what to let go.

The boundary is directional and enforced: the AI layer depends on the deterministic core, never the reverse. The core would work — and does work, when your network drops — with the AI entirely absent. Coaching degrades to "deterministic-only" instead of failing.

The AI picks; it doesn't invent

The most important design decision sits inside the AI half itself. When the coach decides what guidance to give you, the model doesn't freetext coaching advice into your ear. It selects from a curated catalog of directives — pre-written, phase-gated coaching plays, each tagged with the roles it applies to and the game states in which it's even legal to offer.

game state (deterministic, local)
        │
        ▼
eligible directives (gating is code, not model output)
        │
        ▼
LLM picks ≤ 2 — "which true thing matters most right now?"
        │
        ▼
spoken sentence assembled locally, timing rules local

The gating rule — is this directive even offerable in this state? — is deterministic code, computed from live game data. By the time the model sees the menu, everything on it is already true and legal. The model's only job is judgment: which of these matters most in this moment. That's the thing LLMs are actually good at.

Constrain the model to selection, not generation, wherever correctness matters. That one sentence is most of the architecture.

The unreasonable side effects

The split was made for correctness, but it kept paying rent in ways I didn't design for:

Shipping. The desktop app ships without the AI half — no prompts, no vendor keys, no strategy content in the installer. There's a build gate that walks the frozen bundle's import closure and hard-fails the release if any server-only package leaked in. The sensitive IP physically cannot end up on a user's disk, because the packaging step refuses to produce the artifact.

Cost and iteration. Prompts, directive content, and model choice live server-side, so improving the coach's judgment doesn't require shipping an installer. The deterministic half — the part that would be genuinely painful to hotfix — is also the part that basically never needs to change.

Testing. The deterministic core is just functions: game state in, classifications out. It's covered by ordinary unit tests with recorded games as fixtures. The fuzzy half is quarantined behind one HTTP seam, which is the only place where "evaluate the AI" — a much harder problem — even needs to exist.

Where it hurts

Honesty section. The split has real costs:

• The seam is a wire. Every judgment call pays a network round-trip, so the things that must be instant — gank alerts firing at 100 ms cadence — had to live entirely in the local half. You end up designing every feature twice: which side of the wall does it belong on? (The discipline this forces is, admittedly, also a benefit.)
• Two deploy targets. A Windows installer and a cloud service, with version skew between them as a permanent background concern. We ended up building an explicit minimum-client-version floor into the API because of it.
• Offline is a real state, not an edge case. The UI has to be honest about which half is currently powering the coaching, which is more product work than a monolith would need.

Would I make the same call again? Without hesitation. Every painful part of this architecture is a known, bounded pain. The alternative — an LLM improvising claims about a live game — is an unbounded one.

If you're building anything where an LLM's output gets treated as fact, the question worth sitting with isn't "how do I prompt it to be accurate?" It's: which parts of this system are not allowed to be creative? Build those as code. Let the model choose between true things — never let it be the source of truth.

---

Scoutr.gg is in free alpha — how it works, or scout a player right from the browser. I'm Manu; I'm building this solo and writing up the engineering as I go.