End-to-End Testing LLM Features Without Calling an LLM

Picture the Playwright spec: log in, create a job, click Generate, and assert on... what, exactly? The model returns different screening questions every run, so the assertion is either so vague it tests nothing or so specific it flakes tomorrow. Add tens of seconds of generation latency per flow, a real API bill on every push, and a provider key on every CI runner, and you have a suite that costs money to tell you nothing reliable.

That is the wall every AI product team hits, and it is why most of them quietly stop writing end-to-end tests for their AI features at all. We refused to — here is the seam that made it possible.

The Problem

Real LLM calls in an E2E suite are a disaster on four axes:

• Non-determinism — the same prompt returns different output on every run, so assertions flake
• Cost — a CI suite that runs on every pull request burns real money on every push
• Latency — generation flows take tens of seconds, blowing up suite runtime
• Secrets — every CI runner and every contributor needs provider API keys

The usual answer is mocking at the HTTP layer. That is brittle: provider request formats change, streaming responses are painful to fake, and the mock lives far away from the code it stands in for.

The Solution: Fake at the Model Boundary

Our LLM layer is built on a framework-agnostic core package. That gave us a single seam: the point where the application asks for a chat model. We added one environment flag:

LLM_FAKE=1

When it is set, dependency injection hands every consumer a deterministic, in-process fake model instead of a real provider client. The fake honors the same interface — streaming, tool calls, structured output — but returns predictable content synthesized from the request.

Everything else in the stack stays real:

• A dockerized environment: Postgres, a fake GCS server, and the API
• The real SSO login flow, using the identity provider's dev tenant with a fixed dev OTP
• The real web bundle, built in a dedicated e2e mode
• Playwright specs that log in once per role and reuse the frozen session

The specs drive complete generation flows — create a job, generate screening questions, score a candidate — and assert on real UI state, because the fake's output is stable across runs.

Key Lessons

Fake at the boundary you own, not the wire. Swapping the model at the dependency-injection layer means one fake, typed against our own interface, instead of a fleet of HTTP mocks tracking third-party API changes.

Keep everything else real. Database, auth, storage, and routing all run for real. The suite has caught auth-guard regressions and migration issues that a mocked backend would have hidden.

Determinism is a feature you design for. Once outputs are predictable, specs can assert on actual rendered content instead of vague "something appeared" checks.

Zero-key tests change contributor behavior. Anyone can run the full suite locally with no provider account. Tests that are free and fast actually get run.

Results

• Full-stack E2E coverage of every LLM-powered flow with zero provider API calls in CI
• No API keys provisioned to CI runners or contributors for testing
• Generation-flow specs are as stable as ordinary CRUD specs — model variance is no longer a source of flakes
• The suite has caught real regressions a mocked backend would have hidden — auth-guard failures and migration issues, not just UI drift

Related Reading

• Taming LLM Variance: Ensemble Scoring with Outlier Rejection — handling model variance in production instead of in CI
• Prompt Engineering Without Guesswork — measuring prompt quality with evals instead of intuition