Eval Studio — User Guide
Eval Studio is a playground for changing how a prompt behaves — the wording, temperature, or model — and seeing exactly what that change does to quality, cost, and latency before it ever reaches production. Everything on this page is a real feature in the app; nothing here is aspirational.
Getting started
Prod & Experiment panels
Pick a prompt path from the dropdown in the header (e.g. vision/product-identifier). Two panels appear:
- Prod (left, read-only) — the current live prompt, temperature, and model for that path, exactly as configured.
- Experiment (right, editable) — starts as a copy of Prod. Edit the prompt text directly, and change temperature or model independently.
Nothing you do in the Experiment panel touches Prod. There's no "save" or "publish" step — this is a sandbox for trying changes, not a way to edit what's live.
Comparing text
Show Diff
Click Show Diff to see exactly what changed between Prod and your Experiment prompt, highlighted word-by-word in place:
If there's no difference yet, it just says "No diff" next to the button — your prompt text is never overwritten with a placeholder message.
Editing the box again (even one character) drops the highlighting and goes back to plain, editable text. The diff is a snapshot of that one comparison, not a live view.
Before you run anything
Sanitize Prompt
Click Sanitize Prompt to scan your Experiment prompt for four kinds of problems:
| Category | Catches | Redacted? |
|---|---|---|
| Secret | API keys and credentials (Anthropic, OpenAI, AWS, GitHub, Slack, Google, private key files) | Yes |
| PII | Real-looking personal data — names tied to identifying details, emails, phone numbers, addresses | Yes |
| Injection | Embedded text trying to hijack the prompt's instructions ("ignore previous instructions", etc.) | Yes |
| Quality | Garbled or mistyped text that doesn't read as coherent English | No — advisory only |
If anything's found, the box shows the proposed change highlighted in place, and a findings panel appears below with what was flagged and why:
Nothing is applied automatically. You get two choices:
- Accept — replaces your Experiment prompt with the redacted version.
- Dismiss — keeps your original text exactly as it was.
Automatic, not a checklist
Evalset & baseline metrics
Each prompt path has a default evalset — you don't pick one. As soon as you select a path, its evalset name appears and base (Prod) metrics start loading immediately, before you've touched anything else.
The first time a given path + evalset pairing is evaluated, this means a real Prod run happens in the background (shown as "Computing baseline…") — every run after that is instant, because the result is cached. Run Eval stays disabled until the baseline is ready, so there's always something to compare against.
The main event
Running an eval
Once the baseline is loaded, click Run Eval to run your Experiment prompt/temperature/model against every item in the evalset. Each item gets:
- A real response from the model you chose
- A quality judgment from an LLM-as-judge, scored against a rubric or reference answer
- Real token usage → real dollar cost, and real wall-clock latency
This can take anywhere from several seconds to a couple of minutes depending on the evalset size — the page polls automatically and updates when it's done.
Base vs. Experiment
Reading the comparison table
Once a run finishes, a table compares your Experiment's results against the cached Prod baseline — color-coded so you can tell at a glance whether the change helped or hurt:
| Metric | Base | Experiment |
|---|---|---|
| Quality | 0.92 (± 0.08) | 0.97 (± 0.05) |
| Cost | $0.0156 | $0.0231 |
| Latency | 1840ms (± 320ms) | 1210ms (± 190ms) |
The color rule is per-metric, not "higher is always green": quality is better when it goes up, cost and latency are better when they go down. A tie is never colored — it's neutral, not a win. Item count is purely informational and never colored.
One level deeper
Full results page
Click View Results → on any completed run to see every evalset item side by side: the query (plus an image thumbnail for vision items), the Base response, and the Experiment response — each with its own score and judge breakdown:
1.00 ✓ matches_reference — hover any criterion to see the judge's full explanation.
This is the page to open when the summary numbers alone don't tell you enough — e.g. quality looks about the same, but you want to see whether the actual wording changed in a way you care about.
Finding your way back
Recent Runs
The main page always lists your most recent eval runs — path, evalset, quality/cost/latency, status, and a link straight to that run's results page. This list comes from the server, not your browser, so it's there regardless of which tab or link you used to get to a result.
Nothing gets lost
Coming back later
If you view a run's results and then click ← Back to Eval Studio, you land back exactly where you were — same prompt path, same edits in the Experiment box, same comparison table on screen. You don't need to re-select anything or re-run anything just because you navigated away to look at details.
Good to know
FAQ
Why did selecting a path immediately cost money / take time?
The first time you evaluate a given path against its evalset, Prod has to actually run so there's something real to compare against. That result is cached — every visit after that is instant and free until the underlying prompt or evalset changes.
What does [REDACTED:secret] mean in my prompt?
Sanitize found something and you clicked Accept. The bracketed tag tells you what category was removed (secret, pii, or injection) — the original text is gone from the box, but nothing outside that exact span was touched.
Can I edit what "Prod" shows?
No — Prod is always read-only, sourced live from the prompts registry. If it's wrong, that's a registry change, not something to fix from this screen.