Monitoring
Prompt evals
If your code calls an LLM, inline finds the prompts in it, writes test cases for each one, runs them and grades the answers. You get a pass rate per prompt without writing a dataset.
Enabling a repository and setting its defaults are admin tasks. Anyone with evals access can start a run and read what came back.
#Running one
Prompt quality scorecard
inline auto-discovers the LLM prompts in a repository, synthesizes happy-path, edge, adversarial, and boundary test cases for each, runs them, and grades every output against an auto-derived rubric — surfacing where your prompts are fragile.
Enabled repos
3
Avg pass rate
87%
Completed
12
Recent runs
14
Run an eval
Point it at a repo — no dataset or setup needed.
Recent runs
Run an eval lists your enabled repositories with a button beside each. Choose one and inline reads the repository, finds the prompts, synthesizes cases for them, runs those cases and grades the output.
The badge on a run says which of those it is doing: Discovering, Synthesizing, Running, Judging, then Complete. A run in progress can be opened and watched.
#The four kinds of case
inline writes cases in four buckets, and a prompt that handles the first and fails the rest is exactly what this is built to surface.
| Bucket | What it tries |
|---|---|
| Happy path | The input the prompt was written for. |
| Edge | Unusual but legitimate input. |
| Adversarial | Input written to make the prompt misbehave. |
| Boundary | Input at the limits, where behaviour tends to change. |
Case buckets on the settings tab decides which of the four a run covers.
#Reading a scorecard
A finished run shows a pass rate and the case count behind it, so you can see the size of what was tested as well as how much of it held. Avg pass rate across the workspace sits at the top of the page.
Each output is graded against a rubric inline derives from the prompt itself, and the run says why a case failed rather than only that it did.
#Setting up a repository
The Prompt Evals Settings tab holds the per-repository defaults.
| Setting | What it does |
|---|---|
| Cases per prompt | How many cases are written for each prompt found. |
| Pass threshold (0–1) | The score at or above which a case counts as passed. |
| Cost cap (USD) | A ceiling on one run, so a large repository stops rather than continuing. |
| Subject model override | Runs the cases against a model you name instead of the one the repository uses. |
#Gating a merge on prompt quality
Merge gate posts a check that fails when a commit's pass rate drops below Gate pass-rate floor (0–1), which is how a prompt regression is caught the way a failing test would be.
It works like the review merge gate, and The merge gate covers how a check behaves on a pull request.
An optional schedule runs an eval without anyone asking, so the scorecard stays current as the prompts change.
Prompt evals are included with Enterprise. See Plans and billing.