inline
Sign in
Get started

Test the prompts in your code

If your code calls an LLM, inline finds the prompts in it, writes test cases for each one, runs them and grades the answers. You get a pass rate per prompt without writing a dataset, and a prompt change that makes things worse can hold the merge like a failing test.

Found, written,
run and graded

Choose a repository and inline reads it, finds the prompts, synthesizes cases for them, runs those cases and grades the output. The run's badge shows where it is: Discovering, Synthesizing, Running, Judging, then Complete. A run in progress can be opened and watched.

Prompt evals7 prompts discovered
Graded

142

Mean

3.8/5

Failing

9

review/system-prompt.ts
48 cases · faithfulness 4.4 · safety 4.9
4.6
triage/classify.ts
31 cases · precision 3.1 on edge inputs
3.4
walkthrough/layers.ts
24 cases · ordering drifts on big diffs
3.9
memory/distill.ts
39 cases · no adversarial failures
4.4

How it works

Four kinds
of case

Happy path is the input the prompt was written for. Edge is unusual but legitimate input. Adversarial is input written to make the prompt misbehave. Boundary sits at the limits, where behaviour tends to change. A prompt that handles the first and fails the rest is exactly what this is built to surface.

A scorecard that
says why

A finished run shows a pass rate and the number of cases behind it, so you see the size of what was tested as well as how much held. Each output is graded against a rubric inline derives from the prompt itself, and every failed case says why it failed.

Gate a merge on
prompt quality

The merge gate posts a check that fails when a commit's pass rate drops below the floor you set, so a prompt regression is caught the way a failing test is. Per repository, choose the cases per prompt, the pass threshold, a cost cap per run, and a model to test against instead of the one the code uses.

Included

No dataset to writecases synthesized from each prompt
Four bucketshappy path, edge, adversarial and boundary
Derived rubricseach output graded against the prompt's own intent
Cost capa ceiling per run, so a large repository stops
Subject model overridetest the prompts against another model
Merge gatea check that fails below your pass-rate floor

Related

Put it on one repository

Connect GitHub or GitLab, enable a repo, and the next pull request opened gets a review.

Get startedSee everything inline does