# Observability

# Observability

inline reviews code before it ships and watches the infrastructure it runs on. Observability closes the loop: it catches what your users actually hit in production.

The mechanics are familiar if you've used an error tracker — an SDK reports exceptions, they're grouped into issues, you triage them. What's different is what happens next. inline already has authenticated, indexed access to the repository your app was built from, so an issue doesn't stop at a stack trace: an agent opens the real source **at the commit that produced the failing build**, follows the value that went wrong, and writes the patch.

Optional per-workspace module. Nothing runs until you create a project.

## The shape of it

```
your app  ──▶  POST /api/ingest/v1/<publicKey>
                 │
                 ├── symbolicate frames against uploaded source maps
                 ├── fingerprint → group into an issue
                 └── new or regressed?
                       ├── Haiku triage      → priority, category, one-line summary
                       └── agentic root cause → the file, the line, a unified diff
```

Grouping happens **after** symbolication, so an issue keeps its identity and its history across deploys even though every chunk hash changed.

## Getting started

1. **Create a project** at *Observability → Projects*. One project per application.
2. **Link a repository.** Optional, and it's most of the value — without it, analysis is limited to what the stack trace alone shows.
3. **Install the SDK** (or post the envelope directly — see [the protocol](./observability-protocol.html)).
4. **Upload source maps from CI**, if you ship minified browser code.

### JavaScript / TypeScript

```bash
npm install @pendral/inline-observe
```

```ts
import { init } from "@pendral/inline-observe";

init({
  dsn: process.env.NEXT_PUBLIC_INLINE_DSN!,
  release: process.env.APP_VERSION,
  releaseCommit: process.env.GIT_SHA,   // unlocks repo-grounded analysis
});
```

That single call installs global error and unhandled-rejection handlers and starts collecting breadcrumbs (console, fetch/XHR, clicks, navigation). Everything else is opt-out via `integrations`.

**React** adds the component stack, which is what tells you *where in the UI* it broke:

```tsx
import { InlineErrorBoundary } from "@pendral/inline-observe/react";

<InlineErrorBoundary fallback={({ error, reset }) => <ErrorScreen error={error} onRetry={reset} />}>
  <App />
</InlineErrorBoundary>
```

**Next.js** wires the server, edge and browser through the framework's own instrumentation seams — no config wrapping, nothing to break on upgrade:

```ts
// instrumentation.ts
import { registerInline, captureRequestError } from "@pendral/inline-observe/nextjs";

export function register() {
  registerInline({ dsn: process.env.INLINE_DSN! });
}
export const onRequestError = captureRequestError;
```

```ts
// instrumentation-client.ts
import { init } from "@pendral/inline-observe";
init({ dsn: process.env.NEXT_PUBLIC_INLINE_DSN! });
export { onRouterTransitionStart } from "@pendral/inline-observe/nextjs";
```

**Node** covers the process and your HTTP framework:

```ts
import { installNodeHandlers, inlineErrorHandler } from "@pendral/inline-observe/node";

installNodeHandlers();          // uncaughtException + unhandledRejection
app.use(inlineErrorHandler());  // mount after your routes
```

### Everything else

There is no first-party SDK for Python, .NET, Go or Ruby — there's something better: a documented JSON envelope over plain HTTP, and complete copy-paste integrations for each language generated with your real DSN on the project page. The smallest valid report is:

```bash
curl -X POST "https://your-inline-host/api/ingest/v1/ink_pub_…" \
  -H "Content-Type: application/json" \
  -d '{"events":[{"message":"nightly import failed"}]}'
```

See [the protocol reference](./observability-protocol.html) for the full schema.

## Source maps

Minified production stacks are unreadable without them, and the AI's fix quality collapses along with them. Upload on every deploy:

```bash
npx @pendral/inline-observe sourcemaps upload \
  --url https://your-inline-host \
  --key $INLINE_WRITE_KEY \
  --release $GITHUB_SHA \
  --commit $GITHUB_SHA
```

The `--key` is the project's **write key** — the secret half of the credential pair, minted on the project page and shown exactly once. It is not the DSN and must never ship in a bundle.

Pass `--commit`. Without it, symbolication still works, but the root-cause agent falls back to your default branch instead of reading the exact code that crashed.

> Next.js needs `productionBrowserSourceMaps: true`; Vite needs `build.sourcemap: true`.

## Grouping

One issue per distinct bug, not per crash. The fingerprint is:

| Situation | Grouped on |
| --- | --- |
| Stack with in-app frames | Exception type + top in-app frames (module + function) |
| Stack with only vendor frames | Exception type + top frames of any kind |
| No stack | Type + normalized message (numbers, UUIDs, hex, URLs and quoted strings replaced) |
| `fingerprint: [...]` supplied | Exactly what you passed |

**Line numbers are deliberately excluded.** Adding an import at the top of a file shifts every line below it; if line numbers were part of the key, that one-line change would re-group every existing issue in the file and destroy its history — right when you most need it, during a regression.

One consequence worth knowing: grouping runs on the **symbolicated** location, so the first deploy where you start uploading source maps re-groups the issues it affects once — `chunks/checkout-8f2c1a.js` and `src/checkout.ts` are genuinely different locations. It settles immediately after, and it's the right trade: every group from then on is keyed on a real file that survives rebuilds.

Build hashes are stripped from paths for the same reason (`main-8f2c1a9b.js` → `main.js`), but only when the segment looks like a hash (≥8 characters *and* contains a digit), so `checkout-summary.js` is never collapsed into a neighbour's group.

## AI analysis

Two tiers, so cost tracks impact:

**Triage** (Haiku) runs on every new or regressed group. It assigns a priority 1–4, a category, and a one-sentence summary. It is opinionated about priority 4 — browser-extension frames, cancelled requests, cross-origin `Script error.`, offline users and bot traffic are noise, and saying so is what keeps the list readable.

**Root cause** (Sonnet, agentic) runs on high-impact groups when the project is linked to a repo. It uses the same read-only repository tools as inline's code reviewer (`read_file`, `search_code`, `find_files`, `list_directory`), plus the [code-graph index](./codegraph.html) when one exists, pointed at the release commit. It produces:

- the root cause, written in terms of the code it actually read
- the file and line, with the real source shown inline
- a unified diff you can apply
- a confidence score, and verification steps

An issue qualifies when triage rates it P1/P2, or when volume promotes it (enough events or enough distinct users). Volume can only promote, never demote — a rare crash on the payment path is exactly what a P1 verdict is for. You can always force a pass with **Re-analyze**.

The card shows every tool call the agent made. If the analysis says a value is undefined at `checkout.ts:88`, you can see that it opened `checkout.ts` and read line 88.

### When it can't answer

The agent is instructed to say so rather than guess: no in-app frames, a fault inside a dependency, or a file it couldn't localize all produce an explicit "here's what I can tell you" with no diff and low confidence. An overconfident wrong fix costs more than an admitted unknown.

## Lifecycle

- **Unresolved → Resolved** manually, or automatically when an issue hasn't been seen for 14 days.
- **Resolved → Regressed** the moment it's seen again. This re-opens the issue, marks it, and re-queues analysis.
- **Ignored** stops it appearing in the default list without deleting anything.

## Retention and cost control

| Control | Default | Where |
| --- | --- | --- |
| Minimum level | `info` | Settings (workspace) or per project |
| Event retention | 30 days | Settings or per project |
| Events kept per issue | 200 newest | Fixed |
| Daily event cap | 50,000 per project | Per project |
| Client sample rate | 1.0 | SDK `sampleRate` and per project |

Retention only deletes raw event **bodies**. Issues, counts, and the hourly history buckets are never removed by retention, so a year-old regression still shows its true first-seen date and its full chart.

Past the daily cap, occurrences still count toward the issue and its charts — only the individual bodies stop being stored. Your numbers stay honest; your storage stays bounded.

## Security

The DSN carries a **public** key. It ships in browser bundles, so it is not a secret, and it grants exactly one capability: appending events to one project. It cannot read issues.

Uploading source maps requires the separate **write key**, stored SHA-256-hashed — reveal original source and you need a real credential. Reading anything requires a portal session scoped to the workspace.

The ingest endpoint is CORS-open by necessity (browser SDKs post from your origins) and credential-free by design — the SDK sends `credentials: "omit"`, so there is no cookie to protect. Abuse is bounded by a per-project-per-IP rate limit, a body-size ceiling, the daily cap, and length limits on every field in the schema.