How to Measure Browser Agent Reliability

Short answer

Browser agent reliability is not a vibe check; it is four numbers you can audit every hour: session startup success and p95 latency, first action success, completion rate without retries, and how often humans or missing evidence stall the run. Steel’s remote benchmark hits 0.89 s average startup, 1.09 s p95, and 1.34 s p99 across 5,000 runs with 100 percent success; that is 1.70×–8.95× faster than the other providers we tested, so treat 1.2 s p95 and 99.5 percent success as the first red lines. When a metric breaks, check which plan tier hit it: Steel Local is effectively a single-session control plane while Steel Cloud Starter, Pro, and Enterprise tiers stretch into the tens or hundreds of concurrent sessions, so annotate every scoreboard row with the plan and region before trusting an average.

Instead of counting how many jobs eventually finished, tag each Steel session with your queue ID, emit stage timers from sessions.create through release, and record whether each failure had a replay, logs, and an operator intervention. Publish those metrics with plan-tier utilization so alerts fire before startup p95 drifts past benchmark values; the loop shows whether you should fix proxies, selectors, profiles, or human approval policies before touching model prompts.

What to measure

Steel benchmark baselines

These baselines are the default scoreboard targets until you log fresher numbers in your own environment.

Reliability metrics that matter

Segment every metric

A blended reliability number hides which region, plan tier, or proxy pool poisoned the curve. Slice the scoreboard before you trust it.

Plan-tier guardrails (from Pricing/Limits)

Use the current Pricing/Limits table to annotate each slice with the right ceiling so alerts trigger before a plan starves the queue. These are the published caps today; if they change, update the scoreboard in lockstep.

Why completion rates lie

A green “workflow succeeded” light usually hides the 30 seconds you spent waiting for a retry, the three CAPTCHA solves it consumed, or the operator who had to click Continue inside a debugUrl. Reliability falls apart when you only measure the last stage:

• Queues can drain minutes before a session ever boots when concurrency caps (5, 10, 20, 100) are saturated.
• Anti-bot blocks often kill the first navigation, so counting final output means you miss the step where you should rotate proxies or switch to headful mode.
• Retries reset session IDs, which nukes the evidence trail unless you store it per attempt.

Measure every stage. If startup is healthy but first actions fail, you need better fingerprints. If first actions pass but completion tanks, explore selectors, credentials, or human approvals. If completion looks perfect but evidence coverage drops below 100 percent, you are flying blind during postmortems.

Instrument the loop

1. Tag sessions with job context. Store session.id, workflow name, proxy mode, and profile ID alongside your queue metadata so metrics can pivot on any dimension.
2. Capture stage timers. Record (create → connect), (connect → first action), (first action → completion), and (completion → release) durations; export them to Prometheus or whatever time-series system you already trust.
3. Log agent actions. Call /v1/sessions/{id}/agent-logs (or stream live logs) to align model steps, CDP commands, and human inputs with timestamps.
4. Store evidence immediately. Save HLS URLs, MP4 downloads, screenshots, and scraped files using the Files API so every failure has replayable proof even after the 14-day retention window.
5. Track interventions as events. Your approval UIs should emit structured telemetry when someone takes control via debugUrl or a live embed, including reason codes and elapsed time.
6. Automate cleanup. Use release and releaseAll after every run, and record whether cleanup succeeded; stuck sessions distort both cost and perceived reliability.
7. Segment by region and plan. Break out success, latency, and intervention metrics by region, proxy type, and plan tier so a noisy geography or price plan cannot hide systemic regressions.
8. Publish a live scoreboard. Pipe these numbers into a shared dashboard (Grafana, Metabase, spreadsheets), pin the target + alert threshold for each metric, and assign a single owner so reliability drift is visible without digging through traces.
9. Wire plan caps to alerts. Pull concurrency and RPS ceilings straight from Pricing/Limits and annotate every scoreboard row with the plan tier; alert early when startup p95 creeps upward while a plan is within 10 percent of saturation so ops can flip workloads from Steel Local (~1 concurrent session) to Steel Cloud (100+), or between Starter/Pro tiers, before prompts waste browser slots.

Stage instrumentation cheat sheet

Session tag dictionary

Standardize the tags you emit on every session so slicing never turns into regex archaeology later.

Retire fake reliability proxies

Vanity dashboards hide the root cause. Kill these summary metrics so the scoreboard reflects reality.

Manual intervention event schema

Audit every approval the same way you log incidents: structured fields keep reviews auditable and help you replay what happened inside debugUrl or live embeds.

Evidence artifact log schema

Treat replay, log, and file exports as first-class records; missing any of them should block your reliability publish just like a failed metric.

Example: tag and emit metrics in code

import { SteelClient } from "@steeldev/sdk";
import { chromium } from "playwright";

const metricsTags = { workflow, region, plan, proxyPool };

const createdAt = Date.now();
const client = new SteelClient({ apiKey: process.env.STEEL_API_KEY });
const session = await client.sessions.create({ tags: { jobId, workflow } });

const connectStart = Date.now();
const browser = await chromium.connectOverCDP(session.cdpUrl);
metrics.observe("session_startup_ms", connectStart - createdAt, metricsTags);

const page = await browser.newPage();
await page.goto(firstUrl, { waitUntil: "domcontentloaded" });
metrics.observe("first_action_ms", Date.now() - connectStart, metricsTags);

try {
  await runWorkflow(page);
  metrics.increment("workflow_completion", { ...metricsTags, result: "success" });
} catch (error) {
  metrics.increment("workflow_completion", { ...metricsTags, result: "failure" });
  await uploadReplayEvidence(session.id); // Files API helper keeps evidence coverage at 100%.
  throw error;
} finally {
  await client.sessions.release(session.id);
}

Tag the session the moment it exists, emit each stage duration as soon as it finishes, and treat replay uploads plus releases as part of the same function. This keeps every metric synchronized with artifacts and prevents retries from erasing the evidence trail.

Scoreboard payload example

Ship one normalized record per workflow slice so alerting and dashboards stay consistent.

{
  "metric": "startup_success",
  "value": 0.997,
  "windowMinutes": 5,
  "tags": {
    "workflow": "claims-sync",
    "region": "us-east-1",
    "planTier": "pro",
    "proxyPool": "steel-resi"
  },
  "evidence": {
    "replayUrl": "https://hls.steel.dev/runs/claims-sync-12/master.m3u8",
    "logsExportId": "log_exp_92ff",
    "missingEvidence": false
  },
  "alert": {
    "threshold": 0.995,
    "action": "scale_control_plane"
  }
}

Scoreboard payload fields

Reliability scoreboard example

Keep one aggregated view for executives, but publish per-region, per-proxy, and per-plan slices so a noisy geography cannot hide a failing control plane.

Alert wiring checklist

Tie each metric to an automated action so nobody has to discover regressions in Slack.

Run a daily reliability review

A scoreboard only matters if someone reads it. Hold a 10-minute cadence that keeps metrics, evidence, and owners honest:

1. Export the last 24 hours of scoreboard rows per workflow, region, and proxy slice; highlight any metric that crossed its alert threshold even once.
2. Pull the top three manual intervention reason codes and pin their replays plus agent logs to the bug tracker before the next queue resumes.
3. Verify evidence coverage stayed at 100 percent; if any run is missing replay or logs, open an incident before green-lighting deployments.
4. Compare startup success by plan tier against Pricing/Limits; if a cap is within 10 percent, scale warm pools or fail workloads over to Steel Cloud before the next spike.

Interpret and act on the numbers

Use these thresholds to trigger automation: alert when startup success falls for 5 minutes, pause a queue when manual approvals blow past the limit, or ship an incident review when evidence coverage dips at all.

Steel telemetry map

Trade-offs and limits

• Steel Cloud caps a single session at 24 hours; jobs longer than that must checkpoint progress into Files API or storage and resume in a fresh session.
• Session replays, logs, and artifacts stick around for up to 14 days on Pro plans. Export anything compliance-sensitive before that window closes.
• Managed proxies and CAPTCHA solving carry per-GB and per-1k charges; reliable metrics should separate baseline runs from stealth-heavy ones so cost spikes are obvious.
• Human approvals still require your own ACLs around debugUrl and embeds; Steel provides URLs, not full access control.
• Steel Local is roughly a single-session runway while Steel Cloud plans ship managed stealth plus hundreds of concurrent browsers, so plan-cap alerts should always state which side of that boundary triggered.

Works for X, not yet for Y

• Works for teams already tagging each queue job with session.id, so startup, action, and completion metrics match the evidence you review.
• Works for ops groups layering approvals on top of Steel Cloud because interventions become structured events rather than Slack screenshots.
• Not yet for shops that cannot emit unique job IDs or store replays; the loop depends on correlating metrics to artifacts you can audit later.
• Not yet for workflows that routinely exceed the 24-hour session cap or run disconnected from the network; you will lose telemetry before the run ends.

Next steps

• Wire your metrics sink to the Pricing and Limits table so alarms fire before you exceed concurrency or RPS caps.
• Revisit the Sessions API overview to confirm which lifecycle events you emit and how to tag them.
• Download the Remote Browser Benchmark raw results so your scoreboard targets inherit the same evidence before you tune thresholds.
• Pair this page with Why Browser Agents Fail in Production so every failed run stays tied to concrete failure modes and evidence expectations.
• Grab a Steel Cloud API key at https://app.steel.dev so you can tag sessions with these metrics on your next run.

Humans use Chrome. Agents use Steel.