Building a web-first agent evaluation platform

The Problem

AI agents that handle customer support, answer questions, and perform tasks need systematic evaluation. As agents grow in number and complexity, manual testing breaks down. You need a way to:

• Replay real conversations against agents to detect regressions
• Simulate new conversations using LLMs as stand-in users
• Score agent responses automatically using metrics like trajectory matching, task completion, and faithfulness
• Track performance over time across agent versions and configuration changes

The goal: build a platform that runs hundreds of evaluation sessions with reproducible results — replacing ad-hoc manual testing with a systematic, automated pipeline.

Note: Manual testing was insufficient. Agents needed evaluation at scale — across hundreds of conversation sessions, with reproducible results and quantitative metrics.

---

Architecture Decisions

Three foundational decisions shaped everything that followed: the monorepo structure, the data layer choice, and the serverless deployment target.

Monorepo with Turborepo

The project adopted a pnpm + Turborepo monorepo from day one, splitting concerns across focused packages:

Monorepo Package Structure

The monorepo allowed sharing TypeScript types and utilities between packages while maintaining clear boundaries. Build caching reduced incremental build times to under 1 second.

Spreadsheets as the Data Layer

Rather than building a database backend, the project chose Google Sheets as both input and output:

• Input: Users define test datasets as structured sheets (session ID, event type, message text)
• Output: Simulation results, metric scores, and agent snapshots are written back to the same spreadsheet
• Tracking: Transient tracking sheets coordinate chunked processing

Why Spreadsheets?: The target users — agent builders and QA engineers — already lived in spreadsheets. Using Sheets as the data layer eliminated the need for a database, admin UI, or data import/export features. A template system auto-creates properly formatted spreadsheets with seed data.

Trade-offs discovered along the way:

• No transactional guarantees — concurrent chunk writes can race
• Column-position parsing instead of header-name parsing caused fragility
• Metric merge logic required 200+ lines of fragile row-index tracking
• Row limits were hardcoded and discovered late

---

Compliance by Design

A critical early decision: ensure sensitive data (customer conversations, support transcripts) never touches the server infrastructure. This wasn’t a feature bolted on later — it was an architectural constraint from the start.

Loading diagram…

The Browser-Only Architecture

• Simulation runs entirely in the browser — the client orchestrates everything
• API routes are “dumb proxies” — they forward requests with auth headers but never inspect conversation data
• Encryption at rest — OAuth tokens and credentials use AES-256-CBC encrypted localStorage

This was implemented as a dedicated 4-phase branch with 8 commits, transforming the architecture from server-executed to browser-executed simulation.

Before vs. After: Data Flow

War Story: The browser-only refactor touched every layer of the stack. It required creating a browser-compatible HTTP client interface, removing all filesystem and environment variable dependencies from the core library, building a proxy client that routes all API calls through relay endpoints, and handling Date serialization across the browser-server boundary. Eight commits over a single day — one of the most intense refactoring efforts in the project.

Tip: The “dumb proxy” pattern is worth adopting early. When your API routes never inspect payload content, compliance becomes an architectural property rather than an audit requirement. The proxy forwards, it never reads.

---

The Evaluation Framework

Seven metrics were implemented, each following a common interface. Five use an LLM as judge for nuanced evaluation; two are calculated from request metadata.

Loading diagram…

Evaluation Metrics

LLM-as-Judge

Five of seven metrics use an LLM to evaluate agent quality. A shared judge abstraction provides structured prompt-to-score evaluation: the judge receives the conversation context and a rubric, then returns a numeric score with an explanation. This enabled nuanced scoring that rule-based systems couldn’t achieve.

Turn-Level Innovation

The most significant metric design decision was supporting per-turn evaluation in multi-turn conversations. Rather than only scoring sessions as a whole, the framework evaluates each individual Q&A pair. This was critical for identifying which specific questions an agent struggles with — not just that it struggled somewhere.

The Hardest Problem: Merging per-turn evaluation scores back into the correct spreadsheet rows was the most complex challenge. The flow: simulation writes results (each turn = one row), evaluation runs metrics on each AI response, scores must land on the exact row of the AI message they evaluate. This required reconstructing turn indices from flattened sheet rows, handling multiple output formats, and preserving metadata across the merge. The solution spans ~200 lines and required 6 bug-fix commits over 3 days.

---

Serverless Constraints

Deploying on a serverless platform (Vercel) introduced constraints that shaped the runtime architecture: no filesystem, execution timeouts, and shared-process concurrency risks.

Loading diagram…

Chunked Processing

Running 500+ simulation sessions sequentially would exceed any serverless timeout. The solution: a smart chunking algorithm that targets 80% of the platform’s execution limit.

• Each chunk is sized to complete within the timeout
• A tracking sheet coordinates chunk progress for fault tolerance
• Failed chunks retry independently
• Results accumulate incrementally in the output sheet
• All tracking is in-memory — no filesystem operations

Adaptive Rate Limiting

External APIs have strict rate limits. Parallel execution triggers 429 errors. The rate limiter adapts in real time:

Rate Limiting Strategy

A minimum 300ms delay between requests provides burst protection regardless of quota. The rate limit tracker singleton monitors response headers on every request.

OAuth Token Management

OAuth tokens expire during long simulation runs. The solution uses proactive refresh (5-minute buffer before expiry), a background keep-alive interval, concurrent refresh protection (preventing duplicate simultaneous refresh requests), and graceful degradation — if refresh fails but the token is still valid, it keeps working; if expired, it clears and prompts re-authentication.

---

Development Velocity

The platform went from first commit to production-ready MVP in 10 days, built by a single developer.

The First 3 Days

Day 1: Monorepo scaffold with build tooling, web application shell, core simulation library, and multi-profile management.

Day 2: Two major features — chunked processing with tracking sheets for fault tolerance, and the entire browser-only compliance refactor (8 commits). Plus adaptive rate limiting.

Day 3: The entire metrics evaluation framework built from scratch — 20 commits in a single day. TypeScript metrics interfaces, LLM-as-judge integration, E2E and task completion metrics, turn-level evaluation, Sheets integration for results, state management, profile management with encrypted storage, and configurable API endpoints.

Post-Sprint Stabilization

After the initial 10-day sprint, development shifted to maintenance and polish: formatting abstractions, latency calculations, OAuth improvements, hierarchical profile architecture, dashboard visualization, and deployment fixes. Five months later, a major cleanup removed the CLI package and all filesystem-dependent code — deleting 11,000 lines while preserving all web functionality.

---

Iteration Log

Each phase revealed new constraints and forced architectural pivots.

---

Takeaways

Meet users where they already work

Spreadsheets were not elegant, but they eliminated weeks of infrastructure work and let users start evaluations immediately.

Compliance is an architecture decision

The browser-only proxy pattern meant sensitive data handling was solved by design instead of added later as an audit project.

LLM-as-judge unlocks evaluation at scale

Rule-based metrics catch format and timing, while LLM judging assesses answer quality, faithfulness, and clarification behavior.

Monorepo boundaries pay off at refactor time

Package isolation made the web-first cleanup safe because core types, adapters, and evaluation logic were already separated.

Ship fast, then refine the hardest logic

A working end-to-end flow created the feedback loop needed to discover and fix the fragile metric merge path.

Serverless constraints are design constraints

Timeouts, filesystem limits, and shared-process risks pushed the system toward chunked processing and explicit recovery paths.

The Bigger Picture: This platform started as a need to test AI agents systematically. It became something broader: a framework for understanding agent behavior at scale. When you can replay hundreds of conversations, score each turn, and track performance across versions, you move from “does the agent work?” to “where exactly does it fail, and is it getting better?” That shift — from anecdotal to systematic — is what makes evaluation infrastructure worth building early.

Tip: If you’re building AI agents, invest in evaluation infrastructure early. The cost of building it is measured in days. The cost of not having it is measured in regressions you don’t catch, quality problems you can’t quantify, and confidence you can’t earn.