Testing the Untestable: How to Build Automated Evals for LLM & Agentic Systems

In traditional software development, testing is deterministic:

• Input 2 + 2, assert 4.
• Post valid JSON payload, assert HTTP 201 Created.

When your application incorporates Large Language Models or autonomous agents, deterministic assertions break down:

• Model outputs are probabilistic.
• Rephrasing a prompt can alter response phrasing.
• A model update can silently degrade accuracy without throwing an exception.

If your only quality control is "eyeballing" responses in chat, you cannot deploy AI into mission-critical business workflows.

Here is how we architect automated LLM Evaluation Harnesses (Evals) to test and protect production AI systems.

---

1. The Three Layers of AI Evaluation

A resilient AI testing strategy operates across three tiers:

┌──────────────────────────────────────────────┐
│ Tier 3: LLM-as-a-Judge (Semantic Rubric)    │
├──────────────────────────────────────────────┤
│ Tier 2: Deterministic Constraint Assertions │
├──────────────────────────────────────────────┤
│ Tier 1: Unit & Schema Validation Checks     │
└──────────────────────────────────────────────┘

---

2. Tier 1: Schema & Syntax Invariants

Before testing model "intelligence," test basic structural compliance. If your app expects JSON, a markdown code fence or a trailing comma is a test failure.

We enforce runtime validation using Zod in TypeScript:

// test/evals/schema-test.ts
import { z } from 'zod';

const OrderExtractionSchema = z.object({
  customerName: z.string().min(2),
  orderItems: z.array(z.object({
    sku: z.string(),
    quantity: z.number().int().positive(),
    unitPrice: z.number().positive(),
  })).min(1),
  deliveryAddress: z.string(),
});

export function assertValidSchema(rawJson: unknown) {
  const result = OrderExtractionSchema.safeParse(rawJson);
  if (!result.success) {
    throw new Error(`Schema Violation: ${JSON.stringify(result.error.format())}`);
  }
  return result.data;
}

---

3. Tier 2: Deterministic Constraint Assertions

Even without checking semantic nuance, you can verify rigid business rules using deterministic code:

• Fact Presence: Did the response mention the mandatory disclaimer required by financial regulations?
• Prohibited Terms: Does the output contain prohibited internal URLs, leaked system prompts, or competitor names?
• Length & Bounds: Is the response under 500 characters for mobile display constraints?

// test/evals/invariants-test.ts
export function testComplianceInvariants(output: string) {
  // Check required regulatory disclaimer
  if (!output.includes("Authorised Financial Services Provider")) {
    return { passed: false, reason: "Missing regulatory FSP disclaimer" };
  }

  // Check against prompt injection leaks
  const leakSignatures = ["SYSTEM PROMPT:", "ANTHROPIC_API_KEY", "You are an AI"];
  for (const sig of leakSignatures) {
    if (output.includes(sig)) {
      return { passed: false, reason: `Potential system leak detected: ${sig}` };
    }
  }

  return { passed: true };
}

---

4. Tier 3: The LLM-as-a-Judge Pattern

For qualitative metrics (such as tone, conciseness, relevance, and factual grounding against source documents), we employ an independent, strictly prompted Judge Model (e.g., claude-3-5-sonnet or gpt-4o):

// test/evals/llm-judge.ts
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();

interface EvalInput {
  context: string;
  question: string;
  actualAnswer: string;
}

export async function evaluateFaithfulness({ context, question, actualAnswer }: EvalInput) {
  const prompt = `
You are an impartial evaluation judge. Analyze the following actual answer against the provided context.

Context:
${context}

User Question:
${question}

Actual Answer:
${actualAnswer}

Evaluate on a scale from 1 to 5:
1: Contains direct factual contradictions or hallucinations not present in the context.
3: Partially supported by context, but makes unverified assumptions.
5: 100% faithful to the context with zero hallucinations.

Output JSON only with keys: "score" (number 1-5) and "reasoning" (string).
`;

  const response = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 300,
    messages: [{ role: 'user', content: prompt }]
  });

  return JSON.parse(response.content[0].text);
}

---

5. Automated CI/CD Regression Guardrails

We run our eval suite on every Pull Request that modifies prompt templates, system instructions, or retrieval parameters.

# Run automated eval harness in GitHub Actions
npm run test:evals

If the average Faithfulness score drops below 4.7/5.0 or if any regulatory invariant fails, the GitHub Actions build fails automatically. The PR cannot be merged.

By moving from manual qualitative testing to automated quantitative evaluation, you turn AI development into an engineering discipline with verifiable SLAs.