Open-source LLM evaluation pipeline. A personal project, designed and built end-to-end.
EvalLab
An open-source evaluation harness for LLM products: define test sets, define agents, run them against an LLM-as-judge, compare prompt versions with reproducible scored results. The idea: ship prompts with numbers, not vibes. Built after working on production LLM agents and noticing how thin the eval discipline usually is.
Implements LLM-as-judge scoring across four dimensions (accuracy, completeness, tone, safety), failure clustering that groups low-scoring cases by failure mode, and judge calibration with Cohen's κ and a 5×5 confusion matrix between human and LLM judge scores. Runner uses a token-bucket rate limiter, exponential-backoff retries, per-case error isolation, and immutable agent/prompt versioning so historical scores stay reproducible.
PythonFastAPISQLAlchemy 2.0PostgreSQLNext.jsTypeScriptTailwindCSSshadcn/uiGroqLLM-as-judgeRAG-adjacentRenderVercelNeon
