Benchmark any LLM across latency, accuracy, hallucination, and cost, in one automated pipeline.
Choosing an LLM for production is not just about accuracy. It's a multi-dimensional trade-off between latency, hallucination risk, and cost. Most evaluations are ad hoc: a few prompts, subjective scoring, no reproducibility. LLM Eval Harness replaces that with a fully automated, configurable pipeline that evaluates any combination of OpenAI, Anthropic, and Groq models in parallel against a structured JSONL dataset. The key architectural insight is that each evaluation dimension (latency, quality, hallucination, cost) requires a different measurement strategy, and fusing them into a single comparable report is what makes the framework useful for real deployment decisions. The NLI-based hallucination detector (deberta-v3-small cross-encoder) and RAGAS suite provide grounded, reproducible quality signals far beyond simple accuracy metrics.
A JSONL dataset of Q&A pairs (with id, question, ground_truth, context, and metadata fields) is loaded by the CLI entrypoint. The context field supports RAG-style evaluation where hallucination detection is grounded against retrieved passages.
asyncio-based runners for OpenAI (gpt-4o, gpt-4o-mini), Anthropic (claude-sonnet, claude-haiku), and Groq (llama-3.3-70b, llama-3.1-8b) execute concurrently with a configurable concurrency limit (default: 5). Each runner implements a shared BaseRunner interface, making new providers trivial to add.
After each model response is collected, the metrics engine computes five dimensions: (1) Latency: p50/p95/p99 percentiles, TTFT, and tokens/sec; (2) Accuracy: exact match and semantic similarity via sentence-transformers; (3) LLM-as-Judge: a secondary model scores responses 1–5 for quality; (4) Hallucination: a deberta-v3-small NLI cross-encoder checks whether model claims are entailed by the ground truth context; (5) Cost: tiktoken counts tokens and applies per-model pricing tables.
The RAGAS library evaluates the full RAG quadrant: faithfulness (do claims follow from context?), answer relevancy, context precision, and context recall. These scores complement the NLI hallucination check with a semantically richer signal.
Results are output as a Rich-formatted terminal table, a self-contained HTML report with Plotly visualizations, and a JSON export for programmatic use. A Streamlit dashboard enables multi-run comparison with KPI cards, latency bars, accuracy/hallucination/cost charts, a RAGAS radar plot, and CSV export.
llama-3.3-70b achieves P50 latency of 1,956ms and TTFT of 245ms, 16× faster TTFT than llama-3.1-8b
llama-3.1-8b delivers comparable semantic similarity (0.852 vs 0.880) at 10× lower cost per query
0% hallucination rate on the AI/ML Q&A benchmark for both Groq models via NLI cross-encoder
95 automated tests across metrics, runners, and reporters ensure evaluation reproducibility
Docker + HuggingFace Spaces deployment allows the dashboard to run anywhere without local setup