AgentLens — AI Agent Evaluation Platform

Project Overview

AgentLens helps developers understand not just how AI performs, but why it fails. It is an AI Agent evaluation platform designed to analyze and diagnose the quality of AI responses in real-world conversations. By taking a dialogue and a task objective as input, the system evaluates performance across five key dimensions: task completion, accuracy, relevance, user experience, and safety. The platform goes beyond scoring — it identifies critical issues such as hallucinations, missing information, and intent misunderstanding, and presents them in a structured diagnostic report. With instant feedback and clear visualization, AgentLens enables fast iteration and debugging for AI-powered products.

What I Built

• Designed the full evaluation framework for AI Agents with five key scoring dimensions.
• Built an automated evaluation pipeline that converts conversations into structured scoring results.
• Implemented hallucination detection by identifying factual inconsistencies and incorrect reasoning.
• Designed a strict JSON output format to ensure stable parsing and system reliability.
• Developed an interactive dashboard including score cards, radar charts, and diagnostic reports.
• Integrated OpenRouter API to support real-time evaluation with LLMs while optimizing cost.
• Implemented usage limits and BYOK (Bring Your Own Key) system for scalability.
• Created test scenarios (e.g., hallucination cases) to demonstrate system capability clearly.

Reflection

Building AgentLens revealed that evaluating AI is fundamentally harder than generating responses. Simple scoring is not sufficient — developers need structured insights into why AI fails. One key insight is that hallucination detection is essential for trust but difficult to define. By combining task context and response analysis, the system achieves more reliable detection than naive approaches. Another learning is that usability matters: concise explanations and visual dashboards significantly improve how users interpret evaluation results. In the future, AgentLens can evolve into a full AI quality monitoring system with continuous evaluation, feedback loops, and production integration.