Open-source AI evaluation and observability framework for LLMs, RAG apps, AI agents, and ML models with 100+ built-in metrics and continuous monitoring.
Evidently AI is an Apache 2.0 open-source framework for evaluating, testing, and monitoring AI systems including LLMs, RAG applications, AI agents, and ML models in production. It provides automated evaluation with 100+ built-in metrics (hallucination, PII detection, retrieval quality, toxicity), synthetic test data generation including adversarial inputs, and continuous monitoring with live dashboards to detect drift and regressions. Evaluations can combine rule-based checks, classifiers, and LLM-as-a-judge approaches. Evidently AI is a product of Evidently AI.