Evidently AI: An Open-Source Platform for ML Observability
Machine learning (ML) is becoming increasingly important and ubiquitous in various domains and applications. However, developing and deploying ML models is not a one-time process. It requires continuous evaluation, testing, and monitoring to ensure the quality, reliability, and fairness of the models. This is where ML observability comes in.
ML observability is the ability to monitor, analyze, and explain the behavior and performance of ML models in production. It helps data scientists and ML engineers to identify and resolve issues such as data drift, model degradation, bias, or anomalies. It also helps to improve the transparency and accountability of ML systems.
However, achieving ML observability is not easy, as it involves collecting and processing large amounts of data, metrics, and logs from various sources and stages of the ML lifecycle. It also requires a set of tools and techniques that can handle the complexity and diversity of ML models and tasks.
Evidently AI is an open-source platform that aims to provide a comprehensive solution for ML observability. It offers various features and functionalities for evaluating, testing, and monitoring ML models from validation to production. It works with tabular, text data, embeddings, LLMs³, and generative models. It is built for data scientists and ML engineers who want to reliably run ML systems in production.
How Evidently AI works
Evidently AI has a modular approach with three components on top of the shared metrics functionality: Reports, Test Suites, and a Monitoring Dashboard. They cover different usage scenarios: from ad hoc analysis to automated pipeline testing and continuous monitoring.
- Reports: interactive visualizations. Reports calculate various metrics and provide rich interactive visualizations. Users can create a custom Report from individual metrics or run one of the Presets that cover a specific aspect of the model or data performance. For example, Data Quality or Classification Performance. Reports are best for exploratory analysis, debugging, and documentation.
- Test Suites: batch model checks. Tests perform structured data and ML model quality checks. Users can set the conditions manually or let Evidently AI generate them based on the reference dataset. Tests will return an explicit pass or fail result. Users can create a Test Suite from 50+ tests or run one of the Presets. For example, to test Data Stability or Regression Performance. Tests are best for automated batch checks.
- Monitoring Dashboard: continuous tracking. Monitoring Dashboard collects metrics from deployed models using Prometheus and Grafana. Users can monitor every aspect of their data, models, and test results over time. Users can proactively catch and resolve production model issues, ensure optimal performance, and continuously improve it.
Evidently AI has a simple, declarative API and a library of in-built metrics, tests, and visualizations. Users can use Evidently AI as a Python library or as a self-hosted web application. Users can also integrate Evidently AI with popular ML platforms and tools such as Mlflow¹, Airflow², LangChain¹, BentoML², etc.
Why Evidently AI is useful
Evidently AI is useful for several reasons:
- It helps users to achieve ML observability with minimal effort and resources. Users do not need to build their own monitoring solutions from scratch or rely on expensive third-party services.
- It supports various types of data and models with consistent API and metrics. Users can use Evidently AI for tabular data as well as text data, embeddings, LLMs, generative models, etc.
- It provides comprehensive monitoring functionality that covers various aspects of data quality, model performance, drift detection, explainability, fairness, etc.
- It provides rich interactive visualizations that help users to explore and debug their data and models. Users can compare multiple models or model versions using heatmaps, charts, or tables.
- It provides alerting functionality that can notify users when something goes wrong with their data or models. Users can set thresholds or conditions for triggering alerts based on their monitoring metrics or tests.
How to get started with Evidently AI
To get started with Evidently AI, users can follow these steps:
- Install Evidently AI as a Python library using pip: `pip install evidently`
- Import Evidently AI in Python: `import evidently`
- Choose a Report or Test Preset or create a custom one using individual metrics
- Run the Report or Test on one or two datasets as pandas DataFrames or CSV files
- Get the output as an interactive HTML file, JSON file, Python dictionary, or Jupyter notebook widget
- Optionally, set up a self-hosted Monitoring Dashboard using Prometheus and Grafana
- Optionally, integrate Evidently AI with other ML platforms and tools using the Python SDK or the REST API
For more details and examples, users can check out the Evidently AI documentationor the GitHub repository.
Conclusion
- Evidently AI is an open-source platform that provides a comprehensive solution for ML observability. It offers various features and functionalities for evaluating, testing, and monitoring ML models from validation to production. It works with tabular, text data, embeddings, LLMs, and generative models. It is built for data scientists and ML engineers who want to reliably run ML systems in production.
If you want to learn more about Evidently AI or try it out, you can check out their website documentation.
https://github.com/evidentlyai/evidently
I hope you liked my article. If you have any feedback or questions, please let me know.
Let’s Create Value!!!!
