AI Summary:
COMPL-AI helps test generative AI models for alignment with the EU AI Act using 27 benchmarks and a regulatory-to-technical mapping.
As the EU AI Act moves towards implementation, developers and organizations working with generative AI need to start thinking seriously about compliance. COMPL-AI, an open-source framework, offers a structured way to evaluate how well large language models meet the law’s requirements.
COMPL-AI’s two main featues
The COMPL-AI framework is built around two core components that work together to support AI developers and regulators:
1. EU AI Act compliance
COMPL-AI translates the EU AI Act’s legal requirements into measurable technical criteria. This mapping defines what compliance should look like in practice, offering developers a clear path from legal language to system design. It serves as a working guide for aligning AI models with European regulatory expectations.
2. Benchmarking 27 risk dimensions
The framework includes a suite of 27 benchmarks designed to test AI models against areas the EU AI Act identifies as high-risk. These benchmarks cover: Transparency, Fairness, Safety, Robustness and more.
This evaluation method has already been applied to models from Mistral, OpenAI, Meta, Google, Anthropic and others. Most models perform well at limiting harmful content, but many remain vulnerable to bias and cybersecurity issues.
Available open source and for self-assessment
COMPL-AI is openly available, allowing anyone to run evaluations and interpret results. This makes it especially useful for internal audits or early-stage product assessments. It gives developers and product teams a concrete sense of where their models might face scrutiny under the AI Act.
The project is led by ETH Zurich, INSAIT and LatticeFlow AI. Its development reflects a broader effort to ensure that AI in Europe is not only innovative but legally and ethically grounded. As compliance becomes non-optional, frameworks like COMPL-AI will be an essential part of the technical toolkit.
For more information and to access the framework, visit compl-ai.org.