Make AI Compliance Assessments Evidence-Based and Risk-Tiered

Summary

The column argues that AI compliance questionnaires should use evidence-backed, measurable questions scaled to system risk. It recommends reusing controls across frameworks and standardizing model cards to make vendor assessments more useful and efficient.

Key points

• The author says lengthy free-text questionnaires can reward persuasive writing over evidence and may not meaningfully reduce risk.
• For changing AI systems, the column recommends measuring and logging behavior over time rather than relying on point-in-time attestations.
• It proposes five tests for assessment questions: evidence-based, risk-tiered, measurable where possible, decision-relevant, and reusable across frameworks.
• Suggested assessment topics include data flows and retention, access, logged inference parameters, evaluation coverage and cadence, human oversight, incident response, and model changes.
• The author proposes a standardized model card with fields such as model lineage, data provenance, retention, evaluation scores, mitigations, inference parameters, and change history.