Organizations increasingly need to show that their AI systems treat people fairly — for legal compliance, customer trust and simply to avoid harm. A fairness audit doesn’t require a research lab. It requires structure, the right questions and honest documentation. This checklist draws on established practices such as the NIST AI Risk Management Framework, model cards and datasheets for datasets.

Step 1 — Define the scope and stakes

  • What decision or output does the system influence? Hiring screening, credit, content moderation, customer service routing?
  • Who is affected, and how serious is a wrong decision for them?
  • Which groups could be treated differently? Consider legally protected characteristics and context-specific groups such as language speakers or people with disabilities.
  • Which laws and policies apply?

High-stakes decisions about people deserve the deepest audits.

Step 2 — Review the data

  • Where did training and evaluation data come from, and when?
  • Are relevant groups represented in sufficient numbers?
  • How were labels created, and could they reflect past biases?
  • Are any features proxies for sensitive attributes?

Documenting answers in a datasheet — a practice proposed by Timnit Gebru and colleagues — makes data issues visible to everyone involved.

Step 3 — Choose fairness criteria

Decide, with legal and domain experts, which fairness definitions matter for this use case: similar selection rates, similar error rates, calibrated scores, or others. Write down why. Different criteria can conflict, so the choice must be explicit.

Step 4 — Test with disaggregated metrics

Measure performance separately for each group, not just overall:

  • accuracy, false-positive and false-negative rates;
  • selection or approval rates;
  • for generative systems, quality and refusal rates across languages, dialects and demographic references.

If you lack group information, consider carefully designed test sets, synthetic counterfactual tests (changing only a name or pronoun and comparing outputs) or privacy-preserving methods.

Step 5 — Investigate gaps

When disparities appear, find the cause: data imbalance, label problems, proxy features, thresholds or deployment context. Then test mitigations — rebalancing data, adjusting thresholds, changing features, adding human review — and re-measure.

Step 6 — Check the human process

Fairness also depends on how the system is used:

  • Do people understand the system’s limits?
  • Can they override it, and do they?
  • Can affected people get an explanation and contest decisions?

Step 7 — Document

Publish or record a model card — a practice introduced by Margaret Mitchell and colleagues — describing intended use, evaluation results by group, limitations and ethical considerations. Documentation supports accountability and is increasingly required by regulation.

Step 8 — Monitor continuously

Populations, data and models change. Re-run disaggregated metrics regularly, track complaints and incidents, and re-audit after significant updates.

Who should be involved

Effective audits combine data scientists, domain experts, legal and compliance staff, and representatives of affected groups. Independent external auditors add credibility for high-stakes systems.

Related: algorithmic bias explained and AI in hiring and HR.

Bottom line

Audit before launch and keep auditing after.

What to watch next

  • Mandatory assessments. Laws such as the EU AI Act require risk management and, for some deployers, fundamental-rights impact assessments for high-risk systems.
  • Local audit laws. Some jurisdictions require bias audits for automated employment decision tools, a model other places may follow.
  • Standards. International standards on AI management and bias, such as ISO/IEC 42001, give audits a common structure.
  • Generative AI audits. Methods for testing chatbots and image generators across languages, dialects and demographic references are maturing.
  • Third-party assurance. A growing market of independent auditors and certification schemes.

Key takeaways

  • Start by scoping the decision, the stakes and the groups who could be affected.
  • Review data sources, representation, labels and proxy variables, and document them.
  • Choose fairness criteria explicitly with legal and domain experts.
  • Measure performance separately for each group and investigate the causes of gaps.
  • Document results in model cards and monitor fairness after deployment.

Frequently asked questions

Do I need demographic data to audit fairness?

It helps a great deal. Where you cannot collect it, use carefully designed test sets, counterfactual tests or privacy-preserving approaches, and be transparent about limitations.

How often should we audit?

Before deployment, after significant changes to the model or data, and periodically in production — more often for high-stakes systems.

Can an external firm do the audit?

Yes, and independent audits add credibility. Internal teams still need to provide access, documentation and the ability to act on findings.