AI Ethics Checklist: 10 Questions to Ask Before Deploying a Model
Most model failures are quiet. They look like a support queue filling up with complaints nobody routed to engineering, or a screening tool steadily rejecting the candidates who least resemble the people who built it. By the time someone connects the dots, the system has already made thousands of decisions, and undoing them is far harder than pausing them would have been.
What follows is not a compliance ritual. It is a set of questions worth putting on the calendar before launch, while answers are still cheap and egos are still manageable. If your team can answer most of them out loud, without hedging, you are in good shape. If a question produces silence, that is the useful part.
Who answers when it goes wrong
"The team" is not an answer. Accountability means a named person who understands the system well enough to defend it and has the standing to switch it off. That person should not be the same engineer who is measured on shipping it, if you can avoid the conflict — someone in product, risk, or operations usually works better.
Decide early who can pause the system, how quickly they can do it, and what happens in the hours afterward. A useful exercise: write the incident note before you launch. If you cannot describe what a bad week would look like and who would be paged, you have not finished designing the deployment.
If nobody owns the outcome, everybody will assume somebody else does. That assumption holds until the first serious complaint.
Fairness is a measurement problem
Fairness arguments rarely get resolved in the abstract, so keep them concrete. Build a short list of the people your system touches: applicants, patients, borrowers, drivers, tenants, whoever the decision lands on. Then ask which of them could be systematically disadvantaged, and how you would know.
Average performance hides a lot. A model can look solid overall and still be meaningfully worse for one region, age band, or language group. Disaggregate your evaluation metrics by the groups that matter, and be honest about which ones have too few examples to measure — a small sample is a finding, not a pass.
Watch the labels, not just the outputs
Bias often enters through the training data. Historic decisions recorded as labels carry historic judgments; a proxy like postcode, device type, or employment gap can smuggle in information you never intended to use. Ask who created the labels, under what incentives, and what the data would look like if it had been collected today.
Thresholds deserve the same scrutiny. Moving a cutoff from 0.7 to 0.5 changes who gets help and who gets rejected, and that choice is a policy decision, not a technical one. Somebody should be able to explain why it sits where it sits.
Transparency without overpromising
Complete explainability is rarely available, and pretending otherwise erodes trust the first time the explanation does not hold up. What people actually need is narrower: that they know an automated system is involved, that they understand what information influenced the outcome, and that they have a route to a human who can act on their behalf.
Explain to the right audience. A regulator, a support agent, and an affected customer need three different levels of detail. Reason codes and plain-language summaries usually do more good than a feature-importance chart nobody outside the team can read.
Questions people forget
- Can a person contest a decision without having to guess how the system works?
- Does the fallback path exist, and is it staffed?
- What happens when two systems disagree about the same person?
- Are the limits of the model documented where operators will actually see them?
Privacy questions that arrive late
Purpose matters more than consent forms. Data collected to improve recommendations is not automatically appropriate for performance evaluation or fraud scoring, even if it technically sits in the same warehouse. Write down the intended use, then check whether the data you have would be defensible under that purpose if someone asked.
Retention and deletion deserve a real answer. If a user asks to be removed, can you trace their data through the pipeline, including any copies used for training or evaluation? For models trained on personal data, consider whether the system could reproduce identifiable details, and test for it rather than assuming it cannot.
If you handle health, financial, employment, or other regulated decisions, get qualified legal advice for your jurisdiction instead of relying on a general checklist. The rules vary considerably, and the penalties tend to follow the sector, not the technology.
Ten questions to answer before launch
- Who is accountable by name for this system's outcomes?
- What is the most realistic harm, and who would bear it?
- How does performance differ across the groups affected?
- What data was used, and do we have a clear right to use it for this purpose?
- Can someone affected get an explanation and a way to appeal?
- Where do humans stay in the loop, and can they overrule the system?
- What happens when the model is wrong in the direction that hurts people?
- How will we detect drift or degradation after launch, and who reviews the alerts?
- What is the rollback plan, and how long does it take to execute?
- Who has reviewed the documentation and signed off on this version?
Make it a habit, not a formality
Run this exercise as a short review with the people who built the system, the people who will operate it, and at least one person who is not invested in launching on time. Give every question an owner and a date. Record the answers, including the ones you could not resolve, so the next review starts from evidence rather than memory.
Then set a date to revisit. Models drift, data sources change, and a system that was reasonable at launch can quietly become unreasonable a year later. The checklist is not a gate you pass once; it is a habit that keeps the uncomfortable questions in the room, where they are far easier to handle than in a press release.
Photo: Pavel Danilyuk / Pexels
