How to Conduct a Data Privacy Impact Assessment for AI Projects

The churn model had been live for five weeks when someone finally asked where its training data came from. The answer involved two spreadsheets exported by a marketing vendor, a field of free-text support notes, and a data-sharing agreement nobody in legal had ever seen. Nobody had done anything malicious. They had simply built a useful thing without asking the questions that a data privacy impact assessment exists to force into the open.

If you work on privacy alongside a product team, you know this shape well. Assessments get treated as a formality to clear before launch, when they work best as a design activity that runs beside the build. What follows is a practical sequence for running one on a machine learning project, from the first scoping conversation to the review you schedule six months later.

Know when an assessment is required — and when it is simply wise

Under the GDPR, a data protection impact assessment is expected when processing is likely to result in a high risk to individuals. Certain patterns push a project toward that threshold: systematic and extensive evaluation of people, automated decisions that produce legal or similarly significant effects, large-scale handling of sensitive categories of data, tracking behaviour across contexts, or work involving children and other vulnerable groups. Machine learning projects land in these categories more often than their teams expect.

Novel technology is not a trigger by itself, but it is a prompt. Where a model's behaviour is hard to predict or explain, where outputs rank or score people, or where data is repurposed from the purpose it was collected for, write the assessment. Even when a lawyer confirms it is not strictly mandatory, the document pays for itself: it is the artifact that answers a regulator, a customer's security questionnaire, or your own post-incident review.

Privacy obligations vary by jurisdiction and sector. Treat the steps below as an operational method rather than legal advice, and confirm requirements with qualified counsel.

Scope the assessment before you document anything

Scope creep is the most common reason assessments stall for months. Write a scope statement that fits in a paragraph and answers four things: which system, which data, which people, and which decision. For example — a resume-screening model that ranks applicants for warehouse roles in two countries, trained on three years of internal hiring outcomes, affecting external candidates only.

Then state what is out of scope and why. If the model may later be extended to promotion decisions, note that now. The extension will need its own review, and a sentence today saves an argument next year.

Assign one owner and one deadline

Assessments die in committee. Name a single author, usually the privacy lead or an embedded security engineer, and tie the due date to a real milestone such as the design freeze. Everyone else contributes expertise; one person assembles and owns the document.

Describe the processing and justify it

Now write down the facts. What problem is the system solving? Why is a statistical model the right tool rather than a deterministic rule? What data flows in, including logs, prompts, feedback signals, and vendor-supplied enrichment, and what flows out? Where is it stored, for how long, and who can query it?

Then apply a necessity and proportionality test. This is where most projects discover something uncomfortable. Does the model need a full date of birth, or is an age band enough? Do you need individual-level records, or does an aggregate pattern work just as well? Could a smaller training set produce the same quality? Every field you remove is a field you no longer have to protect, explain, or defend in an audit.

Map the risks to individuals — and rate them

Risk registers written for the business focus on fines, downtime, and reputation. A privacy assessment asks a different question: what happens to the person inside the dataset, or on the receiving end of the prediction? Work through categories like these:

• Inference and re-identification. Combining datasets can reveal sensitive attributes — health status, union membership, religion — that were never collected directly.
• Function creep. Data gathered to improve a service quietly becoming an input to an eligibility decision.
• Unequal outcomes. A model that performs well on average but systematically denies opportunities to one group.
• Overreach in automation. Removing human review from decisions that materially affect someone's housing, credit, or employment.
• Exposure through the pipeline. Training data, prompts, and outputs leaking through logs, debugging tools, or third-party APIs.
• Opacity. A person cannot understand or contest a decision made about them.

Describe each risk in terms of harm to a named group of people. "Data breach" is not a useful entry. "A rejected applicant cannot learn which attributes drove the outcome, and there is no route to appeal" is.

For each one, rate likelihood and severity on a simple scale and write down your reasoning. The number matters far less than the conversation, and the record of where the engineer, the lawyer, and the product manager disagreed. That disagreement is the most useful content in the whole document, because it shows a reviewer exactly where the uncertainty sits.

If you cannot explain the harm to an affected person in a sentence they would recognise, the risk analysis is not finished.

Mitigate with owners, dates, and a residual risk decision

A mitigation without an owner and a date is a wish. For each risk, record the action, the person accountable, and when it will be verified. Measures that do genuine work:

1. Data minimisation. Drop unnecessary fields before training, not after. It is cheaper and it eliminates whole risk categories.
2. Purpose locking. Use contractual and technical controls to stop training data being reused for a different class of decision.
3. Access and retention limits. Short retention on training sets and inference logs; role-based access with periodic review.
4. Output controls. Confidence thresholds, mandatory human review for consequential decisions, and a working appeals route.
5. Testing for disparate impact. Measure outcomes across relevant groups before launch and again after every retrain.
6. Vendor terms. Confirm whether a provider retains your inputs, trains on them, or can delete them on request.

Then reassess. What remains after mitigation is the number that matters. Someone senior needs to accept that residual risk in writing, with the reasoning attached.

Consult, sign off, publish — then keep it alive

Consult your data protection officer, security, legal, and the engineers who built the pipeline. Where feasible, talk to affected people. User research counts as consultation and routinely surfaces risks no internal reviewer would think of. If residual risk stays high after mitigation, the law may require consulting your supervisory authority before processing begins; that call belongs to counsel, but the assessment should make it obvious when you are close to it.

Publish a short, plain-language summary. It builds trust, forces clarity, and saves you rewriting the same explanation for every customer audit. Then keep the assessment alive. Reopen it when a new data source appears, when the model is retrained on different data, when you enter a new country or user group, when outputs find a new use, or when you change model provider. Put a review date in the calendar and treat it as a living record rather than a launch gate.

A short checklist you can run this week

Before you call the assessment done, confirm you can answer these in one line each: who owns it; what decision the system influences; which data is genuinely necessary; what happens to a person on the wrong end of a prediction; what you removed to reduce risk; who accepted the residual risk; and when you will look again. If any answer is missing, you have another afternoon of work ahead of you — which is far cheaper now than after deployment.

Photo: jplenio / Pixabay