22 Ethics in the Age of Artificial Intelligence: Algorithmic Decisions

Learning Objectives

You will be able to:

Explain why algorithmic decisions raise ethical problems that human decisions of the same kind do not.
Identify the points at which bias enters an automated decision system.
Explain the accountability gap and relate it to the diffusion of responsibility.
State the core obligations of the Indian data protection framework and what they require of a manager.
Apply the three ethical frameworks to an algorithmic decision and design the controls it needs.

22.1 Introduction

This chapter closes Module III with the newest of the manager’s dilemmas, and the one where the law is furthest behind the practice. Chapter 15 made the point in general terms: law lags conduct, and a genuinely new practice remains unregulated until harm accumulates. Automated decision-making is that case at present.

The subject is not the technology. A manager does not need to know how a model is trained in order to reason about it. The subject is a shift in how decisions about people are made: from a person deciding with reasons that can be asked for, to a system producing an output at a scale and speed no person could match, on grounds that may be unavailable even to those who built it (Manuel G. Velasquez, 2018).

Firms deploy such systems to decide who is shortlisted for a job, who receives credit and at what price, which claims are paid, which transactions are blocked, which employees are flagged for review, and which customers see which offer. Each of these was already an ethical decision. Automation changes its character in four specific ways.

flowchart TD
    A["<b>A decision about people</b>"] --> B["<b>Automated</b>"]
    B --> C["<b>Scale</b><br>one flawed rule applies<br>to everyone at once"]
    B --> D["<b>Opacity</b><br>reasons may be unavailable<br>even to the builder"]
    B --> E["<b>Feedback</b><br>outputs become tomorrow's<br>training data"]
    B --> F["<b>Apparent objectivity</b><br>the output is trusted<br>because it is computed"]
    C --> G["<b>The manager's questions</b><br>consequential? representative?<br>explainable? who answers?"]
    D --> G
    E --> G
    F --> G
    style B fill:#ede7f6,stroke:#7E57C2
    style G fill:#e8f5e9,stroke:#388E3C

22.2 What Automation Changes

Four Differences That Matter Ethically

Difference	Why it changes the ethical situation
Scale	A biased recruiter affects the candidates they interview. A biased screening model affects every applicant, consistently, at once. Consistency is usually a virtue, and here it converts an individual failing into a systematic one
Opacity	A person can be asked why. Many models cannot supply reasons in that sense, only correlations, so the affected person cannot contest the ground of the decision
Feedback	Decisions generate the data used to train the next version. A model that declines a group of applicants never learns whether they would have repaid, so its own output confirms its own assumption
Apparent objectivity	An output that emerges from computation carries an authority a human judgement does not, and is questioned less, even by those who know it is a statistical estimate

The fourth is the one managers most need to guard against in themselves. A number produced by a system is an estimate carrying uncertainty, built on choices somebody made, and the interface almost never displays either.

22.3 Where Bias Enters

Bias in an automated system is rarely introduced by anyone intending it. It enters at identifiable points, and knowing them is what allows a manager to ask useful questions of a technical team.

Point of entry	How it happens	The question to ask
The training data	The data records past decisions, which carry past patterns. A model trained on who was hired learns who was hired, not who would have succeeded	What behaviour does this data actually record, and whose decisions produced it?
The target variable	The thing being predicted is a proxy chosen by someone. Predicting “employee retention” is not predicting “good employee”	What exactly are we predicting, and is that the thing we care about?
The features	A variable may stand in for a protected characteristic without naming it. Postal code, name, schooling and device type can all serve as proxies	Which features could act as a proxy for something we would not use directly?
Representation	Groups thinly represented in the data are modelled worst, and the error is invisible in an aggregate accuracy figure	How does performance differ across the groups affected, not just overall?
Deployment context	A model built for one population, purpose or period is applied to another	Is this being used on the population and for the purpose it was built for?
The human in the loop	A reviewer who defers to the output adds no protection, and a reviewer measured on throughput will defer	What does the reviewer actually do, and what are they measured on?

Common Misconception: Removing the Sensitive Attribute Makes the Model Fair

The intuition is that a model which never sees gender, caste or religion cannot discriminate on them. It is wrong, and the reason is worth understanding precisely, because it is the commonest technical answer given to a governance question.

Sensitive attributes correlate with other variables. Remove the attribute and the model reconstructs it from what remains, because those correlations are exactly what a model is built to find. A hiring model denied gender may learn from the name, the college, the sport, or a gap in employment history. The attribute is gone from the input and present in the output.

Two consequences follow for a manager. Testing must be done on outcomes across groups, not on inputs, which means the firm must retain the group information it is not using for prediction in order to measure whether it is discriminating. And “we do not collect that data” is not an answer to a fairness question, it is a reason the question cannot be answered.

22.4 The Accountability Gap

Chapter 15 set out three conditions for moral responsibility: causal contribution, knowledge and freedom. Automated systems weaken all three at once, in the same way and for the same structural reasons that large organisations do.

Contribution is divided among the team that specified the system, the vendor that built it, the data engineers who assembled the training set, the manager who approved deployment, and the operator who accepted the output. Knowledge is divided: the person who understands the model does not see the affected individuals, and the person who sees the individuals does not understand the model. And freedom is reduced for the operator, who is measured on speed and has no authority to override.

The result is a decision that harmed someone for which no individual is responsible, each pointing accurately to a limited role. That is not a new problem. It is Chapter 15’s diffusion of responsibility with a technical vocabulary, and the remedy is the same: a named accountable owner for each deployed system, who is answerable for its outcomes whether or not they understand its internals, in the way a director is answerable for accounts they did not prepare.

Practitioner Insight: Four Questions Before Any Deployment

A manager does not need technical depth to govern these systems. Four questions, asked before deployment and answered in writing, do most of the work.

Is the decision consequential for the person? A recommendation of what to watch next and a decision on a loan application are not the same problem, and applying the same governance to both wastes effort on one and under-protects the other.

On what population was this built and tested, and how does performance differ across the groups it will affect? An aggregate accuracy figure is not an answer.

Can we explain an individual outcome to the person affected, in terms they can act on? If the honest answer is no, the firm must decide whether to accept that for this decision type, and should record that it decided.

Who is accountable, and what can the affected person do? A named owner, and a route by which a person can contest an outcome and reach a human with authority to change it.

A vendor unable to answer the second and third questions about its own product is telling you something about the product.

22.5 Privacy and the Indian Framework

Automated decisions run on personal data, and India’s framework is set by the Digital Personal Data Protection Act, 2023 (Government of India, 2023). The Act uses its own vocabulary, and a manager should know it.

The Core Structure

Concept	What it means
Data principal	The individual to whom the personal data relates
Data fiduciary	The person who determines the purpose and means of processing. The word fiduciary rather than controller is deliberate, framing the relationship as one of trust
Consent and notice	Processing generally rests on consent, given for a specified purpose, preceded by a clear notice describing the data, the purpose and the rights available
Purpose limitation	Data may be used for the purpose for which consent was given, and not repurposed silently
Data minimisation	Only data necessary for the specified purpose should be collected
Rights of the principal	Access to information about processing, correction, erasure, grievance redress, and nomination
Significant data fiduciary	A class subject to additional obligations, including data protection impact assessment, audit, and appointment of a data protection officer
Obligations on breach	Notification of a personal data breach to the Board and to affected principals

Two implications reach directly into algorithmic practice. Purpose limitation constrains the common practice of collecting data for one reason and later discovering it is predictive of something else; that later use requires its own basis. And the impact assessment obligation for significant data fiduciaries is the statutory hook on which the four questions above can be hung, which is why a firm approaching that threshold should build the process before it must.

The word fiduciary repays attention. It signals that the holder of data exercises a discretion on behalf of someone who cannot supervise it, which is the same structure Chapter 20 described for conflicts of interest, and it invites the same question: would the person whose data this is agree to this use if it were explained plainly.

22.6 Applying the Three Frameworks

Framework	What it asks of an algorithmic decision	What it detects
Consequentialism	What outcomes does this system produce across everyone affected, including errors?	The aggregate case for automation, and the harm concentrated in the groups it models worst
Deontology	Is anyone being used merely as a means, and can they contest what is done to them?	The unexplainable adverse decision, and the consent that was obtained but not understood
Virtue ethics	What kind of organisation deploys this, and what does deferring to the output make of the people operating it?	The reviewer who has stopped reviewing, and the team that stopped asking

The frameworks pull hardest against each other on the trade-off between aggregate accuracy and individual explainability, which is the characteristic dilemma of the field. A more accurate model that cannot explain itself produces better outcomes on average and leaves individuals unable to contest. Chapter 17’s rule applies: where the frameworks disagree, an interest is being sacrificed, and the response is disclosure, consent, contestability or compensation rather than a better score.

22.7 Case Studies

22.7.1 Case Study 1: The Screening Model That Learned History

The following situation is constructed rather than reported.

A large services company automates the first stage of graduate hiring, training a model on ten years of applications and the outcomes of those who were hired. Gender and caste are excluded from the inputs. The model performs well against its target, which is defined as the probability that a candidate would have been shortlisted by the firm’s own recruiters. Shortlisting from certain colleges and from candidates with continuous employment histories rises sharply. Nobody involved intends any of this.

Discussion questions

Identify every point at which bias entered, using the table in this chapter. Which is the most serious and why?
The target variable was the firm’s own past shortlisting decision. State precisely what the model was therefore built to reproduce.
Design the test that would have detected the problem before deployment, and say what data the firm would need to run it.

22.7.2 Case Study 2: The Reviewer Who Agreed

The following situation is constructed rather than reported.

A non-banking finance company uses an automated system to assess small-ticket loan applications, with a human reviewer required for every rejection. Reviewers handle roughly three hundred files a day and are measured on files cleared. Ninety-eight per cent of automated rejections are confirmed. When a rejected applicant asks why, the reviewer can see the score and the top three contributing factors, expressed as variable names. The company reports that all adverse decisions receive human review.

Discussion questions

Is the statement that all adverse decisions receive human review true? Is it honest? Distinguish the two and relate your answer to Chapter 9.
Apply the accountability analysis. Where does each of the three conditions of responsibility weaken, and for whom?
Redesign the review step so that it provides real protection, and state what it would cost in throughput.

22.7.3 Case Study 3: The Data That Found a New Purpose

The following situation is constructed rather than reported.

A retail chain collects purchase histories under a loyalty programme whose notice describes the purpose as personalised offers and billing. A data science team discovers that purchase patterns predict which customers are likely to be pregnant, and proposes using this to time promotions for baby products. The commercial return is significant. The legal team is asked whether this is permitted.

Discussion questions

Apply purpose limitation. Is the new use within the consent obtained, and what would the firm have to do to use it?
Set aside legality. Apply the disclosure test from Chapter 17: would the customers agree to this if it were explained in one plain sentence?
The information is inferred rather than collected. Does that change the analysis, and should it?

22.8 Applying the Chapter

Exercise: Governing an Algorithmic Decision

An Indian health insurer plans to deploy a system that scores claims for likely irregularity and routes high-scoring claims to a manual investigation queue, delaying settlement by two to six weeks. The model is trained on eight years of claims and past investigation outcomes. It will use claim amount, hospital, procedure codes, policy age, geography, and the claimant’s prior claim history. Investigators are measured on cases closed per week. The insurer expects to reduce fraud losses materially and has not yet decided what claimants will be told.

Working in a group, produce a governance assessment. CO-3 requires comparison of frameworks and evaluation of implications for stakeholders, so both must appear.

Map the entry points for bias using the table in this chapter, and for each state one specific way it could occur with this particular data set. Pay attention to the target variable and to geography.
Answer the four questions before deployment, in writing, as the accountable owner would have to. Where you cannot answer, say what you would need.
Assess against the data protection framework. Identify the obligations engaged, whether the intended processing sits within the purpose for which data was collected, and what a data protection impact assessment would have to cover.
Evaluate through the three frameworks. For claimants whose legitimate claims are delayed, for claimants whose fraudulent claims are caught, for honest policyholders generally, and for the investigators, state what each framework says. Identify the point of sharpest disagreement.
Assess the accountability gap. Apply the three conditions of responsibility to the data science lead, the claims head, the investigator and the vendor. State who should be the named accountable owner and what they must be answerable for.
Design the controls. Specify the pre-deployment tests, what claimants are told, the contestability route, the monitoring for drift, and the review cadence. State one control you would insist on even though it reduces the financial benefit, and say why.

Present the assessment as a governance memorandum of no more than three pages, ending with a recommendation to deploy, to deploy with conditions, or not to deploy.

22.9 Summary

Chapter Summary

Automated decisions about people differ ethically from human ones in four ways: scale converts an individual failing into a systematic one, opacity denies the affected person the ground on which to contest, feedback means today’s outputs become tomorrow’s training data so a model confirms its own assumptions, and apparent objectivity means computed outputs are questioned less than human judgements even by people who know they are estimates (Manuel G. Velasquez, 2018).

Bias enters at identifiable points rather than through anyone’s intention: the training data, which records past decisions and the patterns in them; the target variable, which is a proxy somebody chose; the features, which can stand in for characteristics never named; representation, since thinly represented groups are modelled worst and the failure is invisible in an aggregate accuracy figure; the deployment context; and the human reviewer, who adds no protection if measured on throughput. Removing a sensitive attribute does not produce fairness, because a model reconstructs the attribute from correlated variables, which is why testing must be on outcomes across groups and why not collecting the data is a reason the question cannot be answered rather than an answer.

The accountability gap is Chapter 15’s diffusion of responsibility in technical dress, weakening contribution, knowledge and freedom simultaneously, and the remedy is the same: a named owner answerable for outcomes whether or not they understand the internals. Four questions govern deployment: is the decision consequential for the person, on what population was it built and how does performance differ across affected groups, can an individual outcome be explained in terms the person can act on, and who is accountable with what route to contest.

India’s framework rests on the Digital Personal Data Protection Act (Government of India, 2023), with data principals and data fiduciaries, consent preceded by notice, purpose limitation, minimisation, rights of access, correction, erasure and grievance, and additional obligations including impact assessment for significant data fiduciaries. Purpose limitation constrains the discovery that data collected for one reason predicts something else, and the word fiduciary signals a discretion exercised for someone who cannot supervise it.

The three frameworks pull hardest against each other on accuracy against explainability, and Chapter 17’s rule applies: disagreement locates a sacrificed interest, calling for disclosure, consent, contestability or compensation rather than a better score.

Key Terms

Algorithmic decision · Scale · Opacity · Feedback loop · Apparent objectivity · Training data · Target variable · Proxy feature · Representation · Aggregate accuracy · Deployment context · Human in the loop · Rubber-stamping · Fairness through unawareness · Outcome testing · Accountability gap · Named accountable owner · Contestability · Data principal · Data fiduciary · Notice and consent · Purpose limitation · Data minimisation · Significant data fiduciary · Data protection impact assessment · Accuracy against explainability

Summary

Concept	Description
What Automation Changes
Algorithmic decision	A decision about a person produced or materially shaped by an automated system
Scale	The difference converting one flawed rule into a consistent failure affecting everyone at once
Opacity	The unavailability of reasons, which denies the affected person the ground on which to contest
Feedback loop	The loop by which today's decisions become tomorrow's training data, confirming their own assumptions
Apparent objectivity	The unearned authority of a computed output, which is questioned less than a human judgement
Where Bias Enters
Training data	The record of past decisions, which teaches a model what was done rather than what would have worked
Target variable	The proxy someone chose to predict, which may not be the thing the firm actually cares about
Proxy feature	A variable standing in for a characteristic that is never named, such as postal code or schooling
Representation	The condition in which thinly represented groups are modelled worst
Aggregate accuracy	The single figure that conceals differing performance across the groups a system affects
Deployment context	The mismatch when a model built for one population, purpose or period is applied to another
Human in the loop	The reviewer whose presence is claimed as a protection
Rubber-stamping	What that reviewer becomes when measured on throughput and given only variable names
The Fairness Fallacy
Fairness through unawareness	The mistaken belief that excluding a sensitive attribute prevents discrimination on it
Outcome testing	Testing across groups on results rather than inputs, which requires retaining the group information
We do not collect that data	The statement that is not an answer to a fairness question but a reason it cannot be answered
Accountability
Accountability gap	The situation in which a decision harms someone and no individual is responsible
Divided contribution	The division of causal contribution across specifier, vendor, engineer, approver and operator
Divided knowledge	The split by which the person who understands the model does not see the people it affects
Reduced operator freedom	The operator's position, measured on speed and lacking authority to override
Named accountable owner	The remedy: a person answerable for outcomes whether or not they understand the internals
Governing a Deployment
Is it consequential	The first governance question, separating a recommendation from a decision that changes a life
Built on which population	The second question, which an aggregate accuracy figure does not answer
Can it be explained	The third question, asking whether an individual outcome can be explained in actionable terms
Contestability	The fourth requirement: a route by which a person reaches a human with authority to change the outcome
The Indian Framework
Data principal	The individual to whom personal data relates
Data fiduciary	The person determining purpose and means, framed as holding a discretion in trust
Notice and consent	The requirement that consent be specific and preceded by a clear description of data, purpose and rights
Purpose limitation	The constraint preventing silent repurposing of data collected for another reason
Significant data fiduciary	The class carrying additional obligations including audit and a data protection officer
Impact assessment	The assessment that provides the statutory hook for the four governance questions
Accuracy against explainability	The characteristic dilemma, where a better model on average leaves individuals unable to contest