The Ethics of Predictive Learning Systems

Any system capable of anticipating an outcome is also capable of contributing to it. A student described as at risk may be treated differently, may be told, may internalize it, and may be routed toward a narrower set of options. The prediction participates in the reality it describes.

01Four risks we design against

• 01Labeling. A modeled likelihood is a description of conditions at a moment, not a property of a person. Language and interface must make that unmistakable.
• 02Self-fulfilling prophecy. Predictions influence behavior. Systems should be evaluated on whether flagged trajectories improved, not on predictive accuracy alone.
• 03Surveillance. Ambient measurement without meaningful consent damages the trust that learning depends on. Participation should be visible, understandable, and genuinely optional.
• 04Inherited bias. A model trained on historical outcomes learns historical inequity. Disparate impact must be tested for continuously and published internally.

Intelligence should expand what a learner can see about themselves, not narrow what an institution expects of them.

02Purpose limitation as architecture

The most durable protection is structural rather than procedural. Signals collected to support learning should be architecturally incapable of becoming inputs to punitive processes. That is a schema and permissions decision made early, not a policy commitment made later.