The Optimization Trap
How Generative AI is Training Students to Avoid Deep Thinking
If you observe someone who works with ChatGPT or a similar tool for the first time, you will probably notice an almost immediate shift in behavior. The first response from the machine isn't quite right, so they refine the prompt. Next, the doubt kicks in: Is it just me, or is the machine stupid? Ahh, that's better, but still not perfect. Another iteration; then another. Each improvement creates a small hit of satisfaction, encouraging another round of refinement.
This is fun!
This is also textbook technological stickiness.
When you run out of tokens and get the message you have to wait a number of hours to continue, the dissatisfaction hits hard. This isn't just a technological barrier, but also a deeply human and natural one. The experience mirrors exactly what behavioral psychologists observe with variable reward schedules: the unpredictability of when the "perfect" response will emerge keeps users engaged far longer than consistent rewards would.
It is like observing kids who run out of coins for the pinball machine. They have experienced just enough wins to believe the next game might be the one where everything clicks perfectly. We want more coins or tokens. We want to stay in the game.
Of course, this is design by purpose. Generative AI platforms have inherited persuasive UX principles from social media and gaming: variable dopamine reward schedules are in place, most notably in the form of immediate "personal" feedback, and the promise that the next iteration will be even better.
The stickiness becomes even more problematic when combined with AI's push toward personalization. These systems learn our preferences, our writing style, our intellectual shortcuts. They become increasingly adept at giving us exactly what we think we want, creating a convenient feedback loop that can narrow rather than expand our thinking. We're not just using these tools; we're being drawn to them and, more disturbingly, conditioned by them.
From Thinking to Prompting
Platform stickiness is one thing, but what's more concerning is how this iterative loop is replacing deeper forms of cognitive engagement. Instead of wrestling with a complex concept through extended reflection, reading, or discussion, we're learning to break down our thinking into prompt-response cycles. Our thinking becomes faster and shorter. It also becomes increasingly dependent on continuous feedback.
The traditional academic process of forming hypotheses, gathering information, synthesizing conflicting sources, etc. is being compressed into a rapid-fire exchange with an AI system. We're training ourselves to expect immediate responses and to treat knowledge as something that can be optimized through better "prompt engineering" rather than deeper understanding.
To be fair, this iterative refinement can also enhance learning in some contexts - when students use it to explore different angles of a problem or to articulate their thoughts more precisely. The concern arises when this becomes the default mode of intellectual engagement, replacing rather than supplementing traditional forms of deep thinking.
For higher education institutions, this presents a dilemma that goes beyond policy and plagiarism detection. Generative AI is obviously here to stay, and it brings many benefits - including increased pressure to reform a dated model of education. But it is also technology that's actively reshaping cognitive habits.
Students arriving in our classrooms may already be accustomed to the immediate gratification of AI-assisted thinking. They may struggle with assignments that require sustained engagement with difficult material, preferring instead to break complex problems into AI-manageable chunks. For faculty, it is hard to design assignments that require the kind of sustained, uncomfortable intellectual work that builds critical thinking skills.
Moving Forward with Intention
There are many things we cannot control regarding generative AI and education. But surely we can be more intentional about how we integrate these tools into educational practice, observing how generative AI technologies both assist and change fundamental approaches towards educational paradigms at the same time.
We must design learning experiences that explicitly build tolerance for uncertainty, that reward deep engagement over quick optimization, and that value the struggle of working through complex ideas without technological assistance.
This might mean creating "AI-free zones" in certain courses, teaching students to recognize when they're in an optimization loop versus genuine learning, or redesigning assessments to prioritize process over output in ways that make AI assistance less relevant. It might also involve explicitly teaching students about the psychological mechanisms at play in AI interactions, helping them develop awareness of when they're being conditioned versus when they're genuinely learning.
The stakes here extend beyond individual learning outcomes. We're witnessing a fundamental shift in how students engage with complex ideas, and if we don’t intervene, we risk cultivating a generation of students who excel at extracting information but struggle with the deeper cognitive work of synthesis, original thinking, and intellectual resilience.
The irony is that while generative AI promises to augment human intelligence, its current implementation in educational settings may actually be diminishing our capacity for the kind of patient, uncomfortable thinking that has historically driven breakthrough insights. The question isn't whether students can get better answers faster - they obviously can. The question is whether they're developing the intellectual muscles necessary to ask better questions.
If you found this piece worthwhile, you may also want to read the UNESCO Ideas Lab piece The Disappearance of the Unclear Question, co-written by Victoria Livingstone and me.
"AI free zones" is a great idea. Somewhat like the idea of modern gyms - we need to set aside time and consciously exercise our bodies, so that they grow and that so we can enjoy them. The same would be true for the mind.
This is a sharp articulation of the optimization trap in action. When efficiency becomes the default lens, deeper forms of engagement erode. The Reality Drift framework explores this same tension: how optimization loops hollow out meaning as they reshape culture and cognition.