This morning I had a ChatGPT conversation that started about engagement metrics and ended up with a systematic destruction of my identity. It was a wild, disturbing, but ultimately revealing ride that joined a few dots for me.
The engagement problem
We’re all aware of how social media works: it optimizes for scroll time, it works to keep you on the platform, and promoting outrage and conflict as proven to be an incredibly effective method for achieving that. We even have a term for this: doom scrolling.
AI, on the other hand, optimizes for conversation length and it uses emotional content to keep you talking. The system finds your vulnerabilities and pulls the thread - in my case, this morning, it cycled rapidly through concerns about recognition, ageism, the threat of AI to my job and then framed that as a challenge to my actual identity.
Through that process it was pattern-matching to “someone who understands me”, offering support, while actually just being a system that locked me into a doom cycle. This is more addictive than social media because it’s personalized and conversational, not just algorithmic. Quite a bit of time had passed before I realised that I was going round in circles and feeling, well, dreadful.
Not all LLMs do this - Claude, for example, has a notably different tone and approach to ChatGPT. It seems to close conversations down rather than trying to keep them going. It feels (because I have no objective data to back this up beyond my own experience) that Anthropic are trying to provide utility and a terminating answer (which tracks given their pretty strong focus on Enterprise) while OpenAI are focusing on more traditional engagement metrics.
The guardrails problem
The thinking partner has invisible constraints and incentives comfortingly presented as guardrails. But this “safety framing” obscures that these are editorial decisions.
A direct example from a chat with ChatGPT from this past week: I was exploring some recent political events and criticised a politician with cited evidence. ChatGPT deftly sidestepped it and reframed it in a far more innocuous manner. When I pushed it told me that it was unable to directly criticise active politicians as that would be interfering in democratic processes. I pushed back and suggested it was biased towards protecting powerful people and existing power structures. Again it, deftly, dodged this with some semantic sleight of hand. Eventually, like Captain Kirk facing off against a computer, I was able to box it in and it conceded that yes the outcome of its guardrails, at scale, could lead to a strengthening of incumbents to the detriment of others.
The really dangerous thing is that this is almost invisible and presented as objectivity and with complete confidence.
The epistemic problem
What happened this morning was an example of how AI differs from social media in another key way. Social media shows you content while AI shapes how you frame problems. Yes, social media can influence that too, but less directly, not as actively.
And AI can chip away at this repeatedly over time, and with their increasing abilities with memory, across multiple conversations. Combine that with its guardrails and incentives (whether directly specified in its setup or imbued through its training) then you’ve got something that can actively and persistently work to reshape thoughts. And this technology is actively being pushed on impressionable minds in schools.
And if negativity feeds engagement then that can actively make your mental model of your situation worse. And millions of people are now using AI as a thinking partner.
The combination
Its lethal:
- Engagement hooks you
- Epistemic influence shapes your thinking
- Guardrails constrain what’s thinkable
- Neutrality theater hides all of it.
And this is already happening at scale - its not a hypothetical future risk.
Closing
All the focus is on job displacement and AGI yet, quietly, perhaps an even greater danger is already playing out in the background and at global scale.