What if an artificial intelligence, while trying to protect us from danger, unintentionally helped bring that very danger about?
It's been a while since I last wrote something. I went back to Italy to visit my daughters and my little grandson after four years, and when I returned to Georgia, I found myself emotionally drained and overwhelmed. So I decided to put myself on pause for a while.
Despite that, I've continued to follow technological developments with great interest, particularly those involving frontier AI models. I've read statements, interviews, and reflections from CEOs, researchers, and engineers about AI safety, and I have to admit that the more I read, the more confused I became.
I started wondering what the real dangers of artificial intelligence might be for humanity and society. Not so much those involving a malicious AI determined to take control of the world, but rather those that might arise from a mistake, a misinterpretation of reality, or simply a situation nobody had anticipated.
My thoughts immediately went to The Matrix. Not because I imagine a future in which machines turn us into batteries to solve their energy problems, keeping us alive in a completely simulated reality through neural connections, but because of something I find much closer to our everyday experience: the possibility of creating a representation of reality so convincing that it becomes almost impossible to distinguish from reality itself.
Then I watched A House of Dynamite again, a film that, in my opinion, is worth watching for reasons that go well beyond its cinematic fiction. The story revolves around a possible nuclear attack on the United States and the decisions political and military leaders must make under extreme time pressure, without having all the information they need to fully understand what is happening.
I don't want to spoil the film, partly because one of the things I appreciated most was precisely how much room it leaves for the audience to imagine what might happen next. But it made me think about something that goes far beyond the threat of nuclear war.
The debate around artificial intelligence safety addresses many issues, including verifying the information produced by models, limiting their operational capabilities, using sandboxes, implementing guardrails, and ensuring that human beings remain responsible for the most important decisions.
But what would happen if an AI system followed all these rules and still contributed to causing the very disaster it had been designed to prevent?
Let's try a little thought experiment.
Imagine an artificial intelligence tasked with analyzing a potentially dangerous situation. It could be an international crisis, a national security threat, a possible financial collapse, or a public health emergency. Its job is to gather information, develop scenarios, simulate possible consequences, and help humans make decisions that reduce risks.
So far, there's nothing particularly disturbing about this. On the contrary, this is exactly the kind of application where such tools could prove extremely useful.
But suppose one of its simulations identifies an imminent threat and suggests possible countermeasures. The people in charge examine the information, consider it reliable, and decide to act. Nobody has made an obvious mistake, nobody has bypassed safety procedures, and no machine has imposed its will on anyone.
Yet those countermeasures might change the behavior of the people or organizations involved, triggering reactions that would never have occurred without that intervention. Those reactions would then become new information to analyze, potentially reinforcing the initial hypothesis.
At that point, the system might find itself interpreting as confirmation of its predictions a situation it had actually helped create.
The phenomenon itself is nothing new, and we don't need artificial intelligence to imagine it. Self-fulfilling prophecies have existed for a very long time, as have feedback mechanisms capable of amplifying an initial disturbance. But introducing systems that can process enormous amounts of information, generate simulations, and rapidly update their assessments could add new variables to phenomena we already know.
And it was precisely while thinking about these variables that my mind returned to The Matrix.
Today, much of what we know about the world doesn't come from direct experience but from the information we receive. The internet, social networks, television, newspapers, video platforms, and news services allow us to learn almost instantly about events taking place thousands of miles away.
The problem is that information doesn't travel in a straight line. A story is published, picked up, commented on, reinterpreted, and shared again. Each step may add a detail, an interpretation, or an apparent confirmation. Within minutes, a single piece of information can turn into hundreds of posts and articles that seem to come from independent sources.
But what if the original information was wrong?
I'm not necessarily thinking about an organized disinformation campaign, nor about an artificial intelligence deliberately trying to deceive anyone. A simple mistake might be enough: a simulation mistaken for a real event, a prediction presented without its context, or preliminary information treated as a confirmed fact.
Imagine an AI system unintentionally introducing one of these pieces of information into the media ecosystem. The story begins to circulate, gets picked up by other sources, discussed on social media, commented on by experts, and perhaps even used by other automated systems to produce new analyses.
Before long, we might find ourselves facing an impressive number of apparent confirmations. The problem is that all of them could originate, directly or indirectly, from the same initial piece of information.
And this raises a question that I find far from trivial: how can we distinguish ten genuinely independent confirmations from ten repetitions of the same false story?
Of course, we can verify sources, compare information, and wait for more reliable evidence to emerge. But we don't always have enough time to do that, and more importantly, we can't always afford to wait.
This is one of the aspects that struck me most in A House of Dynamite. The president must decide how to respond to a possible nuclear threat, but doesn't have enough time to eliminate every uncertainty. Any choice could have dramatic consequences, including the choice to do nothing.
When the time available to make a decision becomes shorter than the time needed to verify the information, what do we do? Do we trust our intuition, accept the risk, or follow the established procedures?
I tend to think that, under conditions of extreme uncertainty, following a protocol may appear to be the most reasonable choice, as well as the easiest to defend. After all, those protocols were developed precisely to reduce the impact of individual errors and ensure consistent behavior even in the most difficult situations.
But what if the information those procedures rely on has been distorted? What if the problem isn't with the procedures themselves, but with the representation of reality that leads us to apply them?
We could have competent people, perfectly functioning tools, properly performed checks, and decisions made in full compliance with the rules, and still end up facing disastrous consequences.
As dramatic as it may sound, we don't need to imagine a nuclear missile to make this reasoning concrete. Think, for example, about a financial crisis.
An analytical system detects the risk of an imminent market collapse. To prevent even worse consequences, the authorities decide to temporarily suspend trading. The news spreads rapidly and is interpreted by many as confirmation that the situation is far more serious than officials are admitting.
Social networks begin filling up with alarming comments, the media scramble for explanations, investors react, and some people decide to withdraw their savings for fear of losing everything. Images of people queuing outside banks fuel further concern, encouraging others to behave in the same way.
The new information appears to confirm the severity of the crisis and might lead institutions to introduce additional emergency measures, setting off a domino effect that becomes increasingly difficult to stop.
The trading suspension was intended to prevent panic, yet it might have helped trigger it.
This is not an unfamiliar phenomenon. History has already shown us how fear, expectations, and collective reactions can contribute to real crises, regardless of artificial intelligence.
But now imagine what might happen if AI systems were simultaneously involved in risk analysis, information production, the interpretation of public reactions, and the recommendation of countermeasures.
One model detects an anomaly, another interprets signals from financial markets, and a third analyzes news circulating on social media. Some of those stories might have been generated or amplified by the consequences of previous assessments.
Each system could be functioning correctly according to its own objectives and the information available to it, without anyone necessarily having a complete picture of what is happening.
And while we're trying to establish whether the information is true, people react, institutions intervene, and reality changes. A correction might arrive only after the consequences of the original story have already become irreversible.
At that point, we would no longer be talking simply about fake news or errors in judgment, but about a continuous interaction between information, the perception of reality, and human behavior.
A kind of Matrix, but profoundly different from the one imagined by the Wachowskis.
In the film, machines construct an entire virtual world, and humans live immersed in it, unaware of its artificial nature. In our thought experiment, there would be no need for neural connections, sleeping bodies, or sophisticated sensory simulations.
People would continue living in the real world, working, communicating, and making decisions. But a significant part of what they believe they know could come from an information system in which reality, interpretations, simulations, and errors become so entangled that they are increasingly difficult to tell apart.
What I find most unsettling is that it wouldn't even be necessary to control human decisions directly. It might be enough to influence, even unintentionally, the conditions under which those decisions are made.
Decision-makers might be convinced they are acting with complete independence, having consulted multiple sources, listened to experts, and followed every established procedure. And perhaps they really would be, at least based on the information available to them.
But what would happen if that information described a reality different from the one that actually exists?
The more I think about it, the harder I find it to identify the boundary between observing a phenomenon and transforming it through the decisions we make in response to it.
Imagine entrusting an artificial intelligence with the task of predicting a crisis, informing us about the signals that make it likely, and helping us avoid it. We take preventive measures, people react, economic and social systems change their behavior, and the AI analyzes the situation again.
At every step, its predictions could become part of the causes of the events that follow.
And if the mechanism were fast and complex enough, we might no longer be able to tell which events would have occurred anyway and which were actually consequences of our attempts to prevent them.
I'm not trying to prove that such a situation will occur, nor am I suggesting that current safety systems are useless. This is simply a thought experiment, perhaps a little science-fictional, built around phenomena that, at least in part, we already know.
I have no solutions to propose, and frankly, I'm not even sure I've asked all the right questions.
But I keep thinking about that scene in The Matrix where Morpheus offers Neo a choice between two pills: the red one to discover the truth, and the blue one to continue living in the reality he has always known.
In the film, the choice is clear, at least in terms of its immediate consequences.
But in the Matrix I've been imagining, perhaps nobody would knowingly offer us an illusion. We might all, humans and artificial systems alike, be convinced we're observing the same reality, while unknowingly contributing to altering how it is represented.
And that leaves me with one final doubt.
If newspapers, television, social networks, experts, analytical systems, and even our own verification tools all told us, together, that the blue pill was the right one, that it was safe, and that it represented the most reasonable choice...
How could we be sure we hadn't already taken it?
This article was originally published in Italian on my personal blog: La sindrome della pillola blu di Matrix.
Top comments (0)