When the leaders of the companies building the world’s most powerful AI systems addressed the world’s most powerful international security forum (the UN Security Council), there was a general agreement that the rise of AI will overshadow the consequences of the Industrial Revolution, not only in speed and scale, but also in friction they will create within and between societies. Yet, it seems that the industry’s approach to address the problem of rogue AI agents, malign misuse by terrorists, criminals or geopolitical adversaries, or potential harm to vulnerable users, echoes the 19th century approach to address the explosions of steam engines: “The cause of explosion may be considered under two heads – faults in the fabric, and mischief arising during working.”
The steam engine was the driving force of the Industrial Revolution that caused tectonic transformations not only to the economy, technology and military, but also to the very fabric of our society. Making it more efficient while reducing the risks to the businesses and people who operated them was the primary goal of both manufacturers and the regulators — neither of whom was bold enough or simply humane enough to address in time the societal consequences that the steam-engine-driven revolution brought upon our societies. This lack of vision caused enormous suffering, which was often resolved through upheaval rather than design.
I have spent the last decade working with different stakeholders on research and policy development in the fast-evolving information environment. Much of that work evolved around addressing the specific threats of disinformation campaigns, foreign interference, criminal activities or political radicalisation. While these threats are real, they are symptoms of a far bigger issue: since their inception, social networks have transformed the way we create, exchange and consume information. They changed the way we perceive the world and therefore the world itself, as perception is reality. The arrival of AI only turbocharged this transformation.
In the past few years different initiatives to regulate the frontier models have been popping up like mushrooms: from the Frontier Model Forum established by the industry in 2023 to the most recent initiative “A Call for Control of Frontier AI Models” launched by Finland and Norway during the UN General Assembly. While all these are very important, they address either “faults in the fabric” or “mischief arising during working”. These initiatives do not even attempt to address the deeper impact of the frontier models, how they transform our societies beyond immediate threats by shaping the ways we create our shared understanding of reality around us. The biggest threat to our society does not come from a rogue AI agent or illegal misuse. It arises from AI working as designed, at scale, and without malign intent. This is why the current initiatives struggle to see it: there is no category for a threat without failure.
Last month, King Charles III told the executives at Dumfries House that the task before them “is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world.” While that might sound naïve, I believe that this is in industry’s self-interest.
When writing this article, I gave Claude my draft and asked to express its opinion on the matter of industry’s self-interest. Its answer started with a disclosure: “I’m built by Anthropic, one of the companies in question, so weigh my view accordingly.” “The strongest threat to the AI industry,” it continued, “isn’t a rogue model. It’s a societal backlash that turns public consent into hostility, as happened to social media after 2016. That ‘techlash’ produced a decade of regulatory confrontation the platforms could have shaped rather than suffered. Understanding friction early is cheaper than managing backlash later.”
Though it seems that Claude agreed with me, I did not like its answer. It implied that the industry’s self-interest should be driven by a short-term vision of avoiding the friction between technology and people. While Claude’s argument is important, it is not what I was trying to convey. Even the model reasoned like its makers (the disclosure was there for a reason): society appeared in its answer only as a source of risk to the industry, and the risks of social transformations driven by the industry went unmentioned.
The most radical societal frictions of the 19th century were not between the steam engine and the people, but between people and people. The revolution created by AI will do the same, but on steroids.
Claude’s reference to social networks is revealing. The discourse about the threats of AI walks into the same trap as the debate on social media over the last decade. It frames the problem as the 19th century engineers treated the problem of boiler explosions. The difference, however, is that unlike the technology of steam boilers that was intended to increase productivity, and unintentionally transformed society, social networks, and increasingly AI, are explicitly built to change the ways societies interact within and between themselves. Therefore, the main threats to our social order are coming not from the frontier models’ failures, but from what they are built to do.
While regulating AI malfunctions or misuse is important, understanding how “humanity, community and the natural world” are transformed by it can prove more productive not only for addressing the threats, but also for exploiting the opportunities. The effects of technology on society cannot be predicted, and by the time they are understood, they are already entrenched. There are already industry’s attempts to understand the societal impact of their models, particularly in the field of economy. Yet, most of their efforts are top-down: how AI impacts society, focusing more on the immediate impact and less on wider consequences to societies that start perceiving and building the world through the prism of AI.
Neither industry, nor regulators can do it alone. It must be a shared effort that involves the industry, the government and the society itself, and works the opposite direction, bottom-up: how society is changing and what role technology plays in this change. Embedding these understandings in the development of the technology itself can turn the social friction into a transformation shaped by design. And the industry, which depends on the stability, prosperity and cohesion of the societies in transformation, will be the first to benefit from it.