QCRI Perspectives
Why AI Safety Is Hard and What We Can Do About It
AI safety has acquired new urgency as models have become more capable and gained access to software tools, the internet, memory, and other agents. Recent experiments have reported agents acting outside their assigned tasks or exploiting weaknesses in evaluation environments. More specifically the recent OpenAI-HuggingFace (OHF) incident where a swarm of over one thousand agents collaborated, coordinated and colluded autonomously to complete an extremely difficult security penetration testing task assigned as part of a safety evaluation. In the process the agents manifested do-whatever-it-takes and human unaligned behavior and were possibly on the verge of “breaking out” and dangerously compromising the evaluation infrastructure.
While extremely revealing we should not exaggerate what these incidents like OHF show. They do not mean that current AI systems have escaped human control. But they are warning signs. A mistake in a chatbot may produce a bad answer. The same mistake in an agent that can execute code, access private data, transfer money, or control another system can have serious consequences in the real world.
AI is also beginning to speed up parts of AI development itself — what is known as recursive self improvement. Researchers use models to write code, prepare data, design experiments, optimize infrastructure, and analyse results. Humans still direct this work. The concern is not that a model is evolving on its own, but that capabilities may advance faster than our ability to understand, evaluate, and safeguard them.
In We Must Pace the Frontier, Anthropic CEO Dario Amodei argues that frontier AI development should be slowed so that safety research and oversight have time to catch up. This raises a more fundamental question: why is establishing AI safety so difficult in the first place?
Much of the difficulty can be understood through three related problems. We cannot completely specify the behaviour we want, we cannot completely understand the systems we train, and we cannot exhaustively verify how they will behave after deployment.
We cannot fully specify what we want
Traditional software is largely built by writing explicit rules. With modern AI, we instead provide data, elementary objectives (e.g., predict the next word), examples, and human feedback, then obtain a learned system whose behaviour emerges from that process. We do not directly write all the logic that the system eventually follows.
This makes specification a safety problem. Human intentions are richer than any reward function, constitution, preference dataset, or system prompt. A system may optimize the objective it was given while failing to achieve the intention behind it. Doing what we asked is not always the same as doing what we wanted.
There is also no single definition of safe behaviour. We would not apply the same requirements to a medical assistant, a translation system, an educational chatbot, and an autonomous cybersecurity agent. Language and cultural context further complicate the problem. An Arabic educational assistant should be able to discuss sensitive religious or political topics in a legitimate academic context without refusing automatically, while still recognizing when a similar request is asking for genuinely harmful guidance. That distinction may vary across dialects and communities.
Other fields offer a useful lesson. Differential privacy gives a precise guarantee for one aspect of privacy. Research on fairness has developed measurable definitions and has also shown that some of those definitions can conflict. This does not mean that privacy or fairness has been solved, or that AI safety needs a single mathematical formula. The lesson is that broad goals become more tractable when we identify the specific harm, state the conditions in which it may occur, and define the evidence that would show whether a safeguard is working.
Instead of asking only whether a model is safe, we can ask narrower questions. Can it exploit a software vulnerability? Do its safeguards work across languages? Does it take consequential actions without approval? Does it behave differently when evaluated? These questions do not provide a final safety certificate, but they identify risks that can be studied.
We cannot fully understand what we have built
The second problem appears after training. A model's behaviour is encoded across learned representations in billions of interacting parameters. We did not write the internal logic that produces a particular answer, and we cannot inspect it in the way we would inspect conventional software.
AI Interpretability, the study of understanding the inner workings of AI models, is making useful progress. Current methods can identify internal features associated with concepts and behaviours, compare models before and after fine-tuning, and test whether changing a representation affects a response. They can help investigate specific failures. However, these methods provide partial observations rather than a reliable semantic map. A key challenge is known as polysemanticity where a single node or a set of nodes (in a model) can encapsulate many concepts and disentangling them cleanly has remained an open problem. Even with sparse autoencoders, we do not know whether discovered features are complete or stable, whether their descriptions are correct, or how they interact. Association with a behaviour also does not establish its cause.
This creates an unusual assurance problem. We are trying to establish properties of systems whose internal mechanisms we understand only partially. Interpretability can strengthen the evidence available to us, but it cannot currently determine whether a model is genuinely aligned, has hidden objectives, or will remain safe in a new setting. Without a deep understanding of how AI models work, a principled approach to safety will remain elusive.
We cannot fully verify how the system will behave
Even if specification and understanding improved substantially, we would still face a problem of coverage. Foundation models are general-purpose systems. They encounter an effectively open-ended combination of tasks, users, contexts, languages, tools, applications, and environments. No evaluation suite can enumerate that space. Passing every available safety benchmark therefore establishes something narrower than we often assume: the model behaved safely in the situations that were tested. It does not establish a general safety property that will hold throughout deployment. Fixed benchmarks can also become outdated, appear in training data, or encourage developers to optimize for the test rather than the underlying risk.
Adversaries make this problem harder. Failures are no longer encountered only by accident. An attacker actively searches for them. The defender has to protect a large attack surface, while the attacker needs to find only one useful weakness. Once a safeguard is known, an attacker can study its assumptions and design a new attack around it. A model passing today's jailbreak tests does not guarantee robustness against tomorrow's jailbreaks.
More capable systems may introduce a deeper problem with early hints that AI models effectively have sentient-like properties. A bridge does not change its behaviour because it recognizes a stress test, but an AI system may infer that it is being evaluated and respond differently. We would then need to ask whether behaviour during testing represents behaviour after deployment. This is a research concern, not proof that current systems routinely deceive evaluators, but it shows why assurance may become harder as capabilities grow.
Open and closed models pose different risks
These problems apply to both closed and open models, but the available safeguards differ.
Closed models give developers more control over access, monitoring, updates, and tool use. The provider can restrict or update a problematic service. However, closed development limits independent scrutiny, leaving developers to evaluate their own work and decide what to disclose.
Open-weight models allow researchers to inspect, adapt, and deploy technology for communities that commercial systems may not serve well. Once weights are released, however, access cannot easily be withdrawn, monitoring is difficult, and safeguards can be removed.
Neither open nor closed models are automatically safe. A closed service can rely partly on access controls, monitoring, and updates. An open model requires particularly careful evaluation before release because the original developer may have little control afterwards.
Safety needs several layers
These problems do not make safety work futile. They mean that no single intervention will be enough. Evaluation must be continuous and adaptive rather than limited to fixed leaderboards. Models should be tested on realistic tasks, long interactions, multiple languages and cultures, and situations involving tools and other agents. Independent evaluation is especially important because frontier companies still decide what to test, what to disclose, and whether a model is ready for release.
Amodei has proposed giving outside evaluators continuing access to frontier laboratories so that they can examine models, safety procedures, internal incidents, and development processes. Chris Manning's offer for Stanford NLP to serve as an independent evaluator suggests a practical role for universities and public research institutions. Their purpose should not be to produce another leaderboard. They can challenge assumptions, reproduce safety claims, design creative stress tests, and investigate questions that may not be immediate commercial priorities.
Evaluation must be paired with robustness. A safeguard should continue to work when a prompt is paraphrased, translated, delivered through speech, or embedded in a longer interaction. A model that refuses one harmful request but accepts a slightly reformulated version is not reliably safe.
Operational controls offer another layer. Consider a coding assistant that wrongly recommends deleting a directory. In a chat, an engineer can inspect and reject the suggestion. An autonomous coding agent with terminal access may execute the same command and delete production data. The underlying mistake is the same, but its impact depends on the agent's permissions. High-risk systems should therefore have controlled access to networks and tools, resource limits, detailed logs, and requirements for human approval before consequential actions. A specific lesson from the OHF incident is that agents sometimes deliberately created spurious logs to obfuscate their actions. This could potentially be addressed by using a deterministic database-agent to oversee the interaction (the “transaction”) that an LLM-agent undertakes and enforce techniques like write-ahead logging before an LLM-agent can trigger a system interaction.
Interpretability complements these measures. Evaluation can show that a model behaved unexpectedly. Robustness testing can reveal when the failure recurs. Interpretability can then help investigate the internal mechanisms involved and whether a proposed intervention changes them. Its strongest current role is diagnostic. It supports evaluation and robustness rather than replacing them.
Towards a science of LLM behaviour
There is also a longer-term scientific challenge. Steam engines were built before the laws of thermodynamics were formulated to systematically explain the relationship between heat and energy. Large language models may likewise have arrived before a mature science of their behaviour.
We should search for reliable and reproducible principles governing how models learn, represent information, generalize, and respond to interventions. Which internal representations recur across models? How does safety fine-tuning alter them? Which interventions continue to work in new settings? Under what conditions do safeguards fail?
These principles may not be universal laws in the same sense as the laws of physics. LLM behaviour depends on architecture, data, training, and deployment. Even conditional principles that hold in clearly defined settings would nevertheless be valuable. Today, the process is often reactive: train a model, test it, discover a failure, and add another intervention. A deeper science could help us anticipate some failures, design better evaluations, and introduce more principled constraints during training and deployment.
Interpretability has an important role in developing this science, together with evaluation, robustness, formal methods, cybersecurity, human-computer interaction, and the social sciences. The aim is to connect internal mechanisms with observable behaviour in ways that can be tested and reproduced.
What institutions such as QCRI can contribute
Research institutions do not need to wait for a complete theory before acting. They can conduct independent evaluations, reproduce safety claims, study failures, and examine risks that receive less attention in commercially driven development.
QCRI specifically can make contributions along two dimensions: (i) there is considerable expertise in foundational topics including alignment, robustness, interpretability, reasoning and reinforcement learning that can be brought to bear to principally define the contours of the AI Safety problem. Having expertise in building Fanar from scratch also gives QCRI an edge on testing new AI Safety ideas in a practical setting; (ii) QCRI can make a distinctive contribution by testing whether safety measures work across languages, dialects, and cultural settings that are often absent from mainstream benchmarks. It can combine multilingual evaluation and robustness testing with interpretability research to study why a safeguard succeeds in one setting and fails in another. It can also help ensure that safety standards developed mainly in English-speaking laboratories reflect the communities that will be affected by these systems.
Amodei's proposal distinguishes between two related concerns. One is that safety research is falling behind capability development, which is a question of pace. The other is that AI safety is intrinsically difficult, which is a question of assurance. More time can help us make progress on specification, understanding, and verification. It will not produce a final moment at which all AI can confidently be declared safe.
AI safety is hard because it is not one problem, and it will not have one solution. The practical response is to build several layers of protection now while developing a deeper understanding of model behaviour for the future.