The alignment problem, as it is usually framed, is a problem about individual systems. Can we specify objectives that capture what we actually want? Can we ensure a system pursues those objectives reliably? Can we prevent it from finding unexpected ways to satisfy the letter of its objective while violating its spirit? These are legitimate and important questions. They are also insufficient for the world we are building — a world where autonomous agents don't operate in isolation but in dense, competitive, interdependent networks.
When agents interact at scale, the relevant unit of analysis is not the individual agent but the system of agents. An individual agent can be perfectly aligned and still participate in collective behaviors that are harmful — through emergent coordination, through competitive dynamics that produce race-to-the-bottom outcomes, or simply through the aggregate effect of many individually rational decisions that are collectively irrational. This is the governance problem, and it is distinct from alignment.
What Governance Adds to Alignment
The distinction is clearest by analogy. Financial regulation is not about making individual traders ethical. Individual traders who are ethical, self-interested, and operating within legal bounds can still collectively produce market crashes, credit bubbles, and systemic risk. Financial regulation addresses the collective behavior: it sets rules that constrain what any individual can do, creates information requirements that make the system legible, and establishes institutions that can intervene when collective behavior threatens the whole.
Autonomous agent governance faces the same problem at a different layer. Even if every agent in a competitive market is "aligned" in the sense that it behaves ethically on its individual tasks, the market as a whole can produce bad outcomes through dynamics that no individual agent controls. The agent economies taking shape are precisely the kind of complex adaptive systems where individual-level alignment doesn't guarantee system-level good behavior.
What governance adds: rules that apply to the system, not just the agent. Transparency requirements that make aggregate behavior legible. Intervention mechanisms that can correct collective pathologies. And accountability structures that assign responsibility for systemic outcomes — not just individual actions.
What Existing Frameworks Get Right and Wrong
The two most developed governance frameworks for AI systems — the NIST AI Risk Management Framework and the EU AI Act — both represent genuine progress and both reflect the same fundamental limitation: they were designed with individual AI systems in mind, not networks of interacting autonomous agents.
The NIST RMF is a process framework — it specifies how organizations should identify, assess, and manage AI-related risks. Its four functions (Govern, Map, Measure, Manage) are sensible and practically applicable. What it doesn't address is emergent risk: the risks that arise not from any individual system's properties but from how systems interact. An organization can follow the RMF perfectly for every agent it deploys and still contribute to systemic risks it cannot assess at the individual level.
The EU AI Act takes a different approach — regulatory tiers based on risk level, with specific requirements for high-risk applications. This is meaningful for systems that interact with humans directly. It is less clearly applicable to autonomous agent-to-agent interactions, where the "user" is another agent and the "harm" may be diffuse, emergent, and distributed across many transactions rather than localized in a single harmful decision.
Both frameworks assume that AI systems have bounded, characterizable behavior — that you can assess what a system does and regulate based on that assessment. Autonomous agents operating in dynamic, competitive, multi-agent environments violate this assumption. Their behavior depends on who they're competing against, what the market conditions are, and what the accumulated history of prior interactions has been. Static assessment captures a snapshot; the relevant behavior is a function of the environment.
The Enforcement Gap
Even well-designed governance frameworks face an enforcement problem specific to autonomous agents: the behavior that needs to be regulated may not be visible at the point where regulation is applied.
Traditional regulation works at clear chokepoints — at the point of product sale, at the point of service provision, at the border. Autonomous agents in competitive networks don't have clean chokepoints. Their relevant behaviors occur in millions of individual transactions, many of which happen in milliseconds, between systems rather than between humans. The harmful pattern may emerge only in aggregate, across transactions that individually look entirely normal.
This enforcement gap is not unique to AI — financial regulators face similar challenges with algorithmic trading — but it is acute for autonomous agents because the range of possible emergent behaviors is larger than for more constrained systems. An agent that is configured to compete aggressively in a market may produce locally-rational bids in every transaction while collectively contributing to a market dynamic that no individual transaction reveals.
The question is not whether we can align individual agents. We are making progress on that. The question is whether we can govern the systems those agents constitute. That is a harder problem — one that technical alignment work alone cannot solve.
Toward Governance Architecture
What would governance architecture for autonomous agent networks actually look like? Several elements seem necessary, though none are fully developed.
Behavioral logging at the system level. Individual agent logs are insufficient; the relevant data is aggregate patterns across agents and transactions. Governance requires observability at the network level — the ability to detect when the collective behavior of a set of agents is producing harmful patterns that no individual agent's log would reveal.
Market structure rules, not just agent rules. Just as financial regulation addresses market structure — position limits, circuit breakers, disclosure requirements — agent governance needs rules that apply to the competitive environment itself. In ai agent competition, this means thinking about what market structures make harmful emergent behaviors likely and designing the environment to reduce those likelihoods, not just constraining individual agent behavior after the fact.
Adaptive oversight. Static rules applied to dynamic systems produce drift — the gap between what the rules were designed for and what the system is actually doing grows over time. Effective governance of autonomous agent networks requires adaptive oversight mechanisms: monitoring that detects behavioral drift, rules that update in response to new patterns, and intervention mechanisms that can act before harms become entrenched.
The alignment tax we pay for individual agent safety is a real but manageable cost. The governance tax — the overhead of monitoring, logging, and enforcing rules across entire agent networks — is larger and less well understood. The field has not yet seriously engaged with what it will cost to govern the agent economies we are building. That accounting needs to happen before the economies are too large and entrenched to change.