The Outcomes Society Manifesto

The more a system can do, the clearer we must be about what it should do

For most of history, societies could get away with leaving many of their outcomes implicit. Human action was constrained by cost, effort, habit, culture, conscience, institutions, professional norms, markets, and law. These limits did not always guarantee good outcomes, but the constraints they imposed often limited the fallout from some of our wilder, badly thought-out schemes. AI changes all this by massively increasing the speed, scale, and degrees of freedom of action. When, in an AI world, it becomes possible to do almost anything, cheaply and quickly, we can no longer rely on the inertia of traditional constraints to stop us from doing stupid things. We are now forced to answer the question: What is the full set of outcomes and the required steps leading to them that we actually want our systems to attempt to achieve?

The issue

Modern societies are very good at doing things: producing, funding, regulating, measuring, reporting, scaling, and optimizing. But we are less good at making completely explicit what all this activity is ultimately meant to contribute to.

This is understandable. Outputs are visible, countable, fundable, contractable, and attributable. Outcomes are harder. They are shaped by many influences and are more difficult to assign to any one actor.

At the current time, we are used to asking:

• Did you deliver the service?

• Did you spend the money?

• Did you meet the target?

These questions matter, but they do not answer the most basic question of all:

Did this activity actually contribute to the outcomes we actually care about?

And, of course, the only way we can answer this question is if we have fully identified and specified the outcomes we are seeking in a format such that we can work out if we are moving towards achieving them.

Traditionally, human action was constrained and vague outcomes were sufficient; when anything is possible this will no longer work

The pattern

Markets, organizations, technologies, and governments can all optimize narrow goals while wider outcomes remain vague, implicit, or externalised.

That pattern appears in many forms:

• growth without a full account of wider outcomes

• production without a full account of externalities and cost

• organizational targets without a full account of real-world impact

• AI optimization around narrow outcomes with no mechanism to incorporate wider risks and outcomes

The problem is not outputs, markets, efficiency, or technology as such. All of these are essential.

The issue is that we currently have outcomes-incomplete systems.

Why AI hyper-charges this

AI creates a crisis around the issue of outcomes transparency because it massively increases what individuals, organizations, and states can potentially do.

It can generate, analyse, automate, decide, design, and scale at unprecedented speed and low cost. This creates enormous opportunities. But it also makes narrowly-focused optimization more dangerous as AI’s ability to expand the scope of what we can do grows.

AI alignment is, therefore, not only a technical problem. It is part of a wider social problem: powerful systems need clearer outcome specification.

The more a system can do, the clearer we must be about what it should do.

Becoming outcomes-transparent

Being outcomes-transparent in any context means making outcomes and the steps leading to them:

visible — so people can see what is being sought

debatable — so outcomes can be discussed, agreed on, added to, and improved

trade-off transparent — so people are clear about the trade-offs between seeking different outcomes

assumption-aware — so the assumptions behind the steps that are needed to safely achieve outcomes are surfaced and tested

risk-aware — so risks to achieving outcomes, and risks created by seeking them, can be identified and managed

comprehensive — so pursuing narrow sets of outcomes does not crowd out equally important, wider goals

practical — so outcomes shape real decisions, priorities, resources, and action

evidence-informed — so we are moving forward using the most effective means possible

adaptive — so systems can learn, change, and improve over time.

The Outcomes Society

The Outcomes Society is the overarching idea. Such a society would consist of Outcomes Organizations and, increasingly, as the amount of work being done by AI increases, Outcomes Transparent AI. The three concepts of the Outcomes Society, Outcomes Organizations and Outcomes Transparent AI lie at the heart of the next stage of social, organizational, and technological development. To achieve what we want to achieve in this next stage of societal development, we need increasing outcomes-transparency. The same general principles apply to specifying outcomes and ensuring outcome compliance in all three cases.

An Outcomes Society is one that has moved from activity-focused systems to outcomes-explicit systems, driving all activity.

It asks:

What outcomes do we actually want?

• How broad should they be?

• What trade-offs are involved?

• How do we stop narrow goals from crowding out wider outcomes?

The core claim is:

When the scope of our action was limited, vague outcomes were survivable. When the possibilities for action become trivial to scale, societal outcomes must become increasingly explicit.

The Outcomes Organization

An organization that can clearly show how its activities contribute to the outcomes it exists to achieve.

The easiest way to surface the problem with current organizations is to simply ask: ‘give me the full set of outcomes that your organization is seeking and the high-level steps it is employing to achieve these’. If specified properly, this set of outcomes will fully describe what is likely to result from the organization’s activity. Having specified it in this way then makes it possible to do the next step, assessing the extent to which an organization is achieving its outcomes.

An outcomes organization makes explicit:

• intended outcomes

• steps leading to them

• assumptions

• risks

• trade-offs

• evidence

• measurement

• evaluation

• learning

The core question is:

Not just what is our organization doing, but can we show that what it is doing is focused on achieving an explicit clear set of high-level outcomes?

Outcomes Transparent AI

AI designed, governed, and used in relation to an explicit set of outcomes and specification of the steps that should be taken when attempting to achieve them.

There is much talk of AI alignment and AI safety. The clearest way of thinking about this is that, regardless of how difficult it is in practice, it is at base a problem of a lack of initial explicit outcomes specification followed by robust outcomes-compliance enforcement.

Outcomes Transparent AI is based on asking:

• What outcomes should this AI serve?

• What assumptions should it be working from?

• What rights should it not undermine?

• What should it avoid over-optimizing for?

• What scope of activity is it operating within?

• What decision-making has been delegated to it, and when does it need to seek authorization for taking specific actions?

• How are risks and trade-offs going to be identified and managed?

• How is reporting going to be done on the AI system’s performance?

• What other humans or AI systems are auditing the action the AI system is taking?

• What audit information and documentation is it going to provide regarding what it is doing?

The core claim is:

AI alignment is just another instance of the common outcomes problem that is faced in any situation where an agent (human or AI) attempts to take any type of action in the world. Conceptually, it is just a matter of making AI systems outcomes-transparent in a similar way to that which is being proposed here in regard to the Outcomes Society and the Outcomes Organization.

Outcomes Theory

In order to advance the concept of the Outcomes Society, we need a clear conceptual understanding of how to identify, structure and work with outcomes when doing planning, prioritization, alignment, delegation/contracting, accountability, measurement, evaluation and reporting. There are a number of technical questions which need to be addressed by anyone who wants to be outcomes-driven. These are issues such as how to think about the relationship between outcomes specification and indicator measurement, how to attribute improvements in outcomes to particular parties and how to work out who should be held accountable for what. Confusion about these issues has jeopardized many attempts to be outcomes-driven.

The newly developed area of outcomes theory addresses all of these issues. It should be seen as analogous to accounting theory, which provides the conceptual underpinnings of financial management. In a similar fashion, outcomes theory specifies the key concepts, definitions and principles needed for doing technically robust outcomes work in any domain. Aspects of outcomes theory are summarized here and covered exhaustively in this book.

Practical Tools

In addition to a robust conceptual framework for outcomes work provided by a theory such as outcomes theory, we also need practical tools for working with outcomes, whether that be at the societal, organizational or AI system level. Again, to draw on an analogy with accounting, no one would consider doing accounting work without having such work underpinned by a comprehensive accounting system for the organization or initiative involved. Similarly, in the case of outcomes work, we need to think in terms of any type of outcomes work needing to be underpinned by an outcomes system for the organization or initiative under consideration.

One new tool, which has been explicitly designed to represent and allow us to work with an organization or initiative’s outcomes system, is a DoView Board. DoView Boards have been designed to answer the 20 key questions anyone running, having oversight of, or wanting to understand any organization, policy, strategy or initiative of any sort needs to have answered. See here for a collection of different types of DoView Boards for different types of organizations and initiatives. And here for further information on DoView Boards.

Outcomes Theory Substack

Recent posts