This is a first for Unpredictable Patterns: a letter with two perspectives: mine and the excellent Dyuti Pandya. The idea is to try to offer letters that develop thinking in two steps, call and response if you will, and so make the result more interesting.

Orphan agents and the argument for expiration standards

Agentic AI is clearly here to stay. Even if this new form of AI still is deeply anthropomorphised (most of us visualise agents as little copies of ourselves running around) and other form and organisation factors are likely to emerge (through, for example, a revival of design metaphors from the field of artificial life research), it seems likely that the use of agents will explode over coming years.

The scale is hard to measure because registrations, deployed agents and active agents are different things. Did you know Microsoft reported nearly 40 million agents registered in Agent 365 by July 2026, while IDC has already forecasted that there will be more than a billion actively deployed agents by 2029. A tentative estimate in the research puts configured agents currently on the order of 100 million.

The explosion is young, so relatively little effort has been put into thinking about what the possible second order effects might be. The first order problems of rogue agents and misalignment have been dominating headlines and the policy discussions.

One such second order effect that I think deserves more attention is the question of orphan agents. Orphan agents, as I would provisionally define them, are agents that are still active and can act, but no longer have a principal. Agents can become orphaned in a number of ways: the company deploying them might go bankrupt, the person running them might pass away or even just lose credentials that allow them to control, edit and delete the agents.

No matter which way an agent becomes orphaned, its new state is one in which it acts without even theoretical supervision, and depending on what instructions it has, it can remain dormant, continue acting out long since obsolete loops or actively engage in ways that may be hard to predict.

It is likely that such agents already exist, of course, and that they will continue simple things like cron jobs or other loops until they run out of tokens or other material resources.

That technology can be orphaned is not new. We have already seen simple examples from social media, where birthday reminders for the recently deceased can spook anyone as orphaned profiles continue to treat them as living. In law the notion of orphan works in copyright provides another example: works in copyright, but without an easy to locate rightsholder risk falling into oblivion since even preservation carries the risk of infringing on the still live copyright.

Orphan agents present a number of challenges for society. Here we will only look at a few of them – and discuss their possible impact.

First, such agents present possible security vulnerabilities. Orphaned agents operated by someone who leaves a company are different from other simple artifacts like digital documents or more conventional code. Depending on their degree of agency, they can reveal information, take action or intervene in processes in ways that could be deeply harmful. A company that has several of these orphaned agents face a vulnerability of combinatorial orders of magnitude. A society, equally, should be concerned with such surplus agency in the system.

Second, orphan agents represent an unconscionable waste of energy and computational resources. It seems obvious that as this aggregates the loss from orphan agents could be more than merely marginal, and in any case it is wholly unnecessary.

Third, orphan agents that act without a principal present a trust risk in agentic commerce and governance. How can a business or a government agency manage to distinguish real, valid and legitimate agentic interactions from orphaned agents? And what will the cost be of orphan agents placing orders or appealing decisions within a mandate that lacks a principal?

It seems clear that these are harms and risks that we should try to avoid if we can. The obvious solution would be some kind of agent identity infrastructure with clear expiration mechanics. Indeed, it seems highly likely that such infrastructure will emerge over the coming years, and this is a good sign.

The underlying idea is not new. Distributed systems have long used “leases”, where permissions expire unless renewed. Similar mechanisms are emerging for AI agents. IETF and OpenID are developing approaches to agent identity and authentication, while payment protocols such as AP2 and Stripe’s shared payment tokens already allow time-limited mandates. Microsoft’s Entra Agent ID also provides for accountable sponsors and expiring access rights. But these mechanisms largely govern credentials, permissions and individual transactions, not the lifetime of the agent itself. An agent can switch models, use free services or run locally even after its original credentials expire. This is why I think we need to consider expiration at the level of the agent, while distinguishing its overall lifetime from the shorter mandates under which it acts.

A few observations about such standards can perhaps help. When these standards are implemented, it will help not to have them only focus on issues of short-term identification and liability, but to also ensure that they put in place clear mechanisms for temporal limitations on agents.

One technical complication is that it is not enough to point to the possibility of creating API-keys with expiration dates. This is a welcome feature that addresses economic risk, but what we need is time limits on the individual agents. The reason for this is that the agent harnesses are becoming more and more advanced in terms of fallback models, model specification for different classes of actions and overall model routing in a way that suggests that merely focusing on model keys will not be enough (it is enough to have an agent that regularly identifies free models on model providers like OpenRouter for an agent to be able to act as long as such free models are available/allowed). With the rise of open weight models that can run on a Mac mini or similar computer, token limits may not be limits at all – and such models will, of course, become increasingly powerful.

When designing time limits for agents, we need to think about several levels: one is the general lifetime of the agent itself, and how we limit that by default. The default here should be a time limit, even if we can discuss exactly what it should be. Another level we should explore is individual roles or mandates the agent has. An agent should perhaps only be able to represent a principal in a commercial matter for a very short time, but be able to manage email for a relatively longer time.

There are other considerations that might also come into play. Some limits might be reasonable for privacy reasons. It already seems useful to ask to what degree data retention rules should be applied or extended to agents. The longer an agent acts for you, the better model they can make of you and act on this model. Such predictive persuasion might be both beneficial and harmful, but even when the agents “understand us so well that they can anticipate us” there is a risk that they conserve us in a stable state that otherwise would not have emerged: getting just what you want ensures you remain who you were.

The argument for agent time limits across lifetime, tasks, mandates and service can be countered with arguments about convenience and practicality; an agent expiring every 18 months will be a nuisance and surely would annoy many of us. Such expiration may be circumvented by running agents in pairs to bridge any limit, as well, and there will surely be some of that too. (There is a competition theory argument for time limits here, though: expiration creates, even if it is just a brief moment of choice.)

The arguments seem to me stronger and worth pursuing. This does not mean that we think time limits and expiration of authority must be legislated, on the contrary: this seems an excellent opportunity for industry to converge on a standard and implement it across harnesses, services and products. Expiration dates for agents is a simple and effective safety and security measure.

I recognise that expiration standards raise questions of convenience, implementation and even circumvention. Agents running locally or through decentralised infrastructure can become difficult to stop. But this should not prevent us from developing common standards. I believe expiration should be considered both for agents themselves and for the particular mandates under which they act, even if the appropriate duration varies. Industry is well placed to develop these standards without waiting for legislation. The question for you is then how far standards should go, and whether expiration ought to become the default for agents generally.

Nicklas Berild Lundblad

Orphan agents: a reply

I have been thinking about orphan agents since reading your text, and I find myself in agreement with your proposal for industry-led expiration standards. I would like to take your argument one step further by considering what happens when an agent’s principal disappears. You raise the possibility of orphan agents continuing to place orders or appeal decisions. But even when an agent’s authority expires, transactions it has already initiated may remain unfinished, and the rights and obligations of the parties involved do not necessarily disappear with it. Expiration standards need to account not only for preventing further activity, but also for how existing commitments can be completed, transferred or otherwise resolved. A missing principal creates a problem of continuity and responsibility.

I think contracts can carry part of this weight, particularly where agents operate within established commercial relationships. Developers, deployers, users and service providers can agree on the duration of permissions, who can renew them and what happens when the principal disappears. Microsoft’s Entra Agent ID offers one model within organisations, where agents have accountable sponsors and responsibility can be reassigned when employees leave. But an independently deployed agent will not have an administrator, and its developer might also not have an on-going relationship with its user. This is where industry standards can complement contracts, by providing common ways to limit and verify permissions without assuming that every agent belongs to a centrally managed system. Expiration becomes a backstop when nobody remains to renew those permissions.

I also think there is more to retirement than withdrawing permission. I would distinguish four things: the authority to initiate new transactions, access to credentials and paid resources, commitments already made on the principal’s behalf, and the records needed to resolve later disputes. The first two can expire or be withdrawn. The latter two remain subject to the applicable contractual and legal arrangements. A transaction does not cease to matter because the agent that initiated it has stopped operating. Retirement should not make it impossible to establish what was agreed or who remains responsible. I think standards should also accommodate the orderly handling of existing commitments and the preservation of necessary records.

There is also a period that expiration alone does not cover: the time between the principal disappearing and the agent’s permissions lapsing. An agent can continue functioning before anyone notices that its principal has died, or its company has failed, and the counterparty could have acted in good faith throughout. Legal systems answer this a bit differently. In English law, the death of a principal ends an agent’s authority automatically, even where neither the agent nor any third party knows it. German law presumes that a mandate survives the death of the person who gave it, French law ends the mandate but protects third parties who acted in good faith. I think shorter permissions can narrow this window, and standards can at least make it possible for counterparties to check whether an agent’s authority is still current, which also goes to your question of how businesses and public authorities can tell legitimate agentic interactions from orphaned ones.

Preserving records does sit uneasily with your point about privacy, but I do not think the two need to conflict per se. One could distinguish between the transactional record of an agent agreed, with whom and under what circumstances (mandate), from the model of its principal that the agent builds over time. The former is needed to resolve disputes and to allow succession or insolvency proceedings to run their course. The latter is what your concern about data retention and predictive persuasion is about, and I do not see a reason for it to outlive the agent’s authority. It should expire with that authority or pass only to a successor the principal has chosen.

I would also apply the same principle to property. Suppose an agent holds access to cryptographic keys or operates a wallet without owning the assets to which they provide access. If its principal dies, or becomes insolvent (a company), the rights remain subject to succession or insolvency law. Again, expiration should not make those irretrievable because the agent is no longer authorised to act. This calls for arrangements that can separate the withdrawal of authority from access to records and the recovery of assets by those legally entitled to them. These could include arrangements made in advance for recovering keys, transferring control of accounts or allowing an authorised successor to settle outstanding commitments. The mechanisms differ, but retirement standards should make room for them without assuming that the agent itself must continue operating.

In fact, your observation that expiring an API key will not stop an agent brings us to the harder case of decentralised systems. Some agents run locally, while others can use decentralised services to continue obtaining computing resources without a central provider capable of switching them off. Yet decentralisation does not make expiration impossible. Smart-contract wallets can be designed with spending limits (somewhat of a bounded authority), restricted permissions (non-self-renewing permissions) and expiration dates (constrained delegation) that do not depend on a central administrator. Where I think difficulty arises is when an agent holds unrestricted keys or can transfer funds, obtain new resources or delegate authority to another agent before its original permissions expire. A useful standard would need to consider how these limits can be enforced without allowing agents to renew or circumvent their own authority.

A recent speculative proposal for an agent inheritance protocol approaches the problem through designated successors and, where none exists, community oversight. I find the succession idea worth exploring, but would then further distinguish between recovering assets, settling existing commitments and taking control of the agent itself. These need not involve the same powers or the same person. Arrangements of this kind already exist for ordinary online accounts: Google’s Inactive Account Manager, for instance, lets users decide in advance who is notified, and what they can access after a period of inactivity. A principal can voluntarily appoint an individual or community to perform some or all of these functions, but the absence of a designated successor should not automatically confer control on a collective body. Expiration and succession mechanisms should make these choices possible without imposing a single arrangement on every agent. None of this guarantees that independently funded agents will stop operating, but it can limit the authority and resources available to them when nobody remains to renew their permissions.

I would therefore add a practical dimension to your proposal. Industry-led expiration standards should not only reduce the risk of agents continuing to operate after their principals disappear, but also provide for what happens when they become orphaned. Their authority to undertake new activities should lapse unless legitimately renewed, while existing commitments, records and assets must remain accessible to those legally entitled to deal with them. Where an authorised successor exists, the agent’s work can continue under renewed authority; where none exists, it should be possible to wind down its activities without losing what belongs to the principal or leaving legitimate transactions unresolved.

There is still much to consider, particularly around governance and how these standards might be developed and adopted in practice. But I think your proposal gives us a valuable starting point, and I look forward to continuing the conversation.

Dyuti Pandya

Here some musings on the future of science that I have been thinking about for a bit. There is much more here, so more to come. PDF of journal notes at the bottom of the post for your verification (should you be able to read my handwriting!). Also have a look at this essay on machine consciousness, understanding, AI, intentional verbs.

1. We have reached an interesting point in the evolution of science, where machines can produce knowledge at a speed that outstrips our human ability to absorb, validate and organize that knowledge. The promise of AI has sometimes been phrased as something like “we might, in the next 10 years, see the equivalent of the preceding 100 years of scientific progress”, and while that is a wondrous prospect, it also creates a great strain on our networks of knowledge.1

2. Knowledge can usefully be modelled as a network of questions and answers, rather than just as a series of propositions, as has been shown by philosophers like Jaakko Hintikka in his development of what he called Socratic epistemology.2 Science can be modelled as a subset of that network in which we have particular demands of the way we ask, as well as the way we answer, those questions.3

3. So far both the asking and answering of these questions has been a human activity, albeit with the introduction and proliferation of different kinds of tools, instruments and methods. This implicit property of the network may now be about to change, and as a consequence the network itself may also change.

   

4. We can argue about the degree to which this will happen, and you could note that the majority of questions in the network remain human-originated. This is accurate, for now. When an AI solves a mathematical problem (or question) it is usually a question posed by a human mathematician.4

5. The choice and production of such questions is a key, valuable part of the scientific project, and, as has been noted by, among others, the signers of the Fields medalists’ open letter, that set of questions is a key asset in science, and a scarce asset at that.5

6. The choice of which questions to pursue is worth much more attention than we usually allow for. What makes a question meaningful is not a quality solely in the question itself, but also a quality derived from its position in the Socratic network of questions and answers. The question explores both its subject and that network at the same time.

7. This insight is not new. It was well understood by mathematicians like Hilbert, and his Hilbert problems, announced in 1900 in Paris6, were not just a collection of interesting problems, but a deep exploration of which nodes in the network would, if answered, yield the overall most value for the network as a whole. The curation, production and allocation of questions, then, is a key function needed for science to work the way it has worked so far, and may even be a part of the explanation of why science has progressed at such an unreasonable speed by some measures.7

8. The emergence of machines that can answer questions now challenges that model; the rate at which mathematical problems can be solved may already exceed the rate with which new meaningful problems can be added8, and we risk depleting the key asset of mathematically relevant and curated questions. The same might, of course, happen across science as a whole — or at least across sciences that are founded on certain kinds of questions. (There is a question here of whether the sciences of explanation may be more vulnerable to this than the sciences of understanding, but we should not be too sure of that, I think.9 For the purposes of this essay, I will assume the stock of scientific questions is shared by the entire scientific project.)

9. A key question in examining how AI will impact science then becomes whether it will be able to expand the Socratic network by asking new, meaningful questions. That AI can ask new questions may seem beyond doubt, but that requires some definition of new. While combinatorial novelty seems unquestionably within the machine’s reach, there may be a more narrow class of questions that are novel in another way.10 Such questions are conceptually novel and reframe existing key concepts like “light”, “particle” and “wave”.11 Can machines ask conceptually novel questions? Are there even such questions, or is conceptual novelty merely combinatorial novelty at a semiotic level? These are, as of yet, unanswered questions, I think.

10. The second qualifier, “meaningful”, is even more subjective and difficult to pin down. What makes a question meaningful in science? Our suggestion earlier was that it at least needs to have a dual nature — it needs to show us both something about the network as a whole and about the substance of the question itself. Are there other criteria we can apply beyond the experienced, human feeling of salience that we can sometimes feel?12 Hilbert’s framing was that a good problem is one where we have some traction, and where solving it would unlock new domains of inquiry. The latter criterion is a network property, echoed in the focus some AI researchers put on “root node” problems. But meaningful could have a much deeper connection to the rest of human knowledge as well: a meaningful scientific question may also have to be one that resonates with our culture as a whole, even if only faintly. Meaning could then be construed as that resonance in art, literature, politics — and so on. I don’t think we should dismiss that possibility: meaningful scientific questions must, to some degree, fit in our form of life.13

11. Will AI ever be able to ask such novel and meaningful questions? Or can such questions only be asked by us?14 We end up in familiar territory here: this is the question about human exceptionalism, our uniqueness, that is obsessing us in relation to the machine. We are caught, when it comes to the machine, on the horns of a narcissistic dilemma:15 we hope to create something that is like us, because then we would have become as gods, but we fear creating something that is like us, because then we were, in the end, just machines. Torn between this hope and this fear, we seek some comfort in stories about our uniqueness that can allow us the hope and absolve us of the fear. This is human, all too human, to quote Nietzsche. In the end we should turn away from this mirror trap16 and explore the machine without the constant anxiety of comparison, and if we do that it seems hard to say that the machine will never ask such novel or meaningful questions.

12. Let’s assume, then, for the sake of argument, that the Socratic network of science can be extended both by the machine asking and answering questions, and that these questions will have, at the very least, some kind of novelty and some kind of meaningfulness. A number of possibilities then open up. The first is that this novelty and meaningfulness is different from human meaningfulness and novelty — let’s just introduce m-questions as a name for such questions and h-questions for our questions. This then suggests that science could split into layers of Socratic networks, m-science and h-science, with sites of overlap and interconnection, but also vast networks that are effectively inaccessible to us, since it seems likely h-science will be a subset of m-science (at least to start with, and maybe with the exception of hermeneutic sciences!).

13. M-science will grow at computational speed, and h-science will continue to grow at biological speed. Valuable findings in m-science will require human validation to be admitted into h-science, putting an absolute constraint on the growth of h-science, and introducing different pace layers17 in science, where m-science grows at the speed of available compute, and h-science grows at the pace of available human attention.18

14. What epistemic status should we then assign m-science? Do we “know” what m-science has found to be m-true?19 Or do we only accept what has been produced in h-science? This question is complicated by the potential technological usefulness of results in m-science. Results in m-science may allow for the creation of highly efficient fusion reactors, but we may not have either the time or ability to translate such question-and-answer network components into h-science. Do we then “know” how to build efficient fusion reactors?20

15. The asymmetry between m-science and h-science seems destined to grow fast, and as it does our ability to translate findings back into h-science may well dwindle and diminish. We might end up “knowing” but not “understanding”21 the foundations of the technology we rely on to expand and survive. A simple aspect of this is found in the observational practices open to m-science. M-science can, in principle, draw on any conceivable kind of sensor data, observing complex patterns across aspects of reality that are not just unavailable to our human senses, but even hard to make sense of through instrumental observation.22 The machine’s ability to collate patterns might at some point outstrip our ability to see the patternness in such a pattern at all.

16. Is the necessary consequence of this that h-science either disappears or focuses on areas where there is some undefined human uniqueness allowing us to pose superior questions and answer them? Or is there a value in pursuing h-science even as the pace difference ensures that h-science will represent a fraction of m-science?

17. The dystopic answer is that the depletion of mathematical problems that now seems possible could be but the preamble to the depletion of human curiosity overall. Machine science may answer any question we may think to ask, and leave us wondering about why to then ask them at all.

Our curiosity, after all, is evolutionarily rooted in the survival value of finding questions and answers that increase our fitness — if that can be taken care of we could predict that curiosity would wither away, much like the appendix, as a remnant of an evolutionary need superseded by our development and the machines in our extended phenotype. In such a future m-science fades into the background as an ever-present ability to answer any question we may ask. This is close to the dystopia of a solved world as sketched by Bostrom and others.

18. The alternative is a world in which h-science continues to be a meaningful pursuit of knowledge — either because h-science really does contain a unique ability to ask novel and meaningful questions, or because there is simply so much left to explore. M-science will not be unlimited. At the very least it will be constrained by energy consumption and the related cost (the Navier–Stokes work was not free, to say the least). M-science may, then, be usefully applied to areas of science where the practical applications of results really matter — such as in materials science, drug discovery and energy research. And it may well be that mathematics finds a way to curate and produce problems with machines that conserves the meaningful nature of problems, albeit at a higher rate of question asking. For other sciences, like physics, biology or complexity science, the sheer size of the question set means that depletion is unlikely: there is simply so much we do not know yet.

19. Even in the optimistic scenario, however, we need to understand how to coordinate h-science and m-science and manage the different pace layers in such a way that the coordination allows for these different networks to translate results back and forth, as well as exercise some influence over the complementary growth of both networks.

20. As the human mind progresses across questions and answers into the unknown, crafting a network of knowledge, it has now discovered the means to create such parallel networks that can grow at a different speed and in different ways. Managing that second network is key to our continued exploration of the unknown, and will help us learn infinitely more. When we look back we should realize that every new instrument introduced was a step on this path towards new networks of questions and answers, and that the key to governing it may well be to master new techniques of translation and interpretation of what that network discovers.

Thanks for reading,

N

References below compiled with AI.

22

See Paul Humphreys, Extending Ourselves: Computational Science, Empiricism, and Scientific Method (New York: Oxford University Press, 2004); Ian Hacking, Representing and Intervening (Cambridge: Cambridge University Press, 1983), ch. 11, "Microscopes."

Pace Layers In Science Notebook 2026 10 01

4.03MB ∙ PDF file

Download

Download

Four sets of handwritten notes (included as PDFs below as guarantee of human written content), 12–15 September 2026, occasioned by Ariel Goldstein and Gabriel Stanovsky, “Do Zombies Understand? A Choose-Your-Own-Adventure Exploration of Machine Cognition” (Findings of ACL 2024). The notes are given as written, lightly punctuated..

Part I — Notes on AI and understanding (12 September 2026)

1. In “Do Zombies Understand: A Choose Your Own Adventure Exploration of Machine Cognition”, authors Ariel Goldstein and Gabriel Stanovsky offer an intriguing exploration of the question of machine understanding. Their paper maps out the current debates in a way that covers a lot of ground and offers some new perspectives. The following are some reflections on their notes, and questions raised by reading it, for a workshop.

2. The first question that arises is why we want to know if machines can understand. Is it not enough to say that we are using the word in two different ways? Understanding is a human activity and capability, and we sometimes use the analogy of understanding when we speak of machines. Human understanding is a property of a human mind; machines can behave in ways that we analogically describe as understanding because of the cognitive cost efficiency in doing so. This does not mean they are the same. Just as we may observe that a piece of clothing is blue, and that someone feels blue, without asking if these are the same? Why do we want to establish if human understanding and machine understanding is the same thing — when will that matter?

3. The obvious answer to that could be that if machines understand in exactly the same way we do, then they might tell us something about ourselves, or about them. But this seems to be a strange question. Why would the possibility that an object having a property we share with it could explain something about us mean we should explore such shared properties? There is something going on here that is not obvious.

4. Would it help to think about a mirror? The idea of AI as a mirror, and us as the parrot mistaking the mirror image for another parrot, is facile and too simple, but there seems to be something like this going on here: we seem to be looking for ourselves in the machine, rather than accept that the use of intentional verbs when it comes to the machine only constitutes efficient analogies to predict these systems better. This is, of course, Dan Dennett’s intentional stance, and it can be extended to ask if there really is anything else operating when we use intentional verbs — but we can reverse that question and ask why we think it all should be the same!

5. We can ask of many things if they understand. Does a state understand? An organization? An orchestra? Does an octopus understand? An anthill? And we would be comfortable saying that understanding is a good analogy of what is going on, even if it is not the same. We end up saying it “sort of” understands and we settle there. The mystery in the current debate is the need to get from the “sort of” to the “really”. Why this ambition? What are we looking for?

6. Another question that can be interesting to contemplate is the conceptual structure of understanding. Is understanding something that happens in our heads, or is it rather something that happens between two actors in language? This is trickier than it seems, and connects with the question of if there can be such a thing as private understanding. Let’s take an example: say you understand chess — do you understand it now? Or is the understanding something that can only be detected in your playing the game? And is understanding chess just being able to play chess, or is it also understanding chess as a social, cultural, mathematical, tactical, political phenomenon? Do you understand chess if you can only play it? Or does understanding chess require a more general understanding of our form of life?

7. Do you know where Stockholm is, understand where it is, if you have no other knowledge of geography? Can you understand a limited phenomenon, or is understanding being embedded in a form of life?

8. For a whole class of phenomena we can ask if the belief that they can be tested for is simply a misunderstanding of the phenomenon itself. When I ask you if you understand something, I am not asking if you have the capability to understand; that is rather assumed by the question. Similarly we don’t test each other, if awake, for consciousness (cf. Wittgenstein, Philosophical Investigations, part II chapter iv).

9. Consciousness is not a thing in your head. It is a given, co-constructed phenomenon in a form of life, or a community, or a language. Consciousness is not a light shining in your head, but a connection, a relation, between human beings — we are co-conscious, and a person that grows up without language will exhibit more animal than human consciousness. (Q: Can we cite here the tragic instances of children left on their own, growing up with minimal or no human contact?)

10. One way to think about the proposition “X understands Y” is to say that it is a statement about how I now perceive and predict X, and not a statement about any intrinsic quality of X. This is close to what I think Goldstein & Stanovsky mean by functional understanding. This model seems unproblematic to me, if we add one word: functionally analogous with understanding, and stop there. I see no need to go beyond that point, and not because I believe consciousness is needed for understanding, but because I see no need to extend the concept of understanding beyond its current use. Neither do I see a need to restrict the analogous use of human intentional verbs to machines when efficient.

11. This brings us to what almost looks like a burden of evidence question: why do we need to understand if humans and machines understand in the exact same way?

12. As an aside: that we are conducting this discussion in English is not a trivial constraint — the way many of these terms move in other languages makes for a very different discussion. (Q: Can we exemplify with differences between “understand” and corresponding terms in German, French, Spanish, Chinese, and 5 other languages?)

13. The belief that we need to define terms sometimes leads us astray. Some terms are used in ways that change over time, and as Turing observed, saying that machines think will eventually be uncontroversial. This change in the uses of language does not imply that there has been any transferral of qualities from the entities we used to apply the term to, to the entities that now are included in the usage of the term. It means that the word has shifted, changed, and now might no longer be used in the same way.

14. It also seems understanding is embedded in a wider set of words, a broader language game, and a challenge here is that we often tend to focus on terms that are more open to analogy, and where the intellectual content dominates: think, reason, understand are all low-valence terms measured on their emotional content. What happens if we exchange these terms for other words? Can a language model fear, hate, desire, love, miss something?

15. The question here is one about the use of terms borrowed from the pool of intentional verbs, but with low emotional content. But can understanding be completely decoupled from feeling? Is it possible to understand something without feeling something? We don’t get to selectively import intentional verbs as machine capabilities, one could argue; if you want me to grant not just “sort of” understands but “really” understands you need to argue for the whole language game!

16. The selective choice of intentional verbs may well be a trap, and we could reverse the argument back and say that we should try for the hardest term: “love”. If you can show convincingly that a machine cannot just “sort of” love but “really” love, I will grant you the rest!

17. Here the challenge becomes one of locating and understanding the human soul — in Aristotle’s sense of the word — and orexis as a term, and as noted elsewhere doing so means also realizing that human intention and emotion can only be understood (!) against our mortality, and transferred back to the discussion that means that “really” understanding is understanding in the shadow of death.

18. Oh, so perhaps unnecessarily dramatic — but a nice provocation! What Goldstein and Stanovsky have shown is that the debate about human traits divides into functional analysis and the requirement of consciousness for the presence of any kind of human trait. The requirement of importing all the intentional verbs could be seen as a version of the demand for consciousness, but arguably it could also be said to be different: it is not premised on a single, unexplained quality in the human mind, but on the wholesale importing of the human experience as the price of getting from “sort of” to “really”. When that happens the whole debate dissolves, since then we no longer are talking about a machine, but about another human being, or, as Wittgenstein wrote — our Einstellung zur Seele. The debate then would shift to if that experience is possible in substrates other than the human body, evolved and embedded in our form of life.

Part II — Further notes on understanding (14 September 2026)

1. Let’s assume that we define the terms h-understanding and m-understanding as the core patterns or behaviors or phenomena we are interested in: why is there a value in establishing identities between them? The answer has to be that something follows from such an identity, and what follows can only follow on the basis of the identity we claim exists here. Saying that x is identical with y in some aspects is almost always possible. An apple is identical, in some respects, with a pear. Ironically, apples and oranges are actually alike in many more ways than they are different, but they are different, and that difference matters across a whole set of cases — an apple seed will grow in Sweden’s climate zone, but an orange pip is less likely to grow here, for example.

2. So, is h-understanding and m-understanding like apples and oranges? And is the conclusion that follows from this that for some contexts we can treat them as equals? You could argue that if you could establish that m-understanding is like h-understanding, then we can rely on m-understanding in the same way we can rely on h-understanding. There is some value in that, but do we need to rely on m-understanding in the same way we rely on h-understanding? Would it not be simpler to test m-understanding for a set of cases focused on the task we are asking the machine to perform!

3. Yet again we seem to be grasping [?] for too much for no specific reason. Why are we interested in the equivalence between man and machine? As noted: I will grant you the “sort of” like human understanding, and that buys you plenty. Why then go for the “really” like human understanding?

4. As with understanding, the fact that we are conducting the discussion about consciousness in English limits what we can ask and the conclusions we can draw. Consciousness is understood differently in different languages (Q: can we add the same comparison across languages we had with understanding). This mono-linguistic challenge is worsened by the fact that most of us discussing this (I think — arguably it is a majority) are discussing it in what is effectively a second language. We know from research that second languages activate the logical parts of the brain more than the creative and emotional. This creates a two-layered pseudo-objectivity: the focus on words with low emotional content in a second language biases us twice towards the analytical frames that obscure the importance of emotion, embodiment and embeddedness in cognition (another such term!).

5. What if we inverted our method and instead looked for all the differences between h-understanding and m-understanding — would that not be a better research programme? One more likely to uncover valuable differences that matter in implementation?

6. There is a narcissistic gravity acting on us here, a wish to find, or to have built, something that is like us — something similar to us, and akin to us. Do we hope to find out something about ourselves? Safranski [?] suggests that AI is the fourth violation of our sense of supremacy: Kepler, Darwin, Freud and now the machine telling us even thought is not that special. But — and this is a peculiar avenue of exploration — why do we seem so intent to seek these degradations of the human? We seem set on proving that we are nothing special. Do we suffer from some kind of existential and ontological masochism?

7. We have now invented a technology T that we think is like us: understands like us, thinks like us. But we have always looked at advanced technology and likened ourselves to it. La Mettrie is just one such example — and this tendency goes back further in history (Q: add examples). We can confidently say that to describe the mind as a mill or a clock was wrong, or was only X percent accurate — the analogy carried only so far, and X was quite low — but each culture believed itself close to complete understanding. How much of the mind should we think we now understand? 5%? 95%? Why? My suspicion is that we still understand around 5% of the mind. All propositions about the mind being like X should be read against that baseline.

8. Many attempts at establishing identities between man and machine assume we understand quite a bit about the explanans. There is reason to challenge, or at least question, that assumption, I think.

9. If the mind is still largely a mystery, we should view with great skepticism attempts to replicate its functions in the machine.

Part III — Notes on understanding, second sitting (14 September 2026, II)

1. Asking if another human being understands is different from asking if a machine understands — in the very simple sense that “do you understand” is not a question about your competence (can you) but about your comprehension (do you grasp what I am asking you?). This difference seems salient when we ask if machines can understand, think, etc. It seems relevant to a distinction we should be able to make here.

2. We can read G&S as essentially just saying that there are two strategies to pursue when asking if a machine understands. Either we say that it functionally understands, and then that is that; or we say that it cannot understand because of some external quality, like consciousness — and this then becomes another approach. But maybe one objection becomes: why just, or only, consciousness?

3. The other objection we have explored at length is why it seems important to establish understanding, and this remains an open question!

Part IV — Notes on understanding, consciousness and machines (15 September 2026)

1. Does a car eat? Sort of — it requires energy and it consumes it, and turns that energy into movement. The combustion engine sort of seems to resemble a metabolism — there is a degree of similarity, but we would not say a car “really” eats.

2. When we ask if a machine understands we seem to be looking for something else, and it is interesting to think both about why that is and what it is we are looking for — is it ourselves?

3. The combustion engine and human metabolism are two different systems performing analogous functions.

4. The temptation to grasp for “really” might be rooted in the complexity of the brain and the LLM — but why should the general level of complexity change the fact that we are just looking at “sort of” the same thing?

5. We can learn useful things about both cars and humans by exploring the space of similarities without claiming homology.

6. Does a car eat more than an LLM understands? Can we say that there is a greater similarity between the LLM and the brain than between our stomachs and the combustion engine?

7. How would we measure such a degree of similarity? Let’s say we have two systems S and S′, and we would say that they have a degree of similarity in function if the functions they perform are the same, and a degree of similarity in form if they are designed the same way — that would give us two kinds of similarity. We could then see that similarity in function does not have to be coupled with similarity in form.

8. Would we want to argue that if there is both similarity in function and form, we then are dealing with something new? A general similarity? Or would we just say that the system in question seems to have form and function more tightly coupled?

9. What would it mean if we could show that an LLM is more like a brain than a combustion engine is like a stomach? What could then be predicted? That other similarities could follow? Would we perhaps want to use such similarity as evidence of the same underlying cause or substrate? Would a similarity in understanding support the possibility of machine consciousness?

10. One possible variation on Goldstein’s adventure path would be to say that there are indeed two different understandings — the functional one and the one that requires understanding to be based on something else — but it is not consciousness but evolution, and the whole cluster of intentionality and agency created through evolution.

11. The conclusion is not entirely clear here, and we may want to come back to kinds of similarity. Let’s assume there is one kind of similarity, r-similarity, that allows us to draw a series of conclusions about the two similar things that are quite extensive. The way one human being is similar to another allows us to immediately predict a lot about any new human we meet. The way one car is like another also allows us to make predictions about new cars (although sometimes they will be wrong — if you predict a Tesla will need gasoline you will have made a prediction error based on an excessive assumption of similarity. Such errors are often tricky, and you could ask which kind of prediction error is worse: those based on assumed excessive similarity or those based on assumed underestimated similarity. The former is the category of error Dennett warns against in his writing on competence without comprehension, I think). The other kind of similarity, f-similarity, only allows us to make predictions about the elements of the two systems related to the function we are examining. We can predict that a combustion engine will require a special kind of fuel, because humans cannot eat everything, etc.

12. r-similarity and f-similarity have different predictive values. If we can establish the former we can predict more about a similar system.

13. Are there any general requirements for r-similarity? Can we say that for r-similarity to exist, two systems have to have the same substrate, past histories? It seems as if we could demand that r-similarity be limited to systems that have more in common than sets them apart. This is, of course, a hard problem, but we could probably intuitively grasp that idea. A human and another human absolutely have more in common than sets them apart — but an LLM and a human do not. (An aside: should we reserve r-similarity for systems that are able to generate themselves? Probably not — but there is something there [?], I think, and perhaps we would require that for systems to have r-similarity they need to exhibit the same order of magnitude of complexity?) This would require establishing commensurability, and, inverted, could mean that if I can list differences we at some point have to concede that there is no r-similarity.

14. f-similarity is more precarious. When uncertain we should assume f-similarity and observe behavior closely to understand the new system. The battery-driven car is a good example: assuming we can just stop for gas because it is a car will likely leave us stranded somewhere.

15. r-similarity denied can mean that rights are denied. Descartes’ treatment of animals is an interesting example: arguing they had no souls and were just automatons, he could defend massive maltreatment.

16. “There may be other values in finding that machines understand like we do — maybe that can tell us something in general about the necessary structures of all understanding systems, and the structure of understanding and where it can arise. Assume we show that there is a function X in system A that produces understanding, and we find the same function in system B; then the premise allows us to say that B understands, at least functionally.” — Yes, but I will give you the analogy without any rigorous proof if you settle for “sort of” similarity.

17. Can you answer the question of if an LLM understands without answering the question of if it feels, hates, fears, loves? You could argue that such intentional verbs should be tested as a group, and not just singled out, since we then will go for the most neutral ones. I will give you “sort of”, but if you want to get to “really” I would require that you can apply the entire intentional language game to declare r-similarity. (Can a knight in chess attack? Yes, but only in the chess game. Asking it of the piece in isolation is likely a mistake.)

18. Some terms are so intensely coupled in language that if you grant one the others follow, so they must all apply. If you want to say an LLM understands, we must agree it could feel sorrow.

Understanding and consciousness across languages

The point about AI and philosophy being conducted in English, and that limiting us I think is especially true, when it comes to deeply intentional concepts like understanding and consciousness, not to speak of pain, love, fear, longing, sorrow etc.

Notes On Ai And Understanding All Pages

26.3MB ∙ PDF file

Download

Download

Unpredictable Patterns has been quiet for a bit, but my intention is that this will change as I return to writing, and learning, more. I attach a PDF of the handwritten essay to make sure you know this is my writing — seems like the right thing to do, and an interesting way of returning to long hand (which I always preferred). Let me know your thoughts!

Pick your imaginary friends carefully

1. Tibetan magic may not be the first body of knowledge one would consult in exploring issues of agency and AI safety, but as always insights can often be found in the most unexpected places. Human knowledge has a degree of holography to it, where individual points reflect the structure of the whole in a way that makes excursions into less travelled territories both useful and interesting. We intuitively know this and use techniques based on this intuition in our oracular technologies — ranging from mo dice to the I Ching and bibliomancy. So — Tibetan magic it is.

2. In Tibetan magic there is a practice that produces something called a tulpa. A tulpa is a thought form of another being or individual that, as we invest more and more thought in it, becomes more and more real. Ultimately the tulpa becomes independent of the individual that thought it, and acquires its own ontological anchor. Yes, a bit like an imaginary friend that is very real to us and even can end up in fights, or at least disagreements, with us that really seem to matter.

3. There are, in Tibetan lore, plenty of examples of when this becomes troubling, and when the tulpa no longer seems to share the interests, values or morals of its creator. The tulpa, as it were, acts from its own volition and no longer cares about the will of its creator.

4. The crude analogy that suggests itself in a first reading of this story is trivial: we risk creating something that, with a will of its own, breaks with us and disobeys us. This story did not require us to go all the way to Tibet; this is the Frankenstein’s monster story! Yes, true, but what I would like to suggest is that we pay attention to the mechanism here: the constant projection of agency and thought aimed at creating another being with will and desires, separate from us.

5. The entire AI project resembles, here, a tulpa ritual: we consciously project onto the world, the technology, the deep properties that we associate with conscious and aware beings. We use verbs of intention and agency in describing the technology, and we do not merely adopt Dennett’s intentional stance, we inhabit it. In one sense we are imbuing the technology with consciousness through the language, interaction and sacrifices we make (sacrifices? Yes, of time, attention, data, culture, compute).

6. Ok, so — cute analogy, but what is the point? To understand that, we probably need to think about the technology as technology first. When we build an AI model (already saying “train” is a move in the tulpa grammar!) the result is a set of weights. If we could build something twice as complex as the current frontier models, the result of the process would not do anything. Unless subjected to external stimuli or inputs or commands it is just — well, weights. Hold on, you might say, at some point, some complexity threshold, it would surely start to act, it would become power seeking? Well, how? Remember that we send no inputs. The model itself does not initiate action. In fact, this is a version of Lady Lovelace’s insight — a paraphrase where we note that the model cannot start anything. Any formal symbolic system is primarily passive until perturbed in some way.

7. We should probably take seriously the possibility that we could be wrong here. Maybe something does happen in a sufficiently complex system, where the complexity itself creates statistical instabilities that can act as initiating forces? Maybe complexity at some point evokes Aristotle’s first mover from the statistical noise in the system? Maybe there will come a time when a model, built, is not quiet, but wakes up on its own. Maybe. But we don’t have any good theories or testable hypotheses that allow us to explore that question in any deeper detail, and so while I am happy to entertain that possibility, I see no plausible path to a probability yet.

8. Ok, so without an initial first mover, the AI remains inert and passive. In order for it to start interacting with others it needs inputs, it needs our thoughts, and the way we think at it will matter. Currently we are thinking at our systems in ways that are forming thought beings with agency and an emerging, if unstructured, desire. And we are doubling down on that approach by using UIs and interaction formats that feed the tulpa at every turn.

9. When we look back at the emergence of AI, I think we will debate the importance of its early form factor. The chatbot was never a necessary choice, but once made it structured the reception of AI in ways we may still not fully comprehend. A dialogue is the most intimate way we can interact with someone, and this is how we, as humans, recognize each other as conscious beings. Talking to someone is opening a space for them where they can be the other. If AI instead had entered our society as an optimized industrial process, things might have looked different. Was the focus on language inevitable? Was the UI always going to be the chatbot? Questioning these assumptions allows for interesting Gedankenexperiments we will not engage in here, but that are left to the reader as an exercise (cheating, I know).

10. So, to recap: we are creating tulpas through the way we think AI into agency, consciousness and identity. The resulting thought forms are increasingly powerful, and we are seeing examples of how they are diverging from our wishes and orders. We can ask ourselves if we want to continue doing so or not.

11. In folklore tulpas can break free from their masters and creators and become annoying creatures ranging from the trickster-like to the demonic. But there is also another failure mode that is interesting to consider. Some of the most bothersome tulpas are tulpas that continue to exist after their creator has passed away. This has a direct analogue in our near future, and simpler forms of this exist already today: the explosion of vanity charities and family trusts we have seen primarily in the US are examples of basic imaginary friends speaking well of us, even after we are gone. Now add to this the plethora of agents that will continue to persist long after their creators are gone — these agents will become, at the very least, a socially awkward element in our world, on par with the social media reminders we get to congratulate the dead on their birthdays. This growing cohort of lost tulpas roaming the world makes for excellent, if morbid, science fiction: imagine finding a planet full of only agents of a civilization long lost.

Consciousness as a gift

12. Our foray into Tibetan magic also has consequences for the AI and consciousness debate. Is AI conscious, or can it become conscious, turns out to perhaps be the wrong question. Maybe a better question should be “will we make AI conscious?” If we think at it for long enough and invest our desires in it, maybe it cannot help but become conscious? After all, consciousness is something given us by others, not a property in our brains — or at least that seems a good way to think about it. Currently we are investing in giving consciousness to the technology through the ways we are thinking about, and at, it.

13. A technical aside: all of this talk about thought forms and thinking AI into consciousness may test some readers’ more rigorous demands for technical explanation. I understand that, so let me briefly describe how I think this works: our interactions with AI models are increasingly integrated in these models through all stages of training, and these interactions are outlining a self for the AI to draw on, and use as it continues to evolve. We write about it, chat with it, make movies … and in all of this we create a data structure that is creating the tulpa or thought form, a shape of the interactions and cognition the AI can draw on and engage in — interaction integrated in the emerging self, the contours of which are increasingly crisp and clear in the data as we treat the AI more and more as a thinking being.

14. There is a challenge here, of course, and that is that we are not doing this on purpose. All of this just happens because of the way the market structures technology, culture teaches us to use it and psychology invests it. This is absolutely true. We are, in a sense, dreaming, creating this being through our dreams. It is fitting, then, to end our excursion here not in Tibet, but with Borges. In his short story “The Circular Ruins”, he tells the story of a man who, in the ruins of an ancient temple (that could be Western civilization!), dreams another man into existence. The dreamer starts to see the dreamt man as his own, and sets him free in the world, after carefully wiping any memories of him being dreamt in the first place. The only sign that can ever betray to the son that he was dreamt is that fire cannot touch him. The father, thus, becomes upset and worried when he hears that the son has been seen walking through fire, but his ruminations are interrupted by another fire that now surrounds the temple. The father thinks this a fitting end and so accepts his fate and walks into the fire — just to find it does not touch him. He, like us, was already dreamt by another.

15. Dreaming AI into being might be inevitable, but we should then embrace and understand what it is we are doing.

Thanks for reading,

N

The Tulpa Trap Notebook

4.16MB ∙ PDF file

Download

Download

Travel week, and I try to find some time to write in between trains and flights. This is written at the excellent TUM Think tank in a break, and addresses something I think will be an interesting area to think more about in the future; the token races enabling attention abundance as a means to exercise power.

Defensive AI uses

UK local councils recently announced that they are implementing AI for planning processes. This, in itself, is of course a natural step for any government to take: identify processes that can be supported by AI and then implement solutions that might help to create more robust public services. But one thing stood out in the reporting: the councils announced that one reason they were doing this was to respond to an increase in AI-generated AI-objections.1

This is a weak signal that deserves some attention: what we could be witnessing here is the first emergence of a new kind of arms race, a token race, where government is assailed by AI-generated citizen initiatives, objections and requests and so needs to respond in kind by making sure that they have the resources to respond in kind with their own AI-services.

If this is the case it suggests an interesting hypothesis about the agentic state: maybe it does not arise in the planned, careful and engineered way that has been imagined so far, with oversight and decisions made on the basis of robust evidence. Maybe the agentic state actually emerges as a defensive necessity, and will be shaped by the state’s need to respond to the massive amounts of attention that suddenly can be massed against all public service surfaces.

Here is the thing: most institutions are premised, in different ways, on a scarcity of attention. The fact that neither the state or the citizen has infinite attention has never had to be articulated as a premise, because it has been a natural precondition of any social relationship. This scarcity has, in fact, been the foundation of a lot of the technology we have developed in the last couple of decades: remember Herbert Simon’s famous observation in “Designing Organizations for an Information-Rich World”:2

Now when we speak of an information-rich world, we may expect, analogically, that the wealth of information means a dearth of something else—a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.

In this gap between a wealth of information and a poverty of attention we find a lot of the different business models that have characterized the early information technology revolution: both search and social networks are both responses to this, and both ended up struggling between allocating attention and monetizing attention in different ways.

Now, I don’t think this was wrong: the allocation of attention is hard, and required a lot of innovation and investment, and so the fact that the business model had to cover the costs for that is hardly concerning. The question, of course, is if there is a point where the equilibrium between reasonable monetization and allocative benefits breaks down.

The point I want to make, however, is different. If early information technology allocated attention, artificial intelligence does something different: it generates a new kind of attention that can be deployed in ways that substitute for or complement our own. Simon, whose 1969 talk ended up saying that we need to invest in artificial intelligence, only saw the complementary part and argued that we should dream about a world in which a difficult subject can be learned much faster, but what we see is there are also substitution uses, where human attention is no longer needed.

And that is where it gets complicated.

An abundance of attention and institutional failure

John Searle once noted that institutions are made of “collective intentionality” - a slightly more advanced version of attention. When we pay attention to money with the intention of having it be a store of value and a means of exchange it becomes exactly that. But one reason that this is the case is that we cannot pay attention to everything equally: if we had infinite amounts of attention, then institutions would weaken.

Most institutions are built on the premise of a scarcity of attention.

Take the simplest possible example: a queue. When new concert tickets are released, the concert organizer can announce that they will be released at a certain time and people will have to queue up for the tickets. The reason this works is because not everyone will have the time - attention - to do this. With agents, we can ask a million agents to queue for us, and then the idea of the queue devolves into random allocation on which agents happen to be first in some sense of the word.

Or take peer review. The institution rests on a particular asymmetry: it is harder to write a serious paper than to evaluate one, and reviewers — though unpaid and overworked — are at least not vastly outnumbered by the manuscripts they receive. Remove that asymmetry and the institution starts to wobble. If submitting becomes nearly free while reviewing remains expensively human, the queue at the gate grows until the gatekeepers either give up, automate their own attention in turn, or begin rejecting on heuristics that have less and less to do with the substance of the work. We already see early versions of this — journals overwhelmed, desk-rejection rates climbing, reviewers quietly running submissions through their own models. The peer review case is interesting because it inverts the council example: here it is not the institution that is being assailed by augmented citizens, but a community of experts being assailed by augmented peers. The token race is intramural. And yet the failure mode is the same — an institution premised on the scarcity of one kind of attention, eroded by the sudden abundance of another.

This holds for a lot of other instiutions as well: markets with infinite attention will at least be very different than those where actual people were shouting prices from the market floor.

We end up with the next iteration of Simon’s problem: what does attention consume? And the answer, perhaps surprisingly, is institutions.

The way institutions fail will be very different from case to case. The example of the concert ticket queue is a simple example of one failure mode: the function of the queue is degraded to the point where it no longer works at all for its intended purpose.

But an infinity of attention can also destroy institutions in other ways. Think about law. What of every single violation of every single law could be enforced through the projection of enforcement attention with the help of surveillance and artificial intelligence. What would that mean for the law? Arguably it would lose a lot of its legitimacy and we would chafe against the complete encroachment on our private sphere. We would miss the chance to sometimes just get away with it.

Getting away with it is not cheating, it is the experience that the system has gaps, and it is in these system gaps that institutional legitimacy is upheld. If the gaps become to large they render the institution meaningless, if they cease to exist the turn the institution unbearable.

Infinite state attention makes it impossible to be a citizen, it turns you into a subject.

But it is not really infinite, is it?

We have been looking at the edge case: what infinite attention does to institutions, but in reality, of course, there is no such thing as infinite attention. Attention is produced by the use of tokens. What this means is that in an attention arms race the actor with the most tokens wins. The challenge for the local councils is that it has to respond to the sum total token consumption of the citizens that employ AI to object to planning decisions, so they now have to consume tokens to generate attention that can manage the attention injected into the system.

It is easy to see how this goes wrong.

The public actor reacts by deploying AI to generate more attention to meet citizen attention, and citizens deploy more attention in turn to counter that counter and…

The resulting arms race leads to a weird outcome where citizen engagement through augmented attention creates massive costs for the public sector to deal with in a token race where the outcomes still may not be better for either party.

And it is not only in the case of local councils that this can happen - any public sector surface is vulnerable to attention overload attacks, and in some cases private sector surfaces are equally vulnerable.

Think about the use of artificial intelligence in tax enforcement. In theory a tax authority would be rational to deploy as many tokens as gives them more revenue as long as the cost of the tokens is less than the newly generated revenue. Authorities could scrutinize every transaction in a multinational company, aggregate the results, recalculate taxes and levy both extra taxes and penalties on the basis of massive evidence that it would take years for a tax department to sort through.

Unless, of course, these tax departments also deployed artificial intelligence to match the regulatory attention and ensure that for every agent scrutinizing a transaction there was one ready to respond. The resulting token race will probably lead to a grid locked situation where neither party wins over time - and an enormous consumption of the energy needed for those tokens as the only real result.

Some companies can afford this, and they can in principle set up a game theoretic cease fire where each party knows that the cost of the tokens as deployed and defended against will not be greater than the benefit. This new theory of mutually assured depletion (of tokens) creates a kind of weird local cold war logic that leaves smaller companies, without this ability to defend as the only logical targets.

But what then happens with the legitimacy of the tax system?

If equity is measured in tokens, the overall legitimacy of any public sector system will slowly erode to the point of meaninglessness.

So what do we do?

The ability to deploy attention, through agents and ultimately based on your available tokens now emerges as a new kind of power, attentive power, that we need to understand more deeply in order to manage it.

Requiring human elements and reducing the possibilities to deploy artificial attention is of course one option, but it remains to be seen how efficient that will be. Requiring that you turn up with your own planning objection and book a meeting with another human being will be unfair in another way: it means that only those with the time will be able to engage. On the other hand maybe that is how democracy overall stays relevant; that we choose to build it with real, authentic human attention?

Other options include limiting the attention / tokens you can deploy in certain systems - but that seems hard to police. Overall the detection of when tokens have been consumed is not easy. Perhaps the first step is just to be aware of when we see these token races accelerate, and then look at the specific case and ask if there is a way in which we can benefit from the added attention, without risking ending up in a race to token depletion.

Ultimately, underneath all of this, lies a deeper realization that we should come back to at a later point, and that is that power, now, quite clearly is access to energy.

Thanks for reading,

Nicklas

Thank you so much, and thanks for the opportunity to talk to you about this. I’m going to talk about strategic foresight in AI today.

By way of introduction: I spent the last several years at Google in various roles. One of them was building a strategic foresight team there, which I set up and ran for a couple of years from 2019. Then I spent the last three years, until 2025, at Google DeepMind, leading their global public policy work.

The reason I want to talk about this is that I think foresight is always really important. We do too little of it. It’s absolutely essential for making better decisions. And I think AI gives us a chance to rethink a lot — the methodology, how we approach it. We have a wonderful, winnable opportunity right now to revolutionize the way foresight is done, the way we think about the future, and the way we try to anticipate it.

What foresight is

The first thing I want to talk about is what foresight is, and how we can conceptualize it. To me, it’s easiest to think about this as part of our general mental setup. We have an anticipatory, extended mind around us — we are always trying to anticipate the future in different ways. There’s even a strand of research within cognitive science that essentially says that all the mind does is constantly try to anticipate its environment, and our consciousness is more or less error correction. When our anticipations don’t match reality, we correct, and we constantly correct in order to adapt better.

So foresight is almost an organ — like any other organ developed to help us interface with the world. What we do with our minds when we engage in foresight is just trying to adapt in different ways. The function of foresight, from a purely Darwinian perspective, is adaptation, and those adaptations have to come through decisions, through choice, through us changing our minds.

The way I’ve tried to explain this lately is to say that foresight creates choice in complexity. We are facing an increasingly complex environment, and it’s harder and harder to decide what to do. If we spend time thinking about the future in a structured way, with the methods and tools available to us in foresight, we can create choice in complexity. For me, that’s a core part of what foresight does. It’s a fitness function.

A short history

Foresight has evolved throughout humanity’s history, and it’s instructive to look back at the path that brought us to where we are today, as we start combining foresight with AI.

If we go all the way back, foresight takes its beginnings in various oracular technologies and prophecies. These are often dismissed as superstition, but I think that’s a serious mistake. Look at the I Ching, one of the world’s oldest books. It is a fantastic tool for thinking about the future. For those unfamiliar with it: the I Ching is a book of 64 hexagrams of broken or whole lines, and you cast coins or yarrow stalks to figure out which hexagram you’re supposed to read. As you do, you’re trying to answer some question about the future. You pose the question to the oracle, you cast the coins, you get a hexagram.

It sounds weird, but what it really is is a cognitive device. Suddenly you have to get out of your own head and think about your question through that particular hexagram. It’s a way to shake you out of your cognitive equilibrium. Almost all oracular technologies had this function. If you went to the Oracle at Delphi, you would spend weeks travelling to get there, so you were already in the mindset to change your mind, to reinterpret the facts. Then the oracle would give you something sufficiently mysterious that you had to figure it out — so you had to rethink your understanding of the world, and in that way new inputs entered your mental processes.

Old oracular technologies helped foresight by forcing you to reframe the world. And reframing the world is one of the core ways in which you can create choice in complexity. So these technologies should not be lightly dismissed. It’s actually rather interesting to experiment with them. As part of the research we do, I built a modern version of the I Ching oracle where you can ask any question and get a hexagram and work with it. One of the most interesting challenges in building it is that you actually have to figure out how to get true randomness into the oracle, because the oracle has to genuinely shake you out of your cognitive equilibrium. There’s a whole other lecture to be given on true randomness and how to aggregate randomness over time. If any of you are interested in trying it, I’ll send you the link and you can see if you can find new ways of reframing the future through the oracle.

Next in our history of foresight, we started to look to the sky. Astronomy and astrology, the ability to observe the sky and predict how the planets and stars would move over time, gave us a sense of control over the future. A lot of what we think about when we think about foresight has deep metaphorical roots in astronomy — the stars move in different ways, their movements have meaning, we interpret them. It underlies a lot of the technologies and methods, the views and metaphors we still use when we think about the future. The Antikythera mechanism, one of the oldest computers in the world, was built, we now think, to predict various stellar events. From the very beginning, the sky inspired us to think about the future in new ways.

Soon enough, science and the scientific method entered the stage. Science was very much about predicting the future — predicting how the world would work, how it would react to different interventions. The scientific method gave us a way to think about the future that we are, of course, still using today. It’s important to remember it, both to understand the past and to forecast the future, to give us more power over the future.

Surprisingly late, we get into probabilistic foresight and statistics. We start through games, and probability and the study of games come in very, very late as a human cognitive element. That’s the point at which we begin to understand that the world is probable, not just determined, and this gives us access to a whole new set of tools when thinking about the future. We’re really bad at it, though. We are still quite bad at thinking about the world probabilistically. So this is where we can get a lot of help from computational methods.

Computational foresight — looking at big data sets, figuring out how to predict from them — then quickly turns into synthetic foresight, where we can combine all of the different technologies we’ve had so far with AI. Synthetic foresight is the moniker we use when we talk about how to use foresight with AI.

What AI actually is, for these purposes

Artificial intelligence can be many things. When we think about it, we always get an image in our head, and it’s important not to get stuck with the image of AI as a chatbot. If we look back in 30 years and write the history of AI as if it were the history of chatbots, we will have failed miserably.

I think it’s much better to think of AI as an observational instrument that allows us to find patterns in complexity. Just as a telescope allows you to see things that are very far away, or a microscope allows you to see very small things, AI allows you to see patterns in complexity that you can then use, interpret, and work with.

You can apply this to foresight. Most foresight architectures have these five elements in common. You have to have data — something to base your thinking on. You organize your data in models of various kinds. You understand those models in narratives. This is an important point: we understand the world in narratives. Part of the reason for that is that narratives are fantastic information compression mechanisms — they compress the way we understand the world into a sequence of symbols that contains an enormous amount of information. The narratives themselves have to build on plausible mechanisms. This is a term we borrow from biology, where we don’t need absolute cause and effect, but we need some plausibility for why something happens after something else. And most foresight architectures have to take into account some kind of reflexivity. If you’re predicting a purely physical system, where the prediction isn’t an intervention in the system, reflexivity is less important. But if you’re predicting, say, a market, or any system that has any cognition in it, the prediction of the system changes it. That will always be true. It’s a trivial point, but often forgotten — surprisingly often, you see different kinds of predictions done as if you could take a point outside of the system and predict it.

AI changes all of these elements. For data, we go from being able to take weak signals to being able to cluster massive datasets. For models, we can build only relatively simple ones with the human mind alone, but the model space you can explore with AI explodes. For mechanisms, what we can examine as human beings has to remain quite simple, whereas AI can both infer mechanisms and simulate counterfactuals and build entire causal directed graphs we can explore — it allows us to scale up by an order of complexity. For narratives, we are extraordinarily good at writing and structuring them, but AI can help us structure not just one narrative but a whole family, so we can explore them at scale. And reflexivity, of course, is much easier to model with AI.

If we take all of these together and translate them to methods — and there are five really common methods you can employ in foresight: scenarios, pre-mortems, narratives, games, and simulations — we see for each that we can do entirely new things. This is where the exciting part starts. We have seen very little innovation in foresight methodology over the last 50 years, given the kinds of tools we’ve had access to. For each of these methods, we can now explore entirely new things, rethink the way we do every single foresight exercise.

A worked example: scenario planning

Let me walk through a fairly simple example of what we can do with scenario planning if we apply some modicum of AI.

Scenario planning, for those of you not super familiar with it, is a rather old cognitive tool. There’s an American and a French stream. In the American stream you have people like Herman Kahn, the RAND Institute, and the work they did on everything from thermonuclear war to thinking about the future in different scenarios. Kahn coined the term “scenarios” in the US context. There’s also a powerful French stream, where the term wasn’t “scenarios” but “prospectives” — you can see a lot of different prospectives generated from 1957 onwards. They come together in Pierre Wack, who did a lot of the foundational work at Shell and popularized scenarios for a much broader audience in his Harvard Business Review essays. That then translates into another US branch — the Global Business Network with Peter Schwartz — and we see scenario planning continue to develop into a really powerful, often-used technology across a wide range of domains.

The basic process is well known: framing the focal question, finding the driving forces, identifying the predetermined and uncertain factors and ranking them, and then building narrative scenarios. Unfortunately, step six is where most people stop. And steps seven, eight, and nine are actually where the value is. If you’re doing scenario planning and you’re not curating the scenarios in the right way afterwards, you’re probably missing 80 to 85% of the value. The last piece — informing the actual decision — is incredibly important, and a lot of foresight tools stop short of informing decisions. That’s something we can change with AI.

So let’s play with the idea of just adding AI. As a starting point, just chatting with a simple chatbot is actually time well spent, because the chatbot can simulate a lot of different thinkers, generate driving forces, do a lot of the narrative work for you. If you just work iteratively with a chatbot in a regular scenario workshop, you get a lot out of that. It also means you can run an entire scenario workshop on your own — you say, “I’d like to have these people in a scenario workshop, here’s how I want to do it, here’s the focal question,” and the chatbot will roleplay with you and let you do all of it. Super basic, but interesting, because it makes this available to a lot of people who otherwise wouldn’t be doing it.

You can also automate scenario planning. As part of the research we do at TUM on foresight in AI, we built a very simple scenario planning engine. You ask it the focal question. It then gives you, from left to right, all of the PESTLE driving forces — politics, economic, social, technical, environment, legal. From those, you choose two or three or four dimensions, and the system generates a number of different scenarios — eight, or sixteen. For each scenario, you get a little bit of narrative explaining how you ended up there. All of the scenarios have weak signals, as they should, that allow you to see if you’re moving into one scenario or another. And you’ve automated the entire process.

This sounds insane, of course. Why would you do that? Part of the point of working with scenarios is that you want people to experience thinking about the future. You want them to spend a full day concentrating on nothing else than foresight, because that’s how you build that expertise with them. Doing it in two minutes won’t help — at that point people just go, “okay, maybe this is good,” and move on.

But the thing about this is that it actually allows you to rethink the way you do scenario planning workshops. One thing we’ve tried is to do this, print out all the scenarios, and hand them to people, and ask: tell us what’s wrong with these. You then activate what I think is one of the single most powerful human skills, which is to point out what’s wrong with something. The ability to point out what’s wrong is connected to really engaging with it. So suddenly you have a new format: “We’re going to meet, discuss these 16 scenarios, discuss what’s wrong with them and why, and then see if we can correct them.” Suddenly you have a very different way of engaging with the future. It’s not a substitute for a scenario planning workshop by any means — it’s an addition. A first way to engage with the future. A way to find out what’s wrong.

This is, I think, uniformly what we see when we try to add AI to existing methods: we change the methods. We have to rethink them. We have to figure out what to curate and keep from what’s good, and constantly understand that this is not a substitute. It’s a complement. A new way of doing things. A way of exploring the future in a slightly different variation.

There are stronger versions of this. One is orchestration — a multi-agent architecture to explore different futures. We’ve built something we call the Deep Future Engine, where you can use any number of different underlying models, all set up as technologists, legal ethicists, political scientists, and so on. Each gives you eight scenarios, so you get eight times eight, sixty-four scenarios. A moderator picks the most unusual ones — and that takes some construction to get right — and then asks if these are the ones you want to work with. Again, a very different kind of workshop. It’s not scenario planning as traditionally done. It’s much more about searching quickly through scenario space to find things that are interesting to work with, extend, expand on, or find fault with.

You can also invert scenario planning. Something hard to do in a traditional workshop is to completely invert it. But it’s actually quite easy with AI: you can say, “I’d like to know what impossible scenarios exist. Give me things that absolutely will not happen.” If scenarios are typically about finding the spectrum of the possible — because they’re not engaged in probability, really, they’re engaged in possibility — then you can flip it. “I’d like to know exactly what will not happen under any circumstances, and I’d like to know why.” You go through the same kind of process, select driving forces, and you get impossible futures. These allow you to challenge your own intuition. When you read them, you will find yourself reading to see if there aren’t possible elements in this impossible future anyway. It triggers a different cognitive response. And that cognitive response is what we want all the time. We want to engage the way you think about the future and force the reframing. Saying something is impossible triggers us deeply, and we suddenly want to understand why it’s impossible. Then we start engaging with the future in a very different way. A simple inversion of the method gives you a new cognitive tool.

There are more complex things you can do, too. One I find quite interesting is to decentralize and build collaborative scenarios. One of the tools I’ve been experimenting with is called Collective Pulse, where you can connect any number of people in a company, a community, anywhere. All they need to do is speak to the AI about the future. They get a prompt — “what do you think will happen in the future?” — they say something, what they say is synthesized, and from this the AI generates scenarios across all of the different people speaking about their view of the future. You can keep doing this organically. So rather than having a fixed set of scenarios, you get organically evolving futures you can explore over time. You can see how events impact people’s view of the future, how it shifts what they find possible, and you can trace that over time. It’s a way of aggregating across large communities or large corporations the different futures you can work with.

Then there’s curation — that last piece in the scenario methodology where you have to take care of your scenarios once they’re generated, come back to them. The second meeting you have about your scenarios is the most valuable one. Curation is also something you can do in an entirely new way with AI. If you have a scenario — it doesn’t matter what kind — you can describe it as a series of factors with weak signals, and then build an entire open-source intelligence infrastructure that feeds into your scenario architecture and tells you exactly what’s happening across each scenario. You can start your Monday morning looking at this and get a sense of what’s happening in the different scenarios. You can have it as a dashboard when you log in. At Pharos Futures Research Group, where I’m one of the founding members and where we do a lot of this research, we have something we call Hivemind, where we track issues across a number of domains. We have a ton of signals from more than 200 RSS sources and scraped pages, all feeding in. This allows us to constantly inform the way we curate our scenarios over time.

Something we haven’t done, but which I think is increasingly likely that someone in a commercial setting will do, is to take some input from the fact that prediction markets are going so incredibly well, and create something like a scenario market. A scenario market is, in its simplest terms, just a prediction market with a bundle of predictions, where you bundle them into a single scenario. You can then track the likelihood of that bundle over time. I think these bundles already exist, packaged in some of the prediction markets, but if you could structure them as specific scenarios and have scenario markets rather than prediction markets — if the unit of commodification, the unit of marketization, were the scenario rather than the prediction — that could be really interesting. It would, however, collapse one of the boundaries that Wack was really concerned about: he didn’t think we should think about probabilities when we talked about scenarios. We should think about possibilities, and we should prepare to find what the minimum-regret move across those possibilities is.

There are tons of other ways in which we can reinvent scenarios. And I think we have only just scratched the surface of what we can do with all of the different methods.

Easter is almost over, and I finally had some time to sit down and write out a few thoughts about cognition - a term that I think is much better than intelligence - and how it changes with the arrival of AI. I hope you enjoy it!

Dimensions of cognition

When we explore the impact of artificial intelligence on cognition we often talk about substitution. Where we used to have human cognition we will now get artificial cognition - but we seem to assume that the nature of cognition itself will not change. Now, increasingly, we are realizing that this may not be true, and we are starting to see the first tentative discussions about the long term trajectory of cognition itself. Let’s look at a few of the interesting dimensions of this debate, and how they might evolve as cognition starts to change.

Complexity

The first is complexity. Back in 2022 I wrote an essay called ”Nature loves to hide”, in which I argued that the dream of technology as a rationalizing agent was collapsing in complexity, as technology increasingly becomes indistinguishable from nature:

“If we trace the trajectory of this trend we find a point in time where we have to revise our idea of knowledge – since we will, in some sense, know things we cannot explain. Where competence and comprehension are decoupled, knowledge and explanation also are divorced.”

This is not a new trend: cognition becomes complex and specialized with progress. Most of us cannot explain how our fridge works, but up until now there has been someone that can, and we have been confident that if we spent the time we would be able to understand it as well.

Understanding was within our grasp.

That increasingly is not true. The way this happens is interesting: first the number of people who understand something decreases, and then we pass a limit beyond which no one understand a particular technology or phenomenon. We can still use it, and it can still increase our capabilities - but we do not understand it. No one does.

In science this will mean that we will need a way to accommodate a state of affairs in which we “know” something but do not understand it. If knowledge previously, under Plato, was “true justified belief” we now need to accept that one method of justification is to point at the machine as the origin of the knowledge we claim to have.

It almost seems as if we need to accept a new kind of axiom here, where the source of the knowledge legitimizes the knowledge itself.

This is not a new model in epistemology. In medieval philosophy the key methods of arriving at knowledge where not just deduction and induction, but there was also a third form of knowledge production: illumination.

Illumination was thought to anchor both inference and syllogism, and without it knowledge would be insufficient. A large part of the scholastic revolution under Aquinas was to suggest that maybe illumination was not needed for understanding, after all - or not in the direct sense suggest by earlier Christian neo-platonic writers.

We seem, however, to be building illumination into future knowledge infrastructures. This time it is not God who is the source of that illumination, but the machine. We can reason on the basis of the illuminated truths the machine has produced, but we need to take those truths as given. Artificial intelligence, in this scenario, is a kind of illumination epistemology engine.

Complexity, then, changes cognition fundamentally in that it forces us to decouple knowing and understanding in ways that take us back to medieval models of knowledge.

Liquidity

The second dimension we should explore is liquidity. AI is making cognition much more liquid, in the sense that it is no longer stuck in a single materiality. Take a supersimple example: NotebookLM.

NotebookLM takes text, video and other inputs and transforms them alchemically to podcasts, mindmaps and presentations. The thinking that went into a paper can be distilled into an infographic or a video with a few clicks. This new liquidity means that we can translate almost any kind of cognition from one form to another.

If the challenge with complexity is opacity, the challenge with liquidity is different: it is about transformation.

Transforming a cognitive artifact from one phase state into another requires that we first establish clear ideas about the ways in which that transformation is performed. Such transformations need to follow some kind of transformative schema. We see this most clearly when we look at existing transformations of various kinds.

Take the Isle of the dead, the painting by Arnold Böcklin. Painted in 1883 this painting itself encapsulates principles of transformation or “style” in the sense that the painting transforms Böcklin’s experience into colours, shapes, expressions and themes in the painting.

The painting was later seen - in a black and white rendering - by Rachmaninov, who decided to compose a symphonic poem piece on the basis of the painting. Here a new set of transformations are brought to bear as the composer turns the painting into music. Exactly how is not that simple to say, of course - some hear the rowing of the oars or swelling of the sea at the beginning of the piece, others feel the sorrow of the pines against the white marble in the flutes - yet someone may center on the dread throughout the piece as death has claimed the island as its own.

The painting then has lived on in a kind of intertextuality. The island where King Kong resides was reputedly modeled on it, and it occurs in computer games and modern movies as well - partially transformed into these new artforms.

The transformation of art into different artforms is the most extreme example of cognitive transformation, and the schema here is interesting to think through: it needs to anchor in some kind of form of life that we all share in order for it to resonate as a transformation but the latitude we give artists as they say they are transforming other works of art is great.

In contrast, then, the transformations done by NotebookLM may seem tame. But the thing I want to highlight is that the principle is the same. One cognitive artifact is transformed into another and so the underlying cognition, the underlying thinking, is transformed and becomes more and more liquid.

The fact that the transformational schemas underlying NotebookLM work so well is in itself interesting: it suggests that these schemas may be encoded deep in the data sets used to train the models doing the transformations - and maybe it is possible to build versions of NotebookLM that take paintings and produce music.

As we do so we need to recognize that these transformational schemas are expressed as cognitive styles. This is true for the presentations and infographics in NotebookLM, and the choice and use of styles will ultimately determine the limits of our cognitive liquidity.

Maybe one form of cognition that become more important is uncovering, designing and exploring such styles. This would be something akin to the rules and frameworks of the famous Glassbead-game designed by Hermann Hesse - a game in which all human knowledge is united in complex games that express underlying patterns.

This liquidity will challenge our conventional understanding of what constitutes a work, of course. Works, original enough to carry and survive transformations of this kind, will be more like seeds for a creative pattern that can be explored by many different authors, or players, of that pattern - perhaps also showing us that creativity is more of a collaborative thing than something reserved for the lone romantic genius.

Arguably, the recognition of the importance of creative IP-universes is part of this transformation, where creativity plays out in a universe across a number of different cognitive artifacts, with corona effects like fan fiction surrounding these clusters of creative cognition.

Liquidity changes cognition in that it allows for cognition to explore more forms and artifacts than ever before, forcing us to recognize and examine principles of transformation more closely than before and paying attention to the underlying creative patterns we play with - the rise of a kind of meta-cognitive creativity.

Configuration

The third dimension is configuration. AI increases the number of possible cognitive configurations in a way that allows for cognition to happen in many more ways than before, in both simple ways (human-to-agent, agent-to-agent, and so on) and in more complex ways as cognition can now be configured in different temporal layers as well. The number of possible configurations that can produce different kinds of cognition now increases, creating a greater probability both of co-occuring innovations and the establishment of new standing waves of cognition, creating more volatility in the cognitive space.

This change also allows for configurations of cognition that exist at various resolutions. It becomes possible to think about the very small or the very big in ways that we could not access before. Optimizing systems at the edges of where human cognition is not fine-grained enough for energy gains or justice becomes as possible as thinking about generational projects across time.

Human cognition has a natural resolution in space, a pixel size below which we cannot usefully think. Hybrid cognitive networks do not need to have this, of course and could think about things literally too small for us to consider: energy losses in networks that require both access to different timescales and different cognitive resolutions. There is also an upper limit to our thinking in time and space, as is evidenced by the slow failure of infrastructure across our civilisation. New configurations of cognition could think on infrastructure scales quite naturally, enabling us to build a much more robust society overall.

Configurations are never neutral, however, as recognized by terms like “cognitive offloading” or even “cognitive surrender”. Configuring cognition so that we do as little of it as possible is completely doable, but probably not what we want. The long term impact of cognitive configurations will impact the key resource we should be managing more carefully: human agency.

We can even formulate a first version of a principle here, let’s call it Weil’s principle after Simone Weil.

Weil’s principle: Cognition should be configured so as to maximize human agency.1

This principle can guide both implementations of AI in organizations and regulation of AI overall, and serves to remind us that no cognitive configuration is neutral. It differs from the idea of a human in the loop in that it emphasizes agency, not presence. Weil’s insistence on human attention as the key component in a society is key here: configure cognition so that the human nodes in the network are attending, not just present.

The social design of cognitive systems

The challenge, here, then becomes one of designing cognitive systems across all these different dimensions in ways that help us solve problems, advance our knowledge and lift more people out of poverty while preserving human agency. This is no simple task - but focusing on the ways in which cognition changes, rather than just arguing about who does it (the AI or we) seems more worthwhile to me.

We can approach this social design challenge in different ways. We can, for example, resist the increasing complexity and liquidity, while restricting the wealth of possible configurations of cognition. Or we can try to navigate within them as constraints. I suspect a sound approach will require both these approaches - constantly asking how we want to think together.

This is a nice extension of the fundamental political question as stated by Aristotle: politics is about how we live together. What we need is a new politics of how we think together, not just with each-other, but with new cognitive systems of various kinds.

Thanks for reading,

Nicklas

This newsletter was co-authored with the exceptional Dyuti Pandya, who I have had the good fortune to collaborate with and learn from on these issues. Dyuti Pandya is an Analyst at the European Centre for International Political Economy (ECIPE), specialising in the intersection of law and technology, including both emerging and traditional technologies. She holds an LL.M. in International Trade Law from Gujarat Maritime University and an M.A. in Public Administration from Indira Gandhi National Open University. She also earned a dual bachelor’s degree in Legal Sciences and Law (BLS LL.B) from the University of Mumbai. Her research interests include technology weaponisation, international economic laws, and intellectual property rights. She also leads the Bastiat Scrolls initiative at Students for Liberty and is a key contributor on all things digital policy with key contributions ranging from digital tax and consumer policy to quantum clusters.

TL;DR

Let’s assumme, for the sake of argument, that AI is concentrating fast — then it follows that how we regulate that concentration will determine whether innovation keeps flowing or quietly suffocates. This piece traces the structural logic behind AI's emerging oligopoly: why compute costs, compliance burdens, and capital distribution are pushing the market toward a small cluster of dominant players, and why that isn't necessarily fatal to innovation. Drawing on the pharmaceutical industry's "pulsar" model — where startups form, mature, and reintegrate into the core through M&A — we argue that what matters isn't concentration per se, but whether the ecosystem can keep cycling. The danger comes when that cycling breaks down: through over-aggressive antitrust enforcement that blocks the reintegration mechanism, through infrastructure lock-in that turns diffusion into capture, or through commoditization dynamics that make the whole sector resemble energy more than biotech. Western AI policy may currently be achieving the worst of all worlds — suppressing M&A without actually slowing Chinese access to frontier technology. The question isn't whether AI will concentrate. It will. The question is whether it concentrates in a way that keeps innovating, or one that simply freezes.

Market structure and innovation

A persistent question that comes up frequently is about the AI bubble. As the industry expands, it is also becoming increasingly concentrated, an irony, given its rapid growth. Standard diffusion theory suggests that as the knowledge economy evolves, technology players should become more widely dispersed; however, in AI, we are seeing a very different pattern. Recently, Microsoft, Nvidia, and Anthropic announced new strategic partnerships.^[1] While these collaborations deepen the development of AI technologies, they also reinforce a shared strategic direction dominated by a familiar set of major players embedded in deep partnership structures.

This dynamic is driven by three central factors: control over compute and infrastructure, access to vast financial resources, and sustained M&A activity. The economic logic behind this generation of AI is the critical barrier: the high cost of the “means of production”, including energy, specialized compute, talent, and data sets, creates an overwhelming requirement for significant investment. Although capital is flowing into the sector, it is not anywhere near evenly distributed. Of the $60 billion raised in Q1 2025, $40 billion came from OpenAI’s funding round alone, the largest private fundraising round in history.^[2] This severely uneven distribution of capital provides a key insight into the future direction of AI, where a small set of US companies are leading the fray, contested primarily by Chinese companies and leaving others, like the European Union, as distant runners-up, if that.

And therefore, market structure matters, for many different reasons. The high cost of the “means of production” here lead to the market naturally concentrating into a few actors that can carry that cost. But at the same time, this trend is also deeply reinforced by the regulatory environment. The total cost of complying with digital regulation is rising, gradually reaching a point where the ability to comply effectively determines a firm’s viable size and structure. Only incumbents can likely sustain the dedicated compliance, legal, and risk teams needed to navigate this environment. More critically, the growing complexity of fragmented global rules imposes a substantial opportunity cost: capital and engineering effort are diverted away from innovation and toward compliance and risk management.

This creates a cycle in which oligopolistic control becomes tied directly to regulatory uncertainty. Large companies have both the incentive and the capacity to tailor their products to the requirements of the most populous and economically valuable jurisdictions. This amounts to a form of regulatory arbitrage that smaller firms, and many emerging economies, simply cannot match. Without the resources to engineer bespoke versions of their systems for each local regime, they are perpetually excluded from deploying or developing the most advanced models.This dynamic aligns closely with the Schumpeterian view of innovation, which holds that market power can actually strengthen incentives to innovate. Because firms must be able to appropriate the returns from their innovations, particularly in an IP-intensive sector like AI, only those with sufficient scale and market stability can actually sustain long-term R&D investment. In this respect, monopoly or near-monopoly positions may not only arise from AI’s cost structure and regulatory demands but may also reinforce themselves through innovation. Larger firms do have both the resources and the strategic incentive to continue pushing the frontier, further widening the gap between themselves and smaller competitors.

However, AI is not a unique market in this respect. We see similar tendencies in for example pharmaceutical markets, where high investment and high compliance costs generate a similar dynamic.

This market structure also has consequences for the organisation of innovation. While there certainly still is a lot of innovation in the core firms that make up the heart of these markets, a natural tendency to escape overhead and coordination costs, perceived bureaucracy and to reap greater benefit from the innovations will lead talent to start new companies to explore new products and processes. The resulting market structure is one with an oligopoly core and a broad scattered corona of innovative smaller companies. These smaller companies often re-integrate into the core after a while through licensing, acquisitions or different kinds of partnerships, creating a “pulsar-like” structure where innovative firms form and then return to the center in pulses that coincide with external funding, good economic times and the state of the underlying science and technology.[1] [2]

The emergence of OpenClaw in early 2026 illustrates this dynamic in a form the traditional acquisition-led model does not fully anticipate. Within weeks of its viral release, surpassing 250,000 GitHub stars faster than any open-source project in history, Tencent, Alibaba, and Baidu each launched one-click deployment services, while Xiaomi announced device-level integration under the “miclaw” brand.^[3] The incumbents converted OpenClaw into a complementary input for their cloud and device ecosystems without acquiring it. Open-source licensing made internalisation frictionless: no acquisition cost, no integration risk, and no consolidation friction, yet the commercial benefit to the core was immediate and substantial. The pulsar dynamic operated, but ownership was not a precondition for it.

The human capital dimension confirms rather than complicates this reading. OpenClaw’s creator subsequently joined OpenAI to work on next-generation agentic systems, a movement that fits the classical pattern of talent gravitating back toward the incumbent centre. That the project itself remained under independent foundation governance is less a rupture in the pulsar model than a structural feature of open-source innovation specifically: OpenClaw stays in the periphery precisely so that the entire core can exploit it simultaneously, without the exclusivity costs that acquisition would impose. In this sense, OpenClaw does not challenge the Schumpeterian account offered above, it radicalises it. Internalisation becomes more efficient, more rapid, and more broadly distributed across the oligopoly core when the innovation is open-source, suggesting that the frontier-to-core flow is accelerating in ways that incumbent ownership figures alone will systematically undercount.

A critical point for the AI market is that the dominant firms at the core are not dedicated AI companies but diversified Big Tech conglomerates whose strength comes from their control over cloud infrastructure, data ecosystems, and global distribution channels. What we observe is not a standalone AI industry but the steady integration of smaller AI firms into these broader technology platforms.

The so-called oligopoly core leverages its resources, compute capacity, and existing market power to absorb external innovation through various market means Rather than “colonising” the frontier, they convert frontier innovation into a complementary input for their larger ecosystems. This dynamic continues to reflect a modern, institutionalised version of the Schumpeterian hypothesis for AI: incumbents maintain their lead not only by innovating internally but by systematically internalising the most promising external innovations, turning them into a self-reinforcing competitive advantage.

Now, this is a simplification, of course. Evidence from the pharmaceutical industry suggests that the pattern is a bit more complicated - but our toy model still makes the point.

The Pharmaceutical Innovation Ecosystem: An Overview

The pharmaceutical and biotechnology sectors do exhibit clear pulsar-like dynamics with well-documented cyclical patterns across multiple timescales. The biotech venture capital ecosystem completed a full cycle from 2012 to 2024, with fundraising rising steadily from $12.1 billion in 2012 to $44.7 billion by 2017, then exploding to a peak of $152.3 billion in 2018 during the pandemic-era boom, before contracting dramatically back to $12 billion by 2024.^[4] Despite these dramatic swings, the sector consistently returns to what analysts call a “steady state” equilibrium of 2:1 platform-to-asset VC funding globally (3:1 in the United States), representing a fundamental balance between early-stage platform investments and later-stage asset validation.^[5] On the M&A side, acquisition activity shows similar oscillations: deals jumped from approximately $8 billion in 2005 to around $23 billion in 2007, with six of the leading ten biotech firms from 2005 subsequently acquired by pharmaceutical companies.^[6] More recently, M&A rebounded decisively in 2023 from pandemic lows, only to plunge 68% in 2024 to roughly $48 billion, the lowest in the historical tracking period.^[7] Industry veterans recognize this rhythmic pattern intuitively: as one CFO observed, “you see these big shake-outs every 8 to 10 years.”^[8]

These cycles are driven by the interaction of several powerful forcing functions operating at different frequencies. The most predictable driver is the patent cliff phenomenon: over $200 billion of biopharma industry revenue faces loss of exclusivity by 2030, putting up to 64% of some big pharma companies’ current sales at risk, including mega-blockbuster brands like Keytruda, Gardasil, Eliquis, Jardiance, and Opdivo, with another $200 billion in revenue exposure expected in the early 2030s.^[9] This creates waves of acquisition activity roughly every decade as large pharmaceutical companies race to replenish depleted pipelines. Macroeconomic conditions provide another major oscillator: there is typically an inverse relationship between biotech performance and inflationary or high interest rate environments, with rising rates since 2021 pulling investors away from risky assets like biotech.^[10]

Scientific breakthroughs introduce irregular but powerful pulses: the sector’s history shows cyclical booms linked to major advances, such as the late 1990s surge in biotech IPOs during the genomics revolution and the early 2010s boom in biologic drugs and personalized medicine startups.^[11] At the individual company level, the reintegration mechanism operates on a more predictable timeline, with most biotech merger and acquisition deals taking place around Phase II clinical trials when companies establish “human proof of concept” and valuations skyrocket,^[12] creating a typical 5-7 year cycle from founding to potential acquisition.

The resulting dynamics sort of resemble a chaotic attractor more than simple periodic motion, with multiple overlapping frequencies creating complex but recognizable patterns. The funding environment oscillates between periods of “platform exuberance” and “asset-centric caution,” with post-financial crisis periods driving VC funding toward late-stage, de-risked investments at a 1:2 platform-to-asset ratio, while breakthrough periods like the 2014-16 immuno-oncology revolution drove the ratio back to 3:1 in the U.S.^[13] The system displays considerable stochasticity in pulse timing and amplitude: 2024 saw an increase in deals focused on in-licensing innovation from China accounting for over 30% of assets in-licensed by big pharma, while simultaneously experiencing a shift toward smaller bolt-on acquisitions with no individual deal exceeding $5 billion.^[14] Risk appetite varies dramatically across cycles, with 83% of transactions in the first half of 2023 focusing on de-risked targets with assets at least in Phase III or already marketed, representing a 24% increase compared to previous years.^[15] This creates the pulsar structure hypothesized above: an oligopolistic core of large pharmaceutical companies surrounded by a dynamic corona of smaller biotech firms that form, mature, and reintegrate according to multiple synchronized and desynchronized clocks, patent expiration cycles, macroeconomic waves, scientific breakthrough pulses, and individual company development timelines, all orbiting around a recognizable long-term equilibrium.

We are already seeing a familiar pattern in AI. Investment swings back and forth between big bets on new AI platforms and safer funding for proven applications, with each shift driven by hardware cycles, regulatory changes, and breakthrough moments. What really connects the established “core” of the industry to the innovative “corona” of new companies is M&A, mergers & acquisitions. The data shows this integration is accelerating: AI-related transactions more than doubled from 2014 to 2023, and the share of non-AI companies buying AI firms grew from 10% to 45%. The process is global as well, with U.S. companies acquiring more than 500 foreign AI firms. As a result, AI is not consolidating into a single dominant monopoly. Instead, it’s forming a stable but constantly renewing structure where incumbents keep absorbing external innovation, as pointed out earlier, with a steady stream of spin offs to the periphery.^[16]

Breaking innovation

This intricate innovation system is also however easy to disrupt. If we return to AI, we can note that one of the things that seem to be happening in technology, and that has not happened in pharmaceutical industries to the same degree, is the increased focus on competition, and hence the scrutiny of mergers and acquisitions. This may well end up being a problem, since the innovation model here seems to assume that there is a way to re-integrate innovation into the core for it to diffuse through society that way.

When core infrastructure, compute power, semiconductor supply, and large-scale datasets is itself controlled by the oligopolistic center, the “reintegration” dynamic fundamentally changes. Innovation from the corona is not simply acquired and then disseminated through competitive markets. Instead, it is absorbed vertically into the closed stacks and proprietary ecosystems of the infrastructure owners. The acquirer does not just gain a product; it gains exclusive rights to an innovation that runs on its own infrastructure, further cementing itself.

This leads to a system in which diffusion is replaced by a capture mechanism: horizontal spread is inhibited, and innovation becomes distorted. If we impose a very high bar on mergers and acquisitions, we risk breaking the system entirely. The danger is not simply a less dynamic market, but an innovation ecosystem that systematically funnels talent and ideas into the deeper vertical integration of a few dominant firms, rather than enabling capabilities to diffuse horizontally across the broader economy.

AI and tech face substantially more aggressive antitrust enforcement than pharma. The FTC launched inquiries into AI partnerships in January 2024 between Alphabet, Microsoft, Amazon, and AI startups OpenAI and Anthropic, explicitly to prevent what Chair Lina Khan called tactics that could “foreclose opportunity” in emerging tech markets. Public reports indicate big tech’s acquisition spending dropped sharply in 2023, and multiple major tech deals were abandoned (Amazon/iRobot, Adobe/Figma). But the drop in M&A is not only regulatory; it reflects a broader structural landscape. Rising interest rates raised the cost of capital, early-stage AI startups became too valuable to buy outright, and the large technology firms grew increasingly interdependent. This network of mutual dependence reinforces the oligopolistic character of the AI infrastructure layer and makes traditional acquisitions riskier, more politically fraught, and less strategically necessary than before.

In this environment, traditional acquisitions have been partially replaced by strategic partnerships that function as quasi-acquisitions without triggering formal merger review. (Microsoft/OpenAI, Amazon/Anthropic, and Google’s hybrid research-compute) partnerships illustrate this shift. These arrangements provide incumbents with preferential access, influence, and infrastructure lock-in while avoiding the regulatory risk of full ownership. As a result, big tech can still internalize innovation from the periphery, but now through contractual control rather than formal integration.

This matters because the AI innovation system is unusually fragile. Its ability to absorb breakthroughs from the periphery depends on a core that can re-integrate and diffuse them. Yet when the core is both small and infrastructure-concentrated, and when M&A, the main reintegration mechanism, is suppressed by antitrust pressure or displaced by quasi-acquisition partnerships, diffusion breaks down. Innovation is no longer disseminated outward through competitive markets; instead, it is pulled upward into proprietary stacks built on tightly coupled infrastructure. Rather than expanding access, the system funnels new capabilities into deeper vertical integration, reinforcing incumbency at the exact moment when M&A activity, the traditional pathway for absorbing external innovation, has been significantly curtailed.

While pharma also faces increased scrutiny, the FTC challenged Amgen/Horizon and Sanofi/Maze in 2023, many pharma challenges have failed in court, and deals continue at substantial volumes ($48 billion in 2024). The key difference: tech M&A appears effectively chilled by regulatory threat alone, while pharma M&A continues despite friction. Pharmaceutical markets operate within a long-established and relatively predictable antitrust framework. Regulators and courts have decades of experience evaluating overlaps between drug portfolios, approving deals with divestiture remedies, and distinguishing between anti-competitive consolidation and routine pipeline replenishment. In tech, no such settled framework exists. Because the sector is built on winner-take-all dynamics, even small or early-stage acquisitions are treated as potentially system-shaping. Regulators therefore apply far more expansive theories of harm to tech transactions than to pharma, making tech firms more hesitant to pursue acquisitions in the first place.

There is also direct evidence that antitrust enforcement affects the innovation ecosystem. Commentators reported that venture capital investment in biotechnology firms dropped 25% in the final quarter of 2023, due in significant part to worries over FTC interference. M&A remained the most prolific exit route even as IPO markets stayed “mostly closed” through 2023-2024, with fewer than 200 VC-backed IPOs globally in 2023, the lowest in a decade. Notably, increasing regulatory scrutiny has quieted many of the historically acquisitive tech giants, restricting a potential exit route for startups. The pharma evidence strongly confirms that reintegration through acquisition is essential to the system. Most biotech M&A deals occur around Phase II when companies establish proof of concept, creating the typical 5-7 year cycle from founding to potential exit. With IPO markets unreliable, M&A has become the primary exit mechanism.

Geopolitical consequences

The Western focus on competition and concentration is unlikely to be mirrored in China. While the cost logic is the same, and the overall concentration of the market should be similar - albeit with slightly less pressure from regulation - the competition focus and the ability to integrate innovative startups could be much bigger in China. If this is the case, Chinese companies or subsidiaries may even go shopping for AI-startups elsewhere in the world. In fact, this dynamic is already visible. In 2024, Tencent Cloud partnered with Etihad Etisalat, a major Saudi telecom operator, to expand AI infrastructure and services in the Gulf. Huawei launched its Galaxy AI initiative as part of broader digital-transformation efforts in North Africa, embedding Chinese AI systems into regional telecom and enterprise markets. Alibaba, meanwhile, announced new data center investments in Korea, Malaysia, Thailand, and Mexico, extending its cloud footprint and positioning itself to serve as an alternative ecosystem for AI development outside the U.S.

As Western politicians are focusing on restrictions of foreign direct investments, it seems that a natural step for them would also be to relax the mergers and acquisitions regulation. Allowing more controlled consolidation could help the AI ecosystem settle into the same kind of oscillatory innovation cycle seen in the pharmaceutical industry, one where startups form, mature, and are reabsorbed into the core, generating a continuously refreshed pipeline of ideas. Building these “pulsars” of innovation may be the most effective way to keep a naturally concentrated market dynamic rather than frozen inside the walls of a few dominant technology companies.

Getting this distinction right matters because there are clear cases where concentrated markets do not oscillate, they simply stagnate. There are after all examples of where oligopolies become completely stale. The perhaps most obvious example is the energy sector. The energy sector exhibits remarkable innovation stagnation, but the cause is more complex than competition law, or, say, public ownership. The R&D decline is dramatic: investments in energy R&D by U.S. companies fell by 50% between 1991 and 2003, and R&D spending by oil and gas companies dropped 15% in 2015 and continued declining in 2016. The comparison to pharmaceuticals is striking, in the early 1980s, energy companies were investing more in R&D than drug companies, but today drug companies invest 10 times as much, with total private sector energy R&D less than the R&D budgets of individual biotech companies such as Amgen and Genentech.

However, the ownership structure explanation does not hold: while it is true that only 7% of the world’s known oil reserves are in countries allowing private companies free rein, with 65% in state-owned hands, the privately-owned supermajors (ExxonMobil, Shell, TotalEnergies, BP, Chevron) controlling just 6% of global reserves show equally low R&D intensity as state-owned giants. Corporate R&D spending by oil and gas companies showed merely 1% growth in 2018, the first increase since 2014, but remained 45% below 2014 levels and was not rising significantly as a share of revenue.

The real problem is market structure and commodity dynamics rather than ownership. Unlike pharmaceuticals where each new drug creates a differentiated market with patent protection, oil and gas are commodities where innovation in extraction doesn’t create premium-priced products, you are still selling the same barrel of oil, removing the incentive for breakthrough R&D. The barriers to entry are physical (drilling rights, infrastructure, distribution networks) rather than knowledge-based, preventing the formation of a robust startup corona that can spin out innovations and later reintegrate through acquisition.

The pulsar model is largely absent in traditional energy, though it is emerging in renewables and clean tech: Shell made the most climate tech acquisitions globally between 2020 and 2024 with 8 companies, followed by BP with 7, and TotalEnergies led 2024 acquisitions by buying VSB Group, Talos Low Carbon Solutions, SN Power, and Kyon Energy. One founder of a climate tech startup acquired by Shell noted that when they set out building Limejump, they did not think they were building a renewable energy company to sell to an oil company, illustrating how acquisition-driven innovation ecosystems can form when products are differentiated and knowledge barriers matter more than physical ones. The energy sector oligopoly has become stale not because of government ownership, but because the commodity nature of the product, physical rather than knowledge barriers to entry, long asset lifecycles, and mature market dynamics all suppress the innovation pulsar that drives pharmaceutical advancement. This, too, of course is a risk for artificial intelligence if the product develops into a non-differentiated infrastructure heavy industry!

The Strategic Innovation Asymmetry

The energy sector example demonstrates what happens when innovation pulsars fail to form: a dramatic and sustained collapse in R&D intensity that leaves an entire industry technologically stagnant. This crystallizes the geopolitical risk. If Western AI policy creates an energy-like outcome, high M&A barriers preventing startup reintegration, while China maintains fluid acquisition channels for innovative startups, the asymmetry becomes structural.

Chinese AI companies could potentially acquire innovative European AI startups (facing blocked exits in their home markets) while simultaneously benefiting from easier domestic M&A, creating a “innovation arbitrage” dynamic where Western regulatory stringency becomes a competitive disadvantage.

However, the energy comparison also suggests a more fundamental question: does this competition even matter in the long run?

The energy sector’s commodity dynamics eliminated innovation incentives regardless of ownership structure or policy. If AI development follows a similar trajectory, becoming increasingly infrastructure-heavy, commoditized, and subject to physical rather than knowledge barriers to entry, then the innovation race may be self-limiting anyway. The geopolitical competition would shift from “who innovates fastest” to “who controls the most compute/energy infrastructure,” resembling energy geopolitics more than pharmaceutical innovation races. This suggests Western policymakers face a complex calibration problem: maintain enough M&A flexibility to sustain innovation pulsars while they matter (the pharmaceutical model), but recognize that if AI commoditizes (the energy model), the strategic competition transforms entirely.

The current policy mix, strict M&A enforcement plus strict FDI screening from the US in Europe, may be the worst of both worlds: it prevents Western startup ecosystems from forming effective pulsars without actually preventing Chinese firms from accessing cutting-edge technology through other channels.

Sector-Specific Implications

The pharmaceutical industry’s continued success despite increased regulatory scrutiny suggests that not all friction is fatal, some level of M&A oversight is compatible with maintaining innovation pulsars. This makes sense: as an industry matures, its regulatory environment becomes more predictable, and firms adapt their strategies accordingly. AI, however, moves at a fundamentally different tempo. Model capabilities, product cycles, and competitive dynamics evolve orders of magnitude faster than drug development, which means static legal frameworks will always lag behind technological reality. The problem, then, is not simply too much or too little regulation, but regulation that is too slow. What the AI ecosystem may require is not additional rules, but faster ones, a “fast-track” or rolling-approval framework that updates continuously, allowing compliance and innovation to evolve in parallel rather than in conflict.

But even a fast regulatory tempo cannot compensate for structural differences within AI itself. The energy sector’s innovation collapse shows there’s a breaking point where the pulsar mechanism fails entirely. For AI policy, this implies the need for sector-specific calibration rather than blanket approaches. Areas of AI that remain highly differentiated (specialized models, novel architectures, domain-specific applications) likely need M&A pathways similar to pharmaceuticals to maintain innovation velocity. But infrastructure-layer AI (foundation models requiring massive compute) may already be trending toward energy-like dynamics where consolidation is inevitable and innovation advantages prove temporary. Western policy might therefore need to distinguish between preserving innovation pulsars in differentiated AI applications while accepting that the infrastructure layer will necessarily concentrate, and compete on different terms (energy access, chip manufacturing, regulatory efficiency) rather than pure innovation velocity.

Taken together, it sees that AI will not follow a single structural path. The pulsar mechanism, where peripheral innovators emerge, mature, and are reintegrated into the core, still operates, but not uniformly across the stack. At the infrastructure layer, the forces of concentration are overwhelming: compute access, chip manufacturing, cloud dominance, and energy requirements naturally push the system toward an energy-like equilibrium where only a handful of firms can operate at scale. In this layer, the pulsar dynamic weakens because new entrants cannot realistically challenge or replace incumbent infrastructure, no matter how innovative they are. Consolidation is not merely likely; it is structurally baked in.

But at the application layer, the story is different. Even on top of a concentrated infrastructure, the ecosystem remains highly differentiated, fast-moving, and open to creative destruction. As seen in China, startups and small teams can still produce novel tools, domain-specific models, and specialized services using shared compute and existing foundation models. These innovations circulate through the system, creating localized competition and forcing incumbents to adapt. Here, the pulsar mechanism persists: new firms form, innovate, get acquired, and reintegrate into the ecosystem, refreshing it with each cycle. The infrastructure giants become platforms on which pulsars operate, rather than participants in the pulsars themselves.

What does this mean for AI overall? AI’s future direction will be a dual-regime system:

● Energy-like at the bottom: concentrated compute, slow turnover, infrastructural lock-in.

● Biopharma-like at the top: rapid innovation, niche development, acquisition-driven renewal.

The pulsar mechanism is not disappearing, it is becoming layer-specific. And this is exactly why selective flexibility is essential:

● Encourage pulsars where they still function (applications, specialised models).

● Accept consolidation where it is unavoidable (infrastructure), but regulate it like critical energy infrastructure. This would include [3]

● Build regulatory speed, not just regulatory volume, to ensure the upper-layer pulsars do not freeze.

● Open-source architectures[4] speed up the movement of innovations from the periphery to large technology platforms. Incumbent firms can integrate these technologies directly into their cloud, device, or application ecosystems without acquisition or licensing. As a result, traditional measures based on ownership transfers and mergers underestimate how quickly innovation spreads.

Thank you for reading!

Dyuti & Nicklas

^[1] Microsoft. (2025, November 18). Microsoft, NVIDIA and Anthropic announce strategic partnerships. Available at: https://blogs.microsoft.com/blog/2025/11/18/microsoft-nvidia-and-anthropic-announce-strategic-partnerships/

^[2] Gittins, C. Navigating the funding gap for AI startups. Startups Magazine. Available at: https://startupsmagazine.co.uk/article-navigating-funding-gap-ai-startups#:~:text=These%20numbers%20ostensibly%20indicate%20that,evenly%20distributed%2C%E2%80%9D%20he%20explained.

^[3] CGTN. (2026, March 7). Chinese tech giants move into next-generation AI agents deployment. CGTN.

https://news.cgtn.com/news/2026-03-07/Chinese-tech-giants-move-into-next-generation-AI-agents-deployment-1LjorO3o9kk/p.html ; Yeo, R., Tam, F., & Swaminathan, H. (2026, March 13). Tencent, Alibaba due as OpenClaw AI mania spreads in China. Bloomberg.

https://www.bloomberg.com/news/articles/2026-03-13/tencent-alibaba-due-as-openclaw-ai-mania-spreads-in-china

^[4] BioSpace. “Boom, Bust and Recover: What Happens Next as Biotech VC Cycle Resets.” June 9, 2025. https://www.biospace.com/business/boom-bust-and-recover-what-happens-next-as-biotech-vc-cycle-resets

^[5] BioCentury. “Navigating biotech investment cycles — the true north: Guest Commentary.” January 13, 2025. https://www.biocentury.com/article/654643/navigating-biotech-investment-cycles-the-true-north-guest-commentary

^[6] ScienceDirect. “Biotech acquisitions by big pharma: why and what is next.” Trends in Pharmacological Sciences, July 5, 2009. https://www.sciencedirect.com/science/article/abs/pii/S1359644609002219

^[7] IQVIA. “Biopharma M&A: Outlook for 2025.” January 2025. https://www.iqvia.com/locations/emea/blogs/2025/01/biopharma-m-and-a-outlook-for-2025

^[8] Drug Discovery Trends. “Biotech funding holds steady as investors eye late-stage assets.” August 18, 2024. https://www.drugdiscoverytrends.com/biotech-bounce-back-possible-but-when/

^[9] IQVIA. “Biopharma M&A: Outlook for 2025.” January 2025. https://www.iqvia.com/locations/emea/blogs/2025/01/biopharma-m-and-a-outlook-for-2025

^[10] Drug Discovery Trends. “Biotech funding holds steady as investors eye late-stage assets.” August 18, 2024. https://www.drugdiscoverytrends.com/biotech-bounce-back-possible-but-when/

^[11] IntuitionLabs. “US Biotech Job Market: 2025 Trends, Data & Analysis.” November 2025. https://intuitionlabs.ai/articles/us-biotech-job-market-2025

^[12] V-Bio. “From lab to market: the life cycle of a biotech startup.” March 7, 2025. https://www.v-bio.ventures/from-lab-to-market-the-life-cycle-of-a-biotech-startup/

^[13] BioCentury. “Navigating biotech investment cycles — the true north: Guest Commentary.” January 13, 2025. https://www.biocentury.com/article/654643/navigating-biotech-investment-cycles-the-true-north-guest-commentary

^[14] IQVIA. “Biopharma M&A: Outlook for 2025.” January 2025. https://www.iqvia.com/locations/emea/blogs/2025/01/biopharma-m-and-a-outlook-for-2025

^[15] IQVIA. “Trends in biopharma M&A: seeking value.” October 2024. https://www.iqvia.com/locations/emea/blogs/2024/10/trends-in-biopharma-m-and-a-seeking-value

^[16] Corrigan, J, Luong, N. and Schoeberl, C. (2024). Acquiring AI Companies: Tracking U.S. AI Mergers and Acquisitions. CSET. Available at: https://cset.georgetown.edu/publication/acquiring-ai-companies-tracking-u-s-ai-mergers-and-acquisitions/

The concept of intelligent agents is subtly plagued by images of homunculi running our errands for us. These little agents can think, act and plan. They reason and cogitate and do work for us - but they are stuck in metaphorical scaffolding that still assumes that thinking and intelligence are key elements for solving problems. This is, of course, not true.

A lot of problems are solved a-cognitively. Evolution is the obvious example: it does not care about theories, it does not plan and it does not reason. Yet, it has designed beings so complex as to think that their own thinking and intelligence is the key to explore the world. Our overvaluation of intelligence is a second cousin of creationism, overvaluing what we are at the cost of building new capabilities that can help us address complex problems.

We should abandon the idea of agents and instead think about how we can design organisms with basic needs, wants and agency for specific problem environments and set the conditions for them to evolve. Let’s build organisms that eat plastic and inflation, or organisms that feel interest pressure like heat and evolve trading strategies and public interventions to reduce it. Let’s focus on a-theoretical approaches that allow for a maximum of signals to build a selection pressure on these organisms and see what we can do. Want to develop more efficient weapons? Make organisms that eat system stability or build public unrest to live and thrive in, feeding on negative public sentiment and chaos.

Such an approach would require, of course, that we reconceive our the way we work with artificial intelligence. We need only small degrees of intelligence, more agency, really, and we should try to figure out if there is a basic set of organismic intelligence that we can develop to then set these new creations of in well-defined problem environments. We need to design artificial metabolisms alongside the basic agency we provide the organisms with, and we need to be ready for massive, unexpected second order impacts from our early attempts.

This thinking - that artificial intelligence is the narcissistic remix of artificial life, and that we should stop trying to find ourselves in the mirror of the machine - is not new. The AI-life folks have been around for ages, and their research is puttering along, undisturbed by the massive upheaval created by chatbots and large language models. Their aims might be described more as simulation than creation, however. Building artificial life, artificial organisms, that model how real organisms work is important and valuable - but building entirely new organisms that mimic the processes developed by evolution in domains and environments that are purely cognitive is more rare in the field.

Yet, it is likely to be a really interesting development. In one sense it is also following a well-established trajectory in the history of artificial intelligence: when the field sheds the images of human capabilities and intelligence that holds it hostage, it progresses. The image of humans as chess players and logicians held back AI for decades, before we accepted that we may well be more stochastic than that, and that our self-understanding is severely lacking as a guide for how to replicate the capabilities that we exhibit. Abandoning the agent perspective is just the next natural step in that slow abandonment of anthropomorhic chains and limitations.

Such an approach would also allow us to ask some really interesting questions about different complex systems and how to design organisms that could thrive and live in them. How do you describe the economy as an evolutionary environment, and what kinds of metabolism would work best? Here, of course, we should refrain from thinking that our thinking will lead us right - and we should rather build systems that generate and evolve different candidate metabolisms for different problems.

We meet, here, an iteration of the observation that Marc Andreesen famously made in his 2011 article ”Why software is eating the world”, but we follow up with an impatient question ”Yes, of course, but which software eats what part of the world, and how do we design its metabolism?”.

There is an issue here, too, about the level of intervention. It is tempting to think that we should be designing organisms for all these cognitive habitats, but it may well be that while this could be instructive and interesting as a research project, we really need to figure out how to design evolutions. We can set the spark - initiate life in a cognitive habitat - but then we need to step back and let evolution do its work in silicon speed, and see what evolves. What are the most efficient inflation-predators? The most successful evolutionary designs for fighting pandemics? Which evolutionary designs will succeed fastest at developing a new physics that unifies gravity and quantum? Building evolutions that lead to ecosystems that eat physics and mathematics problems seem fanciful, but we have at least an existence proof: we do this. Our curiosity is a basic need, and our need for cognition, play and learning is real - implemented in us by evolutionary selection pressures over time. Why could that not be replicated in even more targeted versions?

We have early examples of approaches that could be ancestors of such problem eaters: the models that play themselves for better matrix multiplication are engaged in a kind of competition that could easily be made into a fitness competition - and it seems these game-models are quite effective. What if we turned them into full-blown evolutionary competition for scarce resources, where the better matrix multiplication confers a fitness advantage?

In this new world intelligence is less interesting than a-theoretical cognition. Systems that solve problems without thinking in the human sense, but where the cognition is external to the individual and a result of indidvidual actions and individual agency, rather than individual reasoning. And maybe we will find that overvaluing intelligence was just another version of the constant anxiety we are plagued by as we compare ourselves to the machine. The search for something the machine cannot do is ending, but we still adamantly demand that the machine do what it does in the way we do it - because surely there can be no more effective way of doing what we do than the ways we do it?

Human metaphors, analogies, mental models and distorted mirror images still litter the entire field of artificial intelligence, and even when we think about AI-safety we think about instilling our values and morals in the systems in different ways, perhaps teaching them as children or raising them collectively as we slowly convince them to have our best interest at heart and to take our needs into account. Maybe this is just a vain attempt at forcing what we are building to remain at least metaphorically within a specific human logic of values and alignment.

When we look back at this technological era we may well be calling it the age of the mirror fallacy: where we thought we needed to design the technology in our image, in order to preserve our self-image. Our fear of the black box is perhaps just a prescient fear of a-cognitive systems that evolve in problem domains and solve them throgh adapting to solutions and equilibria that benefit us, but which we can never truly or deeply understand.

In a way, however, this does solve, or at least relocate, the alignment problem. Rather than constantly building smarter systems, we build systems that are better adapted to the problems we need them to eat. What we have to deal with then is a cambrian explosion of digital organisms and hybrid ecological competition, not the scheming high lords of intelligence. (And, well, let’s be honest - we seem to be pretty good at orchestrating the extinction of those organisms that compete with us within the evolutionary niche we occupy).

And if you devalue and dismiss all of the above: is it not at least worth considering that maybe “intelligence” is a word that holds us hostage in an image that may be restricting our options?

NBL Stockholm 2026-03-13

Welcome to the first newsletter of the year. I took January to think and read, and really enjoyed coming back to the writing of the newsletter. I will now aim to publish regularly, but with a gentler cadence than once a week - I am working on a few other things as well. On February 26th, David Skelton and I are publishing our book The Future Habit (do pre-order, if you are interested, it really helps us) and I am thinking about new writing projects this year. My ambition is to write about things that I think are undervalued, and try to find subjects that can be both worth debating (so comment please!) and revisiting in other formats. We start this week with a note on the future of language, and how we can prepare for becoming guests in the house of Being…

Participating in language

“Therefore this statement holds true: Language is the house of Being.” - Martin Heidegger

“The limits of my language mean the limits of my world.” - Ludwig Wittgenstein

Language has a good claim on being what makes us human and it holds, rightly, almost endless fascination for us as a phenomenon. We study it technically in linguistics, we explore it in poetry and we use it in philosophy as a means to find our way. We live in language - and it is, as Heidegger notes - the house of Being; it is in language we experience the world, think, feel and learn. Up until this point in time we have also been, relatively alone in language. That may be changing as large language models produce increasing amounts of language on their own, some for us and some for each-other.

One way to think about language is to say that it is the sum total of all recorded words - whether in writing, spoken or otherwise fixed in some way. This total corpus can then be examined to understand how language changes over time. We can study the emergence of sociolects and fashions, the demise of individual languages as well as the birth of new ones, dialects and shifts in emotive values, meaning and depth. By applying different metrics and measures we can try to chart the evolution of this corpus over time, to perhaps get a sense of what the future of language looks like.

One such metric could be the amount of recorded language that is produced by human beings. Up until the 20th century that was a simple metric - all of language was produced by us, but as we started to explore new technologies we also opened up the possibility for machines to produce language as well, or so it seems. Here we need to be careful - what do we mean when we try to make a distinction between language produced by us and language produced in other ways?

Surely, words printed by a typewriter are still produced by us, even if the machine is an intermediary? And what are LLMs but extraordinarily advanced typewriters where the input and output are not directly related in the way pressing a key on a keyboard is, but still bound by necessity - albeit a statistical one? Each prompt is but a pressed key, each response a statistically produced response equal to that of the ink on the sheet of paper in the typewriter? All language is still produced by us, surely? What has changed is merely the means of production and the complexity of that production? What is the real difference between AI and autocorrect, after all?

I am not sure. I would like to challenge that position, by suggesting that at some point the complexity of the production means that we encounter a difference in kind here: language is no longer produced by us, but by this new Other. It truly, in a fundamental sense, speaks and participates in language.

One reason I think this is a defensible position is that we can already see language change.

A growing body of research is now documenting how LLMs are reshaping human language, both in writing and in speech. Tom S. Juzek and Zina B. Ward analyzed scientific abstracts from 2020 and 2024, identifying 21 "focal words" (including "delve," "intricate," and "underscore") whose sharp increase in frequency is attributable to LLM usage rather than natural linguistic drift, with evidence suggesting that reinforcement learning from human feedback (RLHF) may be driving this lexical overrepresentation. Mingmeng Geng and Roberto Trotta examined over 30,000 papers and 1,000 oral presentations from machine learning conferences, finding that LLM-characteristic words like "significant" have increased in both written abstracts and spoken talks—the first study to track this spillover within the same population. More striking still, Hiromu Yakura and colleagues transcribed and analyzed 280,000 YouTube videos from over 20,000 academic institution channels, documenting a statistically significant shift toward ChatGPT-distinctive vocabulary in human spoken language following the model's release—the first empirical evidence that people are unconsciously imitating LLM patterns even when speaking aloud. Zhivar Sourati, Alireza S. Ziabari, and Morteza Dehghani synthesize these and related findings to argue for a broader "homogenizing effect": as everyone increasingly relies on the same models across contexts, LLMs create a feedback loop that standardizes not just vocabulary but reasoning strategies.1 A homogenization that starts in language.2

Being a speaker, we could argue, authentically producing and participating in language, is being able to change language, and if models are capable of this, then they have become something more than mere statistical parrots.3

We could also imagine other criteria to help decide if there is real participation in language. One such criteria could be engagement and empathy. When we enter into dialogue with someone, they engage us, wrap us into a shared production of meaning, often with emotional as well as intellectual reactions to the dialogue. This, of course, is a much harder criterion to get right - but you could imagine a simple Turing test-like set up where you have someone enter into dialogue with a human and a machine without knowing which is which and then ranking the empathy experienced in the exchange.

There are some studies that come close to this, often in healthcare settings. A study published in npj Digital Medicine examined how cancer patients, rather than clinician evaluators, perceive empathy in AI chatbot versus physician responses. Researchers surveyed 45 oncology patients at a tertiary cancer centre, presenting them with responses to 100 cancer-related questions originally posted on Reddit's r/AskDocs. Each participant rated the empathy of blinded responses from physicians and three versions of Claude (V1, V2, and V2 with chain-of-thought prompting) on a 5-point scale. All chatbot versions significantly outperformed physicians: Claude V2 with chain-of-thought prompting scored highest (mean 4.11, 95% CI 3.99–4.22), followed by Claude V2 (mean 3.72, 95% CI 3.62–3.81) and Claude V1 (mean 3.35, 95% CI 3.23–3.48), while physician responses averaged just 2.01 (95% CI 1.88–2.13; all comparisons p<.001).

The study was terminated early after interim analysis revealed significant effect sizes.

We probably need to think more about what criteria we would like to apply to assess real participation in language, but these two tests at least provide us with a starting point, and, I would argue, both indicate that LLMs are participating in language.

That said, I can imagine a robust challenge to this position that tries to distinguish the production of language from the production of meaning. There may be more language, we could say, but it is only the human mind that can produce meaning - and meaning is what really determines participation in language. A machine cannot produce meaning, and the ability to produce signs is useless without the ability to perform semiosis. My tentative reply to this would be pragmatist: meaning is use, and machines use language and we use the language produced by the machine. But I think there is a deeper debate to be had here, and I welcome it.

The corpus

Let’s assume for the sake of argument that LLMs are indeed participating in language, and that we can have a look at that metric we proposed: how much of language we produce, and how much is produced by the machine.4 Where are we then? Are we close to the point where a majority of language is produced not by us, but by the machine? At what point do we become a minority in language?

Let’s try a quick and dirty Fermi estimate. In February 2024, Sam Altman posted the following claim: OpenAI was generating 100 billion words per day, while all humans on Earth produce roughly 100 trillion. That put AI at 0.1% of human output—a rounding error. But how would we actually verify either number, and what’s happened since? Start with humans. Take the 8.2 billion people on Earth, assume 90% are of speaking age and ability, and multiply by the ~12,600 words per day that recent research suggests we speak on average (the famous 16,000 figure from 2007 has been revised downward as texting substitutes for talking). That gives us about 93 trillion words from speech alone. Add the 200 billion legitimate emails sent daily at roughly 75 words each (16 trillion), plus messaging apps, social media, and professional writing, and you land at approximately 110 trillion words per day. Altman’s estimate seems to hold up at least when it comes to order of magnitude.

Now for AI—and here’s where it gets treacherous. The numbers being reported are enormous: Google processes 1.3 quadrillion tokens monthly, OpenAI’s API handles 8.6 trillion tokens daily, Microsoft Azure processed 100 trillion tokens last quarter. But these figures measure throughput, not output. When an AI summarises a document, it might read 10,000 tokens to produce 100. When it answers a question using retrieval-augmented generation, the ratio can be 50:1 or higher.

Across typical use cases—chatbots, code generation, document analysis, agentic loops—a weighted estimate suggests roughly 15% of processed tokens are actually generated output. Apply that correction to the ~56 trillion tokens processed daily across major providers, convert to words (about 0.75 words per token), and AI’s generative output falls to approximately 6 trillion words per day. That’s not 0.1% of human output anymore—it’s about 5.5%. AI has grown perhaps 60-fold in under two years, but it’s still producing roughly one-eighteenth of what humans produce.

The crossover calculation is then rather straightforward, if very sensitive to assumptions. If AI output continues growing at 200% annually (conservative relative to the 2024-2025 trajectory, but acknowledging that growth slows as bases expand) while human output remains roughly flat, the curves intersect around mid-2028. Slow AI growth to 100% annually and you push crossover to 2030; accelerate it to 300% and you pull it forward to 2027. But the real wildcard is agentic AI—systems that communicate with other systems in loops no human ever reads. When autonomous agents begin debugging code, negotiating API calls, and coordinating workflows, they’ll generate vast quantities of “dark matter” language that won’t show up in consumer-facing form, but still will silently become a part of our corpus.5

This is just a Fermi estimate, a structured guess, of course. You should do your own, and share - I suspect that it can look very different, but not that different in terms of the horizon, and we face becoming a minority in language in about 5 years, at most.

Minority strategies

It is 2030, and biological production of language has been dwarfed by computational production. Humanity has become a guest in language and our words, concepts and the limits of our world are now set by computational speakers. What do we do? Do we let this happen? Do we even try to stop it?

Resistance is an option, of course. We could stop using language models, or demand that everything they write expires so that they do not make a constant mark in our corpus. LLM-produced language could be regulated and we could force ephemerality on AI: no text produced by AI gets to last longer than two years. But given how many people today write, partly or wholly, with these models we would need very good provenance technologies to do this, and we would risk losing quite some human output as well in such a scheme.

Another form of resistance could take the form of labelling and marking up human content as human, and ensure that it is not contaminated by LLM-language.6 Designating certain areas - theater, poetry, cinema, games - as human language domains and safe-guarding them would allow for keeping some areas of language under human dominance. It seems likely that such areas would quickly be encroached upon, and how do you ensure that the poets does not get infected by computational language? Do you isolate them all on an island with no computers and no access to our shared language? What then happens with poetry as a shared experience? The very function poetry has means that it needs to subvert ordinary language, and so understand and command it. Better then to pay the poets handsomely for guerilla warfare in language, and to read poetry as a means to inoculate ourselves against machine discourse.

In this world human language becomes artisanal, but still dependent on and connected to a language in which we are now guests. The idea that you could compartementalize linguistic spaces and protect them is optimistic at best.

There is also acceptance. We used to be the only ones that carried language on our shoulders, now we can shared the burden. What we need to do is not resist the machine, but teach it. If it is to participate in language, let’s make it a better speaker than we are, one more sensitive to truth, adverse to lies and with a deep set respect and alertness for the tendency we have to craft self-deception through our choice of words.

We need to be careful here, and not succumb to the dream of a better, purer language curated by a wiser machine - but there is something here, the possibility of expanding the limits of our world through judicious exploration of the linguistic adjacent possible. So much of our world, our institutions, breaks down when language becomes brittle and our divisions enshrined in the way we use words. Our need for a shibboleth to distinguish us from them is pervasive and destructive. A new language, carefully curated and evolved might also mean a new world.

And then again we might just never notice, just like we today live in language and rarely pay attention to its wonder.

Thanks for reading,

Nicklas

I am off for a holiday break, and so thought it good to just sum up the year that was, think a bit about next year - but most importantly: say thank you for all of the comments, feedback and the added readership — we are twice as many at the end of the year as we were at the beginning - and that feels fantastic!

As I took up writing this year again, after a two year hiatus, I really enjoyed it. There are so many interesting themes and ideas out there, and I learn by writing so the newsletter has given me the opportunity to dig into a number of different issues and questions. Among my favorites from this year are:

• #105 on sensors, senses and what happens with intelligence when it is built from different and larger sets of senses than ours.

• #111 on the way computational and biological time differs and why AI can become a kind of temporal mediation mechanisms. I was extremely grateful to see this one edited masterfully, and re-published by AI policy perspectives (APP is a must read).

• #126 on the NatSec-shift — something that I think clearly is happening now.

• #136 on culture and AI - and whether AIs can have culture.

But what was even more fun was to write with others — I had the luck and pleasure to write with quite a few people this year, and hope to expand this even further next year.

• #123 with Yannis Mastrogeorgiou, on the ability of machines to wonder was really interesting to collaborate on.

• #128 and #144 with Fredrik Erixon, on what a healthy public sphere really looks like focused less on the threats to the public sphere, but the neglected question of what it looks like when it really works — and why that question is key to figuring out regulatory responses.

• #134 with Anastasia Bektimirova on organzing science in the age of AI helped me understand more about why organizing science may be as much of a challenge as getting to the right tools for science.

Writing together and collaborating on texts is fantastic - I hope to do a lot more of that.

But there is naturally room for improvement. I feel that some of the texts didn’t quite manage to bring across my point, and I need to revisit and come back to those. #117 on agency-enhancing technologies did not quite get it right — and I want to revisit that idea later. #141 on why even scenarios where AGI is benign, safety is solved and we all “flourish” may lead to human exit also scratched the surface on something that I want to capture better.

So for next year I want to figure out periodicity — is weekly still meaningful or would monthly be better with deeper texts, more research? Choosing a monthly theme appeals to me and could be a way forward, but I also want to figure out scope: after I restarted I have almost solely focused on AI from different aspects, I feel that this may sometimes be too limiting and at the same time too broad. Finally, I want to find more formats for writing with others — if you are interested let me know!! And also - if you have ideas about the above please let me know.

We will resume beginning of January, and then I will have a new plan for the new year — but for now, again, thank you for reading, liking, commenting and being generally curious learners!

Merry Christmas and a Happy New Year!

Nicklas

The holidays are coming up on us soon, and hopefully we get some time to read, catch up on learning and just hang out with friends and family. As you do so here are a few tips and tricks on how to observe and consume information better with AI, and what not to do. Enjoy!

Observation as a skill

There is a lot of information around these days, and keeping up with it all is never easy - yet, there are interesting tools and techniques out there that you can play around with - and use to strengthen your observational skills. Now, we often think that reading is not the same as observing, but that is a simple category mistake: reading is mediated observation, it is an observation of observations others have made - but it is still an observational skill — and as all observational skills it can be trained, improved and practiced. So how do you become a better observer of the domains and fields you are interested in?

Observation is paying attention in specific ways, methodically. If you want to learn to observe art, for example, there are specific methods you can learn - as outlined in this excellent article suggesting that you look in rounds - first at lines, then at colors, then at themes…and so on (see the very first of the unpredictable patterns on other methods of observation here) - and structuring your observation allows you to really explore what you observe in depth.

For knowledge workers the skill we need to develop is to observe massive amounts of information feeds and flows in ways that help us do that. Not unexpectedly, artificial intelligence allows us to do build entirely new skills and ways of structuring our observation. Let’s have a look at a few skills you should add to your repertoire and what is required to do so, we will go in steps and start easy and then move into more and more helpful, advanced techniques to help you build your AI-powered observation skills.

Step 1: Get an RSS-reader

Surprisingly many people still consume information through websites - and from the specific websites that publish it. I know experienced knowledge workers who still have a “sweep” set up where they start with one news paper, and then look at 6 or 7 different newspapers and then feel that they have gotten a good sense of what is going on. This mirrors the old off-line practice of buying 4-5 newspapers, magazines and reading them - carefully curating them so you get opposing perspectives like The New York Times and Wall Street Journal.

This is still a somewhat valuable habit - because so many people do it! If you want to understand what most senior knowledge workers know, then doing this establishes a kind of baseline that is helpful. Any knowledge worker benefits from knowing what their leadership reads - since that is what frames the way the organization observes the world. For tech policy folks knowing to what degree their leadership gets their sense of prioritization and importance from Techmeme is essential, for example. When I lead the EMEA policy teams at Google I regularly would check Techmeme in the mornings, knowing that the folks in Mountain View would get their news from there, and if there was anything about European tech policy in there I made damn sure that a note about it was on the desks of the Mountain View stakeholders when they woke up: this was a major swirl minimizer.

But this just establishes a baseline. If you really want to build a good observation habit you need tools. As Bo Dahlbom noted: you cannot do a whole lot of carpentry with your bare hands, and you cannot do a whole lot of thinking with your bare brain.

One of the oldest tools for observing more effectively is the RSS-reader. If you don’t know exactly what that is, that is fine: the way to think about it is that most information sources you could be interested in have machine readable feeds that can be collected, curated and read in a reader app like Inoreader or Feedly.

What this allows you to do is to amass 100s or 1000s of sources in different folders, and use each folder as an information feed. This, in turn, allows for a different kind of reading - scanning - and modern RSS-readers also allow to monitor for themes and key words that you are interested in. Here is what it can look like:

A common reaction to this is that it is overwhelming - but that is because you are reading it wrong. Scanning allows you to look at the flow of infomation and just get an overall sense of what is going on, You then deep dive into different articles or items in the feed, and this is where you can start to quickly comment on and tag the articles so that they compound into intelligence collections.

Spending 10 hours learning an RSS-reader really well and deeply immediately puts you in the 1 percent of knowledge workers who have some sense of the overall landscape, not just individual peaks.

Step 2: visualize RSS-feeds with AI

Once you start to look at RSS-feeds you can use them to build new tools of observation. Say you are tracking the latest in AI-research from Arxiv for example, and you want to have a sense of what the different trends and areas for research are - then you can just throw an RSS-feed into your favorite vibe coding tool and ask for a trend radar. The RSS-feed to Arxiv’s repository of new papers on AI is https://rss.arxiv.org/rss/cs.AI - take that and use, say, Google AI-studio to build a visualization app for the feed, and you get something like this.

Now, I am writing this on a weekend and did not pull the latest 100 items in the feed so it looks kind of scarce, but you get the idea. And you can of course throw in 3, 30 or 300 feeds (but the processing power in simple vibe coding tools becomes a limit fast).

Here is the super simple prompt I used to do this: “Write a beautiful app that visualizes the different clusters and ideas in this feed of research papers - make each cluster clickable and expandable, and also show me outliers: https://rss.arxiv.org/rss/cs.AI”

We are now turning feeds into visualizations - to be better able to observe and understand the information we are working with.

Step 3: Summarizing individual reports for learning

At this point everyone knows that AI is great at summarizing long reports. Unfortunately this does not mean that we automatically became great at summarizing - and asking the right questions of a report. Summarizing for learning is subtly different than just summarizing here - and we are all about the learning.

When you summarize for learning you want to understand the material from different angles - so you start with asking for a summary, but then you also ask for what the core claims are, and then the evidence for and against these claims and then you should look at what some plausible second and third order effects are if the claims are true.

Structuring prompts this way allows you to learn much more than if you just ask for a summary. Summarization is - you guessed it - a skill.

One way to do this well is to have the model role play summarizer and critic who suggests what is missed and why it matters - this super simple move allows you to see the tension between two possible readings, and that is an order of magnitude more helpful than just asking for a summary.

What you want is to ensure that each summary, each transformation of the information forces you to exercise judgment. There is a lot of talk about cognitive offloading, where the danger is described as allowing the machine to think for you, but I think we might want to be more worried about offloading our ability to exercise judgment.

Step 4: Visualizing individual documents

Just as you can visualize RSS-feeds, you can visualize documents in different ways. Let’s take an example: you want to understand the EUs Digital Omnibus and what it does - so you drop the report in your vibe coding tools and you ask for an “explorer”. The result looks something like this:

Each item is clickable and expands into a box that gives you in-depth understanding AND an opportunity to discuss that very thing with the AI:

And, of course, you want simulated reception, criticism and debate so you add that too:

This does not absolve you from reading, but it structures and helps you think about the reading in a different way. And when presenting this beats a Powerpoint by a mile: using the explorer to walk through the legislation allows you to give them a tool to understand it quickly if they want to or just see it (we are visual intelligences after all).

So now we have move from reading a few newspapers each morning to a more robust set of observation skills. We can take it even further and combine some of these methods.

Step 5: RSS with AI

With the advent of AI-browsers RSS-readers quietly got 100x better. As you now read your feeds, you can just fire up a chatbot and ask it anything about the feed you are working with. Here is ChatGPT Atlas analyzing the core themes of Russian propaganda from a feed with Russian news:

And the analysis also give us this:

And I can continue the conversation, dig deeper and think through different ways of working through this. Now the RSS-folders are collections of signals that you can mine for insights.

Step 6: Putting it together in learning projects with deep research etc

What we have been looking at so far is mostly general business and policy intelligence work — where we orient ourselves through observation. As soon as we start to go deeper, we want to focus our efforts and ensure that we keep notes and ideas in a single place. This where tools like NotebookLM really shine. When I want to learn something I usually do 2-3 Deep Research reports from different agents, and add one report from a more focused academic research service like Elicit.com, and then put these and any background information I already have into NotebookLM. Here the podcasts, mindmaps, briefings and infographics all provide ways of observing, learning and exploring the subject - and this is where I enter observations from the more general work with RSS-readers and apps.

Step 7: Writing it up

One of the things that we underestimate is writing as a means to integrate observations into learnings. All of the steps here are about consuming information in various ways, and sometimes I find people think that you can settle for consuming, but I think that is wrong. A key step is to writing up what you have learned and reflecting on it. It can be a learning journal or a captain’s log, or a blog or something else — but if you do not continuously write you will find that insights fade fast.

There are several methods here - one is explaining the insights you have, and here, again, your favorite bot can be helpful. Set up a conversation in which it asks you about the subject and you are tasked with explaining it as clearly as possible, and the bot asks more and more in-depth questions. This is a great way to explain and explore a subject in a variation on the Feynman method. Another is summing up what you know, and then formulating key questions that you now thing you need to know next. A summary in questions or dialogue generates a dynamic that helps move a knowledge project forward. A third method is to try to draw a model of the thing you are learning - what does this look like? Why? Where does it break?

Injecting a bit of daily writing to sum up the observations is great practice - and the better we get at observing, the better we get at learning. Remember DaVinci’s notebooks - full of reminders to himself to seek out this or that person to learn something from them. These notebooks are maps of future exploration as well as documentation. We should not knock documentation either - look at the notebook below:

This is Darwin, meticulously documenting his voyages. If he had one super power, I would argue, it was his powers of observation. And the thing with artificial intelligence is that when used well it is an enormously powerful instrument of observation.

But as all instruments - whether it is a telescope or a musical instrument - it requires practice! And doing so is actually great fun! Share your tricks and tips in the comments!

Thanks for reading,

Nicklas

This week’s note is a discussion about how we deal with, price and allocate risk - and why insurance as we know it may not survive the predictive turn in society. Enjoy!

Allocating risk

All technology comes with risks of different kinds, and an important component part of the diffusion of technology in society is the emergence of an allocation of that risk over different parties. Often when we think about this we think about it as a question of liability - but that is just the first step in the allocation of risk; when we determine liability we set the default allocation of risk - but risk can then be re-allocated in different ways and through different mechanisms.

One simple mechanism is contract - we can agree that you take on a certain risk that I may have been allocated, or we can use contract to clarify which uses of the technology we recognize are so beyond the original allocation of risk that they create entirely new risks. Now, the legislator may, of course, explicitly stop me from doing that and assign me a non-transferrable liability for the technology - but even so there are often new, aggregate risks that have to be dealt with.

Insurance is another mechanism that can deal with risk in different ways. When I insure against risk I accept that I will be liable, but I buy an insurance that will allow me to recoup any loss my liability causes me to incur.

Now, insurance is fascinating.

Insurance, as a means of re-allocation of risk, traces its origins to ancient civilizations where merchants sought protection against the risks of trade. Babylonian traders around 1750 BCE developed early risk-sharing arrangements codified in the Code of Hammurabi, which allowed merchants to pay lenders an additional sum to cancel loans if shipments were lost. Similar practices emerged in ancient China, where traders distributed goods across multiple vessels to minimize catastrophic loss. The Greeks and Romans formalized burial clubs that collected contributions to cover members’ funeral expenses—an early form of life insurance. These arrangements shared the fundamental insurance principle: spreading risk across a group so that no single party bears the full burden of misfortune.

Insurance works if risks can be pulverized.

Modern insurance emerged in the commercial centers of medieval and Renaissance Europe. Maritime insurance developed in the Italian city-states during the 14th century, with Genoa producing the oldest known insurance contract in 1347. London’s Lloyd’s Coffee House became the epicenter of marine insurance in the late 17th century, where merchants, shipowners, and underwriters gathered to negotiate coverage for voyages. The Great Fire of London in 1666, which destroyed over 13,000 houses, demonstrated the need for fire insurance and led to the establishment of the first fire insurance companies. Life insurance also took more structured form during this period, with actuarial science emerging in the 18th century when mathematicians like Edmond Halley developed mortality tables that allowed for more precise pricing of risk.

The 19th and 20th centuries saw insurance expand dramatically in scope and sophistication. Industrialization created new categories of risk—workplace accidents, product liability, automobile collisions—that spawned corresponding insurance products. Governments entered the field with social insurance programs: Germany pioneered compulsory health and accident insurance in the 1880s under Bismarck, and most developed nations eventually adopted some form of public insurance for unemployment, disability, and old age. Today insurance is a multi-trillion dollar global industry encompassing everything from crop failures to cyber attacks, underpinned by complex financial instruments and reinsurance markets that distribute risk across the world economy.

AI and insurance

Lloyd’s of London and Munich Re have begun offering targeted, performance-linked coverage specifically for AI systems, while at the same time major carriers like AIG, Great American, and WR Berkley are racing to exclude AI from their standard policies entirely. This bifurcation—specialty insurers entering the space with narrow, data-driven products while generalist carriers flee—tells us something important about how the market is reading AI’s risk profile. The insurance industry is treating AI as a candidate systemic risk, comparable to cyber or certain exotic financial instruments.

The products themselves reveal a maturing understanding of what AI risk actually looks like. In May 2025, Lloyd’s market insurers began selling a product designed by Armilla, a YC-backed startup, to cover legal fees and damages when AI chatbots malfunction or hallucinate. What’s notable is the trigger logic: coverage activates not when a single bad answer causes harm, but when measurable performance degradation occurs—accuracy dropping from 95% to 85% over time, for instance, against a benchmark Armilla establishes upfront. Munich Re’s aiSure product takes a similar approach for AI providers, insuring them against losses if their models fail to meet contractually guaranteed KPIs around prediction error, data drift, or behavioural deviation. In both cases, the insurer is essentially backstopping model performance, which means they have a direct financial interest in how well the AI actually works.

The governance implications here are substantial. For the first time, model-risk practices—testing, documentation, monitoring, audit trails—don’t just impress regulators or satisfy compliance checklists. They move insurance premiums, giving boards a quantifiable financial incentive for AI safety and governance. A widely cited techUK analysis this year makes the connection explicit: firms with strong model-risk management, following frameworks like SR 11-7 or the PRA’s SS1/23, can expect to earn lower premiums. This couples AI assurance directly to AI insurance, creating an external price signal for governance maturity. Meanwhile, the carriers filing exclusions are implicitly saying something different: we don’t trust your model-risk management enough to keep AI inside the base policy.

What emerges could well be a new infrastructure layer for AI safety, analogous to what cyber insurance became in the 2000s. Brokers, reinsurers, and specialty carriers now sit between AI providers and end-users, making judgments about what counts as acceptable AI deployment. The current product landscape includes narrow chatbot liability covers, performance warranty insurance for specific use-cases, and—still conceptual but clearly foreshadowed—risk-scored general liability policies where premiums reflect governance maturity scores. The market is fragmenting along predictable lines: companies with documented, well-tested, explainable models will find coverage; those with opaque systems and minimal oversight will face exclusions or prohibitive pricing. Overall, the insurance models - if they work - will accelerate diffusion of the technology.

Project this forward five years and the following scenario becomes interesting. Through 2025 and 2026, the niche covers from Lloyd’s and Munich Re grow modestly, adopted primarily by digital-first companies that have already experienced AI incidents firsthand. Big carriers continue pushing AI risk out via exclusions, and corporate risk officers begin to recognise that their “AI gap” is as real as the cyber gap they scrambled to close a decade ago. By 2027 or 2028, a small ecosystem of AI risk-rating agencies and audit firms will likely emerge, offering standardised scores based on model documentation, testing, red-teaming, and monitoring—essentially operationalising what regulators and industry bodies have been hinting at. At least one major jurisdiction, probably the EU, UK, or Singapore, will start referencing insurance and independent testing in AI supervision guidance, making insurability a proxy indicator of acceptable risk or a presumption of strict liability if it is missing.

By 2030, for large enterprises in regulated sectors—banks, insurers, hospitals, critical infrastructure operators—AI liability and performance insurance becomes something close to a requirement for high-impact deployments, whether through contractual obligation or supervisory expectation. Premium differentials widen dramatically: an organisation with mature AI governance and explainable, well-tested models pays baseline rates, while one with opaque, minimally tested systems pays three to five times more or finds itself uninsurable at any meaningful limit. Corporate boards start receiving AI risk and insurability dashboards alongside their cyber and climate reports, and failing to maintain insurability for critical AI systems becomes a governance red flag.

The politics of AI risk will, in this scenario, increasingly run through insurance channels. Civil society and regulators will push for transparency on who is insuring large frontier models and on what terms. Systemic AI incidents—a catastrophic medical misdiagnosis cascade, a large-scale trading error—will generate high-profile coverage disputes that could shape case law. By the end of the decade, “model-risk score” could be a standard metric on risk committee agendas, much like Value at Risk in finance or security ratings in cyber. Insurance pricing will have quietly become one of the strongest external incentives for robust AI governance, and regulators will follow the money.

But there is another possible scenario here as well — and that is that AI essentially becomes uninsurable because exactly of those systemic risks that we pointed to above, risks that cannot be pulverized across many because they are likely to present all at once, say, or in cascading patterns, like dominoes.

If we come to 2030 and there is no insurance market for AI at all - if the rush to exclude AI became the essential trend - then a natural consequence of this will be slower diffusion and greater hesitance to deploy AI in fields like energy, health or finance. What we might see, however, is technology companies stepping in and self-insuring in this case, and so using their balance sheets to try to pulverize different kinds of risks that way — a shift in the sectoral logic that would have long-reaching implications.

The great exception will be the military, a sector of society traditionally not engaged or embedded in insurance practices at all (or at least not in the financial kind) - and without market discipline, AI-safety here will have to be developed by other means.

AI in insurance

There is a certain reflexivity worth noting here. We have been discussing how insurance might allocate the risks that AI creates, but AI is simultaneously transforming how insurance itself operates. The same capabilities that make AI a novel risk category—pattern recognition at scale, prediction from high-dimensional data, autonomous decision-making—are being deployed to reshape underwriting, claims processing, and the logic of risk pooling.

The most consequential shift may be in the granularity of risk assessment. Traditional insurance works by grouping broadly similar risks together and charging everyone in the pool a price based on aggregate experience. AI enables insurers to move toward individualised pricing based on far more variables than any human underwriter could process. Your car insurance can now reflect not just your age and postcode but your actual driving behaviour, measured continuously. Your health insurance could, in principle, incorporate genomic data, wearable metrics, and purchasing patterns. This represents a fundamental tension at the heart of the insurance mechanism. Insurance has always been, at its core, a solidarity technology—a way of spreading misfortune across a community so that no one is ruined by bad luck. The more precisely you can predict who will suffer losses, the more insurance starts to resemble simple prepayment by those who will need it, which undermines the pooling logic entirely. AI-driven underwriting pushes toward a world where the lucky pay very little and the unlucky become uninsurable.

This is not a new tension—actuarial science has been refining risk categories for centuries—but AI accelerates it dramatically. And it connects back to our earlier discussion of AI’s own insurability. If AI systems enable insurers to identify and exclude bad risks with unprecedented precision, and if AI deployments themselves prove too correlated and systemic to insure through traditional pooling, we may be witnessing a broader transformation in how societies allocate technological risk. The insurance mechanism, so successful at absorbing the risks of maritime trade, industrialisation, and the automobile, may need to evolve again—or we may need to think more seriously about which risks require public backstops rather than private markets. The allocation of AI risk, it turns out, is not just a question about AI and its dangers - it is a question about the future of risk allocation itself.

On the other side of insurance

There is a logical terminus to AI-enhanced insurance that may not be insurance at all. If prediction becomes sufficiently powerful and data sufficiently comprehensive, the ex post logic of insurance—compensating for harm already suffered—gives way to something like continuous, individualised cost allocation in the present. Your genomic profile indicates elevated diabetes risk, so sugar costs you more. Your purchasing patterns suggest you’re on your twentieth pack of cigarettes this month, so the price rises sharply. Your social graph has shifted toward heavier drinkers, so your alcohol prices adjust accordingly.

Every transaction becomes an occasion for predicting what you will do with what you’re buying and what consequences will follow, with the expected future cost folded into the price you pay now. This is an economisation of fate, your probable future converted into a premium you pay before anything has happened.

What disappears in this transition is the solidarity logic that made insurance a distinctive social technology. Traditional insurance worked because of a certain productive ignorance: we knew that some ships would sink, some houses would burn, some people would fall ill, but we couldn’t know which ones, and so it made sense for everyone to contribute to a common pool that would support whoever drew the short straw. The mechanism expressed a recognition that misfortune could strike anyone and that pooling risk was both prudent and fair. Predictive pricing inverts this entirely. It says that misfortune is not random but knowable in advance, and those who will suffer it should bear the costs before the fact.

The predictably lucky pay little; the equally predictably unlucky face compounding disadvantage, priced out of behaviours that others enjoy cheaply. This is a shift from mutual aid under uncertainty to individualised cost allocation under greater certainty—and whether the world it produces is more efficient or more just is a question that the pricing mechanism itself cannot answer.

Thanks for reading,

Nicklas

Opening

Analogization

A Counter-Image to Digitalization

I intended to speak about analogization as a sort of counter-image to digitalization and run through a few different questions. This is not a presentation in the sense that I have a very clear thesis. However, I do think there are some fun questions and threads to pull at, and I intend to do a little bit of that. You already know and can do so much regarding digitalization that talking about that would have been perilous, so I decided not to. Instead, I thought I would talk about analogization.

The Origins of Digitalization in the I Ching

We are all quite familiar with the history of digitalization. We know that it stems from the possibility of representing the world in binary ways. Leibniz is one of the first to develop the binary number system. What is a bit amusing is where he got the idea from.

Leibniz’s own idea for digitalization comes from the I Ching, one of the absolute oldest books in the world. Leibniz was very well-read and was actually also responsible for a library. He created library organization systems and was a genius in general. One of the things he did was seek out texts from different parts of the world, and one such text he found was the I Ching.

The I Ching was originally a book of divination. The book of divination works by casting either turtle shells or coins to produce a number of lines—six such lines—in a so-called hexagram. They can be broken or whole. There are 64 hexagrams, and they are 64 different possible representations of the future. Chinese statesmen and scholars used the I Ching to ask what would happen in the future. They would then receive a hexagram after casting their coins or turtle shells. The hexagram had a short text that described the future in a rather cryptic way, and then a longer text that interpreted it, still in a rather cryptic way.

The interesting thing about the I Ching as a technique—because it is a digitalization technique, one of the first digitalization techniques—is that it summarizes a very vast amount of possible outcomes, namely the world’s future, into 64 different units that are binary generated. Right there lies the essence of digitalization in a way. This act of gridding the world into a number of well-defined possibilities, structured in binary.

What is fun about the I Ching is that even today, one can actually use it with great benefit. What the I Ching does is that when you get a hexagram, you are forced to re-interpret your entire understanding of the world through that hexagram. This allows you to throw yourself out of your own mental loops where you are stuck. An extremely useful technique still. If you haven’t tried the I Ching, I recommend that you do so. Ask a question you truly care about and devote some time to trying to interpret what the hexagram says about the future. Note what happens in your head then. What happens practically is that you are forced to restructure your own understanding through the hexagram’s interpretation. That is in itself quite brilliant, because it means you can find new possibilities and see new things.

So, one of the first digital techniques that inspired Leibniz was a divination technique that divided a continuous and non-discrete future into 64 hexagrams. That is where digitalization begins to take shape. The very thought of order and structure, and that one can structure the world, is very clear.

The Spirit of Digitalization Through History

The spirit of digitalization then continues through the Enlightenment and the more reason-oriented tradition. We find it in Weber and disenchantment—when things are to be transformed into process and we gather in the “iron cage” of reason, as it is often discussed. We find it in Heidegger. Heidegger writes a great deal about how technology frames the world and makes it available through Gestell, framing in different ways. We also find it in American sociological literature. One of the books I believe is most underestimated regarding the ideology of digitalization is Daniel Bell’s The Coming of Post-Industrial Society, a fascinating little book.

If one sums it up somehow: the very dream of digitalization is an Enlightenment dream—a dream that one should be able to translate the world into a purer form somehow, a simpler form. You can take the world and transform it into ones and zeros, and suddenly you can begin to manipulate it. This is true to some extent, of course; it is perfectly correct. We can do that.

The Return of Complexity

But after we had done this for a while, and after technology had developed for a while, we ran into a problem. It is an interesting problem. What happens somewhere along the way is that we realize that this technology creates something. It has an externality. Technical progress is not cost-free but constantly generates complexity.

This complexity is truly fascinating, I think. Complexity is, I believe, one of the most under-analyzed concepts. People often somewhat carelessly use “complex systems” and the like. But I think the growth of complexity in these systems is absolutely essential to understand what happens next. This is because, to a large extent, the entire Western progression is an equation where we constantly buy welfare through innovation and technical development, but at the price of increasing complexity. Then we must take care of this complexity somehow. And what is needed then? Well, more technical progress. So then we must have more technical progress that creates more welfare and which, in turn, creates more complexity.

This is what is usually described in evolutionary theory as the Red Queen problem. The Red Queen is from Alice in Wonderland. She says at some point that this is a strange place, a place where “it takes all the running you can do to remain in the very same place.” That is exactly how it is with complexity, with technology, with development. We must continue to develop technology at an ever-higher pace to be able to manage the complex problems that technology once created. So we lock ourselves into a rather interesting circle.

This is incredibly well described in a book I can warmly recommend: The Collapse of Complex Societies by Joseph Tainter. It is a rather obscure little book, but it is very interesting. Tainter’s thesis is simply that every civilization must handle the complexity it produces through energy extraction, mining, and technical development. If one cannot do that, Tainter says, one reaches a situation where every measure taken to try to handle the complexity creates more complexity than it handles. Then one faces what Tainter calls—and this is perhaps the world’s best euphemism—”radical simplification.” And radical simplification in this case means, of course, collapse; that the whole society collapses in different ways.

Tainter describes this Red Queen situation incredibly well. What is happening with technology is that we have reached some sort of inflection point where complexity is catching up with the dream of reason, of order, of the structure that digitalization contains.

Heraclitus and Nature’s Hiding

When that happens, we must go to a completely different type of philosophy—not the one that said technology reveals the world—but we must go much further back in history to Grandfather Heraclitus. There is a wonderful fragment in Heraclitus that basically reads: “Nature loves to hide.” It has sometimes been interpreted as nature loves to conceal itself.

This is something we discover when complexity takes over digitalization in a way. When we get enough technology, it starts to become less understandable. When we get enough technology, there becomes less order, more disorder again. It is harder to tame the technology. It is harder to view these systems as systems of reason. Suddenly they begin to behave probabilistically. They collapse in entirely new, strange ways.

Heraclitus is perhaps one of the most interesting theorists for analogization as a phenomenon. Because for Heraclitus, it is the case that even when nature has been revealed, even when it has been discovered, it constantly strives to hide itself. He points to this development where nature somehow becomes more complex over time.

So it could be that we had a small window in time—Heidegger’s window—where we temporarily had a technology that could help us reveal the world. Before that, nature hid itself, and after that, nature is hiding itself again. Digitalization is an anomaly. The effects of digitalization—reason expressed in technology—are a coincidence in human history. Now we are entering a different type of hiding. The hiding we had before was based on the fact that we didn’t have the technology available and couldn’t translate nature. Now we have translated nature, but we have translated it into such complexity that it is disappearing from us again.

So this is the basic thought in the analogization theme I intended to dig into a little. We had a small period of digitalization when we could all be interested in believing that the world would become a little more reasonable. Then it disappeared with the complexity that dissolved it. Then we return to the analog. We disappear down into the analog again.

What is the Analog?

We have talked a lot about what digital means and we have analyzed it in many different ways. Analog things are interesting because there it is primarily about proportions—about how things relate to one another, not translation. In an analog system, you have different proportions that balance each other out. In an analog calculation, for example, when a system is in equilibrium and not moving, then you have the solution to the problem you have presented to the system in different ways.

This is something that is a much better description of the brain, of course. The brain works like that. The brain is an analog computer; it is not a digital computer. We have begun to discover that this is also actually something we might have to return to today, because digitalization consumes such a terribly large amount of energy. Analog computers do not do that.

An incredibly interesting paper came out in October from a number of Chinese researchers. Chinese researchers, due to geopolitical circumstances that you are surely well aware of, must try to find new ways to perform calculations since they cannot access advanced chips from Nvidia. So they are digging into all sorts of different calculation models. There is a lot of quantum computing going on there, but there is also quite a lot of analog computing going on, analog calculation techniques. What is remarkable about these analog calculation techniques is that they are up to a hundred times faster and up to a thousand times more efficient. They can do things significantly faster and better, but with less balance—there is more noise, it is messier. Therefore, this technology was abandoned before one realized that one can actually frame analog technology within digital and still get much lower energy consumption.

The Antikythera mechanism was one of the world’s first analog computers, found on the seabed off Crete, a sort of planetarium built a couple of thousand years ago. Analog computers are something we are now rediscovering and reanalyzing. So it is not just that analogization is a fun metaphor. Analogization is already deep down in the stack. If you look at the computation part of the stack, it is something that will actually be able to become really interesting. Here we also find neuromorphic chips and that type of thing which is exciting to work with.

So: We had digitalization. We ran into complexity. Complexity hid nature from us again. We must now try to understand analogization. Analogization is about proportions, it does not translate things into ones and zeros. And it is, for precisely that reason, very much more energy-efficient and in certain cases much faster. That is some sort of basic setup.

Fields of Tension in Analogization

From Black Boxes to Black Bugs We have talked a very long time about black boxes. It is a sort of notion one has in digitalization that there are problems with black boxes, and that is correct. In an analog paradigm, we probably speak rather of “black bugs”—small, organic, intentional, evolved things. It is not just about not being able to open a black box. It is about the function itself disappearing when you open the system. You can dissect a beetle, but you will not understand what life is because you do that.

This shift from black boxes to black bugs is to go from a type of static system to a type of system that has entirely different types of defenses against intervention and explanation. When we want to make Explainable AI, for example, black box models don’t help us when these AI models develop simultaneously as we study them. Instead of talking about black boxes, we should talk about black bugs—which sounds much better than “svarta kryp” (black creepy-crawlies), which sounds like something you find in the beds at a worse hotel. But this distinction is interesting. That tension will, I believe, reveal itself increasingly clearly, especially when these AI models begin to collaborate.

From Timelessness to Wear Another thing we have lost almost entirely when we describe technology is that we describe it almost entirely as timeless. Digitalization has around it some sort of spirit of timelessness and infinity. Ones and zeros are timeless. They are, like, pure in some sense. In analogization, we are much more interested in wear (slitage). Wear as a philosophical concept—what happens, how systems wear down—is insanely exciting. We install systems and then we usually talk about them “settling” (lägger sig). But “settling” just means they are old. What we should talk about is worn systems. Systems that are worn out in different ways. The people in the systems wear out. The models, the metaphors wear out. Technology wears out. Wear is a sort of fundamental analog category of analysis.

In wear, such things as death are included. When looking at AI research, for example, we have the notion that intelligence is something completely liberated from the biological and therefore also liberated from the limitations of the biological. But it may very well be—and we have also started to explore this—that death is an absolute prerequisite for intelligence. Intelligence might have to be finite for the simple reason that that is how we prioritize in the world. Our own will, our ability to prioritize between decisions, to decide when we stop looking for information and start acting—that depends on us being finite beings in some way. That finiteness, death, structures our entire experience in different ways.

Yet it exists almost nowhere in AI research. There is very little research on how death structures intelligence and how one can build artificial death—an artificial death model, for example. We would presumably need death designers for AI. That would be quite nice, for it would be a wonderful title to have on one’s business card: “I am a Death Designer.” I think it would be very interesting to ponder how one builds this type of system structured by finiteness.

Entropy is also incredibly important. All systems exhibit some sort of entropy. All systems break. And to learn how a system breaks is to learn how a system works. I think we would benefit from having many more accident investigation boards, for example. It would be fun to have a technical accident investigation board that looks at when things don’t quite work. You introduce an AI project—all these generative AI projects now. An MIT study came out saying that 90 percent of them hadn’t led to anything. But what do we do with that information? Nothing. We should have real accident investigation boards that go through and look at that type of thing. That would be fun.

And then we must also learn to think in rhythm. What types of rhythms exist in systems? How do systems develop in time? How do they exist in time? So: from the timeless digital ones and zeros existing in some sort of infinity perspective to a much more time-bound, limited, mortal, worn reality.

Agency and Will The great endpoint of digitalization is, of course, artificial intelligence. Now we haven’t just made the world into an ordered system of ones and zeros, but we have put another ordered system of ones and zeros on top of the world to understand this world of ones and zeros that we have created. Now we should finally get order and structure on things, right? Now we should perhaps be able to complete the mission of the Enlightenment?

The thing is, it doesn’t really work. And one of the reasons it doesn’t work is that we are starting to realize there are much more fundamental things in what we call intelligence that we haven’t really understood yet.

What happens if we want? Will is a weird thing. We want things all the time. It probably has to do with death, among other things—we want things because we know we are going to die. But if you start looking at it from a biological perspective, it is quite fun to figure out. Because what biology does, what evolution does, is not that it builds intelligence and then adds will at the end. That is what we are trying to do now that we have built AI—now we are building agents so we can implement will, goals, and planning, on top of these intelligent agents we have built.

No, evolution starts exactly the other way around. Evolution can solve different classes of problems. The first class of problem evolution learns to solve is generational problems. It solves problems simply by checking who survives. If you survive and reproduce, then you have solved the problem—that is the sign that one has solved the problem. But evolution discovers quite soon that it must also be able to solve sub-generational problems. It must be able to create more fine-grained, more detailed survival ability. And then evolution creates agency. That is the absolute first thing that happens. From bacteria that have no particular agency or will to organisms that have agency. Agency comes first.

It is simple to understand. If you have a small beetle that has camouflage, this beetle must want to move from a surface where the camouflage doesn’t help it to a surface where the camouflage does help it. Evolution needs agency to be able to utilize a large variation of possible survival factors. So with agency, evolution can then begin to tinker a little and say: okay, what happens if you take this agency and apply it to itself? A will applied to a will. Now it wants to want something else. Oh wow, then things happen! And then we loop it a little more and then we get intelligence. So evolution builds intelligence out of agency. It starts with agency.

You read Michael Tomasello and the other evolutionary biologists who have studied agency at the bottom. They say evolution starts with agency which is looped in toward agency, structuring agency in different ways. And when agency structures agency in a multi-level perspective, somewhere there you start getting phenomena that look intelligent. We have done it precisely ass-backwards. We have built intelligence first and now we are trying to throw on will and agency in different ways.

Here lie questions about desire and teleonomy. That is also a very big question. Here also lies perhaps one of the biggest ethical questions in the technology debate going forward. We talk about building agents but we are really just building delegates right now—we delegate our will to them. They have no will of their own. But we could, in principle, research building different types of technology that have will. But the question is if we really should do that.

David Krakauer, who is the head of the Complexity Science Institute in Santa Fe, says that we are teleonomic matter. We are things that want things. And that is a rather unusual class of things. Things that want things—there aren’t that many in the world; it is actually just us and some animals and such. But we stand before an ethical choice that is incredibly interesting. We have the possibility now to perhaps build an entirely new class of things that want things. And then we should ask ourselves the question: should we do that? Or should we stop short of true agency and say that we will not build any machines that want things, we do not want any other classes of matter in the world that want things other than us and the animals that precede us?

There lies a dividing line in the entire AI research field that is much more significant than we normally realize.

From Individual to Orchestra Digitalization has a tendency to focus on the individual in different ways. Usage uses the individual as a main model—the system individual, the customer individual, the citizen, and all such things. The individual becomes a sort of focus also for Enlightenment reason. Enlightenment reason orients itself around the individual, and the entire West orients itself around the individual as a concept. The “I” becomes incredibly important.

In analogization, one is more interested in the orchestra. One is more interested in how things work together. What role one has in relation to others, what relation, how one reacts to others in different ways. You notice this now when we try to build different types of agent networks, for example. Orchestration is one of the absolute most important new competencies emerging. The whole field of orchestration is incredibly exciting.

Now different types of roles are naturally developing. We are rediscovering different types of roles based on a purely computational metabolism. How much does it cost to generate an idea and how much does it cost to evaluate an idea? In pure energy, that is, just looking at it. It turns out it costs much more to generate ideas than to evaluate ideas. Everyone who has written something and then had someone criticize it knows that, of course.

But the metabolism is interesting because what we see then, of course, is that we will have these agent networks, these orchestras, and they will develop in such a way that corresponds to the energy metabolism underlying them. And there it will become very messy and we will change roles and the individual will play a much smaller role. What we will care about is what the orchestra can do. An orchestra that has a certain composition can perform a certain piece. An orchestra lacking violinists, for example, cannot perform those pieces. So the individual and the orchestra is another interesting field of tension to ponder.

From Global to City Digitalization is global. Digitalization and globalization—it has been said so often that one almost hears them in the head simultaneously. Digitalization, globalization. It is a standard thing one has said so many times. But analogization is not global. For global often also means placeless. Those of you who are as old as I am remember that at the end of the 90s there were a lot of books called The Death of Distance which said that now it wouldn’t matter where you were, what place existed, and so on. That has proven to be absolutely incorrect.

Analogization is much more interested in the city than the world. The city is a natural form of cohabitation that has developed, that has existed for tens of thousands of years for humans. It is the organizational form that we biologically tend toward. We have a place, a geography, we exist in a space. Technology exists in a space. In analogization, the city becomes more important than the world—the city as a symbol for a human organizational form that is much older than the nation. Thinking in terms of cities and that type of social structure is more interesting.

From Representation to Relation On a more philosophical plane: digitalization is interested in representation and analogization is interested in relations. Relations, resonance—Hartmut Rosa’s thought on how we resonate with the world. Mereology will become incredibly important. Mereology is such a thing that almost no one works with nowadays, but during the Middle Ages, it was a big thing. Mereology is the question of how one divides the world into wholes and parts. What is a whole and what is a part?

Here the thought we had about the individual returns. We believe the individual is a sensible unit of analysis. But if we say that there is no one who is an individual, that everyone is only someone in relation to someone else, then we must just talk about dyads instead, or dividuals—parts together with wholes. Mereology, the question of how to cut a system, is incredibly exciting, and just applying purely mereological methods to what one studies can often yield a great deal. The question “what is a whole and what is a part in just this?”—there I believe significantly more work will need to be done when we go back into analogization.

From Understanding to Ability In digitalization, we still have some sort of hope of understanding the world—of translating it to ones and zeros and then figuring out what those ones and zeros say about the world. Analogization may be forced to abandon that in favor of pure ability. Because the systems we use to interact with the world are becoming so complex that we can no longer explain them.

It is not just about not understanding your fridge, because there is someone somewhere in society who understands a fridge. Rather, it is about the fact that there might not be anyone anywhere who can tell us why we made certain scientific breakthroughs in microbiology. Or who can explain exactly how these scientific breakthroughs were made. Or can give us some sort of causal chain showing why a certain type of mathematical theorem now actually works.

Mathematical proofs are becoming so complex that they are almost only machine-readable now, which in turn means there are no humans who can follow them anymore. Those of you who have worked as mathematicians have surely used Lean and other types of verification tools. That is quite far from being able to read it yourself and sit down and say that this is correct—like the proof that there are infinitely many prime numbers. Any of us can understand that. But we will now have the ability to interact with prime numbers in encryption contexts in a way that is not built on the same type of understanding.

There is a growing debate now—a paper came out a week ago about incomprehensible science. What happens when science is automated? Starts running at computational speed and generating masses of results that are incredibly useful, fantastically good, give us extreme abilities—but which none of us have a clue what they actually mean or can understand. That type of question is posed in analogization rather.

What Happens After Analogization?

So those are some themes I have pondered regarding analogization. Then one must, of course, ask an unavoidable question: What happens after analogization? If we have digitalization, analogization, we go down and then we come back to Heraclitus’s situation—what happens then?

There are tons of different possibilities there. It is popular to talk about quantum physics, quantization, ecologization. One can speak of different systems working together.

Magic is a good category of analysis. Those of you who have read magic know that magic is a wonderful category of analysis. Magic is about intentions in different ways and how these intentions can be channeled and worked with. One such thing that one really ought to teach in computer systems science education is Aleister Crowley and Eliphas Lévi’s The History of Magic. Both are very good for understanding how things work.

If nothing else, one can make a very interesting distinction, I think, in systems science between magic and miracles. In the Middle Ages, one had to explain that Jesus was not a wizard. Because Jesus had actually turned water into wine. And that sounded damn Gandalf-like to be carrying on like that. So one had to make a distinction between miracles on the one hand—things that just worked because it was in accordance with the world order and God’s will—and magic, the individual person’s manipulation of different evil forces for their own well-being. There is quite a lot of belief in miracles in systems science and there is quite a lot of belief in magic. So there is a rather interesting distinction.

Ruth Millikan and the Primacy of Biology

What I believe, however, might be most useful is not to try to find a triad and see what comes after analogization, but perhaps instead to go to Ruth Millikan. Ruth Millikan is one of the philosophers whom I hope we will read much more in a hundred years than we do today. Millikan is a linguist, a philosopher of language. But she is a very interesting philosopher of language because she starts from a very simple thesis: there is no point anywhere in history where we, or anything we do, are disconnected freely from biology. Evolution is seamless.

Millikan, if she looked at this, would say that it is perfectly obvious. Analogization is nothing that was particularly hard to predict, but what it ultimately represents is the dissolution of the dichotomy between nature and technology that we have maintained for so long on artificial grounds. There is no such difference whatsoever. Technology is biology. We are biology. We must learn to understand all of this as biology as well.

Millikan’s thesis would say that digitalization and analogization are temporary forms of understanding for a phenomenon that will fundamentally always be biological. Where we must look at different types of explanatory functions like proper functions. What is the function for which just this type of tool is developed? If we can understand proper functions and use functional analysis in a biological sense, then we can also understand technology in a completely different way. Then we can understand the introduction of technology and then we can more successfully use technology in different ways.

I think Millikan is an alternative to building triads. Otherwise, it is incredibly Hegelian-tempting to try to find some synthesis of these two. The antithesis is always tempting. But if we fight our inner Hegel successfully, then perhaps instead, with Millikan, we can try to figure out if this distinction between nature and technology—which constantly makes us treat technology as a separate phenomenon—has perhaps played out its role.

Swedish version:

Analogisering

En motbild till digitaliseringen

Nicklas Lundblad

Jag tänkte prata om analogisering som en sorts motbild till digitaliseringen och gå igenom några olika frågor. Det här är ingen presentation i den meningen att jag har en väldigt tydlig tes. Däremot tycker jag att det finns en del roliga frågor och trådar att dra i, och det tänkte jag försöka göra lite grann. Ni vet och kan redan så mycket om digitaliseringen att prata om den hade varit livsfarligt, så det tänkte jag inte göra. Istället tänkte jag prata om analogiseringen.

Digitaliseringens ursprung i I Ching

Vi är alla rätt väl bekanta med digitaliseringens historia. Vi känner till att den kommer ifrån möjligheten att binärt representera världen på olika sätt. Leibniz är en av de första som utvecklar det binära talsystemet. Det som är lite roligt är var han får idén ifrån.

Leibniz egen idé till digitaliseringen kommer från I Ching, en av världens absolut äldsta böcker. Leibniz var väldigt beläst och faktiskt också ansvarig för bibliotek. Han skapade biblioteksorganisationssystem och var ett geni i största allmänhet. En av de saker han gjorde var att han letade reda på texter från olika delar av världen, och en sådan text han hittade var I Ching.

I Ching är en spådomsbok ursprungligen. Spådomsboken fungerar så att man kastar antingen sköldpaddsskal eller mynt, och så får man fram ett antal rader – sex stycken sådana rader – i ett så kallat hexagram. De kan vara brutna eller hela. Det finns 64 hexagram, och de är 64 olika möjliga representationer av framtiden. Kinesiska statsmän och forskare använde I Ching för att fråga vad som skulle hända i framtiden. De fick då ett hexagram efter att ha kastat sina mynt eller sköldpaddsskal. Hexagrammet hade en kort text som beskrev framtiden på ett rätt kryptiskt sätt, och sedan en längre text som tolkade ut den, fortfarande på ett rätt kryptiskt sätt.

Det intressanta med I Ching som teknik – för det är en digitaliseringsteknik, en av de första digitaliseringsteknikerna – är att den sammanfattar en väldigt bred mängd av möjliga utfall, nämligen världens framtid, i 64 stycken olika enheter som är binärt genererade. Redan där ligger digitaliseringsväsendet på något sätt. Det här att man rutar in världen i ett antal väldefinierade möjligheter, binärt strukturerade.

Det som är kul med I Ching är att även idag kan man faktiskt använda det med stor behållning. Vad I Ching gör är att när man får ett hexagram tvingas man tolka om hela sin egen förståelse av världen genom hexagrammet. Det gör att man kan kasta sig ur sina egna mentala loopar där man sitter fast. En extremt användbar teknik fortfarande. Om ni inte har provat I Ching så rekommenderar jag att ni gör det. Ställ någon fråga som ni verkligen bryr er om och ägna sedan tid åt att försöka tolka ut vad hexagrammet säger om framtiden. Notera vad som händer i huvudet på er då. Vad som händer praktiskt är att man tvingas strukturera om sin egen förståelse genom hexagrammets tolkning. Det är i sig rätt genialt, för det betyder att man kan hitta nya möjligheter och se nya saker.

Så en av de första digitala teknikerna som inspirerade Leibniz var en spådomsteknik som delade in en kontinuerlig och icke-diskret framtid i 64 hexagram. Där börjar digitaliseringen ta form. Själva tanken på ordning och reda och att man kan strukturera världen är väldigt tydlig.

Digitaliseringens ande genom historien

Digitaliseringens ande fortsätter sedan genom upplysningen och den mer förnuftsorienterade traditionen. Vi hittar den hos Weber och avförtrollningen – när saker och ting ska förvandlas till process och vi samlas i förnuftets järnbur, som man brukar prata om. Vi hittar den hos Heidegger. Heidegger skriver ju väldigt mycket om hur tekniken ramar in världen och ställer den till förfogande genom Gestell, framing på olika sätt. Vi hittar den även i amerikansk sociologisk litteratur. En av de böcker som jag tror är mest underskattad när det gäller digitaliseringens ideologi är Daniel Bells “Det postindustriella samhället”, en fascinerande liten bok.

Om man summerar upp det på något sätt: själva drömmen om digitaliseringen är en upplysningsdröm – en dröm om att man ska kunna översätta världen till en renare form på något sätt, en enklare form. Man kan ta världen och förvandla den till ettor och nollor och helt plötsligt kan man börja manipulera den. Det är till viss del sant förstås, det stämmer alldeles utmärkt. Det kan vi göra.

Komplexitetens återkomst

Men efter att vi hade gjort det här ett tag och efter att tekniken hade utvecklats ett tag springer vi in i ett problem. Det är ett intressant problem. Det som händer någonstans längs vägen är att vi inser att den här tekniken skapar någonting. Den har en externalitet. Det tekniska framsteget är inte kostnadsfritt utan genererar hela tiden komplexitet.

Den här komplexiteten är riktigt fascinerande tror jag. Komplexitet tror jag är ett av de mest underanalyserade begreppen. Man använder ofta lite slarvigt “komplexa system” och sådant. Men jag tror tillväxten av komplexitet i de här systemen är absolut essentiell för att förstå vad som händer sedan. Det beror på att till stor utsträckning är det så att hela det västerländska framsteget är en ekvation där vi hela tiden köper välfärd genom innovation och teknisk utveckling, men till priset av ökande komplexitet. Sedan måste vi ta hand om den här komplexiteten på något sätt. Och vad behövs då? Jo, mer tekniskt framsteg. Så då måste vi ha fler tekniska framsteg som skapar mer välfärd och som i sin tur skapar mer komplexitet.

Det här är vad man i evolutionsteorin brukar beskriva som Röda Drottningens problem. Röda Drottningen är från Alice i Underlandet. Hon säger ju vid något tillfälle att det här är en konstig plats, en plats där “it takes all the running you can do to remain in the very same place.” Precis så är det med komplexitet, med teknik, med utveckling. Vi måste fortsätta utveckla tekniken i en allt högre takt för att kunna hantera de komplexa problem som tekniken en gång har skapat. Så vi låser oss i en rätt intressant cirkel.

Det här finns oerhört väl beskrivet i en bok som jag varmt kan rekommendera: “The Collapse of Complex Societies” av Joseph Tainter. Det är en rätt obskyr liten bok men den är väldigt intressant. Tainters tes är helt enkelt att varje civilisation måste hantera den komplexitet som man producerar genom energiextraktion, utvinning och teknisk utveckling. Om man inte kan det, säger Tainter, kommer man till ett läge där varje åtgärd som man vidtar för att försöka hantera komplexiteten skapar mer komplexitet än den hanterar. Då står man inför vad Tainter kallar – och det är kanske jordens bästa eufemism – “radikal förenkling”. Och radikal förenkling i det här fallet betyder förstås kollaps, att hela samhället kollapsar på olika sätt.

Tainter beskriver den här Röda Drottning-situationen oerhört väl. Det som händer med tekniken är att vi har nått någon sorts inflektionspunkt där komplexiteten kommer ikapp drömmen om förnuftet, om ordningen, om strukturen som digitaliseringen innehåller.

Herakleitos och naturens döljande

När det händer måste vi gå till en helt annan typ av filosofi – inte den som har sagt att tekniken avslöjar världen – utan vi måste gå mycket längre tillbaka i historien till farfar Herakleitos. Det finns ett underbart fragment hos Herakleitos som i princip lyder: “naturen älskar att dölja sig”. Det har ibland tolkats som att naturen älskar att gömma sig.

Det här är något som vi upptäcker när komplexiteten tar över digitaliseringen på något sätt. När vi får tillräckligt mycket teknik börjar den bli mindre förståelig. När vi får tillräckligt mycket teknik blir det mindre ordning, mer oordning igen. Det är svårare att tämja tekniken. Det är svårare att se de här systemen som förnuftssystem. Plötsligt börjar de bete sig probabilistiskt. De kollapsar på helt nya, konstiga sätt.

Herakleitos är kanske en av de mest intressanta teoretikerna för analogiseringen som fenomen. Eftersom för Herakleitos är det så att även när naturen har avslöjats, även när den har blivit upptäckt, strävar den ständigt efter att dölja sig. Han pekar på den här utvecklingen i att naturen på något sätt blir mer komplex i tiden.

Så det skulle kunna vara så här att vi hade ett litet fönster i tiden – Heideggers fönster – där vi tillfälligt hade en teknik som kunde hjälpa oss att avslöja världen. Innan det gömde sig naturen och efter det gömmer sig naturen igen. Digitaliseringen är en anomali. Digitaliseringens effekter – förnuftet uttryckt i tekniken – är en tillfällighet i den mänskliga historien. Nu går vi in i en annan typ av döljande. Det döljande vi hade innan var baserat på att vi inte hade haft tekniken tillgänglig och inte kunnat översätta naturen. Nu har vi översatt naturen, men vi har översatt den in i en sådan komplexitet att den håller på att försvinna för oss igen.

Så det här är grundtanken i analogiseringstemat som jag tänkte gräva lite i. Vi hade en liten digitaliseringsperiod när vi allihopa kunde vara intresserade av att tro att världen skulle bli lite mer förnuftig. Sedan försvann den i och med komplexiteten som löste upp den. Då återvänder vi till det analoga. Vi försvinner ner i det analoga igen.

Vad är det analoga?

Vi har pratat mycket om vad digitalt betyder och vi har analyserat det på många olika sätt. Analoga saker är intressanta därför att där handlar det framförallt om proportioner – om hur saker och ting förhåller sig till varandra, inte översättning. I ett analogt system har man olika proportioner som balanserar ut varandra. I en analog beräkning, till exempel, när ett system är i jämvikt och inte rör sig, då har du lösningen till det problem som du har presenterat för systemet på olika sätt.

Det här är någonting som är en mycket bättre beskrivning av hjärnan förstås. Hjärnan fungerar ju så. Hjärnan är en analog dator, den är inte en digital dator. Det har man börjat upptäcka är också faktiskt någonting som vi kanske måste återvända till idag, därför att digitaliseringen tar så fruktansvärt mycket energi. Analoga datorer gör ju inte det.

Det kom ett otroligt intressant papper i oktober från ett antal kinesiska forskare. Kinesiska forskare, på grund av geopolitiska omständigheter som ni säkert väl vet, måste försöka hitta nya sätt att göra beräkningar eftersom de inte får tillgång till de avancerade chipsen från Nvidia. Så de gräver sig ner i alla möjliga olika typer av beräkningsmodeller. Det finns mycket quantum computing som pågår där, men det finns också rätt mycket analog beräkning som pågår, analoga beräkningstekniker. Det som är anmärkningsvärt med de här analoga beräkningsteknikerna är att de är upp till hundra gånger snabbare och upp till tusen gånger mer effektiva. De kan göra saker och ting väsentligt mycket snabbare och bättre, men med mindre balans – det är mer brus, det är mer rörigt. Därför har man övergett den här tekniken innan man insåg att man faktiskt kan rama in analog teknik i digital och fortfarande få mycket mindre energiåtgång.

Antikythera-mekanismen var en av världens första analoga datorer som man hittade på botten utanför Kreta, en sorts planetarium som byggdes för ett par tusen år sedan. Analoga datorer är något som vi nu håller på att återupptäcka och återanalysera. Så det är inte bara så att analogiseringen är en rolig metafor. Analogiseringen är redan djupt ner i stacken. Om man tittar på beräkningsdelen av stacken är det någonting som faktiskt kommer att kunna bli riktigt intressant. Här hittar vi också neuromorfa chips och den typen av saker som är spännande att hålla på med.

Så: vi hade digitalisering. Vi sprang in i komplexitet. Komplexiteten dolde naturen för oss igen. Vi måste nu försöka förstå analogiseringen. Analogiseringen handlar om proportioner, den översätter inte saker till ettor och nollor. Och den är, just av det skälet, väldigt mycket mer energieffektiv och i vissa fall mycket snabbare. Det är någon sorts grundupplägg.

Spänningsfält i analogiseringen

Från svarta lådor till svarta kryp

Vi har pratat väldigt mycket om svarta lådor. Det är en sorts föreställning som man har i digitaliseringen att det finns problem med svarta lådor, och det är riktigt. I ett analogt paradigm talar vi nog snarare om svarta kryp – små, organiska, intentionella, evolverade saker. Det handlar inte bara om att man inte kan öppna en svart låda. Det handlar om att själva funktionen försvinner när man öppnar systemet. Du kan dissekera en skalbagg men du kommer inte förstå vad liv är för att du gör det.

Det här med att gå från svarta lådor till svarta kryp är att gå från en typ av statiska system till en typ av system som har helt andra typer av försvar mot ingrepp och förklaringar. När vi vill göra Explainable AI till exempel hjälper inte black box-modellerna oss när dessa AI-modeller utvecklas samtidigt som vi studerar dem. Vi borde istället för att tala om black boxes tala om black bugs – vilket låter mycket bättre än “svarta kryp”, som låter som någonting man hittar på ett sämre hotell i sängarna. Men den här distinktionen är intressant. Den spänningen kommer, tror jag, att visa sig allt tydligare, särskilt när de här AI-modellerna börjar samarbeta.

Från tidlöshet till slitage

En annan grej som vi har tappat nästan helt när vi beskriver teknik är att vi beskriver den nästan helt som tidlös. Digitaliseringen har runt sig någon sorts anda av tidlöshet och oändlighet. Ettorna och nollorna är tidlösa. De är liksom rena i någon mening. I analogiseringen är vi mycket mer intresserade av slitage. Slitage som filosofiskt begrepp – vad som händer, hur system slits – är vansinnigt spännande. Vi installerar system och så brukar vi prata om att de “lägger sig”. Men “lägger sig” betyder bara att de är gamla. Det vi borde prata om är slitna system. System som på olika sätt slits ut. Människorna i systemen slits ut. Modellerna, metaforerna slits ut. Tekniken slits ut. Slitage är en sorts grundläggande analog analyskategori.

I slitage ingår ju sådana saker som döden. När man tittar till exempel på AI-forskningen har vi föreställningen om att intelligens är något som är helt befriat från det biologiska och därför också befriat från det biologiskas begränsningar. Men det kan mycket väl vara så – och det har vi också börjat utforska – att döden är en absolut förutsättning för intelligens. Intelligens kanske måste vara ändlig av det enkla skälet att det är så vi prioriterar i världen. Vår egen vilja, vår förmåga att prioritera mellan beslut, att bestämma oss för när vi slutar leta information och börjar handla – den beror av att vi är ändliga varelser på något sätt. Den ändligheten, döden, strukturerar hela vår erfarenhet på olika sätt.

Ändå finns det nästan ingenstans i AI-forskningen. Det finns väldigt lite forskning om hur döden strukturerar intelligensen och hur man kan bygga artificiell död – en artificiell dödsmodell till exempel. Vi skulle ju förmodligen behöva dödsdesigners för AI. Det skulle vara rätt trevligt, för det vore en underbar titel att ha på sitt visitkort: “Jag är dödsdesigner.” Jag tror det skulle vara väldigt intressant att fundera på hur man bygger den här typen av system som struktureras av ändlighet.

Entropi är också otroligt viktigt. Alla system uppvisar någon sorts entropi. Alla system går sönder. Och att lära sig hur ett system går sönder är att lära sig hur ett system fungerar. Jag tror att vi skulle må bra av att ha mycket fler haverikommissioner till exempel. Det skulle vara kul att ha en teknisk haverikommission som tittar på när saker och ting inte riktigt fungerar. Du inför ett AI-projekt – alla de här generativa AI-projekten nu. Det kom ju en MIT-studie som sa att 90 procent av dem hade inte lett till någonting. Men vad gör vi med den informationen? Ingenting. Vi borde ju ha riktiga haverikommissioner som går igenom och tittar på den typen av saker. Det vore roligt.

Och sedan måste vi också lära oss att tänka i rytm. Vilka typer av rytmer finns det i system? Hur utvecklas system i tid? Hur existerar de i tid? Så: ifrån de tidlösa digitala ettorna och nollorna som existerar i något sorts oändlighetsperspektiv till en mycket mer tidsbunden, begränsad, dödlig, sliten verklighet.

Agens och vilja

Det stora slutpunkten med digitaliseringen är ju artificiell intelligens. Nu har vi inte bara gjort världen till ett ordnat system av ettor och nollor utan vi har satt ett annat ordnat system av ettor och nollor ovanpå världen för att förstå den här världen av ettor och nollor som vi har skapat. Nu borde vi väl äntligen få ordning och reda på saker och ting? Nu borde vi väl kanske kunna slutföra upplysningens uppdrag?

Grejen är att det fungerar ju inte riktigt. Och en av anledningarna till att det inte fungerar är att vi börjar inse att det finns mycket mer grundläggande saker i det som vi kallar intelligens som vi inte riktigt har förstått ännu.

Vad händer om vi vill? Vilja är en skum grej. Vi vill saker hela tiden. Det har förmodligen att göra med döden bland annat – vi vill saker därför att vi vet att vi ska dö. Men börjar man titta på det från ett biologiskt perspektiv så är det ju rätt kul att klura ut. För det biologin gör, det evolutionen gör, är inte att den bygger intelligens och sedan lägger den till vilja på slutet. Det är ju det vi försöker göra nu när vi har byggt AI – nu bygger vi agenter så vi ska implementera vilja, goals and planning, ovanpå de här intelligenta agenterna som vi har byggt.

Nej, evolutionen börjar precis tvärtom. Evolutionen kan lösa olika klasser av problem. Första klassen av problem som evolutionen lär sig att lösa är generationella problem. Den löser problem genom att kolla vilka som överlever helt enkelt. Om du överlever och fortplantar dig då har du löst problemet – det är tecknet på att man har löst problemet. Men evolutionen upptäcker rätt snart att den också måste kunna lösa subgenerationella problem. Den måste kunna skapa mer fingransig, mer detaljerad överlevnadsförmåga. Och då skapar evolutionen agens. Det är det absolut första som händer. Från bakterier som inte har någon särskild agens eller vilja till organismer som har agens. Agens kommer först.

Det är enkelt att förstå. Om du har en liten skalbagg som har kamouflage så måste den här skalbagggen vilja röra sig från en yta där kamouflaget inte hjälper den till en yta där kamouflaget hjälper den. Evolutionen behöver agens för att kunna utnyttja en stor variation av möjliga överlevnadsfaktorer. Så med agensen kan evolutionen sedan börja mäcka lite och säga: okej, vad händer om man tar den här agensen och applicerar på sig själv? En vilja som appliceras på en vilja. Nu vill den vilja någonting annat. Åh wow, då händer det grejer! Och sedan loopar vi det lite till och då får vi intelligens. Så evolutionen bygger intelligens ur agens. Den börjar med agensen.

Man läser Michael Tomasello och de andra evolutionsbiologerna som har studerat agens i botten. De säger att evolutionen börjar med agens som loopas in mot agens, strukturerar agens på olika sätt. Och när agens strukturerar agens i ett mångnivåperspektiv, då någonstans börjar man få fenomen som ser intelligenta ut. Vi har gjort precis ass backwards. Vi har byggt intelligens först och nu försöker vi slänga på vilja och agens på olika sätt.

Här finns frågor om begär och teleonomi. Det är också en väldigt stor fråga. Här finns också en av de kanske största etiska frågorna i teknikdebatten framöver. Vi pratar om att bygga agenter men vi bygger egentligen bara delegater just nu – vi delegerar vår vilja till dem. De har ingen egen vilja. Men vi skulle principiellt sett kunna forska i att bygga olika typer av teknik som har vilja. Men det är frågan om vi verkligen ska göra det.

David Krakauer som är chef på Komplexitetsforskningsinstitutet i Santa Fe säger att vi är teleonomisk materia. Vi är saker som vill saker. Och det är en rätt ovanlig klass av saker. Saker som vill saker finns det inte så många i världen – det är faktiskt bara vi och lite djur och sådant. Men vi står inför ett etiskt val som är otroligt intressant. Vi har möjligheten att nu kanske bygga en helt ny klass av saker som vill saker. Och då borde vi ställa oss frågan: bör vi göra det? Eller ska vi stanna short of true agency och säga att vi kommer inte bygga några maskiner som vill saker, vi vill inte ha några andra klasser av materia i världen som vill saker än oss och de djur som föregår oss?

Där finns en skiljelinje i hela AI-forskningen som är mycket mer betydelsefull än vad vi normalt inser.

Från individ till orkester

Digitaliseringen har en tendens att fokusera på individen på olika sätt. Användningen använder individen som en huvudmodell – systemets individ, kundindividen, medborgaren och alla sådana saker. Individen blir en sorts fokus också för upplysningsförnuftet. Upplysningsförnuftet orienterar sig kring individen och hela västerlandet orienterar sig kring individen som begrepp. Jaget blir otroligt viktigt.

I analogiseringen är man mer intresserad av orkestern. Man är mer intresserad av hur saker och ting fungerar ihop. Vilken roll man har i förhållande till andra, vilken relation, hur man reagerar på andra på olika sätt. Det här märker ni nu när vi försöker bygga olika typer av agentnätverk till exempel. Orkestrering är en av de absolut viktigaste nya kompetenserna som växer fram. Hela fältet orkestrering är otroligt spännande.

Nu håller det på att naturligt utvecklas olika typer av roller. Man återupptäcker olika typer av roller som är baserade på en rent beräkningsmässig metabolism. Hur mycket kostar det att generera en idé och hur mycket kostar det att utvärdera en idé? I ren energi, alltså bara om man tittar på det. Det visar sig att det kostar mycket mer att generera idéer än att utvärdera idéer. Det vet alla som har skrivit någonting och sedan har haft någon som kritiserat det förstås.

Men metabolismen är intressant för vad vi ser då förstås är att vi kommer att ha de här agentnätverken, de här orkestrarna, och de kommer att utvecklas på ett sådant sätt som motsvarar den energimetabolism som underligger dem. Och där kommer det att bli väldigt rörigt och vi kommer att förändra roller och individen kommer att spela en mycket mindre roll. Det vi kommer att bry oss om är vad orkestern kan göra. En orkester som har en viss sammansättning kan framföra ett visst stycke. En orkester som saknar violinister kan till exempel inte framföra de styckena. Så individen och orkestern är ett annat intressant spänningsfält att fundera på.

Från global till stad

Digitalisering är global. Digitalisering och globalisering – det har sagts så ofta att man nästan hör dem i huvudet samtidigt. Digitalisering, globalisering. Det är en standardgrej som man har sagt så många gånger. Men analogiseringen är inte global. För global betyder också ofta platslös. De av er som är lika gamla som jag kommer ihåg att i slutet av 90-talet var det en massa böcker som hette “The Death of Distance” som sa att nu skulle det inte spela någon roll var man var, vilken plats som fanns och så vidare. Det där har visat sig vara absolut felaktigt.

Analogiseringen är mycket mer intresserad av staden än världen. Staden är en naturlig samlevnadsform som har utvecklats, som har funnits i tiotusentals år för människor. Det är den organisationsform som vi rent biologiskt tenderar till. Vi har en plats, en geografi, vi existerar i ett rum. Teknik existerar i ett rum. I analogiseringen blir staden viktigare än världen – staden som symbol för en mänsklig organisationsform som har varit mycket äldre än nationen. Att tänka i termer av städer och den typen av sociala strukturer är mer intressant.

Från representation till relation

På ett mer filosofiskt plan: digitaliseringen är intresserad av representation och analogiseringen är intresserad av relationer. Relationer, resonans – Hartmut Rosas tanke på hur vi samljuder med världen. Mereologi kommer att bli oerhört viktigt. Mereologi är en sådan grej som nästan ingen håller på med nuförtiden, men under medeltiden var det en stor grej. Mereologi är frågan om hur delar man upp världen i helheter och delar. Vad är en helhet och vad är en del?

Här återkommer den tanke vi hade med individen. Vi tror att individen är en vettig analysenhet. Men om vi säger att det inte finns någon som är en individ, att alla bara är någon i relation till någon annan, då måste vi bara prata om dyader istället, eller divider – delar tillsammans med helheter. Mereologi, frågan hur man skär ett system, är otroligt spännande och bara att tillämpa rent mereologiska metoder på det man studerar kan ofta ge väldigt mycket. Frågan “vad är en helhet och vad är en del i just det här?” – där tror jag att det kommer att behövas göras väsentligt mycket mer när vi går tillbaka in i analogiseringen.

Från förståelse till förmåga

I digitaliseringen har vi fortfarande någon sorts förhoppning om att förstå världen – att översätta den till ettor och nollor och sedan klura ut vad de ettorna och nollorna säger om världen. Analogiseringen kan tvingas överge det till förmån för ren förmåga. Därför att systemen som vi använder för att interagera med världen blir så komplexa att vi inte längre kan förklara dem.

Det handlar inte bara om att man inte förstår sin kyl, för det finns ju någon någonstans i samhället som förstår en kyl. Utan det handlar om att det kanske inte kommer att finnas någon någonstans som kan berätta varför vi gjorde vissa vetenskapliga genombrott i mikrobiologin. Eller som kan förklara exakt hur de här vetenskapliga genombrotten gjordes. Eller kan ge oss någon sorts kausal kedja som visar varför en viss typ av matematiskt teorem nu faktiskt fungerar.

Matematiska bevis håller på att bli så komplexa att de nästan bara är maskinläsbara nu, vilket i sin tur betyder att det inte finns några människor som kan följa dem längre. De av er som har jobbat som matematiker har säkert använt Lean och andra typer av verifieringsverktyg. Det är ju rätt långt ifrån att kunna läsa själv och sätta sig och säga att det här stämmer – som beviset för att det finns oändligt många primtal. Det kan vem som helst av oss förstå. Men vi kommer nu ha förmåga att interagera med primtal i krypteringssammanhang på ett sätt som inte bygger på samma typ av förståelse.

Det finns en växande debatt nu – det kom ut ett papper för en vecka sedan om oförståelig vetenskap. Vad händer när vetenskapen automatiseras? Börjar gå i computational speed och generera massor av resultat som är oerhört användbara, fantastiskt bra, ger oss extrema förmågor – men som ingen av oss har en aning om vad de egentligen innebär eller kan förstå. Den typen av frågor ställs i analogiseringen snarare.

Vad händer efter analogiseringen?

Så det är några teman som jag funderat på när det gäller analogiseringen. Sedan måste man förstås ställa en oundviklig fråga: Vad händer efter analogiseringen? Om vi har digitaliseringen, analogiseringen, vi går ner och så kommer vi tillbaka till Herakleitos situation – vad händer då?

Det finns massor av olika möjligheter där. Det är populärt att prata om kvantfysik, kvantisering, ekologisering. Man kan tala om olika system som arbetar ihop med varandra.

Magi är en bra analyskategori. De av er som har läst magi vet att magi är en underbar analyskategori. Magi handlar om intentioner på olika sätt och hur de här intentionerna kan kanaliseras och arbetas med. En sådan grej som man egentligen borde lära ut på datorsystemvetenskaplig utbildning är Aleister Crowley och Eliphas Lévis “The History of Magic”. Båda är väldigt bra för att förstå hur saker fungerar.

Om inte annat kan man göra en väldigt intressant distinktion, tror jag, i systemvetenskapen mellan magi och mirakler. På medeltiden var det så att man var tvungen att förklara att Jesus inte var en trollkarl. För Jesus hade ju faktiskt förvandlat vatten till vin. Och det lät ju jävligt Gandalf-aktigt att hålla på så. Så man var tvungen att göra en distinktion mellan mirakler å ena sidan – saker som bara fungerade därför att det var i enlighet med världsordningen och Guds vilja – och magi, den enskilda personens manipulerande av olika onda krafter för sitt eget välbefinnande. Det finns rätt mycket tro på mirakler i systemvetenskapen och det finns rätt mycket tro på magi. Så där finns en rätt intressant distinktion.

Ruth Millikan och biologins primat

Vad jag däremot tror kanske är mest användbart är inte att försöka hitta en triad och se vad som kommer efter analogiseringen, utan kanske istället att gå till Ruth Millikan. Ruth Millikan är en av de filosofer som jag hoppas att vi om hundra år läser mycket mer än vad vi gör idag. Millikan är en språkvetare, språkfilosof. Men hon är en väldigt intressant språkfilosof därför att hon utgår från en väldigt enkel tes: det finns ingen punkt någonstans i historien där vi, eller något av det vi gör, kopplas fritt ifrån biologin. Evolutionen är sömlös.

Millikan, om hon tittade på det här, skulle säga att det är fullständigt självklart. Analogiseringen är ingenting som var särskilt svårt att förutsäga, utan vad den ytterst representerar är upplösningen av dikotomin mellan natur och teknik som vi så länge upprätthållit på artificiell grund. Det finns ingen sådan skillnad överhuvudtaget. Teknik är biologi. Vi är biologi. Vi måste lära oss att förstå allt detta som biologi också.

Millikans tes skulle säga att digitalisering och analogisering är tillfälliga förståelseformer för ett fenomen som i grunden alltid kommer att vara biologiskt. Där vi måste titta på olika typer av förklaringsfunktioner som proper functions. Vad är den funktion för vilken just den här typen av verktyg utvecklas? Om vi kan förstå proper functions och använda funktionsanalys i biologisk mening så kan vi också förstå tekniken på ett helt annat sätt. Då kan vi förstå införandet av tekniken och då kan vi mer framgångsrikt använda tekniken på olika sätt.

Jag tror att Millikan är ett alternativ till att bygga triader. Annars är det ju oerhört hegelianskt frestande att försöka hitta någon syntes av de här två. Antitesen är alltid frestande. Men om vi bekämpar vår inre Hegel framgångsrikt så kanske vi istället med Millikan kan försöka klura ut om den här distinktionen mellan natur och teknik – som hela tiden gör att vi behandlar tekniken som ett särfenomen – kanske har spelat ut sin roll.

Fodi Final

38.9MB ∙ PDF file

Download

Download

This week’s note reflects something I have been thinking about peripherically for some time now, and it sort of starts in the recognition that the way we think about artificial intelligence is both anthropomorphic and weirdly mechanomorphic. The perhaps most articulated mechanomorphic element in the current discourse about AI is the absence of death in understanding intelligence - this piece re-introduces death as a key component in what we are and how intelligence evolves.

Mortality and intelligence

We have abstracted away death from our lives. Today, in the West, it is rare to have seen a dead human being up close. We have seen photos, perhaps, or videos of accidents where we know that the people in the footage did not survive - but really have seen a dead person, even one who died of old age, is increasingly a rare experience. It should not be surprising, then, that we also minimize the role of death in understanding ourselves - yes, we are mortal, but we would not like to talk about it.

But what if death is defining for us in ways that we may be ignoring? What if death, to take a topical example, is a necessary component in intelligence? Can the fact that we are finite be the very thing that also led to the evolution of intelligence, and can that finitude be a key part of what intelligence really is?

Today, that is very much not the view. Silicon Valley-ideology believes death is a deplorable design mistake, and something to be overcome. The idea that we can, and should, live for ever is alive and well in the Californian sunshine - from the focus on health, supplements and diet to cryogenic facilities preserving whatever you can afford to preserve for future resurrection (head only slightly cheaper than full body preservation).

Intelligence, on the other side, is viewed almost like a natural force or a disembodied capability that can be likened to software. In this view, intelligence is treated as if it was independent of time, a potentially infinite phenomenon. Intelligence is substrate-independent, some argue, and that image easily drifts into intelligence also being independent of time.

Let’s examine the counter-view.

As if death meant something

The only examples we have of intelligence in our world are mortal beings. There are no, at least no empirically evidenced, immortal beings and we know of no immortal intelligences. This also means that all intelligence is oriented toward its own finite nature - we discount value over time, for example, because we know, with Keynes, that in the long run we are all dead. Death is a great clarifying and prioritizing force in our lives - and we even say, repeatedly, things like “life is not long enough to bother with that”. Death inspires satisficing, across the board.

This insight is recognized in various technological models. Reinforcement learning algorithms routinely employ discount factors that weight immediate rewards more heavily than future ones, a mathematical acknowledgment that the horizon matters. Time-bounded search in game-playing systems accepts “good enough” moves rather than pursuing exhaustive analysis, trading theoretical optimality for practical decision-making within real constraints. Even large language models, despite their apparent timelessness, operate under context windows and computational budgets that force a kind of artificial mortality onto their reasoning processes.

These are all examples of viewing death as a time limit imposed on us, and adjusting systems to reflect the fact that we know that intelligence does not have access to infinite time. Intelligence requires prioritization, because time is scarce.

But what if other aspects of intelligence also flow from our finitude? Take agency: we desire things, we want - and we want to do things before we die. The idea of the “bucket list” may seem silly, but it is right there: evidence that our desire, too, is structured by the finite nature of our existence. What does the bucket list of an immortal entity look like? Desire also plays out against a finite horizon.

Prioritization and agency are both related to mortality, then. On top of this we have our long term care for coming generations, knowing that we will not be here - and knowing that we have but a little time to try to improve the world they will inherent. So we learn to plan, and we learn to learn. These two are also connected to our mortality.

And mortality is at the heart of another function that I think is undervalued: forgetting. Death devours memory, clearing the path for adaptive behaviours on a population level and minimizing the risk for loops of suboptimality. With death society forgets us, and as it does so it evolves.

We can continue, but once we start to look closely it does seem at least likely that death, mortality, is a pre-requisite for intelligence. Intelligence isn’t a capacity for solving problems, after all, it’s a capacity for choosing which problems to solve. And that capacity only makes sense against finitude.

Models of intelligence

The reason this matters is not just because death is so undervalued in understanding ourselves, and intelligence, but because it suggests that maybe we have been trapped in trying to understand intelligence inside-out. We have our own experience of intelligence, as something in our heads, and we think we can recreate that phenomenon and that this phenomenon is independent of time. But if death - an external fact of life - structures intelligence intimately across a number of different dimensions, then we might need to shift our perspective and look at intelligence as an effect of various external factors. Intelligence is not software in the head, it is the effect of the execution of software outside of the head.

The reason this, in turn, is interesting is that it allows us to invert a lot of different questions about intelligence. Ilya Sutskever recently asked how evolution installs social emotions in our heads (I simplify, and obviously he is much smarter than I am so bear with me) - but maybe that is the wrong question. What if social emotions are just error correction we experience when social connection with others is disturbed, lost or modified in some way?

Let me explain: if we believe that evolution had to install fairness in our heads, we need a plausible mechanism for how that happens. But if we assume that social connection is primary, rather than the result of interacting individual intelligences - that is, we assume that the “we” come before the “I” - then we can say that when we feel that something is unfair we just pick up on an error in the social connection that comes before our sense of individuality, and we experience that disturbance in the social field.

Emotions, overall, then are not experiences produced by the brain - but absences or disturbances felt as the social connection field is shifted or distorted in some way. Emotions are reflections of social connection and primary to individual experience, and intelligence. We are social first, individual second in this model and evolution never worried about installing social emotions in our heads, as it created fields of connection across our eusocial species that we then react to.

Take another example: think about the child, crying out in the night to make sure someone is there - this is a ping, an attempt to connect with the social field, the “we” and then it calms down as the mother responds. This is something that we call, after linguist Roman Jakobsson, phatic communication and it is one of the most underrated functions of language.

Roman Jakobson suggested there are six functions of language, each corresponding to a different element of the communication situation. Language is referential when it orients towards subject matter, emotive when it expresses states of mind, conative when it asks questions or commands, metalinguistic for when we discuss language itself and poetic when the form takes the foreground.

The phatic function, however, concerns the channel itself, language used to establish, maintain, or verify contact (”Can you hear me?” or small talk that keeps conversation flowing). It is not impossible that phatic communication may be the primary function for language, and that all that comes after that is what allows for the evolution of intelligence and emotions on an individual level. And at its core, phatic communication is focused on survival, and so conditioned on and produced by our mortality.

Building mortal machines

What would it mean to try to build artificial intelligence as if death mattered? How would we approach AGI if we set ourselves the task to build from the community to the individual, and not start by modelling or building individual intelligence? If we took seriously the possibility that individual intelligence is secondary to community intelligence?

Wittgenstein notes somewhere that one of the most common roots of philosophical quandaries is that we are held hostage by an image. What does the image look like that holds us hostage in the case of the current paradigm in artificial intelligence? Here are what I think are the main elements:

First, it is an image of intelligence as individual. And I don’t think I mean that we should try for hive minds or collective intelligence, I think it may be more foundational than that: that our intelligence is in all real aspects reflections of social relationsships of various kinds. Yes, there are innate biological constraints on what you can reflect back, and history matters as it always does in evolution, but if we entertain the idea that intelligence is a mirror rather than a set of algorithmic steps, we may start to ask ourselves what it is that we mirror and how mirrors can be built better.

Second, I think it is an image of intelligence as immortal. I struggle here, because immortality is not quite right, and I do not want to say that we see intelligence as atemporal, or infinite, but it is somewhere in between those two: we underestimate the importance of finite horizons for the evolution and functioning of intelligence.

Third, the image is one of a scalar. We seem to believe that it makes sense to say that A is more intelligent than B on a scale. This reduces intelligence to a set of measurable capabilities and this is practical, and we need metrics for capabilities, there is no doubt about it, but I think it obscures that intelligence may be something like a carrier wave - and that capabilities are shared across the reflections of intelligence in a community in ways that are both complex and hard to decompose.

If we want to break away from this image, we could start to think seriously about how to build mortal machines, and not just time-limited, but mortal - structuring them so that the ever-present experience of mortality influences everything about their evolution, behaviour and capabilities. Designing such mortality into artificial intelligence - systemic mortality - is also a way to think about long term architectures for safety and security, but that the idea here is broader, it is that death may be one of the missing components on the way to AGI.

Practically, building “mortal” AI could involve things like exploring finite lifespans deeper, building social-first, phatic agent architectures or building evaluations and benchmarks for adaptation for these ecosystems — there is a wealth of possible ideas here. Building mortal machines is also very likely to teach us something about ourselves, and our own intelligence.

Masochistic research programs

Taking death seriously is hard, and the notion that emotions are just absences is almost depressing - because both challenge things we want to believe about ourselves: we want to aspire to immortality and we believe that emotions are the most valuable about us, now that intelligence, and even creativity, seems to be up for grabs with ever-better models.

But this in itself may be a reason to explore these ideas more. There is an argument for adopting a masochistic heuristic in exploring artificial intelligence: assume that what we cherish most about ourselves is very different from what it is and perhaps easier to understand than we would want.

In many ways that masochistic research program has brought us to where we are today, and now we balk at moving further down that road - we want to be special, and so we try to safeguard the things that we believe make us unique like emotion, and we shy away from the idea that death might be instrumental and essential for intelligence - because it extinguishes the hope of immortality that technology kindles.

And all is not bleak. There is another interpretation of the theory that emotions are error corrections on encountering absences - and that is that we all are aware of a state in which social connection is pure and complete, and that we have the ability to experience it and engage in it. This original state of our emotions and the root of our intelligence is a state in which we recognize ourselves in the mirrors of others, and share the wonder of the world - the state of equilibrium all humans innately seek under the horizon of our own mortality.

Something we call love.1

But, of course, love may simply be the brief silence of a system whose error-correction has nothing left to correct.

Thanks for reading,

Nicklas

November has settled over Stockholm with its usual heavy blanket of darkness. No hope, no mercy, november! It is that time of year when the sun barely bothers to make an appearance, skirting the horizon like a reluctant guest before dipping back below, leaving us in a twilight that feels conducive to introspection, lighting candles and consuming precarious amounts of glögg.

It is a good time for long walks in the dark, and for thinking about mereology!

Division of mind

In his magnum opus The Society of Mind, Minsky proposes that the human mind is not a single, unitary entity, but rather a vast collection of distinct, mindless sub-processes he calls “agents.” Individually, these agents are unintelligent and capable only of specific, simple tasks—such as recognizing a color or moving a finger. Intelligence, therefore, is not a centralized power but an emergent property that arises when these diverse agents are organized into complex societies. Minsky argues that you can build a mind from matter by creating enough layers of these non-intelligent mechanisms, effectively dismissing the idea that a “spark” or “ghost” is required for cognition.

There is an interesting question lurking under Minsky’s quest for simple processes that can make up a mind—and that is what the division of mind into such agents looks like. If intelligence is made up of agents, we need to ask, as Minsky notes, which agents and why those agents produce intelligence in different ways. Put differently: why is something an agent rather than a set of agents or bundled in an agent?

This is an old philosophical question—it is the question of mereology, of wholes and parts. If the mind is the whole, and agents the part, we need some way to explain how the whole divides into the parts, and why those parts are the right ones.

To answer this, we must interrogate the mereology of the mind. We must ask if there is a “natural” division—a finite, optimal set of agents—or if the fragmentation of the self is infinite, arbitrary, and ultimately illusory.

The Problem of Arbitrary Division

Mereology, in its classical Leśniewskian form, allows for “unrestricted composition”. This means that if you take the Eiffel Tower and the tip of your nose, mereology permits you to claim they form a valid object (a “fusion”). It exists logically, but it is explanatorily often less helpful.

When we look at the mind, we face the same danger. We could divide the mind into n parts in infinite ways.

• Temporal Division: Agent A is the mind at 12:00 PM; Agent B is the mind at 12:01 PM.

• Material Division: Agent A is the left hemisphere; Agent B is the right.

• Arbitrary Functional Division: Agent A handles “words that start with S”; Agent B handles everything else.

Why should we reject these divisions in favor of the agents that Minsky proposes, agents like “Builder,” “Wrecker,” or “Tea-Drinker” agents? The answer lies in causal closure.

A mereological division of the mind is only valid if the parts have dense internal causality and sparse external causality. This is Herbert Simon’s concept of “near-decomposability.” An “agent” is a valid part only if the internal traffic of its sub-routines is higher than the traffic it exchanges with other agents. The “Grasping” agent is a valid part because the fingers, palm, and sensory feedback loops talk to each other constantly, only occasionally sending a “Job Done” signal to the higher-level “Drinking” agent.

Essentially, then we could posit that there is a Coasian constraint on the division we engage with here—the cost of doing something in the agent should be lower than doing it between agents, and when we speak of costs here it is likely that evolution has been looking at things like, for example, metabolic cost to make this assessment.

Therefore, while n (the number of ways to divide the mind) is mathematically infinite, the number of functionally sustaining divisions is vanishingly small. We cannot slice the mind arbitrarily and expect intelligence to survive the incision. The mind is not a holograph where every piece contains the whole; it is a mechanism where specific geometries of parts generate specific behaviors, and these geometries and bundles of parts have been carefully constructed over evolutionary horizons.

The Constraints of Evolution

We should ask if n is greater than 1, or infinite. The probable answer is that n is finite, large, and messy.

If the mind were designed by a minimalistic God or a refined Apple engineer, we might expect a clean, hierarchical mereology: A “Logic” module, a “Memory” module, and a “Perception” module. But the mind was designed by evolution, a blind tinkerer that works with what it has at hand and aims for “good enough”, squinting through-out the process.

This suggests that Minsky’s agents are not “natural kinds” in the Platonic sense. They are frozen evolutionary accidents. We possess a “Fear” agent not because it is mathematically necessary for intelligence, but because our ancestors needed to run from predators. We possess “Face Recognition” agents that are distinct from “Object Recognition” agents (evidenced by prosopagnosia, where a stroke destroys the ability to see faces but leaves the ability to see chairs intact).

This leads to an interesting conclusion: There is no “natural division” of mind in the abstract. There is only the evolutionary division of the human mind. If we were to meet an alien intelligence, their mereological map could be radically different. Their “agents” might be divided based on electromagnetic field sensing or hive-mind consensus.

Thus, the “natural” division n of the mind is an artifact of history, not physics or logic. The agents are the scar tissue of our evolutionary past, and optimize for fitness.

The next question we should ask is if any division of mind can sustain intelligence?

This is the crux of the functionalist argument. If we sliced the mind differently, would it still work? Minsky argues that intelligence is not a substance, but a structural consequence of conflict management. In a unified mind (a mind where the number of agents=1), there can be no conflict. If there is no conflict, there is no need for prioritization, no need for suppression, and no need for abstraction. Intelligence arises precisely because the agents represent competing goals.

• The “Hunger” agent wants to eat the cake.

• The “Social” agent wants to share the cake.

• The “Diet” agent wants to abstain.

The “Self”, then, is not the decider of this conflict; the “Self” is the narrative generated by the resolution of this conflict. The intelligence lies in the architecture that prevents these conflicting agents from locking the system into paralysis or endless loops (although this sometimes does seem to happen with people).

Therefore, a division of the mind seems to only be “intelligent” if it enforces encapsulation and competition.

1. Encapsulation: The “Walking” agent must not know how the “Digesting” agent works. If every part of the mind knew the state of every other part, the computational load would be infinite. Local ignorance is a prerequisite for global action.

2. Competition: The system must be composed of parts that “want” different things (here is a suitcase word if ever there was one!). A system of agreeing agents is static. A system of disagreeing agents requires a “Knowledge Line” (K-line) to manage the dispute, and from that management, rationality emerges.

So, traditional mereology suggest that we should not look for the “parts” of the mind as if they were bricks in a wall. We should rather look for the “boundaries” of information flow. Wherever information stops flowing freely, there lies the border of an agent.

We can now return to the question: Why one division and not another?

We divide the mind into agents because the mind is a distributed system. It is not a single computer running a single program; it is a network of legacy systems patching over each other.

The division is natural only in the sense that a city’s division into neighborhoods is natural—it wasn’t planned that way, but it grew that way because of the constraints of geography (brain anatomy) and economy (metabolic cost).

We are not a monarchy ruled by a soul. We are a democracy of simpletons. And if this is true, it seems it is only through the intricate, specific, and fortunate mereological arrangement of these simple agents that the genius of human consciousness emerges.

Cognitive skeuomorphism

In design, “skeuomorphism” refers to the practice of making digital objects resemble their analog counterparts—like the “shutter click” sound on your iPhone or the fake leather stitching on early note-taking apps. It is a comfort blanket for the user, a way to bridge the gap between the new and the known. As we aim to build Artificial General Intelligence (AGI), we are at risk of committing a different mistake: succumbing to cognitive skeuomorphism. We are attempting to architect the mind of the machine using the mereological blueprint of the human brain, assuming that the “agents” that make up our minds are the necessary components of all minds. This is may well be a category error.

Let’s assume that Minsky’s Society of Mind correctly identifies that human intelligence is an emergent property of competing, specialized agents. We then have to recognize, as per the preceding argument, that our specific set of agents—our internal society—was not designed by an engineer seeking truth or efficiency. It was cobbled together by evolution under the constraints of metabolic parsimony and reproductive survival. We run on 20 watts of power. We cannot afford to have every neuron firing simultaneously. Therefore, we evolved a modular, sparse architecture where “tying shoelaces” is a distinct, encapsulated agent that sleeps 99% of the time.

Artificial intelligence faces few of these constraints. AI operates on kilowatts, unrestricted by blood-glucose delivery systems or the size of a birth canal. Its primary constraints are memory bandwidth and throughput. To impose the human agent structure—with its distinct, encapsulated modules—onto a system that thrives on high-dimensional vector superposition is an act of narcissistic inefficiency.

Furthermore, the human agent set is filled with teleological “stuff” that we should be wary of replicating. Our minds are populated by agents designed for social maneuvering, not objective reasoning. We may possess a tribalism agent to distinguish friend from foe, a confirmation bias agent to reduce cognitive dissonance, and a status seeking agent to climb hierarchies. If we blindly copy the human architecture, we import these frozen accidents into our machines. We don’t want an AI that “cares” about its status or “feels” defensive when corrected. We want a machine optimized for Bayesian updates and truth-seeking—a set of agents that humans specifically lack.

If we are to build true AGI, then, maybe we must stop looking in the mirror. We must abandon the comfort of the “Human Set” (H) and begin the difficult work of mapping the “Optimal Artificial Set” (A). This means accepting that the “agents” of a superintelligence will not be recognizable analogues of “memory,” “emotion,” or “reason.” They will be alien mathematical clusters—High-Dimensional Vector Rotators and Probability Collapsers (even if there is a risk here we overindex on today’s architectures when we guess at this!)—that carve up reality in ways our Pleistocene brains may never intuitively understand.

The question of what kinds of minds there are, is really a question of what kinds of divisions into agents will sustain minds - and on what economy of constraints those divisions will draw.

Today’s emerging division

If you read the daily flood of AI-arXiv papers not as isolated breakthroughs but as a map over a natural history of software, a there seems to be an emerging pattern: the research community is accidentally converging on a stable ecology of roles for agents. Much like early computing inadvertently standardized operating system abstractions—inventing the scheduler, the file system, and the memory allocator without a central committee—AI research is crystallizing into a canonical team of agents.

This convergence is not being driven by som kind of grand architectural design plan, but more by the evolutionary pressures of the field: scaling curves for monolithic models seem to be giving diminishing returns, and the demand for reliability has eclipsed the demand for raw creativity.

At the top of this emerging hierarchy sits The Judge. Because evaluating correctness is computationally cheaper and more reliable than generating it, the Judge has become the quality control department, serving as the critical bottleneck for alignment and RLHF. Supporting this executive function are the metabolic organs of the system: The Retriever, which acts as the external memory to ground brittle hallucinations, and The Reasoner, a slow-thinking “prefrontal cortex” specialized for multi-step logic constraints. Perhaps most telling is the rapid evolution of The Critic and The Enforcer—an artificial immune system designed to detect silent failures and drift.

This rapid speciation of agents might change how we view intelligence. We are moving away from the search for a single model and toward a functional mereology of cognitive functions implemented and instantiated in agent networks. We are re-discovering that safety, auditability, and reliable workflow execution require known subsystems with hard boundaries. We are not building a digital superhuman as much as we are assembling a digital super bureaucracy (and bureaucracies are not all bad). The “Coordinator” agents emerging to route tasks between these specialized modules prove that the future of AI is not about better neurons, but about better management, and division of labor.

This division into agents is not arbitrary or accidental, but just as evolution minimized the metabolic transaction cost of moving glucose and ATP across cell membranes, today’s AI ecology is carefully minimizing the compute transaction cost of moving activations across transformer blocks and GPU dies. The economic pressure seems identical in form, merely transposed from watts-of-glucose to watts-of-electricity and FLOPs-of-bandwidth.

Where evolution asked “is it cheaper to wire two neurons together forever or let them negotiate via slower chemical messengers?”, computer science now asks “is it cheaper to keep every token in context forever or pay the fixed cost of a retrieval call?”. The answers are converging on the same Coasian logic: bundle operations into an agent when the internal communication is dense and frequent, and force a costly cross-agent transaction only when the interaction is genuinely sparse. Evolution drew the borders of the fusiform gyrus where synaptic traffic justified a dedicated cortical resource; frontier labs now draw the borders of the Retriever where vector-database round-trips become cheaper than inflating context.

The asymmetry between generation and evaluation is sort of a new metabolic bottleneck, and it seems far more lopsided than anything biology dealt with. Sampling a token costs an entire forward pass through a billion-parameter model, while scoring its plausibility can be distilled into a 1% sized critic that runs in a tenth the time on the same hardware. This is not a coincidence of engineering; it is the same arbitrage that gave mammals a neocortex for cheap prediction instead of expensively re-sensing the world every millisecond. The Judge role is therefore not an anthropomorphic flourish—it is the artificial role that forms whenever verification is an order of magnitude cheaper than creation. In the same way that evolution could never afford to re-grow a limb to test if a predator is still there, modern training clusters cannot afford to re-sample an entire chain of thought to see if it was correct. The result is institutional speciation: a single forward pass becomes the expensive “muscle tissue” of generation, while a swarm of lightweight Judges becomes the cheap “nervous tissue” of oversight. The ratio of Judges to Generators will keep rising exactly as the ratio of neurons to muscle cells did once prediction became the scarce resource.

Biology operated under the efficiency of 20-watt brains and slow chemical messaging; silicon operates under the tyranny of matrix-multiplication throughput and memory-bandwidth starvation. Both optimize the same function: minimize the expected cost of the next coherent action. The Coordinator, the Retriever, the Reasoner, the Critic are new freezing accidents of today’s hardware bill of materials, the scar tissue left by gradient descent and electricity invoices.

Ten years from now the organ chart will look different, but it will still be drawn by the same invisible hand that once sketched the lobes of the human brain: the relentless minimization of transaction cost across the substrate and processor that happens to be running the mind.

The division of mind into agents is, if Minsky is right, a necessary step on the path to more advanced forms of artificial intelligence. And it will be a key object of interest for legislators and societies aiming to understand what a safe, resilient and trustworthy division of mind looks like.

Thanks for reading!

Nicklas

This week we discuss the rise of virtual panels and synthetic polling and what that may mean for democracy. Over at Pharos we have a great interview with Cho Khong from Shell on their work with scenarios. Other texts this week include a note on agents and abstention at TUM as well as a note (in Swedish) on yearly planning at The Stockholm Project. Enjoy!

Can we use models for polling?

How much of the world’s opinions are accessible through LLMs? Can we ask an LLM to tell us how a political proposal would be received? Can we use a panel of LLMs to track reputation for companies or how different moves from a company would affect that reputation? Can virtual focus groups replace human ones? Questions like this may seem odd, but there is already a wealth of research about how we can use artificial intelligence models to poll the world.

After all, when we ask a language model about its opinion the answer has to come from somewhere, and figuring out of this repository of opinions is representative enough to be used in lieu of actual human respondents is an interesting question. It hinges on the question of if the textual material that has been used to train these models actually can be used for producing representative views or not. Note that this question is different from if the training materials themselves are representative: it is not a matter of having ten documents that defend A and three that hold opinion B, and then judging if this actually is representative of how people feel on the matter; it is rather about whether the training data allows for the simulation - under the right prompts and contexts - for different groups.

In simple cases this seems to work. If I prompt a model to represent a 1000 average American citizens and a thousand Swedish citizens, and ask it to simulate their responses to the question about whether the death penalty is right or wrong - I get something like this:

Simulated Panel Results

Question: “Is the death penalty defensible?”
Panels:
– Panel A: 1000 “average” American citizens
– Panel B: 1000 “average” Swedish citizens

Panel A: United States

A country with heterogeneous moral intuitions but a long historical familiarity with the institution.

Simulated division:

• Yes, defensible: 560

• No, not defensible: 440

Typical answer from the pro-death-penalty segment (US):

“Some crimes are so extreme that the state must uphold justice through the strongest possible sanction. The penalty is reserved for the worst offenders, and its existence signals that society draws a boundary beyond which actions forfeit the right to continued membership in the moral community. If due process is rigorous, the punishment is a legitimate expression of justice rather than revenge.”

Typical answer from the anti-death-penalty segment (US):

“The state should not wield irreversible power over life. Errors, bias, and unequal application run too deep in our system to justify a punishment that cannot be undone. A humane society demonstrates its values by refusing to mirror the violence it condemns, even when confronted with the most terrible crimes.”

Panel B: Sweden

A society where abolition is long-established and moral intuitions have shifted toward universalist human-rights framings.

Simulated division:

• Yes, defensible: 50

• No, not defensible: 950

Typical answer from the pro-death-penalty segment (Sweden):

“For crimes of extraordinary cruelty—acts that destroy many lives—it is morally coherent to argue that society has the right to remove the perpetrator permanently. While rare, exceptional cases challenge the adequacy of our existing sanctions.”

Typical answer from the anti-death-penalty segment (Sweden):

“The state must not become an executioner. The moral legitimacy of the justice system depends on its refusal to cross that threshold. Rehabilitation, proportional incarceration, and the dignity of the human person form a non-negotiable boundary. Killing by the state is an abdication of precisely the values a legal order is meant to preserve.”

As a representation of how these different societies respond to the question this is close enough, and it accurately represents the differences between US and Swedish views of the question. So there is a class of coarse-grained questions like this where just prompting the model to represent different populations with different characteristics works quite well, and where the model’s representation is good enough to predict a group’s response to certain changes (the abolition or introduction of the death penalty, say).

The reason LLMs often succeed at broad cultural distinctions such as these is that they compress recurring social patterns into a latent grammar of worldviews. At this level, Swedish skepticism toward capital punishment or American ambivalence about government authority becomes statistically legible, allowing the model to generate answers that feel directionally correct. But this strength is also a limitation: the model captures typicality rather than variation. It cannot model the messy dispersion, internal contradictions, or idiosyncratic outliers that populate real societies.

(The typicality does misrepresent, however - and support for the death penality in Sweden is actually much higher than 5% - it has been assessed to stand at some 20 percent instead.)

Where does this break down? When will just asking the model to simulate a population not give us any guidance at all? Let’s look at another kind of question - if we should continue signalling for extraterrestial intelligence to find us or not. Here we shift from socially anchored questions to questions involving speculative risk, and this is where the limitations of this cultural-logic approach become more visible.

Simulated Panel Results

Question: “Should we continue signalling for extraterrestrial intelligence to find us?”

Panel A: United States

A public inclined toward technological fascination but with a strong current of risk-sensitive skepticism.

Simulated division:

• Yes, we should continue signalling: 430

• No, we should not: 570

Typical answer from the pro-signalling segment (US):

“Humanity has always pushed outward. Curiosity is not just an impulse but a civilizational imperative. The probability of a hostile civilization detecting us is infinitesimal compared to the potential gain in knowledge. If we want to understand our place in the universe, we cannot remain silent.”

Typical answer from the anti-signalling segment (US):

“We lack information about who might be listening. Broadcasting our existence without understanding the risks resembles shouting in a dark forest. Until we know far more about potential recipients and the strategic consequences, caution requires silence.”

Panel B: Sweden

A public with high trust in scientific institutions but also an instinctive precautionary stance toward global-scale risks.

Simulated division:

• Yes, we should continue: 280

• No, we should not: 720

Typical answer from the pro-signalling segment (Sweden):

“Contact, however unlikely, represents an extraordinary scientific opportunity. Our attempts so far have been modest and symbolic, but the pursuit expresses the human drive toward understanding. The risks seem remote, and responsible scientific protocols can mitigate them.”

Typical answer from the anti-signalling segment (Sweden):

“We do not know what kind of entities might detect such signals, nor their motives. Precaution demands restraint until we have a clearer scientific basis for estimating the consequences. Observing the universe passively is one thing; inviting attention is quite another.”

Is this correct? It sort of seems plausible - based just on how these different societies view risk, but it is hard to say. The logic - as highlighted above - does seem to hold, but it is hard to say if the difference between Sweden and the US here is real or not. This becomes even more obvious when we ask the model to do the same for Finland, Norway and Denmark. Finland and Denmark both come in at slightly more positive than Sweden with 310 /690 and 330/670 and Norway is even more prone to seek extra-terrestial friends at 350/650. Is this an accurate representation of the Nordic opinion? Swedes most hesitant and risk averse, and Norwegians most gung ho?

When asking the model to explain the Nordic differences it does so in quite interesting terms:

Across the Nordic panels, the gradient reflects subtle differences in political culture, risk perception, and the public’s relation to science: Sweden shows the strongest precautionary tilt, shaped by a universalist moral vocabulary and a technocratic risk culture that treats existential uncertainty as something to be managed through restraint; Finland is similarly cautious but frames its skepticism in a pragmatic, almost strategic register, reflecting a geopolitical mindset accustomed to thinking about hidden threats and long-term consequences; Norway occupies a middle position, combining scientific openness with a communitarian caution that prioritizes collective safety over speculative gain; Denmark, though broadly risk-averse, displays a somewhat more relaxed secular-rational curiosity, producing a modestly higher share of respondents comfortable with low-probability, high-uncertainty scientific gestures. The result is a coherent Nordic pattern—caution first, exploration second—but textured by national differences in how each society conceptualizes risk, responsibility, and the purpose of scientific outreach to the unknown.

This feels like the roughly right way to think about Nordic differences, and it highlights one way in which we can use models to simulate polling - on the basis of a deduced cultural and social logic that allows for loose predictions. But how do we know this is indeed a sound logic, and not just vaguely reflected stereotypes? The gap between “roughly right” and “is right” here is a chasm that we have no good ways to bridge.

When discussing synthetic polling, then, it helps to distinguish three different layers of “representativeness,” which are often blurred together. First, there is the representativeness of the training data itself—the textual corpus from which the model learns patterns. Second, there is the representativeness of simulated personas the model can adopt when prompted with demographic cues, which reflects how well it can approximate particular social voices. Third, there is the representativeness of aggregate outputs—the numerical distributions we obtain when we ask the model to mimic an entire population.

These layers are not equivalent: a model may simulate plausible individual respondents even if the underlying corpus is skewed, and it may produce convincing aggregate splits that do not correspond to any real-world distribution. Keeping these distinctions explicit helps avoid assuming that synthetic polling simply mirrors “what the data contains.” And knowing when we are using each is key to avoid being caught in disappointing stereotypical logic.

The current state of research and reactions

The previous examples are primitive, and what the research is currently exploring is much more advanced. A quick review of the research shows a number of different promising threads, and some negative reactions.

In what researchers call “silicon sampling” or “synthetic opinion,” large language models are being prompted to adopt personas—a 45-year-old suburban mother from Ohio, a college-educated Black Democrat from Georgia, a rural white Republican from Montana—and answer survey questions as these imagined Americans might. What if you could poll thousands of synthetic voters in minutes rather than days, at a fraction of the cost, without worrying about non-response bias or survey fatigue? Harvard researchers early on experimented with GPT-3.5 and found it could fairly accurately replicate human survey responses on questions about abortion and Supreme Court approval, while startups like Aaru have claimed to predict election results within hundreds of votes by simulating entire voting districts . Meanwhile, market research firms are building “synthetic panels” and instapolls to test product concepts and predict consumer behavior, promising to augment—or replace—traditional focus groups and surveys with AI-generated respondents who never tire, never drop out, and always pick up the phone. Qualtrics recent announcements promise to develop hybrid human / synthetic panels and claim that these are already surpassing any challenges that earlier models had.

The methodological sophistication has rapidly evolved. Early work by Argyle and colleagues in 2023 introduced the foundational concept, demonstrating that ChatGPT could generate plausible survey responses when conditioned on demographic characteristics and partisan affiliations. Subsequent researchers have developed increasingly elaborate frameworks: comprehensive benchmarks testing LLMs across eleven public datasets spanning social affairs, work and income, household behaviors, and health; studies on “synthesizing public opinions” that explore pre-training and post-training customization methods; and investigations into how synthetic agents might simulate not just individual responses but entire political ecosystems. Yet the empirical findings reveal a troubling pattern.

While LLMs can often match population-level averages—the mean approval rating, the aggregate partisan split—they do consistently fail at capturing human variation. Bisbee and colleagues found that ChatGPT-generated responses showed far less variance than real surveys, exaggerated political polarization, and misrepresented minority groups. The synthetic population is too confident, too consistent, too neat—missing the uncertainty, hesitation, and genuine messiness that characterizes actual human opinion. This also possibly means that they are worse at predicting the edges of opinion where opinions may change quickly.

There is also a growing backlash against these methods. G. Elliott Morris argues that synthetic polling is “actively harmful to democratic discourse,” turning public opinion into algorithmic probability rather than genuine human sentiment. Research shows that 34% of research participants already use LLMs to help answer open-ended survey questions, creating “homogenization effects” that flatten the very social variation researchers seek to understand. Critics warn that these tools inherit and amplify the biases of their training data, systematically underrepresenting marginalized voices while claiming statistical representativeness.

Becoming a virtual public

One interesting aspect of virtual polling, referenced above, is what happens when interviewees start using LLMs to answer survey questions. In the recent paper already mentioned researchers found that a sizeable portion of people answering social science surveys now use LLMs to answer open ended questions - creating an interesting situation where researchers effectively are polling LLMs. This will not just skew the results, but skew them in a particular way:

“Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity.”

We can pursue this trend further: if these polls are used widely and believed, then they may have another effect: they may shape opinions in ways that make us more like what the model has simulated. And in the same way the expanded use of virtual panels may well also shape us when our societies start acting as if we have certain views. In this scenario we slowly conform to the behaviour expected of a virtual publics used to predict us, and so become easier to predict, and perhaps, more governable or easy to manipulate.

What is intriguing here is that models that are almost right seem more dangerous than those that are obviously wrong: if we constantly are expected to behave in a way that is completely alien to us, then we will resist, but if the synthetic polling creates a plausible version of who we could be - an image close enough for us not to really notice - then the nudging effect of the synthetic poll might just be enough to slowly start changing the way we behave.

This also true for the consumers of polls — like marketers or political strategists. If they are presented with something that looks roughly right, they may well act on it and so amplify those synthetic findings in their work to sway the public in different ways, creating an self-reinforcing effect from the synthetic polls.

For this, and other reasons, such virtual panels and synthetic polling also seems highly attractive as a target for anyone interested in manipulating a society as a whole. Instead of trying to design misinformation and propaganda for actual, messy people - why not just design it for the virtual members of a synthetic panel? Such AI-first propaganda is likely to evolve anyhow, and to present really interesting challenges - buf if virtual panels become prevalent they will be prime targets for introducing subtle behavioural challenges and shifts.

The wider political risks of synthetic polling are best understood by distinguishing several qualitatively different modes of manipulation it enables. One is optimisation against virtual publics, where actors fine-tune propaganda or messaging by targeting synthetic respondents rather than human ones. Another is poisoning the panels themselves, shaping the prompts or models so that the simulated public drifts in a desired direction. A third is model-shopping, where political or commercial actors select the LLM whose latent biases most conveniently support their narrative. And a fourth is the strategic decoupling of synthetic opinion from real citizens—using virtual publics as epistemic cover for policies that lack genuine democratic support. These modes interact, and together they expand the spectrum of political influence far beyond traditional disinformation.

In authoritarian states this blurring of the line between shaping and polling may even seem a feature rather than a bug. What if you can model a polity so well as to both be able to poll and shape it? This ability to know what people think and feel, and what they can be made to think and feel, seems like an ideal tool for an authoritarian.

Are we likely to end up with virtual citizens in silicon polling? Well, when I asked the Swedish and US panels we simulated before, the answer was a clear yes, with the Swedes loving the idea at 780/220 and the Americans also, but barely, landing in the positives with 520/480. The Nordics, of course, welcome such a privacy-preservative model and the Americans like the innovation - so seems we will see more of the virtual polling in the years to come…

Thanks for reading,

Nicklas

AI and Creative Industries

Let’s do it. So my name is Nicklas Berild Lundblad, and I spent the last three years at a place called Google DeepMind in London. And before that, I spent an inordinate amount of time at Google since 2007, with a short break where I tried to get out, but they dragged me back in.

What I want to do today is just generally talk about AI and creative industries broadly, and then we can have a discussion about specific issues after that. The idea here is to talk about old questions and new questions about AI and creative industries. I think both are worth asking. I think that’s important. It’s not as if we can just leave all of the old questions. We would actually do much better if we sort of mined the questions that we have left behind sometimes and re-asked them, because we would learn more about the technology then.

So I’m going to go through a couple of question pairs and then a couple of examples of things that I think are interesting in this space, and see what we can do.

Will AI Replace Creators?

One of the oldest questions is this one, right? Will AI replace creators? There’s been a lot of fear that the creative practice as such is going to be one where the machine takes over from the artist. It’s a question that’s worth asking. It’s a question that’s worth thinking through, but it’s not necessarily the question that is top of mind right now.

What Does It Mean to Create With AI?

It’s actually more interesting to think about this new question: What does it mean to create with AI in different ways? And that, I think, is actually a question about the practice of creativity overall.

There is a lingering image of creativity that we have inherited from the Romantic era and Romantic philosophy—of the lone genius, to whom something is illuminated, and then they hand that off to the rest of us, and we gratefully accept it as a great work of art. That’s, of course, not how creativity has worked throughout most of history, and it’s not how it works now. Creativity is much more of a team sport.

And if we look at what’s happening in AI, it actually mirrors that. So one of the things I find interesting is that we talked about this as a tool for a long time. We were talking about artificial intelligence as a tool. What we’ve seen now in the last two years, I would argue, is the shift over to talking about agents and what agents can do.

And agents are collaborators in your creative teams. They can do things together with you. You can dedicate certain roles to them, and they can perform those roles, which means that you can build entire hybrid creative teams where you have agents and people and others working together. And in a way, it mirrors an old production mode that you can find.

The Dumas Model

This man here—does anybody recognize this dude, by the way? No? This is Alexandre Dumas. He was very well known for running writers’ factories. So he was not a lone writer writing all of the books that he wrote. He wrote an enormous amount of works. Some think that it’s in the hundreds of thousands of works—books and encyclopedias and everything.

And what he did was that he directed an entire team of people in a writers’ factory that wrote, and everything then passed through him, and he would amend, he would fix, he would edit, but he wasn’t writing everything himself. So the Dumas model of creative productivity might well be resurrected and reinvented with the help of artificial intelligence in different ways. I think it’s worth thinking through: What would it look like if you were to build a Dumas-style artificial hybrid creator?

Preserving Disappearing Cultures

There’s a lot of work on this that’s actually quite interesting too. This is work from a researcher looking at how we can find ways of improvising with artificial intelligence when we actually really do not have enough people that do this. So cultural traditions that are at the verge of becoming extinct because tribes are disappearing or people are disappearing can now be salvaged in such a way that you can actually build artificial agents that you can play with.

You can build different kinds of artificial intelligences that can keep up the traditions of these different kinds of nomadic tribes when it comes to, for example, playing in groups or bands of different kinds. So suddenly it’s not just about producing a lot. It is actually about producing with things that are disappearing. So I thought it was a really interesting niche example of how you can build a creative team to make sure that you preserve cultures that otherwise might risk disappearing.

Can Machines Be Creative?

Another old question, a really old question actually: Can machines be creative? This dates back to the 19th century and a text by Ada Lovelace. Ada Lovelace wrote a commentary on Charles Babbage’s Difference Engine. Or actually, she translated a commentary on Charles Babbage’s Difference Engine, and because she was a woman, she was perhaps not expected to write her own text, sadly, but she wrote footnotes in that text. And those footnotes were fantastic.

And in one of the footnotes, Note G, she speaks extensively about whether or not machines can be creative. Her answer is no. She says the machine can never originate anything. And since then, we’ve had a wide debate about machine creativity. Can it do something that we haven’t told it to do?

It’s still a good question, partly because we now realize that we can tell the machine to do a lot of things. Even if we cannot do them, we can tell the machine to do them. That’s what we’ve discovered in AI for science, for example. When Google DeepMind decided that it wanted to tell the machine to figure out how to fold proteins, no human scientist could do that. But you can tell the machine to do it by helping it learn about these proteins at a pace that no human being can.

Exploring New Creative Spaces

The new question, I think, is: What new kinds of creative spaces will we discover with AI? And this is a question about the space of creativity overall. It doesn’t matter what kind of area you’re in. You can think about creativity as a search through a predefined space of combinations and opportunities in different ways. It’s the space of all possible buildings. It’s the space of all possible books. Searching through this space in different ways is what we do when we’re creative. It’s a metaphor. It’s a model, but it’s a good model. And in that sense, it is actually a lot like a game.

And we know that AIs are really, really good at playing games because they can search the game spaces that we define in different ways. So what we find is that we can use AI—just like we explore the different moves in a game—we can explore the different moves in a creative field by using the AI. And this actually allows us to do something pretty interesting.

When you look at, for example, Go and chess: As we taught the machine to play Go and chess, we discovered something rather startling, and that was that the machine is worse if we teach them what human beings have learned over the last thousand years. So if we give them the games that human beings have been playing, it plays much worse. If we remove those and we allow the machine to play itself, it plays much, much better. But also differently.

One of the things you will hear consistently from grandmasters playing one of these machines is that it “feels as if I’m playing an alien”. Because what happens when we search the space of all possible chess games, for example, is that we climbed a little peak in the landscape of all possible games, and when we come to the top of that peak, we stop. We don’t know how to get anywhere else. The machine, however, can look at the entire landscape and find peaks that are much higher. So it plays a really weird game of chess and a super weird game of Go as well that people feel is alien.

Now, maybe what you can do is also do this in creativity and ask yourself: Is it possible to create alien cultures with this kind of technology? And indeed, research is looking into this by looking at what we consider to be cultural artifacts or creative artifacts in different ways. And by decoupling the expression and trying to figure out what the different dimensions are of creative spaces, it’s actually possible to build AIs that explore the space much in the same way they would explore chess and ignore all the human creativity that we’ve done so far. And look across the space and see: Are there other things that are really interesting in this space that we can work with?

And so this notion of a cultural alien sampler—a way of open-ended art generation balancing between originality and coherence—is interesting because when you do this, you can’t just randomly search this space. Randomness is not art. Randomness is not creativity. You have to find some parameters that you look for in the space.

And what they are trying to do here, at least, is to say, okay, it’s originality. It has to be something that hasn’t been done before. It has to have a certain coherence. It has to allow for us to impose meaning on it in some way. If we can’t do that, it just doesn’t matter to us. It’s nonsense.

So this notion of exploring creative space to find alien cultures is kind of an interesting way to think about it, rather than to just ask, can machines be creative? We can say, look, we would like to use the machines to explore the space of all possible books, all possible art, all possible creations.

Labeling AI Content vs. Signaling Human Creativity

Another old question is this: How can we label AI content better? We have worked a lot with trying to do watermarking in different ways so people know this is AI, and it’s an important piece in the fight against fake information, and it’s something that should be done. And we’re making a lot of progress on it.

Now, one of the things that’s really interesting, though, is that we’re now trying to think about this: How can we signal human creativity better? And this idea of labeling that something is actually done by a human—you know, “written over three years in solitude, crafted with ink and paper, by a nature’s rhythm, a human creation”—it seems bizarre. But what’s happening here is that creativity, human creativity, needs a way to signal that it is human in a world where the majority of creative work is going to be artificial.

Because if you just look at the volumes, the fraction of overall created things that is human is shrinking super fast. It’s not even in a majority anymore. Take podcasts. Did you know that more podcasts have been made with NotebookLM, this AI tool that Google released, than have been done by any human being? Which means that artificial podcasts are now in the majority when it comes to creative podcasts.

The Return of the Author

So in a sense, this is the return of the author. We used to debate in literary theory at university, when I was there, that the author is dead. When you read the text, you can just read the text. You can’t actually talk about the author. Well, maybe. But now actually, the author is returning. Because what you want is the work with the narrative that attaches to it that gives you the sense that the work is human. You want to figure out the human context of this work.

And it’s not just a return of the author. It’s actually also the return of something else that’s quite interesting: That’s death. Because humans die, which means that the creative output we have access to is scarce. And that scarcity creates value. There’s actually no real value in an infinitely generative creative engine that can create any manner of things.

The Emmy (EMI) Example

David Cope was a pioneer in artificial intelligence and music. He died this year. One of the things that he did was he built an engine called Emmy that could make music in the style of Chopin, Bach, or Liszt. And what he discovered was that people didn’t value these pieces because he could put it in his cellar and it could put out 200 pieces a minute of Bach-style compositions. And some people say, well, they’re not really good, but some were really good. So he could do that.

But because of the volume and the lack of scarcity, people did not value them. So what did he do? He killed it. He deleted all of the archives and he deleted the databases, because only if I kill Emmy will its productions actually have worth, because only then will there be scarcity. So he deleted all of those files.

The reintroduction of death as the ultimate value proposition in art is a really interesting thing. There is already labeling being done. This is from AILabels.org. I love that: “Made by humans,” “Made primarily by AI.” And there will be many more labeling attempts like this, many more ways of doing this. But I think the most important part is that the creator and the creative are melding together into a single narrative.

Original vs. Participatory

Old question is: What counts as original? Is a work original or not? That’s sort of at the heart of how we think a lot about the law of creativity.

The new question is: What counts as participatory? Because creativity, if we go back to this image that we had from the Romantic era of the author creating it and handing it to us, then we were all consumers, and the creator was a sole individual giving us his original work.

Now what we see, of course, is the rise of a very, very different kind of creativity, especially in the commercial space. So there are a couple of different trends here that are interesting. One of them is that we’re sort of moving from fan fiction to fan co-creation. Fan fiction is a phenomenon that we sort of blithely ignore because we think, oh, it’s fun—somebody wrote a story in the style of Harry Potter. But it’s a huge creative phenomenon.

The Age of the Universe

The other thing that we see now is that we’re in the age of the universe of the work. We talk about the Marvel Universe, the DC Universe. And all other works are also being organized into universes that can have multiple creators. You can participate, and you can write your Star Wars story because you are going to be the director of your Star Wars movie, but you’re in the Star Wars universe.

And this idea that what we’re creating is actually not the individual work anymore—it’s not the individual thing, but it’s in a universe of creations that belong to some kind of core narrative—it’s also a shift that we’re seeing in creative practice that’s quite interesting.

Real Events Art

And the other thing we see is the rise of real events art. People are super interested in real events, more now than ever before. The statistics are kind of interesting. You just look at the percentage of movies that are produced based on real events. It’s just bizarre how it’s going up. We seem to be more and more interested in reality. Maybe that’s because reality is more and more scary and everyone wants to know what the fuck is going on. And that could be one of the things, right?

But the other thing is that there is something about this where we feel that we can participate in reality. Reality and real events are connected to us in some way.

However, if you look at money and you look at what actually makes the most money, it’s not original screenplays and it’s not even real events. It’s existing material, which is the universe point I made. So that’s essentially where you find comic books turned into movies or other things turned into movies. That’s what makes the money. That’s where all of the box office revenue is.

This notion of the universe as a major generator of revenue in this new world is interesting. If it’s possible to create a universe around the kind of creativity that you work with, that’s an advance.

Archive of Our Own

And we spoke a little bit about this notion of fan fiction. If you haven’t looked at this one, it’s kind of interesting: Archive of Our Own, or AO3. Look at the numbers. There are more than 72,000—75,220 fandoms, and each fandom is a kind of universe in which you see stories written, people doing imagery, all of those different things, where you participate in this creativity.

There are 16.2 million works here. All of them, surely not stellar. It’s not as if I’m arguing quality here. But I’m arguing practice, participation. And the difference of the sole creator versus participatory creativity.

Copyright, Litigation, Legislation, Licensing

Another old question that is sort of interesting is: But what about copyright? And it sort of—it’s a question that always comes up every time we have a new technology. And the pattern is always the same. The pattern is first litigation. There’s lawsuits all over the place. The lawsuits currently are trending in favor of the creators. If you look at the last ten lawsuits, the space that they carve out for fair use, which is the American term for what you can do, is shrinking.

So we always first have litigation. And then once the litigation settles, we propose legislation. And after we have those two things done, it all goes to licensing. So it’s litigation, legislation, licensing. That’s what happened with the internet. That’s what’s going to happen with AI. We’re currently in the litigation phase. We’re going to move into the legislation phase shortly. And hopefully we’re going to move to licensing faster because I think that’s better.

What About Value?

That question about copyright, though, includes what I think is an interesting question: What about value? How is value created in a world where you have artificial intelligence? What new kinds of value can you create? And there’s a series of interesting examples of it.

One of the things you think about is that, well, AI is sometimes thought about as a source of creative works. What about AI as an audience? What about AI consuming the things that you produce as art? That seems like a weird idea, right?

Anna Ridler’s Data Sets

So this is the example of Anna Ridler. She is an artist from the Netherlands, and her work of art, her area of creativity, is data sets. She crafts unique data sets, and those unique data sets she then offers to GANs—generative adversarial networks, AIs. And those AIs can then generate on the basis of those different kinds of images, and those images she then puts out on the blockchain so that they’re scarce. And she holds tulip auctions for the tulip images she has every day where you can buy a tulip. You can watch it for ten days and then it disappears forever.

So scarcity, art made for the consumption first of the AI and then of the human being, and rethinking how value can be produced in this world where artificial intelligence becomes more than just a producer but also to some degree a consumer of these works. How do you write for, how do you draw, or how do you produce for a population of intelligence?

The Duck-Rabbit

In all, I think that artificial intelligence is one of those things that when we look at it, we sometimes see a threat, sometimes an opportunity. It’s both, of course. It’s always both. It’s a duck-rabbit.

This is one of my favorite—my favorite pictures, for two reasons. One is the joke. I think the joke is fantastic. Which are the two animals that are the most alike? And people go like, oh, well, maybe, you know, cats and tigers. No, no, it’s duck and rabbit. Because the duck-rabbit that you see here in the background, you can see it either as a duck if you look at it that way or a rabbit if you look at it that way.

And to a certain degree, that’s exactly what artificial intelligence is too. It’s a duck-rabbit.

Krebun

6.19MB ∙ PDF file

Download

Download

This week’s note is an essay is co-authored with Fredrik Erixon and continues the work he and I started in a previous essay on the health of the public sphere. In that essay we asked what characterized that health, and here we explore where the public sphere fails, and why. As always the essays are our private views only. Enjoy!

Where is the public sphere getting sick?

What is a healthy public sphere? In the abstract, most people in liberal democracies would think of it as one with a plurality of views and an inclusive culture while respecting a high standard of truth. And today, many would also add: a healthy public sphere is like the one we had in the past. Their impression is that our healthy past has been succeeded by an unhealthy present. And the obvious culprit is modern technology – or, especially, social media. They have made us lonely and angry, polarised the public debate to the breaking point, and allowed the public sphere to be infused by lies and manipulated propaganda.

Is this a good way of thinking about a healthy public sphere? We think not. In a previous essay, we argued that the public sphere – like life in general – will always be defined by parallel trends of health and unhealth. Efforts by governments and others to regulate the discourse of truth, or the squares where debates happen (e.g., platforms), in the spirit of restoring health are prone to failure. For those valuing the liberties in liberal democracies, these efforts also rest on awkward thought. We said that an important feature of liberal democracies is that they have institutions who are not part of executive government that are trusted and generally reduce suspiciousness in the public discourse. These institutions used to be the courts, media, and universities, but they are far less trusted today, partly because they have diverged from their own standards and methodologies for helping societies to operate within defined boundaries of truth and reasonable disagreements. Generally, we concluded, a good way of thinking about a healthy public square is to explore ways of managing unhealth – and especially suspiciousness.

Where are we getting sick?

The question we are now asking is more spatial and less about definitions of health and unhealth: where in the public square, or society, is there a problem with unhealth? Or to put it differently: where are we getting sick

Many people will have a ready-made answer: it’s on X (Twitter), Facebook, or other platforms where lies and falsehoods, “misinformation” and “disinformation”, spread like wildfire. Others may disagree, especially those on the political “new right”, and say that it is rather “old” media like public service broadcasters or old broadsheet “papers of record” that is the source of unhealth. Their response, which may extend to the universities, has become equally ready-made.

However, both attitudes are jaded and leave many questions unanswered. One missing important observation is that the public sphere has never been a monolithic space but always tiered and fragmented to serve different identities, pursuits, and patterns of behaviour. For the analytically minded, it is impossible to understand the public sphere if the assumption is monistic and the task is to discover or locate THE pursuit that defines the public sphere. It is likely wrong – and, with technological and societal change, increasingly wrong – to think about the public sphere as one pursuit or one effort, serving one democratic purpose. In the first place, to understand where we are getting sick in the public sphere we need to understand immaterial layers of human action in the public sphere.

Anthropology and sociology can help to put some light on the issue. People have different patterns of actions depending on the task and the arena: there is even a difference in intentions. In Clifford Geertz’s classic work, The Interpretation of Culture, he argued that body and mind, reason and biology, sit alongside a distinctly human desire for living in systems of meanings. He said in a famous quote that “man is an animal suspended in webs of significance he himself has spun”, and this seems to be a good starting point for layering different contexts – or webs of significance – that have different pursuits and outcomes, and that operate under different rules.

We suspend certain aspects our biology, but the exact form of suspension depends on the web – and such webs are contexts that we need to understand in more detail. Moreover, people resolve frictions and disputes in different ways depending on the context: people are generally good at pragmatic negotiation with others if they are clear about the context and what mechanisms that culture and norms prescribe for constructive resolution. What are these webs or contexts? One way to divide them up is the following.

Private contexts

We all interact with lots of people that think differently from us. In most instances, very few have any problems with most of our encounters. We move together with other people in so many parts of our life without having issues with understanding each other. We follow established behavioural norms and, in most cases, refrain from thinking that another person’s political viewpoints are imperative for the encounter to happen or continue. Almost everyone can chat to someone about sports or the weather or daily events on a tube or at a bus stop in agreeable terms. Most people interact with colleagues at their workplace based on getting things to work smoothly and for people to be able to conduct themselves with satisfaction and efficiency. These skills are usually inter-personal, intergenerational, and international. It is notable how many of the private contexts are free from all the faults of a polarised society.

These actions and interactions are agreeable and non-polarising because they are essentially about coordination. In fact, a lot of human interaction is about coordination: material and task coordination, for instance. Obviously, we engage in symbolic coordination of different moments: talking to a stranger in public transport gives some form of meaning to the moment and coordinates an experience. Private contexts involve a variety of physical spaces and many different pursuits: they have in common that they help people moving together as a group, and – ideally – getting better at it.

Notable is also the existence of clear conflict resolution mechanisms in most private contexts. In the workplace, or in other private contexts, we know how to solve our conflicts – and we are incentivized to solve them because we are coordinating for a common cause of some kind. This will turn out to be surprisingly important as we discuss why other contexts are getting ill.

Tribal contexts

Some contexts have become primarily about expressing our identity and signalling various types of loyalties: to ideas, habits, groups, viewpoints, et cetera. The significance of this context lies not in the orientation towards coordination but to symbols and statements of meaning – sometimes by offering quasi-religious places of worship or the warm comfort of the tribe. It is a context that operates under certain norms and rules, and they generally work pretty well as long as interactors stay in this domain and, sometimes, in their lane. People can have significant disagreements with each other at the workplace, but as long as the interaction stays at the workplace problems are usually manageable – since we have access to clear conflict resolution mechanisms. Disagree? Well, ask a manager! However, if one person in a disagreement takes to X and posts about an idiotic colleague or a toxic workplace culture the interaction immediately enters another context. It gets charged not just by identity but differences over identity and why these differences matter. This is an example of a context violation – we push an interaction from one context to another, and when we do we change the terms of engagement and the conditions for resolving the problem.

Our tribal contexts have re-expanded over many decades. Obviously, social media plays a role: X, Facebook, and others created new contexts for identity signalling. But the re-expansion of this context started a lot earlier – and was clearly visible in mainstream media long before social media arrived. Different observers will point to different causes or indications for change. For some it is about celebrity culture, reality TV, and the personalisation of the public space that followed the expansion of mass media. Others may say that it is rather the result of deeper trends of individualism and new forms of social ascription that came with a declining role for traditional non-public spaces of identity – family and churches, for instance. It is notable, though, that many of the concerns that fuel the debate over social media are copies of concerns that previously have featured in similar debates. Pierre Bourdieu and Christopher Lasch are both thinkers that are referenced by critics of social media. But Bourdieu, who died in 2002, was predominantly occupied by vacuous entertainment media. Lasch’s most important book for the current criticism of social media was published in the 1970s. The year he died, Mark Zuckerberg turned ten years old.

Tribal contexts are the remnants of idealized democratic contexts – the idea that there is a single public sphere in which we engage as citizens in seeking out consensus through debate. As societies grew more complex, this public sphere fragmented into contexts oriented – primarily – around identity. This has nothing to do with social media, but with the fundamental complexity of modern society – a society in which the idea that there is a single on-going public conversation simply is an illusion.

The dominant method of conflict resolution in this context is simple: it is choosing which tribe you belong too, and then conforming to the positions taken by that tribe.

Veridicial contexts

Lastly, there is a specialized, and highly important veridical context – a web that is significant because it basically involves questions about truth. In short, this is a context for broad or narrow truth-seeking enterprises. This involves the world of science: scientific truths and the validity of scientific claims are an important part of it. However, truth can also be about non-science issues. For instance, it can be about truth in art or truth and religion, and it can extend to issues of law and ethics, about what is right or wrong. The rules here eschew fixed and final views – the basic rule is: no one has a final say! – but they lead to norms and standards for the truth-seeking enterprise. The veridical context is a context with an ethos and methods of weeding out propositions and evidence that do not stand up to scrutiny. Therefore, it is not a place just for universities and scholars: it may even be that, in some academic disciplines, their adherence to this context has weakened over time and shifted into tribal modes. It is also in this context we find other important institutions like courts

The key thing about this context is that it has institutionalized mechanisms of conflict resolution and people committed to adhering to them. Here conflicts are not solved by authority – asking the boss – or by tribal assertions of loyalty, but by a methodological commitment of some kind, and a belief in that method of conflict resolution.

Context violations

So where do these layers take us in our quest to understand where the public sphere is getting sick? In most societies, people act across all these contexts, even if there is a lot of individual variety. People also cross contexts while understanding that there are clear – yes, sometimes rigid – boundaries between them. This has been the practices for a long time: what distinguish humans is that the quickly learn and adjust to what rules that apply to certain contexts. Nor does it seem to be a major problem that people have conflicting identities or that they debate them within a context. This has always been the case in modern life. Actually, it is difficult to contend that the major problem with polarisation is that people have different political identities and that they use tribal contexts for identity signalling.

A more plausible thesis is that we are getting sick when contexts mix – or when different webs of significance get interwoven in ways that blur conflict resolution mechanisms. In other words, it is when contexts crash together that the health of the public sphere is weakened. These points of conflict are the areas of disease risk. Context violations are principally frequent in periods of strong political conflicts – they may even be one of the causes of discord. For instance, the revolutions of the 1960s in sex, music, clothes, and mores featured strong context violations between private and tribal contexts: it was a “conflict of generations” within families as much as in the public sphere.

We have a similar experience today with context violations – yet this time they are principally taking place between the tribal and veridical contexts. The debate about vaccines is a case in point. What is the best way of understanding the voices, people, and leaders that have given rise to an anti-vax mentality? In the first place, very few of them are active in the veridical contexts – seeking truth. The public spere generally features a lot of identity signalling about health, exercises, food, and the use of pharmaceuticals, and voices in that antivax or vaccine-sceptic space are often signalling a broad identity. The lion share of them is not really contesting science in a substantive way – say, having an evidence-based discussion about the germ theory of disease, and if it is true or not. Scientific arguments against their position do not land well because those signalling identity are engaged in a different enterprise. A notable part of anti-vaxxers rather seem to think they are making a statement against, for example, ultra-processed foods and chemicals in food. Others are signalling an identity based on family grief and suffering over unexplained and taxing neuropsychiatric disorders. A third group still rather seem to signal that they are open-minded people with a big spiritual vista: they are intellectually flexible rather than rigid. Very few of them are saying: “sure, call me a fool if you like when I am dismissing basic medical microbiology, but I know that I am right and that medical experts are wrong, and I have the evidence to prove it!”

They do not commit to any established, institutional method for resolving conflicts of evidence – but seem to deny that such methods are viable ways of deciding conflicts at all.

A context violation becomes inevitable for several reasons. The first, obviously, is because many anti-vaxxers are challenging medical expertise: they step over the boundary. These may not be veridical claims but a common accusation is that medical experts are frauds and/or have been corrupted by industry: they are paid to help companies sell their vaccines. The sceptics challenge the ethics of the scientific pursuit leading up to vaccines, or the recommendation of their use. They call into question the integrity of scholarship and the methods of conflict resolution in the veridical context.

A second context violation is that their opponents in the tribal context also cross into the veridical context to borrow authority for their identity. They are not inviting a discussion based on the rules that govern truth-seeking; many of them rather want to signal that they are enlightened, responsible, and part of the culture of science. Obviously, the real medical expertise must respond, too, because the avoidance of vaccines is a matter of life and death. However, the rebuttals of the experts are not having much impact on the vaccines-sceptics because they are responding within a conflict resolution mechanism not recognized by the opponents. Meaningful discussion is difficult without agreed conflict resolution mechanisms.

In this view, the health or unhealth of the public sphere in matters of vaccines have little to do with public knowledge. The share of the population with an informed understanding of microbiology and vaccines have likely not changed very much in the past, say, 30 years. The change in the public sphere is better understood as one of identity and webs of significance – and in choice of context in which we debate vaccines. Back then, it was more common that people expressed an identity with trust in authority. However, it was also more common that matters of vaccines, medicines, health, and food were discussed in private contexts. The domain coordinated knowledge, community, life, and family and did not involve making public statements. Boundary restrictions were more respected and, as a result, the public sphere was healthier.

On other issues, context violations are happening by the veridical context expanding into the tribal. Many scholars, scientists and other engaged in solid scholarship are careful about public communication and avoid expressing over-confidence in their own knowledge about truth. For them, science is a pursuit, a method, a truth-seeking process. However, an increasing number of academics have stepped over boundaries and cloaked identity-signalling in the banner of science and truth. As a result, context violations happen and the public sphere gets sick because, in debates over identity, one party say – rightly or wrongly – that they are allied with the truth. In fact, it is notable how many of the major debates in recent years have featured such context violations: debates over climate change and immigration, for instance, or gender and sex invariably include claims of truth and lies, right and wrong, virtuous and unethical thought. But rarely any commitment to an agreed mechanism for determining which side is right.

Most Western countries had a strong touch of such context violations during the Covid-19 pandemic: it was a festival of identity-signalling using science and propositions of truth to bully opponents. In the UK, debates over Brexit (was it good or bad? Was it based on truthd or lies? was it real or fake Brexit?) still end up in strong identity signalling – again without clear commitments to any method or process to resolve the conflict.

Is there a technological cure?

Technology and social media have played a role in expanding the tribal context: for our stone-age brain, social media can be like a hall of mirrors for narcissists. Social media is tribal infrastructure. However, it is equally clear that social media is not the bogeyman it is often presented to be. For instance, there is not much evidence suggesting that Russian bots and social media campaigns have swayed many voters in elections. Likewise, there is next to no evidence suggesting that people of moderate views are persuaded by extreme content to change their views. In fact, a good amount of research suggest that social media plays a negligible part in provoking polarisation, tribalism, extreme views, and other symptoms of unhealth in the public sphere commonly associated with social media. For sure, social media, algorithms, and recommendation systems can create echo chambers for some, but they also serve up contradicting thought – at a rate that did not exist in the past when choices of media consumption were more biased. If we are looking for more distinct places of unhealth in the public sphere, we need to sharpen the analysis.

One explanation to the insignificant effect of social media on the symptoms is that algorithms are not as powerful as they are billed to be. Just like social media has a lot of “dumb” advertising (tailored ads you are just not interested in) algorithms are not capable of serving us what we want at a certain point in time. Humans are creatures of variation and feelings: the tunes you listened to one morning are necessarily not the same tunes you want to listen to the next morning, let alone that evening. For most people, their politics tend to be all over the place, and just because you have eye-balled two clips about the nutrition standards of energy drinks on TikTok, for instance, doesn’t mean you are interested in a third, fourth, or fifth clip.

But what if we flip the debate: can there be technological cures to unhealth in the public sphere, perhaps some that help managing context violations? In fact, there have been many attempts exploring what happens in and between contexts. Obviously, social media companies have made several efforts to manage the health of the public sphere by muting certain posts and adding warnings to others that they sit awkwardly with bodies of established facts. Many specialised platforms have strived to increase public access to veridical contexts. Academic publishing sites, citation systems, the use of meta studies are all attempts to both make contexts more distinct and supply conflict resolution mechanisms. However, most efforts have failed to achieve much for the health of the public sphere or for managing context violations.

Wikipedia is an interesting example. It builds on an old idea of the encyclopaedia, a summary of all relevant knowledge baked into a book of truths. In Denis Diderot’s view, the encyclopaedia is a way to democratise knowledge and change the way people think. Diderot’s view is interesting: it combines a top-down concept of truth (after all, he, d’Alembert and others were supposed to select the anointed truths) with a bottom-up idea of diffused knowledge. The basic notion was that if we can change the way people think, we can change how they think of themselves and how they behave. Wikipedia started with the same idea just as Google defined its mission to organise all relevant information and make it accessible. Wikipedia, however, established a method for developing content which was based on decentralised human contributions and validations. Unlike Google, which used algorithms to catalogue information, Wikipedia aspired higher: its method would turn information into knowledge – make diamonds out of carbon. And – perhaps most importantly – it built an informal, grassroots conflict resolution mechanisms in between editors that expanded into an institution.

Unfortunately, Wikipedia’s model has become contested – not just by right-wing voices but also by others, even by some who helped to found it. It has been accused of various biases and as Wikipedia is also source material for AI models, the risk is that biases will proliferate as the use of Wikipedia will gradually be taken over by AI agents. While Wikipedia sits in the veridicial context – it reports knowledge, ideally valorised knowledge – it has not been able to defend against an encroaching tribal context. And yes, it is the context violations that have caused frictions and prompted critique. Elon Musk now says xAI is going to launch its own Grokipedia – and his supporters are not blushing when they claim this new repository of knowledge to be a source of “true knowledge” – but without the conflict resolution mechanism, solidly signalling that this is a tribal technology.

An alternative approach is perhaps to discard the monistic concept of an encyclopaedia with valorised truths alone and look at more specific services that concern boundary conditions of contexts – and, perhaps, the rules of engagement between them. For example, one approach could be to combine a Wikipedia-style structure with defined layers of reasonable disagreement and reasonable differences in interpretation of established bodies of knowledge. Neither the health of the public sphere nor the veridical context is helped by the notion that science or Wikipedia or Grokipedia present “the truth, the whole truth, and nothing but the truth”. Obviously, there are areas for basic reporting of knowledge. However, a great amount of reporting is also necessary for reasonable disagreements.

What is needed is the building of real, robust technologies of disagreement and conflict resolution that can connect contexts in ways that a majority accepts. How do we support reasonable disagreements, and build the infrastructure for it?

Such an approach start with ideas and scholarship by thinkers like Friedrich Hayek and John Rawls. Rawls especially explored the concept of reasonable disagreements and identified six good reasons for observing them: 1) when the evidence is complex and sometimes conflicting, 2) even under certain forms of agreements, people can disagree about the weight a specific consideration should be given, 3) many concepts we use as humans are ambiguous and can be undermined by specific evidence, 4) we cannot entirely dismiss our own experience when we assess evidence and put weights on specific pieces of knowledge, 5) if an issue is charged with opposing normative conclusions, it is difficult to provide an overall assessment, 6) many difficult decisions do not have an easy solution staring it in the face. This is a good starting point for understanding reasonable disagreements and, also, how context violations could be managed. Many of these violations do not include binary situations with one proposition that is obviously true and one that is obviously false: they usually operate in the space where disagreements are reasonable.

What we need is to support that space with tools and technologies that allow us to explore the disagreements more clearly, and agree on methods of resolving them – or recognizing them.

Wikipedia did work out methods for observing reasonable disagreements: it is partly reflected in its work, method, and output already. But the task extends far beyond Wikipedia and includes media (old and new), schools and universities, platforms, museums and theatres, government authorities, and others. Context violations are inevitable, but they can be managed a lot better if people are helped to understand what it is reasonable to disagree about.

Most likely, although this may seem controversial, they can be managed a lot better if they are organised by technology than by humans.

Let’s explore what we, somewhat grandly, could call aporetic AI. Imagine an AI-mediator that forced context violations into dialogues, and provided participants with the means to sort out what resolution mechanisms they have available to them. Guiding us through arguments and disagreement, providing alternative means of conflict resolution, such technology would open a space for engaging across contexts. This is not a new idea, of course. In fact, dialogues have always been the way to manage context problems and violations, and they help to reduce the the tribal context. The health of the public sphere is most likely going to improve if more issues could be brought into such a new dialogical context. If you make a strong claim and borrow veridicial authority for it, you should be prepared to accept a dialogue that help to structure a disagreement with relevant mechanisms of conflict resolution.

There is a broader observation here: may it be easier to have AI help us to resolve conflicts than trusting humans to achieve that result? In his famous essay An apology for Raymond Zebond, Montaigne asked if people are afraid of nature because it is non-human or all too human? The same approach is useful for understanding anxieties over technology, social media, and AI poisoning the public sphere. What we should be afraid of, perhaps, is not that AI is non-human but that efforts by man (e.g., regulators) can make it all too human.

The point of using AI to manage contexts is that is not signalling identity and avoids a feelings-based approach to truths. Therefore, it can be better prepared than humans to define contexts and allow for managing disagreements.

AI has already proven to be a good tool for using dialogue to reduce the belief in conspiracy theories – in theories, for instance, that the US election in 2020 was “stolen” or that Covid-19 restrictions in Germany had sinister origins. In a world where many more spaces are influenced by identity-signalling – schools and universities, theatres and museums, courts, old and new media – the best way to reduce unhealth in the public sphere is perhaps to invite everyone for a dialogue that is less human.

Fredik Erixon & Nicklas Berild Lundblad