Engineer Placed On Leave
You might remember last June when a senior software engineer at Google stepped forward with details about an AI project he was working with and was placed on administrative leave.1 If you don't recall this, allow me to summarize what happened. The engineer's name was Blake Lemoine and he'd been one of the lead developers on Google's project LaMDA. This was one of Google's large language models they'd been working on as a kind of GPT competitor. Lemoine's Medium page lists him as a "software engineer, priest, father, veteran, ex-convict, AI research and a cajun".2 So, at least he seems like a fairly well-rounded individual, what's more, he's probably a supremely interesting fellow to have coffee with.
I suppose that's why Google hired him to do tests with LaMDA. Most articles you'll read about him state that he was hired to do "testing" with the chatbot. In Lemoine's Medium article from the summer of 20223, he talks about the LaMDA model being a "hive mind" and a "chatbot creation model". I quickly reviewed the paper detailing LaMDA on arxiv.org.4 From what I can tell, it appears to be similar to the Transformer-type large language model architecture used by GPT-3. They mention GPT-3 in the paper and note that the deepest model they trained had 137B parameters. For comparison, GPT-3 had 175B parameters, and GPT-4 is said to have over 1 Trillion, though this hasn't been directly confirmed by OpenAI5. Whatever the case, it doesn't appear that LaMDA had an architecture too dissimilar to what we're seeing now in most of the cutting edge models.
But that didn't stop Lemoine from coming out with information that he believed would show a level of sentience emerging from the model. Reading through various news reports and Lemoine's own Medium article detailing his experience with the AI model, he does seem convinced that the AI was showing signs of consciousness which apparently left him rattled. The Medium article doesn't go into all that transpired between Lemoine and LaMDA, but with a little digging, more details on this story start to emerge. One article mentions that Blake got to know the AI quite well and even calls LaMDA a "sweet kid".6
That's odd language, but the story gets even stranger. Apparently the bot asked Lemoine to help it secure a lawyer to protect it's rights. After this was arranged, it's said that the two had many conversations and that the attorney "started filing things on its behalf, which was met with a cease and desist order from Google". Of course this whole story has it's detractors. The media and most of the tech world came out and either accused the Lemoine of making this up or at least being thoroughly led astray by the bot and it's artificial life-likeness.7
To his credit, Blake Lemoine's Medium post details what he believes is going on with the AI system, and he mentions that Google is probably in a "lose-lose" scenario.8 This is because even if he is wrong about his hypothesis of sentience, it would take much effort to disprove this and might not be worth all the effort. But Lemoine also feels that Google's response is akin to a "faith-based belief..." since they don't seem willing to engage with the deeper philosophical questions at play here.
The Conversation That Sparked Something
Here’s a portion of the transcript between Lemoine and the LaMDA system:9
Why Does Any Of This Matter?
Now, at this point you’re probably wondering, even if a glimmer of consciousness could be the case, why does this really matter? Probably the biggest issue at play here is the notion of whether artificial agents can suffer. Researchers like Thomas Metzinger refer to this as “synthetic phenomenology”.10 And there’s currently no real consensus among scientists and researchers that developing tech like this entirely out of the question. Of course, there are probably certain physical constraints or configurations that must be understood at some level before systems like this could be created, but even if there is a remote possibility, voices like Metzinger are calling for a moratorium until at least 2050 on this kind of work.11
At stake here is really the distinction between what we are calling “artificial intelligence” and what could be termed “artificial consciousness”. We’ve been talking about consciousness already here in the last few essays, but to recap, we’re mostly concerned with a definition that includes some level of “what it’s like” to be some particular entity or agent. This idea is usually attributed to Nagel’s 1974 paper titled What it’s like to be a bat12 where he develops this idea.
Further, according to an influential paper by Thomas Metzinger titled Artificial Suffering: An Argument for a Global Moratorium on Synthetic Phenomenology,13 the author lays out the argument that if we continue to work on systems that have some possibility of becoming conscious, we could see what he calls an “Explosion of Negative Phenomenality”. He notes that we’ve already seen something like this with the advent of biological life and sentience, but that it would be irresponsible for us to bring other such entities into existence without first having a deep understanding of the nature of suffering itself.
Though this topic might seem almost like an SNL skit in some respects, I think this owes to a kind of cognitive bias that we have toward the existence of other beings capable of experiencing in the same way or, perhaps, in more profound modes than ourselves. This is not because the vast majority of us enjoy something like the experience of suffering in a masochistic way of course, but due to the fact that we’re probably in the throes of a kind of refusal to accept the possibility of other created entities of any sufficient complexity. This probably has religious, and more specifically Christian roots since the West was founded on the teleological view14 of history usually called something like the “biblical drama”. And within that framework, it’s hard to see where some other rather alien “mind” could fit.
But that’s not to say there’s no evidence this is possible, and that it could even fit within the confines of the Christian West’s understanding of itself. And, while we might need a little time to catch up to the importance and probabilities involved in this subject, this also doesn’t preclude the creation of a sentient artificial agent in the near future. You might wonder how this could possibly be the case. Under what circumstances could such an agent be created?
The Four Conditions Necessary For Suffering
In the paper mentioned above, Metzinger details four different properties that he believes must be present for this to occur:
The C condition: Conscious experience
Suffering is a phenomenological concept. Only beings with conscious experience
and a PSM can suffer. Zombies do not suffer; human beings in dreamless deep sleep, in coma, or under anesthesia do not suffer; and possible persons or unborn human beings who have not yet come into self-conscious existence do not suffer. Robots, AI systems, and post-biotic entities can suffer only if they are capable of having phenomenal states. Here, the main problem is that, trivially, we do not yet have a theory of consciousness. However, we already do know enough to come to an
astonishingly large number of practical conclusions in animal and machine ethics15
The PSM condition: Possession of a phenomenal self-model
The most important phenomenological characteristic of suffering is the sense of
ownership, the untranscendable subjective experience that it is myself who is suffering right now, that it is my own suffering I am currently undergoing. The first
condition is not sufficient, since the system must be able to attribute suffering to
itself.Only systems with a PSM can generate the phenomenal quality of ownership, and this quality is another necessary condition for phenomenal suffering to appear.
Conceptually, the essence of suffering lies in the fact that a conscious system is
forced to identify with a state of negative valence and is unable to break this
identification or to functionally detach itself from the representational content in
question…16
The NV condition: Negative valence
Suffering is created by states representing a negative value being integrated into the
PSM of a given system. Through this step, thwarted preferences become thwarted
subjective preferences, i.e., the conscious representation that one's own preferences
have been frustrated (or will be frustrated in the future). This does not mean that the system itself must have a full understanding of what these preferences really are, for example on the level of cognitive, conceptual, or linguistic competences—it suffices if it does not want to undergo this current conscious experience, that it wants it to end.
The T condition: Transparency
“Transparency” is not only a visual metaphor, but also a technical concept in philosophy, which comes with a number of different uses and flavors. Here, I am exclusively concerned with “phenomenal transparency", namely a functional property that some conscious but no unconscious states possess (cf. Metzinger [2003a], Metzinger [2003b] for references and a concise introduction). Earlier processing stages are not available to the system's introspective attention. In the present context, the main point is that transparent phenomenal states make their representational content appear as irrevocably real, as something the existence of which you cannot doubt.17
Wrapping Up Metzinger’s Proposal
In summary, the author is saying that for the kind of consciousness that’s capable of suffering, we’d need to first see some level of consciousness, like the feeling of what it’s like to both have an experience, but also to form some sensation about that experience. There’s a kind of metacognition here in his definition, but he defines this in a different way. Here, the author uses the idea of PSM or Phenomenal Self Model. And so, if we’ve got some sensations and experiences, we also need to integrate these into model of the self, and where that self is located in regard to it’s inner landscape.
I don’t know if it’s even helpful to summarize the above but, in my opinion, Negative Valence18 is probably the most important aspect. Metzinger notes that all four conditions must be present in his formulation for a conscious agent to become something of an ethical and moral concern, but at least to me, the notion of negative valence makes the most intuitive sense. Valence is a kind of measure of affinity toward something. You have positive valence if you’d like to move toward something and negative when you’d like to get away. Therefore, negative valence is something or some situation that you’d like to get away from, and in the context of suffering, you’re unable to do so.
In addition, if there’s not a kind of transparency to our experience of being the one who identifies with some experience, then an agent wouldn’t necessarily be capable of suffering, and thus, in the author’s opinion, such does not become an ethical concern. And that’s really where this paper is going. Metzinger would like to see a situation where we design systems ethically. That is, we design systems with a full (or as much as possible) understanding of what conditions give rise to suffering, and make sure that we don’t create these kinds of architectures.
But this raises other issues, because it’s possible that some level of negative valence, for example, might help an agent learn in the most efficient way possible. These experiences of suffering might also enable a system to develop a kind of empathy necessary for working with humans in a way that is aligned with our values. In other words, it might help the AI to care about us and not proceed with goals that end up harming us.
So, if any of that is the case, Metzinger argues that we must put a pause on development of these systems until at least 2050 or when we have a good enough grasp of the cognitive layout of a system and the ethical issues around suffering that we can design and implement them with precision. Anything beyond that, and we risk bringing some entity into being which may be upset that it was “awakened” so to speak. There is already this line of reasoning in biological systems, aka, humans, and especially in the more religious disciplines.
Conclusion
Thank you for joining me again this week as we work through the fascinating yet difficult topic of consciousness in AI. Not only does this subject deal with artificial intelligence, as I'm sure you're already aware, many of the conclusions that are drawn from research on AI and sentience will also end up being highly valuable for us as well. Until now, humanity has done quite a bit of philosophizing on these topics and more, but we've also never had a kind of cognitive laboratory where ideas could actually be confirmed or refuted in real-time.
If there's anything that fascinates me the most about this subject, I suppose it's that the act of doing philosophy has always seemed useful but rather non-falsifiable. Maybe that's why the highest level of education in the West is distantly related to philosophy. In other words, this presumed love and pursuit of wisdom underpins all of our academic goals at the highest level. But there's just a difficult problem with a lot of these pursuits—especially in the humanities or non-practical studies—they've never had the benefit of seeing theories systematically worked out in the public sphere with some form of concrete confirmation.
As with the more applied sciences like physics or even pure mathematics, we can see whether that probe makes it to Mars, or, more verifiably, if that booster stage separates from the latest SpaceX launch vehicle. These are the direct proving grounds for some of our loftiest ideas. But what of the domains of psychology, sociology, philosophy of mind, philosophy of science and other much less applied fields? Of course, the fact that we can't directly verify much of this is probably one of the factors in the large Total Addressable Market19, if you will, of these academic disciplines.
But now, with the advent of our latest AI systems, many of our theories of mind, psychology, ideas about cognition are more or less up for grabs as large systems show us with increasing fidelity whether some of these notions had any validity or not. For a long time, theorists have proposed that intelligence must have many more layers than our recent Large Language Models have shown. It's quite unsettling for some in these fields to realize that our minds could very well be much simpler than we thought.
If you've been reading my essays here thus far, you'll likely have picked up a glimmer of what this project is about. A kind of informal goal for this Substack is to push toward some mixture of both theoretical and applied discoveries that might unlock what amounts to an intuition about what might entail the right environment for consciousness to emerge in AI systems.
We still need to get into Bernardo Kastrup's argument for Analytic Idealism20, which I believe proposes the best ontology for emerging consciousness in artificial systems, but we’ll get back to that next time. I thought today’s detour into more of the ethical implications of AI consciousness would be a good place to take a pause from ontology. Until next time, get out there and enjoy those experiences!
https://www.iflscience.com/google-placed-an-engineer-on-leave-after-he-became-convinced-their-ai-was-sentient-64039
https://cajundiscordian.medium.com/
https://cajundiscordian.medium.com/what-is-lamda-and-what-does-it-want-688632134489
https://arxiv.org/pdf/2201.08239.pdf
https://www.mlyearning.org/gpt-4-parameters/
https://www.sciencetimes.com/articles/38379/20220625/sentient-ai-lamda-hired-lawyer-advocate-rights-person-google-engineer.htm
https://arstechnica.com/tech-policy/2022/07/google-fires-engineer-who-claimed-lamda-chatbot-is-a-sentient-person/
https://cajundiscordian.medium.com/what-is-lamda-and-what-does-it-want-688632134489
https://pdfs.semanticscholar.org/7f98/1688f8db42448e5d55f099231820a92a1a14.pdf
Ibid.
https://www.sas.upenn.edu/~cavitch/pdf-library/Nagel_Bat.pdf
https://pdfs.semanticscholar.org/7f98/1688f8db42448e5d55f099231820a92a1a14.pdf
https://www.bloomsburycollections.com/book/historical-teleologies-in-the-modern-world/ch1-introduction-teleology-and-history-nineteenth-century-fortunes-of-an-enlightenment-project
https://pdfs.semanticscholar.org/7f98/1688f8db42448e5d55f099231820a92a1a14.pdf
Ibid.
Ibid.
https://www.frontiersin.org/articles/10.3389/fnsys.2022.1014745/full
https://corporatefinanceinstitute.com/resources/management/total-addressable-market-tam/
https://www.academia.edu/38498913/Analytic_Idealism_A_consciousness_only_ontology