Temperature Check: Recursive Self-improvement and the Singularity

A review of the literature on techniques that enable AI takeoff

NEW: AIGC (AI Generated Content) Score for this essay → 97.5% Organic

Going forward, I’ll include a score on all of my essays that detail my use of AI content. In the near-term, I’d love to see more creators adopt something like this for a more transparent look at the origin of the content we consume.

Essay text: 100% Organic

Analysis: 95% Organic, 3% You.com, 2% GPT-4

Intro

I'd like to thank all of the new subscribers and offer a generous welcome to CogitoAlgo. Our society, and indeed most of the developed world, is undergoing a seismic shift in how we understand ourselves and our relation to domains such as labor, production and intelligence on the axis of technology. Not only that, but the very notion of what will be available forms of labor and production in the near future are now being called into question.

But it’s not just limited to this scope. These merely stand in the path of disruption by recent advances in technology. What’s fully at stake is probably closer to the very core of what it means to be human, our future, and how we fit into the cosmos.

Issues like these, and more, stem from recent advances in AI and the ramifications such technology is having now and will continue to have in the (near) future. Never before have questions around this kind of technology been quite so apropos as they are today. You've no doubt found yourself reading this substack and other sources in an effort to stay up-to-date with what's going on.

The rise of ChatGPT caught many off-guard as the quickest application in history to reach 100 million users. After just two months, OpenAI's chatbot reached the 100 million user milestone. To put this into perspective, TikTok reached the same level in over nine months and both Google and Facebook burned through five years each to get that many users.¹

Exponential Growth as Singularity Proxy

This kind of growth is exponential, and its probably a good intuition for what something that’s generally called the “technological singularity” might feel like. You may already aware of this idea, but if not, here's a good intro:

The technological singularity is a hypothetical future event in which technological progress becomes so rapid and exponential that it surpasses human intelligence, resulting in a future in which machines can create and improve upon their own designs faster than humans can. This could lead to a point where machines are able to design and build even more advanced machines, leading to a runaway effect of ever-increasing intelligence and eventually resulting in a future in which humans are unable to understand or control the technology they have created. Some proponents of the singularity argue that it is inevitable, while others believe that it can be prevented through careful regulation of AI development.²

Over the years there have been many predictions about an event, or a series of events, that lead to some kind of technologically-mediated tipping point for humanity. And while there are those that are actively working to bring this into reality eg. futurists and transhumanists, there are others that remain skeptical that such a milestone could be passed without irretrievable, or perhaps, existentially dire consequences for the future of humanity. In other words, if we don't pass this next hurdle very cautiously we might end up either extinct, a race of slaves, or worse³. Much like the Wachowski Brother's vision of humans as nothing more than useful energy producers in The Matrix.

Such a dystopian future is obviously quite bleak, and it does seem as though we might be racing toward a similar outcome with reckless abandon. There are many different thought processes around a future like this, but much of it seems to revolve around intelligent systems that are capable of what's generally called recursive self-improvement⁴. Recursion is an important idea in software engineering, and other fields such as mathematics, but the basic understanding is a function or process that calls itself.

Understanding Recursion

Many software engineering problems are best solved with a recursive function. Calculating the factorial of a number is a good toy example. While this kind of overly simplistic function isn’t too common in most production software, it can be useful to help us see how recursion works. Even if you've never programmed anything before, an easy example in Python⁵ should at least give you the basic idea:

def factorial(n):
    if n == 1:
        return 1
    else:
        return n * factorial(n-1)

Let’s break this down. We start by taking a number as parameter "n", and if it's equal to 1 we'll return 1. Otherwise, the function will call itself again but this time having subtracted 1 from the original number, multiplied by the original number. The function will repeat this until it returns the full factorial for the original number. In other words, this function will multiply all of the positive integers from 1 to n, including n itself.

Does that make sense? Don't let the fact that it's code throw you off. You might be tempted to think this looks a lot like that class you hated in high school, but just taking it at face-value, it's really quite simple. Some of the most powerful concepts we know are also simple—like some of the foundational math involved in training AI⁶. This is recursion—the calling of a function from within itself.

Here's a more in depth definition:

Recursion means "defining a problem in terms of itself". This can be a very powerful tool in writing algorithms. Recursion comes directly from Mathematics, where there are many examples of expressions written in terms of themselves. For example, the Fibonacci sequence is defined as: F(i) = F(i-1) + F(i-2) ⁷

Okay, so that's a relatively easy concept to grasp, but you might wonder how this is related to AI and the singularity? Basically, imagine the idea of the recursive function above, but scaled up to an entire AI system, or group of systems acting together. What if something like ChatGPT were able to run recursive functions on itself?

Frozen at Training Time

Currently the models used in the GPT-type lineup have their programming basically "frozen" by the time the regular users interact with them. If you're already aware of how these systems work, please excuse the following rather naive example. An LLM model like GPT (Generateive Pre-trained Transformer) is basically given a bunch of text data and it tries to "learn" structures and patterns in the data, which it then uses to predict what word will come next when given a prompt.

In other words, ChatGPT is really good at predicting what sounds like the most statistically probable thing to say in response to whatever you ask it. But the way that it "learns" these patterns and structures is by keeping an immensely huge mathematical matrix (multidimensional array) of "weights" that help it to predict what is most relevant. These weights, or basically numbers representing the probabilities of structures that its found during training, are the model. So, when we speak of the system being "frozen", it's because these weights have already been learned and won't be changed again unless it's re-trained.

What this means, practically speaking, is that systems like ChatGPT, Bing, or Google's LaMDA don't continue to "learn" anything once they've been trained, with some small caveats. One of these is a prompting strategy where the agent (like GPT) is told to give an answer and then criticize it's own output. In a recent paper titled Language Models can Solve Computer Tasks from March of 2023, this approach is detailed as "RCI", or Recursively Criticize and Improve. The paper goes on to detail how this approach coupled with something called CoT or "Chain of Thought" can enable the agent to "perform better than either separately"⁸.

What Does It Take to Get There?

But this kind of improvement will not usher us all the way to the singularity. Even though they use the term recursion in their explanation, this is not the same type of recursion that would enable a system to scale itself beyond human reach. I’ll give you a quick second to guess what kind of recursion we would need for an out of control explosion of advancing technology. You got it—the simple recursive-style function we looked at earlier. Now, it’s not that simple, but the basic idea remains the same. If a system is able to recursively run its own code and generate iterative improvements, such a system could theoretically scale to some maximal horizon of capability.

We don’t yet know where this horizon might be or where it leads once encountered, but its probably safe to say that it scales beyond human cognition. And that’s really the heart of the singularity idea. This is the notion that we’ll reach some event horizon⁹, as on the rim of a black hole, where there’s no turning back. Some believe we’ve already reached that point. Others think it’s still decades or even centuries away. I thought I’d look through the academic literature on the subject to see what all the smart people are up to right now.

What the Literature Says

There are a fair amount of papers on the singularity, but these are largely hidden behind paywalls, and that makes it a difficult subject to assess. I’ll include some of them as links¹⁰ in case you’ve got better institutional access than me. But there’s probably a better way to get a read on how close we are to this kind of technology. Looking through the papers on actual software engineering techniques for achieving recursive self-improvement, it's clear this is a much easier path in terms of access.

I’d like to say at the outset that this won’t be an exhastive look at the relevant literature, but I hope it’s sufficient to at least get a feel for what’s out there. I used various sites to gather relevant papers, but probably one of the most interesting searches can be done on the site Connected Papers. This allows for pinning a search to a particular paper and then generating a graph of all connected work. The result is a visual representation of various ideas along with dates of publication.

My initial search was for “Recursive Self-improvement”. I used the following paper as the context for relevancy as it appeared to be both fresh (2022) and most directly related to my query. A Modern Self-Referential Weight Matrix That Learns to Modify Itself specifically looks at self-referential Neural Networks, which are different models than what’s used in ChatGPT, but it’s a good starting point.

The authors show that they’re able to produce a model that “is effectively capable of generating useful self-modifications”. I won’t get into the specifics here, but the close of the paper mentions recursive self-improvement and areas for more research. The authors state “This work may also have to consider standard limitations of NNs such as their lack of systematic generalisation”¹¹. In other words, NNs (Neural Networks) are not the best at doing general work, as they tend to be specialized.

Going further, the authors cite another paper on Transformers. If you’ll recall earlier, these are the models that ChatGPT and others like them are based on. And since these are the kinds of advances that have recently made headlines, it's clear that any major self-improving leaps in this space would be quite important. So what does the cited paper have to say on this subject? It appears there’s currently a problem with these models doing meaningful recursive work.

The authors of Going Beyond Linear Transformers with Recurrent Fast Weight Programmers note that “a major drawback of Transformers is that their computational complexity in time and space is quadratic in sequence length.”¹² In other words, as these models try to pay attention to larger amounts of data, they become much more computationally expensive. And that basically means they're also more expensive in terms of actual funds as well.

Sam Altman, the CEO of OpenAI (ChatGPT), recently said that the era of giant models is already over. But what does this mean exactly? It means that they’ve hit a kind of bottleneck with scaling models purely on more data and compute power. So, what’s next? I’d bet probably something like what’s mentioned in the paper above. The authors go on to say “[The new models] enjoy time and space complexities linear in sequence length with states of constant size. While their performance on some tasks does not fully match the one of regular Transformers, several improvements have already been proposed.”¹³

In non-engineering terms, this means that these models can be scaled easier and cost less when doing so. In other words, they can process more information, keep track of what they’ve already seen in real-time and also use recurrent or “auto-regressive” features. Again, in non-technical-speak, this portends that these models, or something very similar, are what’s next. And they’ve also got some recursive capabilities. At this time it doesn’t appear they’ve even started working on how these types of systems could work on updating and replacing their own code, but the foundations for this are being laid.

The conclusion of this paper states the following: “our novel Recurrent FWPs...compete well with the previous best linear Transformer variants for truncated contexts, and with full contexts, beat regular Transformers. Our RFWPs can also be used as drop-in replacements for problems where RNNs are still dominant.”¹⁴ Here the term acronym RFWP is just the name of their new system. And the fact that they're calling it novel shows that this is a new frontier in the research.

Future Research Directionality

But, again, what does this statement mean, practically? They’re basically saying that they’ve found something even better than the Linear Transformer. Recall that’s what we talked about above as cheaper and more easily scaled version of the model that powers ChatGPT. Okay, so this looks like a promising venue for the researchers to continue adding more self-improvement-like powers to their systems. What else is out there?

Again, this is just a sample and won’t be an exhaustive review. An equally important recent paper titled GENERAL-PURPOSE IN-CONTEXT LEARNING
BY META-LEARNING TRANSFORMERS discusses the “ambitious” goal of using “only black-box (italics mine) models without inductive bias”.¹⁵ But for what purpose? It appears that they propose that general models like ChatGPT or other similar language models can be used as meta-learners to help identify better model creation constraints. In other words, something like what’s being proposed here would learn how to learn. Or, put more succinctly, it would learn how to create better learning models.

Of the papers discussed here, I believe the latter is probably the most relevant for sparking something capable of spinning into a recursive self-improvement machine. Something like this would be able to identify not just systems like the novel mechanism above for a cheaper, more scalable Linear Transformer, but also mechanisms that we haven’t even thought of yet. And these are the sysytems that would probably be proto-recursive improvers, to coin a phrase that maybe gets closer to what these will become.

Conclusion

So, does this mean that we are currently entering into The Singularity? Probably not. Are we close? Yes, based on this current research, I believe we are. But, as recently published in the Free Press, economist Tyler Cowen says that should be wary of making predictions in this environment. Citing other relevant periods of “moving history” he notes the following:

The reality is that no one at the beginning of the printing press had any real idea of the changes it would bring. No one at the beginning of the fossil fuel era had much of an idea of the changes it would bring. No one is good at predicting the longer-term or even medium-term outcomes of these radical technological changes (we can do the short-term, albeit imperfectly). No one. Not you, not Eliezer, not Sam Altman, and not your next-door neighbor.¹⁶

I’d say I’m probably right there with the next-door neighbor in terms of relevancy to this discussion, but either way, he’s correct. Neither myself, nor any of the people you’re likely to read today or talk with tomorrow will be able to tell you when this might happen with any degree of certainty. But, I’d also say that given the circusmstances, the best thing to do is watch. Keeping an eye on what’s being developed is probably the only way to guage where we stand in terms of the sweeping, irrevocable changes described by the technological singularity.

So, let’s keep our heads up. You’ll likely have your job today and probably next month. Beyond that is anyone’s guess. If I were forced to guess how long we’ve got until the next major leap in systems that can recursively better themselves, I’d probably throw out a number that sounds a lot like ~2-3 years, but is probably best left as a statement as powerful as those little metal tongs used to grip a stuffed animal at the arcade. A game that’s somewhat rigged from the start. Until next time, get out there and enjoy being conscious!

1. https://www.zdnet.com/article/chatgpt-just-became-the-fastest-growing-app-of-all-time/

2. https://www.aiforanyone.org/glossary/technological-singularity

3. Is there anything worse? Yes, yes there is. And it’s probably got something to do with bio-engineering and suffering.

4. https://www.lesswrong.com/tag/recursive-self-improvement

5. Python is widely considered one of the best languages for beginners primarily due to its terse but powerful syntax and its lack of static typing

6. More specifically activation functions

7

https://users.cs.utah.edu/~germain/PPS/Topics/recursion.html

8. https://arxiv.org/pdf/2303.17491.pdf p1

9. https://www.britannica.com/topic/event-horizon-black-hole

10. https://api.semanticscholar.org/CorpusID:245663426

https://www.scopus.com/record/display.uri?eid=2-s2.0-85139160940&origin=inward&txGid=a88098e377c8ee58f114eee6e07ab558

https://www.tandfonline.com/doi/full/10.1080/02604027.2022.2050879

https://link.springer.com/content/pdf/10.1007/s12530-022-09431-7.pdf

https://www.semanticscholar.org/paper/Artificial-Superintelligence%3A-A-Recursive-Model-Reddy/a3d9aa604d62ac77bbfc3cd879e3d82064bb7b72

https://easychair.org/publications/preprint_open/dWLr

11. https://arxiv.org/pdf/2202.05780.pdf

12. https://arxiv.org/pdf/2106.06295.pdf

13. Ibid. p2

14. Ibid. p10

15. GENERAL-PURPOSE IN-CONTEXT LEARNING BY META-LEARNING TRANSFORMERS

https://arxiv.org/pdf/2212.04458.pdf

16. There Is No Turning Back on AI