The unrealistic call for a pause
The unrealistic call for a pause
A considerable number of well-known individuals from the tech space as well as researchers, including those from the AI field, are calling for a six-month pause during which no AI models larger than GPT-4 are started/trained.
Pause Giant AI Experiments: An Open Letter - Future of Life Institute
This is also being covered in the daily press.
What is this about?
A few thoughts on this that I could not find in the press. If these insights are too obvious, please forgive me.
The latest generations of large language models have shown that more is better, meaning that a model with more weights (parameters) and trained on more data not only becomes better/more accurate but also produces emergent properties every time. I recommend the observations from the GPT-4 publications as well as an Analysis by MS Research. The focus is on the fact that current large models are developing intelligence in the sense of the ability to recognize and evaluate complex situations and derive actions or draw conclusions. This is encouraging as it significantly increases the usefulness and value of these systems.
Just for context: What is currently known as ChatGPT is based on an older generation of these systems (GPT3). The analyses focus on the properties and capabilities of newer models such as GPT4 (OpenAI) and Palm (Google). Google’s Bard will also not be based on such a system, but rather an older model.
In addition to the risks of the current models for creating disinformation and being used for malicious purposes by humans, there is also concern that these systems may become uncontrollable at some point. This concern is not yet too great for current models, but it is a concern for researchers when it comes to the next generation.
Alignment
The current approach to controlling these systems is called “alignment,” which means that these systems should act “in our best interests.” This includes not lying, not hallucinating, responding without bias, avoiding dangerous and manipulative responses. It’s not like other computer topics where the concern is about correctness or security in terms of program code vulnerabilities; it’s about the behavior of a system.
Alignment is currently achieved by training the system with humans (and actually also with human-copying AIs). The output of the system is evaluated and the evaluation is fed back.
The core of the above request for a pause is that we now have the means and scale to build systems that will have even more capabilities. Researchers are concerned that we do not have the same means to control such a next system. The current alignment methods do not scale to the same extent as the size and capability of the model.
Personally, I am relaxed about this, but explaining it would completely exceed the already long post. if there is interest, I can explain this in a separate post.
And now?
Isn’t this unrealistic? A more capable system is a huge opportunity for any market participant. A researcher who builds it first may receive ridicule but certainly also fame. So even if all companies and institutes now want to take a break, the temptation to build it is disproportionately large in both business and research. It is a prisoner’s dilemma with n participants. Someone will not adhere to it and reap the profits - at least in the military sector, which I trust much less to control, I expect unrestricted research.
It is also unrealistic for quick and uniform state regulation to declare and enforce a moratorium. Here too, the advantage for the individual state is great not to adhere to it. Can the US afford to take its foot off the pedal here? And then I also do not believe that most states are capable of doing so.
For those paying attention, it can also be read the other way around: It is the promise that the next step will be systems that are another order of magnitude larger and will deliver even more capabilities and value. As soon as I read that, I know it’s worth starting and building it.
“There is no bad news” - this letter simply adds to the already immense attention on the topic. And soon we will see even larger systems and the values they create.