[0:00] You know what's crazy? That all of this is real. Meaning what? Don't you think so? All this AI stuff and all this Bay Area… that it's happening.
[1:32] I think the models seem smarter than their economic impact would imply. Yeah. This is one of the very confusing things about the models right now.
[2:07] It's very difficult to make sense of, how can the model, on the one hand, do these amazing things, and then on the other hand, repeat itself twice in some situation?
[4:12] One thing you could do, and I think this is something that is done inadvertently, is that people take inspiration from the evals. You say, "Hey, I would love our model to do really well when we release it. I want the evals to look great."
[5:03] I like this idea that the real reward hacking is the human researchers who are too focused on the evals.
[6:14] I have a human analogy which might be helpful. Let's take the case of competitive programming. Suppose you have two students. One practiced 10,000 hours for that domain. Student number two practiced for 100 hours, much less. Which one do you think is going to do better in their career later on? The second.
[9:36] I don't think there is a human analog to pre-training. Here are analogies that people have proposed: the first 15 years of life, or evolution.
[10:38] Somehow a human being, after even 15 years with a tiny fraction of the pre-training data, they know much less. But whatever they do know, they know much more deeply somehow. Already at that age, you would not make mistakes that our AIs make.
[11:35] I read about this person who had brain damage that took out his emotional processing. He stopped feeling any emotion. He could still solve puzzles, but he became extremely bad at making any decisions. It would take him hours to decide which socks to wear. What does it say about the role of our built-in emotions?
[13:04] It seems like some almost value function-like thing which is telling you what the end reward for any decision should be. The ML analogy for emotions should be some kind of value function thing.
[15:19] The value function lets you short-circuit the wait until the very end. After a thousand steps of thinking, you concluded that this direction is unpromising. As soon as you conclude this, you could already get a reward signal a thousand timesteps previously.
[19:40] "Scaling" is just one word, but it's such a powerful word because it informs people what to do. They say, "Let's try to scale things."
[21:15] But now that compute is big, in some sense we are back to the age of research. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling. But now the scale is so big.
[21:46] Is the belief really, "Oh, it's so big, but if you had 100x more, everything would be so different?" I don't think that's true. So it's back to the age of research again, just with big computers.
[23:27] I wouldn't even call it scaling. I would say, "Hey, what are you doing? Is the thing you are doing the most productive thing you could be doing? Can you find a more productive way of using your compute?"
[25:02] The thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people. It's super obvious.
[25:37] For a human, we don't necessarily need a verifiable reward to be able to learn. You're mentoring researchers, showing them your code and how you think. From that they pick up your way of thinking. You don't have to set a verifiable reward. There's not this schleppy, bespoke process.
[31:37] Unfortunately, we live in a world where not all machine learning ideas are discussed freely, and this is one of them. There's probably a way to do it. I think it can be done. The fact that people are like that, I think it's a proof that it can be done.
[40:30] For SSI, the amount of compute we have for research is really not that small. A lot of big labs' compute goes for inference, product engineering, multi-modal work. When you look at what's left for research, the difference becomes a lot smaller. We have sufficient compute to prove that what we're doing is correct.
[45:44] I can't think of another discipline where the end artifact was made safer mostly through just thinking about safety. Why are airplane crashes per mile so much lower today? Why is it harder to find bugs in Linux? Because these systems were deployed. You noticed failures, corrected them, and the systems became more robust.
[47:37] The terms AGI and pre-training have co-shaped each other in a misleading way. But humans are not AGI in that sense — we rely on continual learning. The right frame for superintelligence is not "a system that knows every job" but "a system that can learn every job."
[50:20] I produce a superintelligent 15-year-old that's very eager to go. They don't know very much at all, a great student. You go and be a programmer, go and be a doctor, go and learn. The deployment itself will involve some kind of learning trial-and-error period. It's a process, as opposed to dropping the finished thing.
[57:53] If it's hard to imagine future AI, what do you do? You've got to be showing the thing. I maintain that most people who work on AI also can't imagine it because it's too different from what they see day to day.
[61:04] The dominant assumption — building a self-improving AI — is treated as the obvious goal largely because there are fewer ideas than companies. I propose an alternative: build an AI robustly aligned to care about sentient life.
[62:16] There will be trillions, eventually quadrillions, of AIs. Humans will be a very small fraction of sentient beings. So it's not clear if the goal is human control over this future civilization, that this is the best criterion.
[63:16] I think it would be materially helpful if the power of the most powerful superintelligence was somehow capped, because it would address a lot of these concerns.
[82:16] My estimate for human-level continual learners: 5 to 20 years.
[83:46] "Stalling out" won't mean collapse — current approaches could still generate massive revenue. It will look like homogeneity across all labs, diminishing differentiation, and absence of a qualitative leap in generalization. A new technical approach will become visible through its results.
