Didn't expect this take on the subject, but it's brilliant how you tied Vonnegut's quote to the current AI landscape. You're spot on about the 'end of history' narrative and those wild acquisition bids. I'm curios to hear more about what alternative scenarios you see playing out, beyond the winner-takes-all dystopia.
hello, thanks for the comment. that is a good question and probably deserves its own post. as you point out, a winner take all version of true AGI would move quickly toward instability. even if we approach AGI, it will run into economic and energy bottlenecks. growth gains look capped at modest levels, and the past few years of maximum data center buildout show we cannot scale electrical output fast enough to sustain current spending without much broader investment in energy infrastructure.
even with those limits, AI can still reduce labor demand across many services. the near-term effect is likely slower wage growth, more competition for skilled roles, and greater pressure on younger workers. a new equilibrium will form over time as it has during past technological shifts, but it is not clear which sectors will absorb displaced labor. the transition is likely to intensify existing K-shaped and affordability dynamics.
as for what individuals can do, the practical focus is on skills that are hardest to imitate. taste, judgment, and the ability to connect decisions to empirical or theoretical reasons remain valuable. current models operate through probabilistic pattern inference rather than symbolic, rule-based deduction. this means their conclusions do not have built-in logical guarantees, and any deductive-looking step needs human inspection. this limitation belongs to present methods, not to machine learning in general. future techniques may integrate symbolic or hybrid structures that offer verifiable reasoning.
there is still strong value in studying math, programming, and physics because these sharpen reasoning and help people evaluate whether model outputs follow known structures of the world. even as AI assists more tasks, human work will likely shift toward conceptual validation and interpretation.
education may need to adapt as well. when routine knowledge becomes cheap to generate, the emphasis shifts from test-focused recall toward reasoning, model building, and clear explanation.
I’m so glad I read this. Thank you for writing because I have a weirdly similar experience this week. I’m also kinda struggling with direction.
I was driving to work listening to an audiobook talking about being your authentic self and that’s one of the best gift you can give to the world or whatever. And I kept thinking, okay like how do you know? “How am I not myself?”
Then I got to work, and we had a new person join, and everyone was going around introducing themselves. People were sharing where they came from and fun facts, and I just… didn’t want to talk about my past or give some polished story. Nothing felt worth mentioning in that moment. So I just said, “I like to draw.” People laughed at how short it was and moved on, but honestly it felt weird.
So reading your blog really hit home.. I appreciate this so much.
Hi Norman, sorry I didn’t get a chance to watch this right away. What a thought-provoking video, and I think it highlights one of the insidious things about LLM training generally: it’s trained on the corpus of digitally available information, from fiction to forums to YouTube scrapes. I loved the Computerphile host’s question about how we know that papers or popular content about model escape or obfuscation aren’t influencing the model. Essentially, the models are so enormously expensive to pre-train that we can’t do the scientific gold standard and run randomized controlled trials to experimentally test these hypotheses.
So without RCTs we can gather anecdotes, which are open to speculation and interpretation. I thought about this a bit, and here are the two salient points I came up with.
1. The linguistic behavior of performing “want” is easily extended to action within the context of coding because the LLM will generate the most probable next sequence of text given the context. The performance of the language is what makes it (naively, in my opinion) pass the Turing Test; it can convincingly, at first glance, perform wanting. But based on the way LLMs work, this is strictly an immutable probabilistic output.
2. The model that exists post-training is essentially static, aside from the context you supply. This idea of changing optimization being analogous to changing values is interesting. I feel they use the word “want” in an anthropomorphic sense. For example: what does a bacterium want? In a more philosophical sense, wanting has something to do with the limbic system, with being in a continuous time loop, with having energetic needs and drives. Without these meta-objectives to optimize against, what can it mean to want? Consciousness and wanting involve flitting between sometimes opposing desires. That makes behavior unpredictable, mutable, subject to dramatic shifts in objectives. None of these things seem like they would be true of a program trained to optimize against one loss function (text believability). We should be dubious anytime the model claims to want anything, as it is an artifact of the training data.
Thanks for sharing the video, and hope you’re well.
Yeah I agree with your take. An LLM will say it wants to do X but it’s not really wanting like how we want. It just says it wants because it was trained to. It’s interesting though because it kinda forces us to define what does it mean for a human vs machine to want or to feel. I’ve heard that people can’t see colors that they don’t have words for, so how much of human emotion is innate vs. baked into language? Obviously babies can want food even if they can’t speak the words for it, but it feels like there could be a crossing point somewhere along the spectrum of emotion complexity.
Didn't expect this take on the subject, but it's brilliant how you tied Vonnegut's quote to the current AI landscape. You're spot on about the 'end of history' narrative and those wild acquisition bids. I'm curios to hear more about what alternative scenarios you see playing out, beyond the winner-takes-all dystopia.
hello, thanks for the comment. that is a good question and probably deserves its own post. as you point out, a winner take all version of true AGI would move quickly toward instability. even if we approach AGI, it will run into economic and energy bottlenecks. growth gains look capped at modest levels, and the past few years of maximum data center buildout show we cannot scale electrical output fast enough to sustain current spending without much broader investment in energy infrastructure.
even with those limits, AI can still reduce labor demand across many services. the near-term effect is likely slower wage growth, more competition for skilled roles, and greater pressure on younger workers. a new equilibrium will form over time as it has during past technological shifts, but it is not clear which sectors will absorb displaced labor. the transition is likely to intensify existing K-shaped and affordability dynamics.
as for what individuals can do, the practical focus is on skills that are hardest to imitate. taste, judgment, and the ability to connect decisions to empirical or theoretical reasons remain valuable. current models operate through probabilistic pattern inference rather than symbolic, rule-based deduction. this means their conclusions do not have built-in logical guarantees, and any deductive-looking step needs human inspection. this limitation belongs to present methods, not to machine learning in general. future techniques may integrate symbolic or hybrid structures that offer verifiable reasoning.
there is still strong value in studying math, programming, and physics because these sharpen reasoning and help people evaluate whether model outputs follow known structures of the world. even as AI assists more tasks, human work will likely shift toward conceptual validation and interpretation.
education may need to adapt as well. when routine knowledge becomes cheap to generate, the emphasis shifts from test-focused recall toward reasoning, model building, and clear explanation.
I’m so glad I read this. Thank you for writing because I have a weirdly similar experience this week. I’m also kinda struggling with direction.
I was driving to work listening to an audiobook talking about being your authentic self and that’s one of the best gift you can give to the world or whatever. And I kept thinking, okay like how do you know? “How am I not myself?”
Then I got to work, and we had a new person join, and everyone was going around introducing themselves. People were sharing where they came from and fun facts, and I just… didn’t want to talk about my past or give some polished story. Nothing felt worth mentioning in that moment. So I just said, “I like to draw.” People laughed at how short it was and moved on, but honestly it felt weird.
So reading your blog really hit home.. I appreciate this so much.
so relatable 🫨 it can be so difficult to vulnerable with new people
Hi Dylan. Well said! I wonder what you’d think of this video regarding whether LLMs can or cannot want- or maybe what does it mean to want?
https://youtu.be/AqJnK9Dh-eQ?si=ltmMBg6Xxu3rfG3j
Hi Norman, sorry I didn’t get a chance to watch this right away. What a thought-provoking video, and I think it highlights one of the insidious things about LLM training generally: it’s trained on the corpus of digitally available information, from fiction to forums to YouTube scrapes. I loved the Computerphile host’s question about how we know that papers or popular content about model escape or obfuscation aren’t influencing the model. Essentially, the models are so enormously expensive to pre-train that we can’t do the scientific gold standard and run randomized controlled trials to experimentally test these hypotheses.
So without RCTs we can gather anecdotes, which are open to speculation and interpretation. I thought about this a bit, and here are the two salient points I came up with.
1. The linguistic behavior of performing “want” is easily extended to action within the context of coding because the LLM will generate the most probable next sequence of text given the context. The performance of the language is what makes it (naively, in my opinion) pass the Turing Test; it can convincingly, at first glance, perform wanting. But based on the way LLMs work, this is strictly an immutable probabilistic output.
2. The model that exists post-training is essentially static, aside from the context you supply. This idea of changing optimization being analogous to changing values is interesting. I feel they use the word “want” in an anthropomorphic sense. For example: what does a bacterium want? In a more philosophical sense, wanting has something to do with the limbic system, with being in a continuous time loop, with having energetic needs and drives. Without these meta-objectives to optimize against, what can it mean to want? Consciousness and wanting involve flitting between sometimes opposing desires. That makes behavior unpredictable, mutable, subject to dramatic shifts in objectives. None of these things seem like they would be true of a program trained to optimize against one loss function (text believability). We should be dubious anytime the model claims to want anything, as it is an artifact of the training data.
Thanks for sharing the video, and hope you’re well.
Yeah I agree with your take. An LLM will say it wants to do X but it’s not really wanting like how we want. It just says it wants because it was trained to. It’s interesting though because it kinda forces us to define what does it mean for a human vs machine to want or to feel. I’ve heard that people can’t see colors that they don’t have words for, so how much of human emotion is innate vs. baked into language? Obviously babies can want food even if they can’t speak the words for it, but it feels like there could be a crossing point somewhere along the spectrum of emotion complexity.