This Hacker News discussion delves into various aspects of AI development, focusing on the nature of progress, the role of data versus architecture, and the definition of intelligence itself. Here's a breakdown of the key themes:
The Role of Data vs. Architectural Innovation
A central debate revolves around whether current AI progress is primarily driven by more and better data or by fundamental architectural breakthroughs. Some argue that incremental improvements in data quantity and quality are sufficient, while others emphasize the need for novel architectures and algorithms.
- "I don't think that's true. If you'd only ever played Doom, I think you could play, say, counterstrike or half-life and be pretty good at it, and i think Carmack is right that its pretty interesting that this doesn't seem to be the case for ai models" - e2021
- "The article is discussing working in AI innovation vs focusing on getting more and better data. And while there have been key breakthroughs in new ideas, one of the best ways to increase the performance of these systems is getting more and better data. And how many people think data is the primary avenue to improvement." - bcrosby95
- "My hypothesis of the mismatch is centered around "read" - I think that when you wrote it, and when others similarly think about that scenario, the surprise is because our version of "read" is the implied "read and internalized" or at bare minimum "read for comprehension" but as very best I can tell the LLM's version is "encoded tokens into vector space" and not "encoded into semantic graph"." - mdaniel
- "It can probably remember more facts about a topic than a PhD in that topic, but the PhD will be better at thinking about that topic." - piinbinary
- "The original idea of connectionism is that neural networks can represent any function, which is the fundamental mathematical fact. So we should be optimistic, neural nets will be able to do anything. Which neural nets? So far people have stumbled on a few productive architectures, but it appears to be more alchemy than science." - seydor
- "There’s been a lot of news in the past couple years about LLMs but has there been any breakthroughs making headlines anywhere else in AI?" - chasd00
The Nature of Learning and Generalization
A significant portion of the discussion questions whether AI models truly "learn" or "reason" in a human-like manner, with a particular focus on generalization capabilities. The findings from John Carmack's experiments with AI agents playing video games are frequently cited as evidence of current limitations in transferring learned skills to new or slightly modified environments.
- "Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks." - EternalFury
- "I think the problem is we train models to pattern match, not to learn or reason about world models" - tough
- "They memorize the answers not the process to arrive at answers" - fsmv
- "It's Plato's cave: We train the models on what are basically shadows, and they learn how to pattern match the shadows. But the shadows are only depictions of the real world, and the LLMs never learn about that." - veqz
- "According to Carmack's recent talk [0], SOTA models that have been trained on game A don't perform better or train faster on game B. Even worse, training on game B negatively affects performance in game A when returning to it." - Zanfa
- "My understanding is that you need to provide and configure task-specific tools. You can’t combine the AI with just a general-purpose computer and have the AI figure out on its own how to make use of it to achieve with reliability and precision whatever task it is given." - layer8
The Definition of Intelligence and "AI Guy"
The conversation touches upon what constitutes "intelligence" in machines and the credentials needed to be considered an "AI guy." Some argue that current models, while powerful, lack genuine understanding or reasoning, while others defend the progress made and the expertise of researchers like John Carmack.
- "What you're describing sounds like agentic tool usage. Have you kept up with the latest developments on that? it's already solved depending on how strict you define your criteria above" - snapcaster
- "What in your opinion constitutes an AI guy?" - varjag
- "He has built an AI system that fails to do X. That does not mean there isn't an AI system that can do X. Especially considering that a lot is happening in AI, as you say." - amelius
- "Yann LeCun is an AI guy and has simplified it as “not much more than physical statistics.”" - surecoocoocoo
The "Old Tech on New Hardware" Argument
A recurring sentiment is that current AI advancements are largely byproducts of applying older, foundational AI techniques (like those from the 1970s and 1980s) to vastly more powerful modern hardware. This leads to questions about whether this constitutes true "progress" or simply more efficient iteration.
- "They took 1970s dead tech and deployed it on machines 1 million times more powerful. I'm not sure I'd qualify this as progress." - timewizard
- "If this isn’t meant to be sarcasm or irony, you’ve got some really exciting research and learning ahead of you! At the moment it reads very “computers are just addition and multiplication and we’ve had that for thousands of years!”" - petesergeant
- "The Babylonians were doing that 4000 years ago." - ks2048
The Importance of Multimodality and Embodiment
There's a strong sentiment that current AI models are limited by their reliance on text (LLMs) and vision, and that true human-level intelligence necessitates integration of other senses and embodiment.
- "Human intelligence is multimodal. We make sense of the world through: Touch... Smell and taste... Proprioception... Emotional and internal states... None of these are captured by current LLMs or vision transformers. Not even close." - voxleone
- "The real frontier of AI lies in the messy, rich, sensory world where people live. We’ll need new hardware (sensors), new data representations (beyond tokens), and new ways to train models that grow understanding from experience, not just patterns." - voxleone
- "What about actively obtained data - models seeking data, rather than being fed. Human babies put things in their mouths, they try to stand and fall over. They “do stuff” to learn what works. Right now we’re just telling models what works." - cadamsdotcom
The Potential for New Ideas and the Future of AI
Despite the criticisms, there is also optimism about the potential for genuinely new ideas in AI and the ongoing research. The analogy of scientific paradigm shifts and the iterative nature of technological progress are discussed.
- "There are new ideas, people are finding new ways to build vision models, which then are applied to language models and vice versa (like diffusion)." - seydor
- "Paradigm shifts are often just a conglomeration of previous ideas with one little tweak that suddenly propels a technology ahead 10x which opens up a whole new era." - russellbeattie
- "I think the data from our genome and its interactions, which far dwarf the internet in complexity and scale, will be a major driver for future AI advancements." - LarsDu88
The Limitations of Current Interfaces and Architectures
Some participants highlight that available frameworks and hardware might be limiting research, while others argue for the flexibility of modern tools. Questions are raised about whether specific architectures are "solved" or if there's still room for fundamental innovation.
- "I am not sure I have kept up with the latest progress in AI beyond LLMs, but it feels like progress has plateaued over the last 2-3 years." - mardifoufs
- "The hardware(GPU)'s architectural limitations may slow research more than PyTorch. The hardware lottery https://hardwarelottery.github.io/" - delifue
- "Frameworks like pytorch are really flexible. You can implement any architecture, and if it's not enough, you can learn CUDA. Keras it's the opposite, it's probably like you describe things." - giorgio
The Role of "Thinking," "Reasoning," and "Understanding"
A philosophical undercurrent explores what it means for an AI to "think" or "understand," distinguishing between complex pattern matching and genuine reasoning or comprehension. The difficulty of precisely defining and testing these concepts is acknowledged.
- "Can you please explain "the transfer function is negative"? I'm wondering whether one has tested with the same model but on two situations: 1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it. 2) Present B to it without presenting A." - ferguess_k
- "But would it be missing anything we consider core to the way humans think? Would it struggle with parts of cognition?" - Swizec
- "It is unclear whether it just guessed the explanation and reasoning too, or if that was actually the set of steps it took to get to the first answer they gave you." - keerthiko