Schmidhuber's Claims of Priority and the "Inventor" Label
A major theme revolves around Juergen Schmidhuber's claims of having "invented it all" or having priority in key deep learning concepts. Many commenters discuss his tendency to point out how modern contributions are often trivial generalizations of his earlier work.
- "Every so often Schmidhuber is brought back to the front-page of HN, people will argue that he 'invented it all' while others will say that he's a-posteriori claiming all the good ideas were his." (belval)
- "A lot of this is because nobody likes braggers - however, in all fairness, his argument is that a lot of what is considered modern ML is based on many previous results, including but not limited to his own research." (kleiba)
- "But as far as I understand, Schmidhuber's claim is more severe: namely that Bengio, Hinton and LeCun intentionally failed to cite prior work by others (including himself) but instead only cited each other in order to boost their respective scientific reputation." (kleiba)
Disagreement on Schmidhuber's Impact and Whether He Was "Held Back"
The discussion also touches on the idea of whether Schmidhuber's work was genuinely groundbreaking and whether he was unfairly overlooked, or whether his inability to foresee the utility of his work on modern hardware hindered the field.
- "But more seriously, I'm not a fan of Schmidhuber because even if he truly did invent all this stuff early in the 90s, he's inability to see its application to modern compute held the field back by years." (goldemerald)
- "I also strongly disagree with the idea that his inability to practically apply his ideas held anything back. In the first place, it is uncommon for a discoverer or inventor to immediately grasp all the implications of and applications of their work." (Vetch)
- "In the interim, when most derided Neural networks, his lab was one of the few that kept research on Neural networks and their application to sequence learning going. Without their contributions, I'm confident Transformers would have happened later." (Vetch)
The Role of Credit, Attribution, and Novelty in Scientific Progress
A recurring theme is the tension between the desire for novelty and the importance of incremental progress, as well as the challenges of assigning credit fairly. Several users argue for a shift in culture away from chasing novelty for its own sake and towards valuing collaboration and communication above all else.
- "This is a culture thing that needs to change... we shouldn't be chasing these notions of novelty and 'impact'. They are inherently subjective and lead to these issues of credit... It is only 'embarrassing' because it 'undermines' the work. It only 'undermines' the work because how we view credit." (godelski)
- "Ironically, I think the problem is we care too much about credit. It ends up getting hoarded rather than shared... From building on open source, it's libraries all the way down, but we act like we did it all alone." (godelski)
- "If a bona fide scientist makes a mistake about missing attribution, they would correct it as soon as possible. Many, however, would not correct such a re-discovery, because it's embarrassing." (jll29)
Rediscovering and Reinventing Ideas: A Common Occurrence
Several comments highlighted the frequency with which ideas are independently discovered or re-invented, emphasizing that new scientific insights are often built upon previous work, and the importance of standing "on the shoulders of giants."
- "We all stand on the shoulders of giants, things can be invented and reinvented and ideas can appear twice in a vacuum." (belval)
- "I have seen many cases where people -- accidentally as well as intentionally -- copied or re-invented the work of others" (jll29)
- "People at work have, I think, gotten tired of my rant about how people who are ignorant of the history of their field have a tendency to either re-invent things that already exist, or to be snowed by other people who are re-inventing things that already exist." (mindcrime)
The Value of Understanding the History of a Field
Several individuals emphasized the importance of understanding the history of a field to avoid re-inventing the wheel and to properly acknowledge intellectual contributions.
- "His 'shtick' to me isn't just about him saying 'people didn't give me credit' but it seems more 'AI people in general haven't credited the history of the field properly.' And in many cases he seems to have a point." (mindcrime)
- "I wonder at times if it stems back to flaws in the CS pedagogy. I studied philosophy and literature in which tracing the history of thought is basically the entire game. I wonder if STEM fields, since they have far greater operational emphasis, lose out on some of this." (voidhorse)
- "...people who are ignorant of the history of their field have a tendency to either re-invent things that already exist, or to be snowed by other people who are re-inventing things that already exist." (mindcrime)
The Role of Infrastructure and Funding in Scientific Advancement
Several comments point out that access to resources and infrastructure can be just as important, if not more so, than the models themselves.
- "Ultimately, models matter much less than infrastructure. Transformers are not that important, other architectures such as deep SSMs or xLSTM are able to achieve comparable results." (nextos)
- "It's not very sexy to say 'Oh yes, we are just using an old Soviet learning algorithm on better hardware. Turns out we would have lost the cold war if the USSR had access to a 5090.' , which won't get you the billions you need to build the supercomputers that push the state of the art today." (noosphr)