Machine Learning

From the ‘Tape-Finding’ Mystique of Geoguessr to the Shortcut Learning Myth in Deep Learning: What Happens When We Over-rely on Certain Features? 1. The Days of Being Dropped in the Middle of Nowhere: The Romance of Vibe Guessing Ever since my exchange student days ended, I’ve unexpectedly become captivated by Geoguessr. Its charm lies in not needing to download any app; you can simply open your browser and be ‘randomly airdropped’ anywhere in the world, anytime. While playing, I not only revisit street views I’ve walked through but also discovered I was gradually developing a superpower – ‘Vibe guessing’. ...

As I delved deeper into LLM applications and Agent systems, I became utterly disgusted by the industry’s “AI sycophant” style of mystical marketing. Vendors now constantly boast that their new models “can think” and “have logic,” as if adding a “Pro” or “Thinking” suffix suddenly makes a neural network sprout a human brain. If we don’t adopt an attitude of skepticism and questioning, we can easily be led astray by these attractive claims. Today, let’s coldly dissect the true differences between Flash models and Thinking models, and what those so-called “thinking processes” actually entail, starting from their underlying operational mechanisms and physical limitations. ...

At the time, I used to think that a Decoder was essentially just playing “word-chain” based on the input, unable to pre-determine what it was going to generate as a whole. But looking back now, if we don’t blindly join the hype of those AI sycophants, spouting pretty phrases like “because GPT is more spiritual,” but instead dispassionately dissect it from the foundational layers of Information Theory and engineering reality, you’ll find that the triumph of Decoder-only architectures is, fundamentally, an inevitability dictated by mathematical and physical constraints. ...

「If we’re on the path to the Turing machine, Seq2Seq itself makes more sense than a Decoder." This is a sentence I wrote in my notes in 2022. Back then, I was experimenting with an early version of GPT-2, and I kept wondering: What exactly is this thing? In comparison, models like BART or T5, based on the Seq2Seq concept, seemed much more reasonable. Unexpectedly, a few years later, in this AI arms race, it’s the Decoder-only architecture that has claimed the MVP title. ...