Decoder-Only

At the time, I used to think that a Decoder was essentially just playing “word-chain” based on the input, unable to pre-determine what it was going to generate as a whole. But looking back now, if we don’t blindly join the hype of those AI sycophants, spouting pretty phrases like “because GPT is more spiritual,” but instead dispassionately dissect it from the foundational layers of Information Theory and engineering reality, you’ll find that the triumph of Decoder-only architectures is, fundamentally, an inevitability dictated by mathematical and physical constraints. ...

「If we’re on the path to the Turing machine, Seq2Seq itself makes more sense than a Decoder." This is a sentence I wrote in my notes in 2022. Back then, I was experimenting with an early version of GPT-2, and I kept wondering: What exactly is this thing? In comparison, models like BART or T5, based on the Seq2Seq concept, seemed much more reasonable. Unexpectedly, a few years later, in this AI arms race, it’s the Decoder-only architecture that has claimed the MVP title. ...