Cactus Engineering Blog·· 27 天前精选评分58
Cactus 受控实验：删掉 FFN 后 Transformer 只损失 0.006 nats
What a Transformer Loses Without Feed-Forward LayersWe deleted the feed-forward network from a transformer and measured what was actually lost, under controls for parameters, compute and depth. At matched parameters the answer is 0.006 nats, and all of it lives on tokens with nothing to look up.Henry Ndubuaku · September 17, 2026 · 12 min read
导读
在控制参数量、算力与深度后测量删除前馈网络的损失：匹配参数下仅 0.006 nats，且集中在无外部信息可查的 token 上；该结论支撑 Needle 2/3 的纯注意力架构，完整研究为 A Controlled Study of Attention-Only Transformers。
推荐理由
对 transformer 架构常识的反直觉实证，方法论干净。
来源：Cactus Engineering Blog · cactuscompute.com