Cactus Engineering Blog·· 27 天前精选评分60
Hadamard MLP 替代 FFN：单层参数 4.7M 降到 25.6K，算力五分之一
The Hadamard MLP for Channel Mixing for Almost No ParametersNeedle replaces the transformer's feed-forward layer with three Kronecker-factored mixes that start as the Walsh-Hadamard transform and learn from there. 25.6K parameters per layer instead of 4.7M, and a fifth of the compute per token.Henry Ndubuaku · September 17, 2026 · 13 min read
导读
Needle 用三个从 Walsh-Hadamard 变换起步学习的 Kronecker 分解混合层替代 transformer 前馈层，d=768 时单层参数从约 4.7M 降至 25.6K，每 token 计算约为原来的五分之一，契合端侧权重读取瓶颈。
推荐理由
量化明确的架构创新，直指端侧推理的内存墙问题。
来源：Cactus Engineering Blog · cactuscompute.com