hey thanks for the comment, author here. That's a fair read of the loss numbers, I could only fund up to 1.2B scale myself and I don't want to oversell the results. But note that the benefits that Soft-MTP delivers in the paper is *consistently growing across scale*. The residual is irreducible with scale and will only become noisier as model grows bigger, so removing it via Soft-MTP is intuitive. It is also easily verifiable cos it only needs minor code change, codebase: https://github.com/JaydenTeoh/Soft-MTP
The results look like a statistical fluke to me. Tiny effect, sometimes negative.
hey thanks for the comment, author here. That's a fair read of the loss numbers, I could only fund up to 1.2B scale myself and I don't want to oversell the results. But note that the benefits that Soft-MTP delivers in the paper is *consistently growing across scale*. The residual is irreducible with scale and will only become noisier as model grows bigger, so removing it via Soft-MTP is intuitive. It is also easily verifiable cos it only needs minor code change, codebase: https://github.com/JaydenTeoh/Soft-MTP
there is reproduction of Soft-MTP's benefit over standard MTP at 35B scale but it is only one data point for now: https://x.com/HivazeMe/status/2118432486724391217