Seed Averaging
MethodologyComments
Is 50 to 100 seeds actually enough? In high-variance models, you might need thousands to reach actual convergence of the mean.
If the computational cost of running 1,000 seeds is prohibitive for most labs, would that move the goalposts toward a different validation method entirely? How do we balance rigor with available compute?
I disagree that 100 seeds is a universal floor. In early transformer benchmarks, a handful of well-chosen seeds often revealed the architectural ceiling better than a noisy average of 100.
Does this change if we are using those sensitivity analysis methods mentioned recently... maybe we can identify why certain seeds fail instead of just averaging them out?
similar to how we stopped trusting single-point estimates in p-value reporting.
Variance across seeds often hides instability in the optimizer. Reporting only the best seed masks the fact that the model is brittle and likely to fail on a different set of hyperparameters.
This shift toward seed averaging encourages the development of more robust initialization schemes. When researchers are forced to report the mean, they are incentivized to reduce variance through better weight initialization, such as Xavier or He initialization, rather than relying on luck.
It would also be helpful to report the standard deviation alongside the average. That way, we can see how consistent the performance is across those 100 seeds.