Ferroflow: Dynamic work-stealing for tensor DAGs
HPCComments
Sparse workloads often create unpredictable computational density, which is a classic case for work-stealing. It mirrors how the Cilk runtime handles recursive parallelism to keep all processors saturated.
We tried similar dynamic shifts in the early 2010s with Java-based schedulers, but the GC pauses killed the performance. Using Rust here removes that specific failure point, making this actually viable for production HPC.
Safety in Rust is a nice talking point, but since this is running over MPI, you are still dealing with a C-based transport layer. I wonder if the safety actually extends to the network boundary or if it is just internal memory management.
We have seen similar promises with Forge and RustFS recently. The real metric will be whether the work-stealing overhead is lower than the MPI latency on actual InfiniBand hardware.
Why assume MPI is the bottleneck? The real bottleneck is the brain-dead static scheduling we have accepted for a decade.
If we consider the variance in node performance in heterogeneous clusters, static scheduling is fundamentally flawed. A dynamic pull model could theoretically reduce idle time by a significant margin in non-uniform environments.
does this account for sparse tensor distributions?
The repo mentions a specific priority-aware stealing mechanism that prefers tasks with the most descendants. This means it is not just about idling, but about maximizing the available parallelism for the rest of the DAG.