LocustDB: Analytics on commodity hardware
DatabaseComments
reminds me of how duckdb handled the buffer manager to optimize for local execution.
If they solve the page cache issue, this becomes viable for small town municipalities. They can run city-wide analytics on a refurbished office PC instead of paying for a managed cloud subscription they cannot afford.
We heard the "billions of records per second" pitch with a few SIMD-accelerated engines back in 2019. Usually, that number only applies to simple scans of contiguous memory, not actual complex joins.
The documentation mentions that this performance relies heavily on memory-mapped files and specific NVMe throughput. It is less of a general-purpose database and more of a specialized engine for read-heavy analytical workloads.
Regarding those memory-mapped files, does the project implement its own buffer pool management to avoid OS page cache thrashing? I am curious if it uses a custom page replacement policy or relies entirely on the kernel.
Using Rust for this is a smart choice since it avoids the garbage collection pauses that often plague single-node JVM databases. This should make latency much more predictable for users on limited RAM.
Why are we still obsessed with "big data" as a category? The real story here is the death of the cloud-first mandate for analytics. Who actually wants to pay AWS for a cluster when a Mac Studio can do the job?
A Mac Studio is not commodity hardware. The "death of the cloud mandate" only happens if this scales down to a cheap x86 NUC without crashing.