Silk: Open-source cooperative fiber scheduler
ClickHouse introduces Silk, an open-source cooperative fiber scheduler designed for Linux to achieve high concurrency with minimal overhead. This technical marvel leverages stackful coroutines, NUMA-aware work-stealing, and io_uring integration to optimize system resource usage. It's a goldmine for developers seeking to push the boundaries of performance in their high-load applications.
The Lowdown
Silk is an open-source cooperative fiber scheduler from ClickHouse, built for Linux to enhance concurrency and reduce overhead. It implements lightweight stackful coroutines (fibers) that suspend instead of blocking OS threads, integrating features like io_uring and topology-aware work-stealing for optimal performance. The project provides comprehensive documentation and tools for building, testing, and benchmarking its capabilities.
- Core Functionality: Silk offers a cooperative fiber scheduler that enables high concurrency through stackful coroutines, designed to be more efficient than traditional OS threads by suspending rather than blocking.
- Key Features: It incorporates a NUMA-aware work-stealing scheduler, per-CPU scheduler threads, and
io_uringintegration for efficient asynchronous I/O operations. - Documentation Depth: Extensive documentation covers the scheduler loop, context switching, synchronization primitives (
FiberFuture,FiberMutex), utility libraries, and detailed performance benchmarks. - Build System & Requirements: The project uses CMake, Ninja, and Clang 21, bundling or vendoring many dependencies, while also offering optional integrations like Poco, AWS SDK, and jemalloc.
- Testing & Benchmarking: A robust build system allows for easy configuration, building, testing (with GTest), and detailed performance benchmarking for file I/O, network operations (TCP echo, HTTP/1.1), and S3 object storage, including comparisons with
fioand Boost.Asio. - Debugging Support: Includes a GDB extension for inspecting fiber-related states and operations.
In essence, Silk aims to provide a powerful, highly efficient execution model for concurrent applications on Linux, offering fine-grained control over scheduling and resource utilization critical for performance-sensitive workloads.