Filtered by: Scheduling × Clear all

Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

Daeyeon Son 2026-07-07
Inference LLM Scheduling ×

The problem is that on-device LLM decoding is a hard-barriered CPU-SIMD computation requiring all cores per token, but preemptive scheduling causes deadlock or silent logit corruption. The method introduces the elastic gang in Anima OS, using an ACK-latched epoch protocol with generation-tagged latches and RCU-style membership consent to allow per-token core membership changes without waiting on named cores. Experimental evidence on AMD Zen 5 shows bit-exact inference under per-token membership changes for 135M and 7B models, with 1.75x, 1.52x, and 1.28x general throughput improvements over static 8-core splits at 25%, 50%, and 75% inference duty cycles, and core return costs of 0.22 us. This matters because it enables safe, dynamic core sharing between LLM inference and OS processes, Pareto-dominating static partitions and auto-sizing the gang online to maximize throughput without stranded cores.

PDF

Performance evaluation of scheduling tasks in many-core systems utilizing processes and threads

Mejgan Dedaj, Argyro Gailla, Theofanis Ioannou, Stamatia Kastrinaki 2026-07-07
Memory Microarchitecture Simulation Scheduling ×

This study evaluates the scalability of process-based and thread-based schedulers for many-core systems using a memory-intensive quick-sort workload on large tensors. The method compares bounded prolific, bounded collective, and three pipe-based producer-consumer schedulers for processes, alongside static, dynamic, guided, chunk-based, chunk-stealing, adaptive chunk, and AIMD adaptive scheduling strategies for threads. Experimental results on a 24-core x86-64 platform show that thread schedulers, particularly dynamic and guided, deliver the highest overall performance, while pipe-based process schedulers demonstrate strong scalability with one-to-one pipes excelling for smaller workloads and many-to-many pipes for larger workloads. This matters because it identifies lightweight thread scheduling as optimal for shared-memory row sorting, while AIMD/adaptive and pipe-based schedulers remain valuable for contention-aware execution and distributed-style heterogeneous workload management.

PDF

GORIO: GPU-Centered Remote I/O for Graph ANNS over NVMe-oF

Gen Zhang, Wenhao Gu 2026-07-07
GPU Search Data Design Cache Scheduling × Workload

GORIO addresses the problem that graph-based approximate nearest neighbor search (ANNS) vector indexes often exceed GPU memory, and existing CPU-centered remote I/O over NVMe-oF is poorly matched to GPU graph traversal. The method keeps all query evolution, page-miss generation, and resume decisions on the GPU, using the CPU only as an NVMe-oF transport proxy, with a two-layer design for GPU-direct remote I/O and ANNS-specific scheduling. On a SIFT1M DiskANN-style workload over RDMA NVMe-oF, GORIO achieves 1.31× speedup over the state-of-the-art remote-I/O reference and 3.73× over the direct remote page-cache path. This matters because it provides a concrete GPU-centered remote I/O substrate that significantly accelerates graph ANNS for vector databases and retrieval-augmented generation services.

PDF