Hypura Breaks Memory Limits: Run 70B LLMs on 32GB Macs with New Tiered Scheduler
- What: Hypura, a storage-tier-aware LLM inference scheduler for Apple Silicon.
- Key Feature: Enables models that exceed physical memory to run by tiering tensors across GPU, RAM, and NVMe.
- Performance: Runs a 40 GB Llama 70B at 0.3 tok/s and a 31 GB Mixtral 8x7B at 2.2 tok/s on 32 GB hardware.
- Availability: Open-source project available via GitHub for M-series Mac hardware.
A new open-source inference scheduler called Hypura has launched, allowing Apple Silicon users to run Large Language Models (LLMs) that significantly exceed their device’s physical memory capacity. By intelligently distributing model tensors across GPU, RAM, and NVMe storage tiers based on access patterns, the tool prevents the "Out of Memory" (OOM) crashes and system "swap-thrashing" that typically occur on consumer Mac hardware when loading massive AI models.
Bypassing the Unified Memory Ceiling
The primary bottleneck for local AI on Mac hardware is memory capacity. While Apple’s M-series chips offer high-bandwidth unified memory, many users are limited to 16 GB or 32 GB configurations. According to Hypura’s documentation, a standard 32 GB M1 Max cannot naively load a 40 GB model without the operating system’s OOM killer intervening.
Hypura solves this by treating the Mac's fast NVMe storage as a functional extension of its memory pool. The scheduler profiles the hardware's GPU working set and NVMe bandwidth to solve a placement optimization problem, assigning every tensor to the most efficient tier possible. This allows users to run a 31 GB Mixtral 8x7B model on a 32 GB Mac Mini—a feat that causes standard tools like llama.cpp to crash.
Technical Architecture: Three-Tiered Scheduling
Hypura operates by categorizing model components into three distinct hardware layers:
- GPU (Metal): This tier hosts attention layers, norms, and embeddings. These are small but accessed every token, requiring the fastest possible speeds.
- RAM: This serves as an overflow layer for tensors that do not fit in the primary GPU working set but still require relatively fast access via memory mapping (mmap).
- NVMe: The remaining bulk of the model is loaded on demand from storage using direct I/O. Hypura uses speculative prefetching to load these weights just before they are needed in the forward pass.
For Mixture of Experts (MoE) models like Mixtral, Hypura utilizes "expert-streaming." Because only two out of eight experts are typically activated per token, the scheduler intercepts the router's decision and loads only the specific expert strides needed from NVMe. This results in a reported 75% reduction in I/O requirements.
Optimized Performance Benchmarks
In technical testing on an M1 Max with 32 GB of unified memory and 5.1 GB/s NVMe sequential read speeds, Hypura demonstrated significant stability improvements over existing solutions.
When running Qwen 2.5 14B, which fits entirely within memory, Hypura achieved 21 tokens per second (tok/s) with zero overhead compared to native Metal performance. However, the scheduler’s true utility appeared with larger models:
- Mixtral 8x7B (Q5_K_M, 30.9 GB): Achieved 2.2 tok/s using expert-streaming, maintaining a 99.5% neuron cache hit rate.
- Llama 3.3 70B (Q4_K_M, 39.6 GB): Achieved 0.3 tok/s using "dense-FFN-streaming," where 31.8 GB of weights were streamed from NVMe while attention and norms remained resident on the GPU.
In both of these high-capacity scenarios, the documentation notes that standard llama.cpp implementations resulted in immediate system crashes.
Impact for Developers and Power Users
For the first time, developers with mid-tier consumer Macs can experiment with flagship-class models that were previously reserved for high-end Mac Studio or Mac Pro configurations. By implementing a neuron cache that tracks loaded expert slices across tokens, Hypura exploits temporal locality to ensure that once a model "warms up," I/O overhead is minimized.
"Hypura turns high-speed NVMe storage into a functional extension of Apple's unified memory, effectively bypassing the 'Out of Memory' killer," the project's documentation states. This shift means that the cost of entry for running 70B+ parameter models locally has dropped from the price of a $4,000+ workstation to a standard $600-$1,000 consumer Mac, albeit at lower token-per-second rates.
What’s Next
As a community-driven open-source project, Hypura is currently focusing on refining its "RESEARCH_INTEGRATION_PLAN." The tool already features automated hardware profiling, meaning users do not need to manually tune memory budgets or prefetch depths. Future updates are expected to further optimize the prefetch lookahead depth, which currently scales automatically based on available system memory.
The project utilizes GGUF files, making it compatible with the most common quantized model formats used by the local AI community. While currently an independent tool, its tiered-storage approach represents a potential shift in how local inference engines may handle the growing disparity between model sizes and consumer hardware limits.
Sources
All technical specifications, pricing, and benchmark data in this article are sourced directly from official announcements. Competitor comparisons use publicly available data at time of publication. We update our coverage as new information becomes available.