RunanywhereAI Speeds AI Inference on Apple Silicon

Y Combinator-Backed RunAnywhereAI Launches MetalRT, Claims Fastest On-Device AI Inference on Apple Silicon

Key Facts

What: RunAnywhereAI released MetalRT, a custom inference engine using Metal shaders that outperforms llama.cpp, Apple’s MLX, Ollama, and sherpa-onnx across LLMs, speech-to-text (STT), and text-to-speech (TTS) on Apple Silicon.
Performance: LLM decode up to 1.67x faster than llama.cpp and 1.19x faster than MLX; STT processes 70 seconds of audio in 101 ms (714x real-time, 4.6x faster than mlx-whisper); TTS synthesis in 178 ms (2.8x faster than mlx-audio and sherpa-onnx).
Open Source Release: RCLI, an MIT-licensed end-to-end on-device voice AI pipeline built on MetalRT, is now available via Homebrew with one-command install and includes local RAG, 38 macOS actions, and a full-screen TUI.
Founders: Sanchit and Shubham, Y Combinator W26 batch.
Technical Approach: Custom Metal compute shaders, zero runtime allocations during inference, unified engine for all modalities, and pre-compiled kernels that bypass traditional framework overhead.

Lead paragraph

Y Combinator-backed startup RunAnywhereAI has open-sourced RCLI and its underlying MetalRT inference engine, claiming it delivers the fastest end-to-end on-device AI performance on Apple Silicon for large language models, speech-to-text, and text-to-speech workloads. The project, announced on Hacker News, promises sub-200 ms voice response times entirely locally without cloud APIs or API keys. By writing custom Metal shaders and eliminating framework layers, the team says it has solved the latency compounding problem that has driven many developers back to cloud services.

Body

The core innovation is MetalRT, described by the founders as the first inference engine to natively support LLM, STT, and TTS modalities on Apple Silicon using a single unified runtime. Traditional inference stacks introduce multiple layers of abstraction — graph schedulers, runtime dispatchers, and dynamic memory managers — between the application and the GPU. MetalRT removes these by using hand-written Metal compute shaders for quantized matrix multiplication, attention mechanisms, and activations, all compiled ahead of time and dispatched directly.

Benchmark results published by the company show significant gains. For LLM decoding on the same model files, MetalRT achieves 658 tokens per second on Qwen3-0.6B compared to 552 tok/s with mlx-lm and 295 tok/s with llama.cpp. On the larger Qwen3-4B model, it reaches 186 tok/s versus 170 tok/s for MLX and 87 tok/s for llama.cpp. Time-to-first-token is reported at 6.6 milliseconds. The team also benchmarked LFM2.5-1.2B at 570 tok/s.

Speech workloads show even more dramatic improvements. MetalRT transcribes 70 seconds of audio in 101 milliseconds, equating to 714x real-time performance and 4.6x faster than Apple’s mlx-whisper. Text-to-speech synthesis completes in 178 milliseconds, 2.8x faster than competing mlx-audio and sherpa-onnx implementations.

RCLI, the open-source reference application built on MetalRT, demonstrates these gains in a complete voice pipeline. The tool supports push-to-talk interaction, local retrieval-augmented generation (RAG) with approximately 4 ms latency over 5,000+ document chunks, 20 hot-swappable models, and 38 native macOS actions controllable by voice. It uses three concurrent threads with lock-free ring buffers and double-buffered TTS to minimize latency. The pipeline falls back to llama.cpp if MetalRT is not installed.

Installation is deliberately simple for developers and enthusiasts. Users can run a one-line curl installer or use Homebrew:

brew tap RunanywhereAI/rcli https://github.com/RunanywhereAI/RCLI.git
brew install rcli
rcli setup
rcli

The rcli setup command downloads approximately 1 GB of models. A full-screen terminal user interface provides per-operation latency readouts, giving developers visibility into each stage of the pipeline.

In their announcement, founders Sanchit and Shubham explained the motivation: “We built this because demoing on-device AI is easy but shipping it is brutal. Voice is the hardest test: you’re chaining STT, LLM, and TTS sequentially, and if any stage is slow, the user feels it.” They noted that latency compounding in a voice pipeline quickly becomes noticeable — three stages each adding 200 ms results in 600 ms of delay before the user hears a response.

Impact

The release arrives amid growing interest in on-device AI on Apple Silicon. Apple has invested heavily in its unified memory architecture, Neural Engine, and MLX framework, which has become a popular choice for local LLM inference. Recent Apple Machine Learning Research highlighted MLX’s ability to deliver substantial speedups on M5 chips compared to M4, particularly for large matrix operations.

RunAnywhereAI’s approach takes a different path by bypassing higher-level frameworks entirely in favor of lower-level Metal shaders. This mirrors earlier community efforts, such as various Rust, C, and Zig-based inference projects discussed on Hacker News, but claims broader modality support and superior end-to-end voice performance.

For developers, the availability of a fast, fully local voice pipeline could accelerate building privacy-focused applications, offline AI assistants, and edge AI experiences on Mac, iPad, and potentially future iOS devices. The MIT license on RCLI lowers the barrier for integration into commercial products.

What's next

The company has published detailed methodology and optimization blogs covering LLM decode, speech benchmarks, voice pipeline design, and on-device RAG. A YouTube demo video is available showing the system in action.

RunAnywhereAI has not announced specific commercialization plans, enterprise offerings, or support for additional platforms beyond Apple Silicon. The team invites the community to explore what they would build if on-device AI were “genuinely as fast as cloud.”

The project is currently in early stages, with the GitHub repository serving as the primary distribution point. Performance claims are based on the company’s own benchmarks and should be independently verified across different Apple Silicon chips and model configurations.

Show HN: RunAnwhere – Faster AI Inference on Apple Silicon

Sources

Original Source

Related Topics

Comments