RunAnywhere (YC W26) Speeds AI Inference on Apple Silicon

RunAnywhere Launches MetalRT, Claims Fastest On-Device AI Inference on Apple Silicon

Key Facts

What: RunAnywhere (YC W26) open-sourced RCLI and released MetalRT, a custom Metal-based inference engine that outperforms llama.cpp, Apple’s MLX, Ollama, and sherpa-onnx across LLM decode, speech-to-text (STT), and text-to-speech (TTS).
Performance: LLM decode reaches 658 tok/s on Qwen3-0.6B (1.67× faster than llama.cpp, 1.19× faster than MLX); STT processes 70 seconds of audio in 101 ms (714× real-time, 4.6× faster than mlx-whisper); TTS synthesis in 178 ms (2.8× faster than mlx-audio and sherpa-onnx).
When: Launched via Hacker News on the Y Combinator W26 batch, with immediate open-source availability.
Availability: Install via Homebrew (brew install rcli) or install script; downloads ~1 GB of models; fully on-device with no cloud or API keys required.
Tech: Custom Metal compute shaders, zero runtime allocations, unified engine for all three modalities, pre-allocated memory, lock-free ring buffers in the voice pipeline.

Lead paragraph

RunAnywhere, a Y Combinator W26 startup founded by Sanchit and Shubham, has released MetalRT, a high-performance inference engine purpose-built for Apple Silicon, along with RCLI, an open-source end-to-end voice AI pipeline that runs entirely on-device. The company claims MetalRT delivers faster inference than existing solutions across large language models, speech-to-text, and text-to-speech by using custom Metal shaders that eliminate framework overhead. The launch aims to solve the latency challenges that currently push developers toward cloud APIs when building responsive on-device AI applications, particularly voice interfaces.

Body

The core innovation is MetalRT, described as the first inference engine to natively support LLM, STT, and TTS modalities on Apple Silicon using direct Metal GPU compute shaders. According to the team, most existing inference runtimes introduce layers of graph schedulers, runtime dispatchers, and memory managers between the application and the GPU. MetalRT removes these abstractions by compiling custom shaders for quantized matrix multiplication, attention, and activation functions ahead of time and dispatching them directly.

Benchmark results published by RunAnywhere show significant gains. For LLM decode on an M4 Max, Qwen3-0.6B achieves 658 tokens per second compared to 552 tok/s with mlx-lm and 295 tok/s with llama.cpp. Larger models follow similar trends: Qwen3-4B reaches 186 tok/s (versus 170 and 87), and LFM2.5-1.2B hits 570 tok/s (versus 509 and 372). Time-to-first-token is reported at 6.6 milliseconds. The same engine delivers 714× real-time performance on speech-to-text, transcribing 70 seconds of audio in 101 ms — 4.6× faster than Apple’s mlx-whisper — and synthesizes speech in 178 ms, 2.8× faster than competing TTS solutions.

RCLI, the open-source reference implementation built on MetalRT (MIT license), demonstrates these capabilities in a complete voice pipeline. It features three concurrent threads using lock-free ring buffers, double-buffered TTS output, local retrieval-augmented generation (RAG) with roughly 4 ms latency over 5,000+ document chunks, support for 20 hot-swappable models, and a full-screen terminal user interface displaying per-operation latency. The tool includes 38 macOS voice actions and falls back to llama.cpp if MetalRT is not installed. Installation is straightforward via Homebrew or a one-line curl script, after which rcli setup downloads approximately 1 GB of models.

The founders explained that they focused on voice AI because it represents the most demanding latency test. In a typical voice pipeline, STT, LLM inference, and TTS run sequentially. Even if each stage adds only 200 ms of latency, the cumulative delay exceeds 600 ms, creating a sluggish user experience. Traditional approaches often rely on cloud APIs to mask this latency. By building a single unified engine with all memory pre-allocated at initialization and zero allocations during inference, RunAnywhere says it has eliminated compounding delays while keeping everything local.

Detailed technical write-ups are available on the company’s blog covering MetalRT’s LLM decode optimizations, speech benchmark methodology, voice pipeline architecture, on-device RAG techniques, and additional performance data.

Impact

For developers and independent builders, the release lowers the barrier to creating fast, private, on-device AI experiences on Macs and, by extension, potentially future iOS and iPadOS applications. The open-source RCLI serves as both a working demonstration and a foundation for new voice-first tools, meeting assistants, and accessibility applications that do not require internet connectivity or send data to third-party servers.

Enterprise interest is also likely. RunAnywhere positions itself as a full-stack infrastructure layer for on-device intelligence, handling model delivery, storage, and multi-engine abstraction. Earlier announcements indicate the company is building toward production-grade solutions for iOS and Android, suggesting today’s Apple Silicon-focused release is an early milestone in a broader cross-platform strategy launched from the Y Combinator W26 batch in January 2026.

The performance numbers, if independently verified at scale, could influence how teams evaluate local versus cloud inference, especially for latency-sensitive applications in healthcare, education, automotive, and consumer electronics where data privacy and offline capability matter.

What's next

RunAnywhere has not yet detailed a specific timeline for iOS or Android SDK availability, though company materials describe plans for a unified platform supporting both mobile operating systems. The team invites the community to experiment with RCLI and share ideas for applications that become practical once on-device inference matches or exceeds cloud responsiveness.

Further optimizations, expanded model support, and additional RAG capabilities are expected as the project evolves. The company has published extensive methodology and benchmark blogs that developers can use to validate claims and build upon the open-source foundation.

The launch has already generated discussion on Hacker News, where the post reached 108 points with 34 comments, reflecting strong interest in practical on-device AI infrastructure.

Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

Sources

Original Source

Related Topics

Comments