or use Xcode to create a new macOS App

Building Real-Time Voice Agents on Apple Silicon with RunAnywhere’s MetalRT

RunAnywhere’s MetalRT lets you run LLMs, STT, and TTS on-device with Metal shaders and zero runtime overhead, delivering 1.67× faster decode than llama.cpp and 714× real-time STT on Apple Silicon.

This changes the game for builders who want low-latency voice experiences that never touch the cloud. You can now ship fluid, private voice agents that feel instantaneous because the entire pipeline (mic → STT → LLM → TTS) stays under 400 ms on an M-series Mac.

Why this matters for builders

Until now, on-device voice AI was either slow or required stitching fragile runtimes (llama.cpp + whisper.cpp + Piper + custom glue). MetalRT is a single unified inference engine with custom compute shaders, pre-allocated memory, and lock-free concurrency primitives. The open-source RCLI demonstrates the full end-to-end pipeline and is MIT licensed, giving you a production-grade starting point you can fork or learn from.

When to use it

• You are building a Mac-native voice assistant, dictation tool, or accessibility product.
• You need sub-500 ms end-to-end latency for conversational voice.
• Privacy is non-negotiable (no cloud, no API keys).
• You want to ship to Apple Silicon users today and keep the option to add iOS/Android later via the RunAnywhere platform.
• You are comfortable editing Swift, C++, or using the provided rcli binary as a reference implementation.

The full process

1. Define the goal (30 minutes)

Write a one-paragraph spec:

“Build a Mac menu-bar voice agent that listens on push-to-talk (or hotword), transcribes locally with MetalRT, runs a small Qwen3-4B or LFM2.5-1.2B model with system prompt for helpful actions, then speaks the response. Support local RAG over my notes folder. Show per-stage latency in a small HUD. Stay under 450 ms end-to-end on M3 Pro. No network calls.”

Decide on model sizes, target latency budget, and fallback behavior.

2. Shape the prompt for your AI coding assistant

Use this starter prompt (copy-paste ready):

You are an expert Mac developer building with RunAnywhere’s MetalRT.

Project: Real-time voice agent using rcli / MetalRT on Apple Silicon.

Requirements:
- Use the official rcli binary or link against MetalRT when available.
- Three-stage pipeline: STT → LLM → TTS.
- Use lock-free ring buffers or GCD for concurrency.
- Pre-warm all models on launch.
- Show live latency numbers for each stage.
- Support local RAG over a folder of markdown files (use the 4 ms retrieval technique from RunAnywhere’s blog).
- Graceful fallback to llama.cpp if MetalRT is not installed.
- SwiftUI menu bar app with optional full-screen TUI mode.

Generate:
1. Project structure (Swift + optional C++ bridging).
2. Exact brew install and setup commands.
3. Initialization code that runs `rcli setup`.
4. Pipeline orchestration with measured timings.
5. Sample system prompt and RAG context injection.

3. Scaffold the project

mkdir voice-agent && cd voice-agent
swift package init --type executable
brew tap RunanywhereAI/rcli https://github.com/RunanywhereAI/RCLI.git
brew install rcli
rcli setup

Create a Sources folder and add a simple Swift wrapper that spawns rcli in interactive mode or calls the underlying MetalRT APIs once the C++ headers are available.

4. Implement the core pipeline

Key architecture (from RCLI):

• Three concurrent threads
• Lock-free ring buffers between stages
• Double-buffered TTS so synthesis never blocks playback
• Pre-allocated Metal buffers (zero runtime allocations)

Example Swift orchestration skeleton:

// Pipeline.swift
class VoicePipeline {
    private let sttEngine = MetalRTSTT()
    private let llmEngine = MetalRTLLM(model: "Qwen3-4B")
    private let ttsEngine = MetalRTTTS(voice: "alloy")

    func processAudioBuffer(_ buffer: AVAudioPCMBuffer) async -> String {
        let start = CACurrentMediaTime()
        
        let text = try await sttEngine.transcribe(buffer)   // ~100 ms
        let sttMs = (CACurrentMediaTime() - start) * 1000
        
        let context = await rag.retrieveRelevantChunks(text)
        let prompt = buildPrompt(userText: text, ragContext: context)
        
        let response = try await llmEngine.generate(prompt)  // ~200-300 ms
        let llmMs = ...
        
        try await ttsEngine.speak(response)                  // ~180 ms
        
        updateHUD(latencies: ["STT": sttMs, "LLM": llmMs, "TTS": ttsMs])
        return response
    }
}

5. Validate performance

Run these checks:

• Measure end-to-end latency with rcli in interactive mode and compare against the 658 tok/s and 101 ms STT numbers in the announcement.
• Test on M1, M2, M3, and M4 devices.
• Verify memory usage stays flat (MetalRT pre-allocates everything).
• Test RAG retrieval time over 5,000+ chunks — should be ~4 ms.
• Record a 30-second conversation and confirm no perceptible lag.

Use Instruments to confirm zero unexpected allocations during inference.

6. Ship safely

• Bundle the models inside the app or use rcli setup on first launch.
• Add a clear privacy policy stating everything runs locally.
• Provide a one-click “Reset models” command.
• Include fallback path: if MetalRT fails to load, switch to llama.cpp automatically (RCLI already does this).
• Release as a menu-bar app first, then consider a full Swift package or iOS version using the upcoming RunAnywhere SDK.

Pitfalls and guardrails

### What if MetalRT is not installed on the user’s machine?
RCLI already falls back to llama.cpp. Detect this and show a non-blocking “Using CPU fallback — install MetalRT for 2-3× speed” banner.

### What if latency compounds above 500 ms?
Profile each stage separately. The most common culprit is cold-start model loading. Always pre-warm at app launch. Use smaller models (Qwen3-0.6B or LFM2.5-1.2B) for the first release.

### What if the user has an older Intel Mac?
Gracefully degrade to a cloud option or show an “Apple Silicon only” message. The current MetalRT engine targets Apple Silicon.

### How do I add custom actions (calendar, email, etc.)?
RCLI ships with 38 macOS actions. Extend the system prompt with AppleScript or Shortcuts integration. Keep the prompt under 4k tokens to maintain speed.

What to do next

1. Ship the menu-bar MVP this week.
2. Add local RAG over the user’s Documents folder.
3. Instrument and publish your latency numbers (compare against the official benchmarks).
4. Explore the RunAnywhere full-stack SDK when it becomes available for iOS/Android.
5. Open-source your agent under MIT so others can learn from your pipeline orchestration.

You now have a repeatable process for turning the RunAnywhere launch into a real product instead of another forgotten demo.

The on-device voice future is here — and it’s faster than most cloud APIs.

Sources

• Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon (https://news.ycombinator.com/item?id=47326101)
• RCLI GitHub repository (https://github.com/RunanywhereAI/RCLI)
• MetalRT LLM Decode Benchmarks (https://www.runanywhere.ai/blog/metalrt-fastest-llm-decode-e...)
• Speech Benchmarks & Voice Pipeline Optimizations (https://www.runanywhere.ai/blog/metalrt-speech-fastest-stt-t... and https://www.runanywhere.ai/blog/fastvoice-on-device-voice-ai...)
• RunAnywhere YC company page and official announcements (January–March 2026)