Swift Speech Recognition and Typed Decisions with Open Models on Apple

Moving Beyond Text Generation for Speech and Classification

Relying on a general-purpose large language model to transcribe audio or route support tickets often introduces unnecessary latency, higher compute costs, and brittle text parsing. Purpose-built speech recognition and classification models can execute these tasks faster and more reliably on local hardware.

Building on the foundation of AnyLanguageModel and swift-huggingface, two new Swift packages—AnySpeechRecognizer and AnyDecisionModel—bring modular, open-weight model support to Apple platforms. Both libraries pull models directly from the Hugging Face Hub and store them in the standard cache so macOS tools and Python environments can share weights without duplicate downloads.

Drop-In Speech Recognition and Speaker Diarization

AnySpeechRecognizer mirrors Apple's SpeechAnalyzer API—including SpeechTranscriber, SpeechDetector, and AssetInventory—while allowing developers to swap the underlying engine. By changing the import and transcriber initialization, apps can switch from Apple's default engine to Core ML conversions of NVIDIA Parakeet or Alibaba Qwen3-ASR.

The package also fills a major gap in Apple's native framework by introducing SpeakerDiarizer. Powered by NVIDIA's Streaming Sortformer (v2 and v2.1) on Core ML, it tracks up to four overlapping speakers on the same timeline and attributes timed words to specific participants.

• Parakeet v3, Ultra, and Unified run on a custom Core ML runtime supporting live audio streams, word timings, confidence scores, and an 8-second live encoder.
• Qwen3-ASR (0.6B and 1.7B) runs on the GPU and covers 30 languages plus 22 Chinese dialects, making it well suited for batch file transcription.
• Each Hub repository pins revisions and records source checkpoints and file checksums to keep app downloads reproducible.

Typed Decisions and Calibrated Probabilities in Swift

When an application needs a yes-or-no flag, an enum routing choice, or a rating-scale score, parsing free-form LLM text hides the model's uncertainty. AnyDecisionModel replaces text generation with a stateful DecisionSession that evaluates independent questions against text or JSON states and returns strongly typed Swift results alongside option probabilities and entropy-based confidence scores.

Developers can run standard language models via MLX, Core AI, or llama.cpp by reading single-token answer probabilities in a single forward pass, or use dedicated decision-head models like Cloudflare's 9B Clef-Flash on MLX and System One endpoints via Jev or Ollama. To improve reliability on local next-token backends, the library supports temperature and bias calibration as well as rotation debiasing to counter positional bias in multiple-choice lists.

Practical Implications and Workflow in HuyIDE

For Swift and macOS engineers, separating audio ingestion and structured triage from heavy chat generation keeps memory usage predictable and makes downstream application logic type-safe. However, tuning probability thresholds, wiring up evaluation scripts like natural-speech-eval.py, and managing multi-backend Swift traits require coordinated edits across Swift packages and Python tooling.

When prototyping these pipelines on an Apple Silicon Mac in HuyIDE, you can use Spec Workflow to define Given/When/Then criteria for your Swift Choosable enums and calibration thresholds before writing code. You can also connect a local Ollama instance through HuyIDE's AI Provider Hub to test local model workflows alongside your codebase, and press left Cmd and right Cmd together to capture your running macOS test app window directly into HuyIDE chat for visual debugging.

• Use HuyIDE's RepoMap indexing and Hybrid Rust N-API Core Engine to navigate across Swift package traits and Python evaluation scripts.
• Inspect real-time Git worktree ChangeSet diffs in HuyIDE before committing updated model revision pins or threshold configurations.