Core ML vs MLX: Why We Chose Core ML for Sequence

When people search Core ML vs MLX, they usually want to know which stack is better for running on-device transcription inside a native Mac app.

We had to answer that question for Sequence, a native Mac app for transcription and editing. We started by exploring MLX and several existing Whisper-based options. We ultimately shipped inference with Core ML. This is the comparison we wish we had when we began, along with the product reasons behind our decision.

Quick answer

	Core ML	MLX
Best at	Shipping inference inside apps	Research, training, fine-tuning, and experimentation
Primary audience	App developers shipping to users	ML engineers iterating on models
Hardware path	Hardware path: CPU, GPU, and Neural Engine options through Core ML	Apple silicon GPU and CPU through an open array framework
Ship format	Compiled .mlmodelc or .mlpackage with explicit I/O	Flexible model code and runtime integrations
Fit for Sequence	Production transcription runtime	Useful for exploration, but not our production path

Short version: MLX is a productive environment for exploring, fine-tuning, and experimenting with models on Mac. Core ML is designed to run models inside Apple-platform apps, with a stable packaging story and the ability to use different compute units, including the Neural Engine when the model and device support it. For a product editors trust every day, we needed the second job more than the first.

The two tools can also be part of the same broader workflow, but there is no automatic MLX-to-Core-ML conversion path that we could simply rely on. A team might explore or train with MLX, then port or re-implement the model in a framework supported by coremltools, validate its outputs, and package the resulting model for an app.

What we were optimising for

Sequence is not a research lab. It is an editing tool for people who cut video and audio for a living. Transcription has to:

• Stay on-device for privacy, offline use, and no API costs
• Hold up during long sessions on laptops as well as desktops
• Be power-aware when someone is editing on battery
• Ship as a predictable runtime, not a constantly changing research workflow
• Match quality we can defend with repeatable measurements

That list changes the Core ML vs MLX decision. Benchmarks matter less than the experience, it must feel solid during many hours of a podcast editing.

Where MLX shines

We did start with MLX, and it earned the enthusiasm around it.

• Fast iteration. If you are adapting models, trying decoding changes, or fine-tuning, MLX feels close to the hardware and close to the research ecosystem.
• Familiar ML ergonomics. Its array-first design and training tools make it a productive place to explore models and experiments.
• Apple silicon native. Unified memory and GPU-friendly execution make Mac a serious place to train and prototype, not just a thin client for a remote GPU box.
• Language flexibility. Python is common in the MLX ecosystem, but MLX can also be integrated with C++, C, Rust, Swift, and likely many other languages through its available APIs and bindings.

If your goal is to understand a model, fine-tune it, test an architectural idea, or iterate on a checkpoint, MLX can be a very good home for that work.

What we tried

Our decision was not based on a theoretical comparison. We tried to fit real transcription systems into the architecture we were building for Sequence.

We trialled MLX alongside Rust-based components, and we also experimented with prebuilt Whisper libraries. Those experiments were useful, but many of the pieces did not fit the architecture we had envisioned. The issue was not that the libraries were bad, far from it; it’s just we had a very specific way to handle inference in Sequence.

We needed a native, predictable runtime that could sit alongside the rest of a macOS video-editing application, behave consistently across supported Apple silicon Macs, and give us control over the boundaries between model execution, decoding, and post-processing. That pushed us away from the prebuilt libraries and towards building a more custom Core ML path.

Where Core ML won for us

Once we evaluated the product path, three things made Core ML the clear winner:

1. Control over the compute path

Core ML gave us a single native model runtime, but it did not mean blindly sending every transcription to every available compute unit. We could decide which parts of Apple silicon to involve based on the size and expected duration of the job.

That matters because the GPU is not always the fastest option. For a very short transcription, the overhead of waking up and preparing GPU execution can outweigh the performance benefit. In those cases, CPU and Neural Engine execution was faster in our tests, so our code avoids triggering the GPU for small jobs.

For longer transcriptions, the balance changes. GPU execution can provide stronger throughput when its startup cost is amortised across a larger workload. This gave us a practical way to tune the runtime around the job rather than assuming that one compute path was optimal for every clip.

2. Packaging we could ship

The second question was how models reach users.

Core ML and Xcode gave us a straightforward way to package the model as part of the app, but packaging was not just about convenience. We encrypt the majority of the model components so they cannot be casually inspected or modified, and so the runtime can trust that the pieces it loads are the ones we validated.

That matters for two reasons. First, it helps prevent external tampering with the transcription pipeline. Second, it gives editors confidence that what they downloaded is the exact product Sequence intends them to use. If one component is changed, the package should stop working and require a fresh download rather than quietly drifting out of spec.

That packaging discipline is boring in the best way. Editors should never have to care which framework we used. They should only care that transcripts show up quickly and accurately.

3. A harness we could harden

We wired together:

• Export parity checks against reference behaviour
• Tensor-boundary validation
• Decode-loop corrections
• Layered post-processing hardening

That stack got us to 2.54% WER on LibriSpeech test-clean with our Parakeet-oriented Core ML path. NVIDIA published 6.05% for Parakeet v3 - a seven-dataset macro average, so it is not a direct comparison with our test-clean result.

Core ML vs MLX by decision, not dogma

Choose MLX when you are:

• Training or fine-tuning on Apple silicon
• Prototyping architectures and decoding strategies daily
• Working primarily in a research workflow, whether that workflow uses Python, Rust, C++, Swift, or another supported language
• Optimising for flexibility and experiment velocity

Choose Core ML when you are:

• Embedding inference in a Mac or multi-platform Apple app
• Shipping compiled models with stable input and output contracts
• Caring about compute-unit selection inside a real UI process
• Optimising for supportability, battery behaviour, and release hygiene

Use both when the workflow supports it. Explore or train in MLX, PyTorch, or another suitable framework, then port or re-implement the model as necessary, convert it with coremltools, and validate the Core ML output before shipping. That is closer to how the tools fit together than a tribal either-or choice.

What we would tell our past selves

1. Separate “can we run it?” from “can we ship it?” MLX answered the first question quickly. Core ML answered the second for Sequence.
2. Measure the product loop, not only tokens per second. Long-form audio, memory pressure next to an NLE, thermal throttling, and battery drain can change the winner.
3. Treat prebuilt libraries as starting points, not guarantees. A library can be useful in isolation and still be a poor fit for the boundaries and lifecycle of your application.
4. Budget for the harness. Export parity and decode hardening moved the needle for us.
5. Keep the story honest. MLX is not worse. It is optimised for a different lifecycle.

The outcome for Sequence

Core ML is the runtime we trust editors with.

It is how on-device transcription stays local, fast and reliable.

If you want the product side of that work, read How to Transcribe Your Footage Locally on Your Mac.