Skip to content
Apple AI Guides
Apple on-device AI
Languages: Swift · Python · Metal

Developer guide series

Apple on-device AI

Foundation Models, Core AI, MLX, Evaluations, Speech, and Metal—from first API call to shipping on device.

Apple on-device AI — a developer's guide series

Covers: iOS 27 · iPadOS 27 · macOS 27 · watchOS 27 · visionOS 27 · tvOS 27 · Xcode 27 Frameworks: Foundation Models · Core AI · MLX · Evaluations · Speech · Metal Performance Primitives

Current snapshot (2026-10-09): 17 parts, 60 reference guides, 1201 indexed symbols, and 1,831 classified warnings (1,469 concrete silent failures). Installed verification environment: macOS 27.0 (26A428), Xcode 27.0 (27A266a). Stable-release guidance leads; beta-only APIs and measurements carry their own evidence dates and platform identity. See current state for latest observed releases, validation dates, and destination-specific baselines. Observed stable releases: Xcode 27, iOS 27.0.1, macOS 27.0.1, ml-explore/mlx 0.32.3, ml-explore/mlx-lm 0.32.0, ml-explore/mlx-swift 0.32.3, ml-explore/mlx-swift-lm 3.32.3, apple/python-apple-fm-sdk 0.2.1. Release availability does not attest untested runtime behavior.

Seventeen parts covering Apple's 2026 on-device AI stack end to end — from a three-line LanguageModelSession call down to a hand-written Metal matmul kernel, and back up through shipping, updating and evaluating a model in production.

Pick your path

New to the stack? Start with Part 1

How the four product lines fit together, how to choose a model backend, and every version, hardware and entitlement gate.

Build an AI feature in a Swift app

LanguageModelSession, guided generation, tools, and failure handling — then Parts 3–4 for agentic sessions and custom backends.

Convert a PyTorch model for on-device use

torch.export to .aimodel, then Parts 9–10 for compression, hardware authoring, and LLM deployment.

Work in Python with MLX

Fundamentals through fine-tuning and serving — and Part 13 when the model needs to ship inside a Swift app.

Debug something that fails silently

The silent-failure index, sorted by the symptom you observe: wrong output, empty output, performance cliff, version drift.

Migrate a pre-iOS-27 app

What changed between iOS 26 and 27, the adapter sunset, error-taxonomy migration, and dual-SDK builds.

Two cross-cutting indexes:

  • The silent-failure index — every warning callout in the series, in one page, sorted by the symptom you observe: wrong output, empty output, performance cliff, version drift, …
  • The API & symbol index — symbols → the guides that cover them, with presence flags against the captured SDK interfaces.

Evidence conventions

VERIFIED identifies the cited source or recorded check; it does not imply every device or release was tested. RECONSTRUCTED marks an inferred composition or spelling. GAP marks an unresolved limitation and its safe default. Runtime claims retain their toolchain, destination, and observation date; community measurements retain their attribution. GitHub closure, released source, and demonstrated remediation are separate observations.

How the series is organized

Each part is a directory:

part-NN-<slug>/
  README.md          ← the part's master guide: orientation, reading order, decision tables
  references/        ← the individual deep-dive guides for that part
    <nn>-<slug>.md

Read a part's README.md first. It tells you which reference guides you actually need and in what order; most readers need two or three of them, not all.1

The one-paragraph map

In 2026 the four product lines stopped being alternatives and became layers. The Foundation Models framework is no longer "the API for Apple's on-device LLM" — LanguageModelSession now sits on a public LanguageModel / LanguageModelExecutor protocol pair, and there are five conformers: SystemLanguageModel (rebuilt, 4,096-token context, now accepts images), PrivateCloudComputeLanguageModel (32K, three reasoning levels, no API keys, per-user quota), CoreAILanguageModel, MLXLanguageModel, and ChatCompletionsLanguageModel — which quietly turns mlx_lm.server, Ollama, vLLM and LM Studio into Foundation Models backends today. So the question stopped being "which framework do I choose" and became "which backend do I choose behind one session API." That reframing is the spine of this series.

Underneath, Core AI is the inference framework that powers on-device Apple Intelligence, now public: a portable .aimodel → specialization → AIModel → InferenceFunction → NDArray. MLX is the open array framework and the fastest path from a Hugging Face checkpoint to something running. Evaluations cuts across everything and is Apple's answer to the fact that there is no model version pinning API. Metal Performance Primitives / TensorOps is the floor both Core AI and MLX stand on.

The parts

Part Title For
1 Orientation and gating Everyone. Read first.
2 Foundation Models: the everyday API Swift app developers
3 Context, profiles, and agentic sessions Swift app developers
4 Beyond the built-in model App developers + package authors
5 Prototyping, profiling, non-Swift access Everyone
6 Evaluations Anyone shipping an AI feature
7 Core AI: the Swift runtime Swift developers shipping a model
8 Core AI: converting a model from PyTorch Python ML engineers
9 Core AI: compression and numeric formats Python ML engineers
10 Core AI: hardware authoring, debugging, and LLM deployment Python ML engineers
11 Metal and TensorOps Kernel authors
12 MLX in Python Python ML engineers
13 MLX in Swift Swift app developers
14 Bridges between stacks Anyone moving a model between stacks
15 Shipping and operating on device Everyone shipping
16 Adjacent capabilities Developers wiring an app into the system
17 Migration from pre-iOS 27 Anyone with a shipping iOS 26 app

Every guide in the series

All 60 reference guides, by part. Each part's README.md is the entry point and tells you which of its references you actually need.

Part 1 — Orientation and gating - The 2026 Apple AI stack, and how to choose a model backend - Every version, hardware, entitlement and runtime-surface gate

Part 2 — Foundation Models: the everyday API - LanguageModelSession end to end - Guided generation and snapshot streaming - The Tool protocol, calling modes, and the required-mode loop - Local RAG with SpotlightSearchTool, plus OCR and barcodes - Image input, and what the model cannot do with pixels - The complete failure taxonomy: availability, errors, guardrails and refusals

Part 3 — Context, profiles, and agentic sessions - Token budgeting, transcript anatomy, and KV-cache economics - Dynamic Profiles, modifiers, and session state - foundation-models-utilities: Skills and history transforms - Baton-pass, phone-a-friend, model routing, and tool-calling control

Part 4 — Beyond the built-in model - Private Cloud Compute: eligibility, reasoning, and quota UX - Core AI, MLX, and any OpenAI-compatible server behind LanguageModelSession - Authoring a LanguageModel provider package - Executor lifecycle, configuration identity, and preserving work across calls

Part 5 — Prototyping, profiling, and non-Swift access - #Playground, scheme simulation, and reading a Foundation Models trace - The fm CLI and the Foundation Models SDK for Python

Part 6 — Evaluations - Building blocks, Swift Testing integration, and evaluation-driven development - Model judges, score dimensions, drift, and Cohen's kappa - SampleGenerator, synthetic datasets, and evaluating tool trajectories

Part 7 — Core AI: the Swift runtime - AIModel, InferenceFunction, NDArray, and the memory model - Specialization, the model cache, and ahead-of-time compilation - States as KV cache, and pipelined execution - Model bundles, the LLM engines, and grammar-constrained decoding - Non-LLM engines: bundles, function structure, warmup, specialization, and caching

Part 8 — Core AI: converting a model from PyTorch - torch.export to .aimodel, and the IO / state / dynamic-shape contract - When an op will not convert: coverage, composites, custom lowerings, externalization - TorchMetalKernel: writing and embedding a custom Metal kernel

Part 9 — Core AI: compression and numeric formats - coreai-opt quantization: configs, GRAPH vs EAGER, calibration and QAT - Palettization, pruning, joint compression, and mixed precision - int4 to MX: which layer supports which numeric format

Part 10 — Core AI: hardware authoring, debugging, and LLM deployment - Authoring for the Neural Engine and for the GPU: two opposite rulesets - The debug gauge, the Core AI Instrument, and the Core AI Debugger - From a Hugging Face checkpoint to a loadable LLM bundle

Part 11 — Metal and TensorOps - TensorOps: matmul2d, tensor types, and what quantization actually looks like - Cooperative tensors, reductions, and building a fused attention kernel

Part 12 — MLX in Python - MLX fundamentals: unified memory, lazy evaluation, transforms, and compile - Numerics, hardware gating, and writing custom Metal kernels from Python - MLX quantization: modes, group sizes, gates, and the corruption bugs - mlx-lm: the CLI surface, the generation API, and KV caching - mlx_lm.server, local agents, and distributed inference over Thunderbolt - LoRA and DoRA fine-tuning, and adding a new architecture

Part 13 — MLX in Swift - mlx-swift-lm in an app: setup, concurrency, memory, and media input - Generation, tool calling, and KV cache management in Swift - MLXFoundationModels and MLXGuidedGeneration: backing LanguageModelSession with an MLX model

Part 14 — Bridges between stacks - Bridges into Core AI: mlx2coreai, swift-lm, and the community zoo

Part 15 — Shipping and operating on device - Shipping models: Background Assets, per-architecture variants, and updates - Memory, jetsam, thermals, energy, and measuring honestly

Part 16 — Adjacent capabilities - SpeechAnalyzer: live transcription, assets, and custom vocabulary - App Schema Domains: the complete map of what Siri can actually do - On-screen awareness: making Siri understand "this" - One index, three consumers: entities, Spotlight, and Foundation Models - DNIKit: auditing datasets and networks before you convert

Part 17 — Migration from pre-iOS 27 - What changed between iOS 26 and iOS 27: the complete checklist - The adapter sunset: migrating off custom LoRA adapters - Error taxonomy migration: GenerationError → LanguageModelError - Building for two SDKs: conditional compilation across 26 and 27 - Core ML to Core AI: what moves, what stays, and how - Toolchain and asset compatibility: when your build artifacts stop working

Editorial conventions

These are load-bearing. A large fraction of the source material for this series is beta-era, and a meaningful fraction of the API spellings in circulation are reconstructions from spoken WWDC narration rather than anything anyone has seen in writing. The conventions below exist so you always know which is which.

Evidence markers

See Evidence conventions for source, reconstruction, uncertainty, and measurement labels. Suggestive evidence was measured outside the target configuration and requires a dated target check before promotion.

Silent-failure callouts

A SILENT FAILURE callout identifies a problem that produces no useful error. Use the silent-failure index to find the symptom and the owning guide.

Version floor

Every guide states its version floor in the first 200 words, and every API is marked with the earliest OS that has it — 26.0, 26.2, 26.4 and 27.0 all matter and are routinely confused. Version confusion is the single largest source of phantom bug reports in the developer forums.

Measurement attribution

Every number is attributed: Apple-published, community-measured, or measured by us — with hosting mode, hardware/device, OS and runtime build, Xcode build, destination, build configuration, exceptions, and date. A community benchmark is never presented as an Apple figure. Where community numbers complicate Apple's claims, both are given. Never carry a measured default between tool-hosted, app-hosted, and device-hosted processes merely because the runtime build matches; contextSize has already disproved that shortcut.

Defect state and resolution

Repository state is mechanical evidence, not a resolution claim. Write “PR merged” or “issue closed” separately from the semantic disposition: fixed, fixed-with-residual, merged-unreleased, closed-unfixed, closed-unmerged, superseded, consolidated, or unknown. A closure without code or maintainer evidence is never described as fixed. Use unknown when the evidence does not establish what happened, and retain the workaround or hazard until a dated target-specific verification retires it.

Known-bad claims

There is material circulating about this stack that is simply fabricated — invented file extensions (.coreaimodel, .aiasset), a coreai-torch convert CLI that does not exist, "iOS 20 / macOS 17", and an on-device LoRA training API that was never shipped. Part 1 carries a known-bad-claims reference so these don't get reintroduced by a well-meaning reader or coding agent.

Precedence when sources conflict

  1. Headers and SDK sources you can read on disk
  2. Apple documentation pages
  3. Apple-staff answers on the Developer Forums
  4. WWDC session transcripts
  5. Community repositories and blog posts

Several WWDC transcript claims are already superseded — custom adapters, PCC eligibility, the model-tier split. Where a forum answer from Apple staff conflicts with a session, the forum wins, and the guide says so.

Known gaps, and what would close them

A 🔴 GAP names an unresolved question and the evidence needed to close it. Keep material uncertainty beside the affected advice; closed investigations remain in Git history. The remaining manual UI task is Instruments recording.

Verification and series status

The generated snapshot above records the current corpus and installed environment. SDK captures bind each artifact to its Xcode, SDK, and optional Metal Toolchain identity. Runtime measurements remain destination-specific, and unavailable SDK targets retain their earlier verification date.3

Corpus

The research behind this series lives in the notes index and research map. Retained primary captures and immutable repository pins support current claims, migration decisions, and regression comparisons.2

  1. The inventory in Every guide in the series links every part and reference. The intentionally shorter MLX references declare their terminal sections in Part 12. Apple's TN3193 specifies the on-device model's 4,096-token context window. ↩

  2. The reproducibility manifest in scripts/clone-research-repos.sh records the 16 repositories and exact full commit SHA for each checkout. ↩

  3. scripts/dump-sdk-interfaces.sh owns the managed capture and capture-manifest.json; scripts/diff-interfaces.sh performs read-only temporary drift captures. The operational steps live in notes/NEXT-BETA-CHECKLIST.md and notes/FRESHNESS-RUNBOOK.md. Apple's Core AI AOT documentation is the primary source for installing the Metal Toolchain component before capturing coreai-build. ↩