Both iOS and OS X come with a wide array of audio APIs, ranging from very low level to very high level. The number of different APIs, developed over time as these platforms have grown and changed, can be quite overwhelming to say the least. This article gives a brief overview of the available APIs and the different purposes they serve.
Media Player Framework
The Media Player framework on iOS is a high-level API for audio and video playback, and includes a stock user interface you can drop into your app. You can use it to play back items in the user’s iPod library, or to play local files or network streams.
Additionally, this framework contains APIs to query the user’s media library, as well as to configure system audio controls like on the lock screen or in the control center.
AVFoundation is Apple’s modern media framework that includes several APIs for different purposes and on different levels of abstraction. Some of these are modern Objective-C wrappers for lower-level C APIs. With a few exceptions, it’s available both on iOS and OS X.
AVAudioSession is specific to iOS and coordinates audio playback between apps, so that, for example, audio is stopped when a call comes in, or music playback stops when the user starts a movie. This API is needed to make sure an app behaves correctly in response to such events.
This high-level API gives you a simple interface to play audio from local files or memory. This is a headless audio player (i.e. no UI elements are provided), and it’s very straightforward to use. It’s not suitable for streaming audio from the network or for low-latency realtime audio applications. If those things are not a concern, this is probably the right choice. The audio player API also comes with a few extra features, such as looping, playback-level metering, etc.
As the counterpart to
AVAudioPlayer, the audio recorder API is the simplest way to record audio straight to a file. Beyond the possibility to receive peak and average power values for a level meter, the API is very bare bones, but might just be what you need if your use case is simple.
AVPlayer API gives you more flexibility and control than the APIs mentioned above. Built around the
AVAsset classes, it gives you more granular access to assets, e.g. to pick a specific track. It also supports playlists via the
AVQueuePlayer subclass, and lets you control whether the asset can be sent over AirPlay.
A major difference compared to, for example,
AVPlayer’s out-of-the-box support for streaming assets from the network. This increases the complexity of playback state handling, but you can observe all state parameters using KVO.
AVAudioEngine is a modern Objective-C API for playback and recording. It provides a level of control for which you previously had to drop down to the C APIs of the Audio Toolbox framework (for example, with real-time audio tasks). The audio engine APIs are built to interface well with lower-level APIs, so you can still drop down to Audio Toolbox if you have to.
The basic concept of this API is to build up a graph of audio nodes, ranging from source nodes (players and microphones) and overprocessing nodes (mixers and effects) to destination nodes (hardware outputs). Each node has a certain number of input and output busses with well-defined data formats. This architecture makes it very flexible and powerful. And it even integrates with audio units.
Audio Unit Framework
The Audio Unit framework is a low-level API; all audio technologies on iOS are built on top of it. Audio units are plug-ins that process audio data. A chain of audio units is called an audio processing graph.
You may have to use audio units directly or write your own if you need very low latency (e.g. for VoIP or synthesized musical instruments), acoustic echo cancelation, mixing, or tonal equalization. But a lot of this can often be achieved with the
AVAudioEngine API. If you have to write your own audio units, you can integrate them in an
AVAudioEngine processing graph with the
The Audio Unit API allows for Inter-App Audio on iOS. Audio streams (and MIDI commands) can be sent between apps. For example, an app can provide an audio effect or filter. Another app can then send its audio to the first app to apply the audio effect. The filtered audio is sent back to the originating app in real time. CoreAudioKit provides a simple UI for Inter-App Audio.
OpenAL is a cross-platform API. It provides positional (3D) and low-latency audio services. It’s mostly intended for cross-platform games. The API deliberately resembles the OpenGL API in style.
On iOS, Core MIDI and CoreAudioKit can be used to make an app behave as a MIDI instrument. On OS X, Music Sequencing Services gives access to playing MIDI-based control and music data. Core MIDI Server gives server and driver support.
- The most basic of all audio APIs on OS X is the
NSBeep()function that simply plays the system sound.
- On OS X, the
NSSoundclass provides a simple API to play sounds, similar in concept to
- All notification APIs — local and remote notifications on iOS,
NSUserNotificationon OS X, and CloudKit notifications — can play sounds.
- The Audio Toolbox framework is powerful, but very low level. It’s historically C++ based, but most of its functionality is now available through the
- The QTKit and QuickTime frameworks are on their way out and should not be used for new development anymore. Use
AVFoundation(and AVKit) instead.