Audio

1. Introduction

Digital audio processing is a complex yet fascinating subject. Understanding it requires knowledge in multiple technical disciplines, including physics, electrical/electronic engineering, and software engineering, especially within the Linux ecosystem. This article aims to demystify the Linux audio stack by explaining the basics of sound, how humans perceive sound, and the workings of digital audio. We will then explore the components of the Linux audio stack and how they interact. Buckle up; it’s going to be a long and informative read.

2. Disclaimer

In this article, I will simplify concepts and ideas to provide basic knowledge for following along. Each chapter could easily expand into a dedicated, lengthy blog post. However, for simplicity’s sake, I have refrained from doing so. If you would like more details about a specific chapter, please feel free to contact me or leave a comment below. I’m happy to elaborate if needed or wanted.

3. Understanding Sound and Sound Waves

Before diving into the technicalities of audio processing on Linux, it’s essential to understand what sound is. In a nutshell, sound is a vibration that propagates as an acoustic wave through a medium such as air. These waves are created by vibrating objects (such as speakers, headphones, piezo buzzers, etc.), causing fluctuations in air pressure that our ears can pick up and perceive as sound. Audio waves can have many shapes; the most common waveform is the sine wave, which is the fundamental building block for all other waves.

All waveforms have two fundamental properties:

• Period (duration): The time it takes the wave to complete a single oscillation.
• Amplitude: This represents the wave’s strength or intensity, which we perceive as volume. Higher amplitudes generally correspond to louder sounds.

The Frequency of a waveform can be derived from its period. Frequency, measured in Hertz (Hz), represents the number of oscillations (waves) per second. Higher frequencies correspond to higher-pitched sounds. Thus, the frequency is the inverse of the period duration, or mathematically:

$f=\frac{1}{\text{period duration}[s]}=(\text{period duration}[s]{)}^{-1}$

Amplitudes can be measured as the pressure variation in the air, known as Sound Pressure Level (SPL). The standard unit for pressure is Pascal (Pa), but in audio, we often deal with very small pressures, so micro pascals (µPa) are typically used. The most common way of describing amplitude is Decibels (dB), a logarithmic unit describing the ratio of a particular sound level to a given reference level. Decibels are preferred because the human ear perceives sound pressure levels logarithmically.

• dB SPL: Sound pressure level in decibels. In air, the reference pressure is typically 20 µPa (referred to as P₀), the threshold of human hearing. It is calculated using the formula:

$\text{SPL}=20\times {\log }_{10}\left(\frac{P}{P_0}\right)$

where P is the measured pressure and P₀ is the reference pressure (20 µPa).

• dBV and dBu: When sound is converted into an electrical signal by a microphone, the amplitude can be measured in volts (V). dBV and dBu are units for voltage measurements in audio systems, referencing 1 volt for dBV or 0.775 volts for dBu. They can be calculated similarly:

$\text{dBV}=20\times {\log }_{10}\left(\frac{V}{V_0}\right)$

where V is the measured voltage and V₀ is the reference voltage.

4. How Humans Process Sound Waves

Sound waves are captured by the pinna, the visible part of the outer ear. The pinna directs sound waves into the ear canal, which travels through and reaches the eardrum (tympanic membrane). The sound waves cause the eardrum to vibrate, corresponding to the frequency and amplitude of the incoming sound waves. These vibrations are transmitted to the three tiny bones in the middle ear, known as the ossicles (malleus, incus, and stapes). The ossicles amplify the vibrations and transmit them to the oval window, a membrane-covered opening to the inner ear. The vibrations pass through the oval window and enter the cochlea, a spiral-shaped, fluid-filled structure in the inner ear. Inside the cochlea, the vibrations create waves in the cochlear fluid, causing the basilar membrane to move. The movement of the basilar membrane stimulates tiny hair cells located on it. These hair cells convert mechanical vibrations into electrical signals through a process called transduction. Different frequencies of sound stimulate different parts of the basilar membrane, allowing the ear to distinguish between various pitches. The hair cells release neurotransmitters in response to their movement, which generates electrical impulses in the auditory nerve. These electrical signals travel along the auditory nerve to the brain. The auditory signals reach the auditory cortex in the brain, where they are processed and interpreted as recognizable sounds, such as speech, music, or noise.

Some “Hard Facts” to better understand the dimensions of sound waves:

• The typical frequency range of human hearing is from ~20 Hz to ~20,000 Hz (20 kHz). This range can vary with age and exposure to loud sounds.
• The frequency of human speech typically falls within the range of ~300 Hz to ~3,400 Hz (3.4 kHz).
• Human ears are most sensitive to frequencies between 2,000 Hz and 5,000 Hz.
• The quietest sound the average human ear can hear is around 0 dB SPL (sound pressure level), equivalent to 20 micropascals.
• The loudest sound the average human ear can tolerate without pain is around 120-130 dB SPL. Sounds above this level can cause immediate hearing damage.
• The range between the threshold of hearing and the threshold of pain is known as the dynamic range of human hearing, which is approximately 120 dB.

5. How Digital Audio Works

With a basic understanding of sound waves in the analog world, let’s explore sound waves in a world of zeros and ones.

When we talk about audio in a digital context, we’re referring to sound that’s been captured, processed, and played back using computers.

5.1. Audio-Input and Sampling

Sampling for computers is analogous to hearing for humans. Sampling is the process of converting analog sound waves into digital data that a computer can process. This is done by taking regular measurements of the amplitude of the sound wave at fixed intervals. The rate at which these samples are taken is called the sampling rate, measured in samples per second (Hz). Common sampling rates include 44.1 kHz (CD quality) and 48 kHz (professional audio).

Each point on the sine wave represents a single discrete sample/measurement. The first oscillation has a higher sample rate, while the second has a lower sample rate for demonstration purposes only. Typically, the sample rate is constant and more than double the highest frequency humans can hear.

5.1.1. Nyquist-Shannon Sampling Theorem

The Nyquist-Shannon Sampling Theorem provides a criterion for determining the minimum sampling rate that allows a continuous signal to be perfectly reconstructed from its samples. A continuous-time signal that has been band-limited to a maximum frequency (f_{max}) (meaning it contains no frequency components higher than (f_{max})) can be completely and perfectly reconstructed from its samples if the sampling rate (f_s) is greater than twice the maximum frequency of the signal. Mathematically, this can be expressed as:

$f_s>2\ast f_{max}$

This explains why common sampling rates are typically a little more than double the human hearing range (e.g., 44.1 kHz for CD quality).

5.1.2. Quantization

In the analog world, signals have infinite precision. For example, if you zoom in on a sine wave, you can always find more detail — the waveform continues to be smooth and precise no matter how closely you zoom in. However, in the digital world, precision is limited. To understand this limitation, consider the question: “How many numbers can you represent between 0 and 1?” If you use 0.1 as the smallest possible increment value, you can only represent 10 discrete values between 0 and 1. If you refine the smallest increment to 0.01, you get 100 values between 0 and 1. In the analog realm, there’s no smallest possible increment; values can be infinitely precise. But in digital systems, precision is constrained by the number of bits used to represent the signal. Digital audio uses a fixed number of bits to encode each sample of sound. For instance, with 16-bit audio, there are 65,536 discrete levels available to represent the amplitude of each sample. This means that the smallest difference between two levels, or the smallest possible increment, is approximately 0.000030518 for 16-bit audio. Professional audio typically utilizes 24-bit depth. Everything higher than that is usually unnecessary, with differences only noticeable by audiophiles.

The process of approximating each sample’s amplitude to the nearest value within these discrete levels is known as quantization. The number of discrete levels depends on the bit depth of the audio. For example:

• 16-bit audio provides 65,536 possible amplitude values.
• 24-bit audio offers 16,777,216 possible amplitude values.

Quantization is essential because it allows us to represent analog signals with a finite number of discrete values, making digital storage and processing feasible, even though we sacrifice some level of precision compared

to the infinite resolution of the analog world.

5.2. Digital Processing and Effects

Once audio has been sampled and quantized, it exists in the digital realm as a series of numbers, which can be stored in a file, streamed over the internet, or processed in real-time. Digital audio processing involves various techniques to modify these numbers to achieve a desired effect. This could include:

• Equalization (EQ): Adjusting the balance of different frequency components.
• Compression: Reducing the dynamic range of the audio.
• Reverb: Simulating the effect of sound reflecting off surfaces in a space.
• Distortion: Intentionally adding harmonic content or other modifications to the signal.

5.3. Audio-Output and Digital-to-Analog Conversion

Just as audio input involves converting analog signals to digital data, audio output involves converting digital data back into analog signals. This is done by a Digital-to-Analog Converter (DAC), which takes the digital audio samples and reconstructs the waveform. The reconstructed waveform is then amplified and sent to a speaker or headphone to produce sound. DACs, like ADCs, have bit-depth and sampling rate specifications that determine the quality and precision of the output signal.

5.4. File Formats

Digital audio can be stored in various file formats, each with its characteristics and use cases. Common audio file formats include:

• WAV: Uncompressed audio format, often used for high-quality recordings.
• MP3: Compressed audio format, widely used for music distribution due to its balance of quality and file size.
• FLAC: Lossless compressed format, which preserves the original audio quality while reducing file size.
• AAC: A compressed format often used in streaming and mobile applications, providing good quality at lower bitrates.

6. Linux Audio Stack Overview

Linux is a powerful platform for audio processing, offering flexibility and a wide range of tools and applications. However, the Linux audio stack can be complex due to the many components and layers involved. Understanding the Linux audio stack requires familiarity with several key elements, including drivers, APIs, and sound servers. Here is an overview of the main components:

6.1. Kernel Space: ALSA (Advanced Linux Sound Architecture)

At the core of the Linux audio stack is the Advanced Linux Sound Architecture (ALSA), which is part of the Linux kernel. ALSA provides drivers for a wide range of sound cards and interfaces, allowing the operating system to communicate with hardware. It offers both low-level access to audio devices and a higher-level API for audio applications. ALSA is responsible for:

• Handling audio hardware devices.
• Providing a unified interface for audio applications.
• Offering direct access to audio hardware for low-latency audio processing.

6.2. User Space: PulseAudio, JACK, PipeWire, and Others

In user space, several audio servers and frameworks offer additional features and capabilities on top of ALSA:

6.2.1. PulseAudio

PulseAudio is a sound server that provides advanced audio features such as:

• Per-application volume controls.
• Network audio streaming.
• Mixing and resampling.

PulseAudio sits between ALSA and user applications, providing a more user-friendly interface and additional features.

6.2.2. JACK (Jack Audio Connection Kit)

JACK is a professional-grade sound server designed for low-latency audio processing, often used in professional audio production environments. It allows for complex routing and synchronization of audio streams between applications and hardware. JACK is ideal for scenarios requiring precise timing and minimal latency, such as music production and live performance.

6.2.3. PipeWire

PipeWire is a relatively new multimedia server for Linux, aiming to unify and replace existing solutions like PulseAudio and JACK. It provides a low-latency, high-performance framework for handling audio and video streams, suitable for both professional audio work and general-purpose audio needs. PipeWire supports advanced features like sandboxed audio processing, which enhances security and flexibility.

7. Practical Aspects and Common Issues

7.1. Choosing the Right Audio Server

The choice of an audio server depends on the user’s needs:

• PulseAudio: Ideal for general desktop use, providing a good balance of features and ease of use.
• JACK: Suitable for professional audio work where low latency and precise control are critical.
• PipeWire: Emerging as a versatile option that can handle both professional and casual audio needs.

7.2. Dealing with Latency

Latency is a crucial consideration in audio processing, especially for real-time applications like live music performance or interactive installations. To minimize latency:

• Use a real-time kernel: Linux can be configured with a real-time kernel to reduce latency.
• Optimize buffer sizes: Smaller buffer sizes can reduce latency but may require more CPU power.

7.3. Audio Configuration and Troubleshooting

Configuring audio on Linux can be challenging due to the variety of hardware and software components involved. Common issues include:

• Device compatibility: Ensuring that your hardware is supported by ALSA and the chosen audio server.
• Conflicting sound servers: Ensuring that PulseAudio, JACK, and PipeWire are configured to work together if they are used concurrently.
• Application compatibility: Some applications may require specific configurations or additional software to work correctly with the chosen audio stack.

8. Conclusion

Digital audio processing on Linux is a complex but rewarding field, offering a powerful and flexible environment for working with sound. By understanding the fundamental concepts of sound, digital audio, and the Linux audio stack, users can effectively navigate and utilize the tools available in the Linux ecosystem for a wide range of audio applications, from casual listening to professional audio production.