Internals

The Audio Library uses 44100Hz, 16BIT sampling.

One of the used inputs or outputs is responsible for the timing. Usually, this is one of the I2S interfaces.

AudioStream

The heart of the library, AudioStream, is part of the Teensy core (Teensy 3 AudioStream.h AudioStream.cpp / Teensy 4 AudioStream.h AudioStream.cpp).
The AudioStream is responsible to call all other parts (such as inputs, outputs, effects, etc).
Simplified, it is a linked list. There is a software interrupt that is called every 2.9 milliseconds  (see below). For each audio object used the update() function is called (example).

AUDIO_BLOCK_SAMPLES (Blocksize)

By default, it uses blocks of 128 Samples. So the duration of block is (128 / 44100) = 0,0029 seconds, or 2.9 milliseconds. This also defines the latency.
If you want shorter blocks to reduce the latency, it is possible to reduce the blocksize on both, Teensy 3.x and Teensy 4.x - either by editing the #define AUDIO_BLOCK_SAMPLES to a multiple of 16 or by adding a define to your IDE (Not possible with the Arduino-IDE) . Reducing the block size comes at the cost of higher computational overhead.

AUDIO_SAMPLE_RATE

On the Teensy 3.x models, there is no official way to change the Sampling frequency - On Teensy 4, this is partly supported. Not all parts of the library will work with a modified sample rate. Things that will not work without additional modifications are USB-Audio and some of effects may not work as expected.

Teensy 4: As with the AUDIO_BLOCK_SAMPLES it is possible to either edit the #define or add it to your IDE (Again, not possible with Arduino-IDE). The outputs will use automatically use the new frequency .

Teensy 3 has the frequency hardcoded to the outputs. It uses 44117Hz. The reason for that is, that there is no hardware-setting  for the analog in- and outputs that allows exactly 44100. As all in- and outs need to be in sync, 44117Hz is used for I2S, too. There is a forum thread however, that shows a way to modify the frequency on Teensy 3.x, too.

Some parts use AUDIO_SAMPLERATE_EXACT, others AUDIO_SAMPLE_RATE. So it is important that both have the same value.

Snippet to change the I2S frequency on T4.x / MicroMod:

#include <utility/imxrt_hw.h>

[...]

void setI2SFreq(int freq) {
 int n1 = 4; //SAI prescaler 4 => (n1*n2) = multiple of 4
 int n2 = 1 + (24000000 * 27) / (freq * 256 * n1);
 double C = ((double)freq * 256 * n1 * n2) / 24000000;
 int c0 = C;
 int c2 = 10000;
 int c1 = C * c2 - (c0 * c2);
 set_audioClock(c0, c1, c2, true);
 CCM_CS1CDR = (CCM_CS1CDR & ~(CCM_CS1CDR_SAI1_CLK_PRED_MASK | CCM_CS1CDR_SAI1_CLK_PODF_MASK))
      | CCM_CS1CDR_SAI1_CLK_PRED(n1-1)
      | CCM_CS1CDR_SAI1_CLK_PODF(n2-1);
}