Recently, I have developed a keen interest in the creation of production and audio equipment. When I was working at Nordic as a student, I learned about transmitting audio via Bluetooth, which intrigued me. Especially the exciting Bluetooth le audio, which is said to get better audio quality with lower bitrates and less latency. When playing an instrument, you want the delay between your movement and the response of your voice to be minimal. I wanted to test if I could take advantage of the nrf5340 to produce a pleasant sound with a simple keyboard interface control, and see if the delay was enough not to affect my playing.
The app consists of a sounding device, a synthesizer, and a receiving headset. LE Audio can simultaneously send two audio streams corresponding to the left and right parts of the headphones.
The application is based on the Nordic NRF5340 Audio demo application, but streamlined and specialized for synthesizer use cases. The main focus of the application is the synthesizer and keyboard input sections.
The structure of the synthesizer is highly modular, making it easy to add or exclude different modules and test different audio synthesis topologies. The settings provided demonstrate different aspects of the synthesizer, such as polyphonic oscillators, effects, and audio sync time correlation.
The synthesizer module receives information from the button module about which note to stop and stop. These notes can be converted into sequencers, or in this case, into arpeggiators. Arpeggiators are time-dependent, so a time source is required. We have to use a time source that is synchronized with the time sense of the processed audio. That's what the tick provider does, which adds time to each block of audio processed. The tick provider sends a tick to the subscription module. This is the same way that MIDI synchronizes different audio sources, so MIDI synchronization can be achieved.
For polyphonic synthesis, we need multiple oscillators equal to the number of notes that can be played at one time. Key Assign keeps track of the currently active oscillator and assigns notes to the oscillator in the way that it is activated first (the one that has been active for the longest time). If there is no inactive oscillator, it will assign the new note to the first active (the longest active oscillator) oscillator. By default, the application is configured with five oscillators. Even if the arpeggiator of ** is more than 5 notes, it is difficult to notice that the last active note is cut off.
In Audio Process Start, the application is configured for mono audio encoding. All modules are also built in mono. It's not difficult to modify the app to handle stereo audio. For example, you can convert an echo effect to a stereo ping-pong delay effect. Before using the echo effect, divide the mono sound into two channels, left and right. These channels are then fed into the echo effect. Make sure that the PCM size is set correctly in SW Codec Encode for stereo encoding.
Audio is processed in n sample blocks, which is started in the audio process. The timer ensures that the processing interval of the new block matches the audio sample rate. This should perhaps be synchronized directly with the Bluetooth connection interval for further development.
Use a 16-bit depth. DSPs primarily use integers in fixed-point format. Therefore, the fixed16 type and related operations are defined in interger math. Since the application is specific to the NRF5340, in some cases the DSP instructions included in the SOC are used and abstracted by the DSP instructions.
The following block (from oscillatorc) is an example of producing a sine wave whose frequency is determined by phase increment. The upper 8 bits of Phase Accumulate[24:31] are used to select which stored samples to use, while Phase Increment[8:23] is used to determine the interpolation between the two samples. Then there is the application amplitude of the signal.
for (uint32_t i = 0; i < block_size; i++)Fixed Interpolate and Scale is provided in Intel Math to efficiently perform this calculation using DSP instructions on the CPU.
In the current application configuration, the NRF5340 application core is approximately 80% utilized when all oscillators are active. When two headset devices are connected, about 40% of the application kernel is used. A large part of this is probably LC3 encoders.
The minimum connection interval for Bluetooth Low Energy is 75 ms. Audio processing every 7Start once in 5 milliseconds to match the connection interval. As a result, the processing process does not add latency. The new LC3 codec in LE Audio, which replaces the SBC codec, should significantly reduce latency. However, it has not been tested in this particular application. The input button has a debounce time of 50 milliseconds. However, this does not cause delays. This is because during implementation, the button state changes every time a new interrupt is triggered. After the debounce time is over, test this assumption by reading the pin value. This will result in an occasional incorrect button state, but it will be corrected again after the debounce time. This did not result in any auditory artifacts in the test**.
The resulting application demonstrates a functional system for transmitting synthesizer audio using BLE LE Audio. For instructions on how to test your application, check out the GitHub repo. Here are the settings used**: 2 x NRF5340 Audio DK as headphones and 1 x NRF5340 DK as synthesizer:
While no quantitative analysis was performed to determine the total latency, I don't think the latency is noticeable when playing the keyboard and listening to the resulting sound. I believe there are a lot of exciting new applications for LE Audio on the NRF5340.