Add Audio effects

[gamblor21]

Foundation to add audio effects to CircuitPython as discussed in issue #8974.

This PR aims to create initial modules, some base effects and utilities to help create future effects. The ones included will serve as a template for future creations.

To do:

• Create modules for filters, delays, dynamics and reverbs
• Create 2 or 3 effects (echo, chorus and TBD open to suggestions)
• Test effects against audio sources that are not signed, 16-bit
• Add "mix" setting for how much of the effect is applied.
• Look into adding dynamic range compression (like synthio)
• Look into BlockInput for inputs so they can be varied by the synth/time
• Consider a change to the processing logic to make it easier to understand

Looking for feedback on the API and anything else that comes up.

When I have time I plan to document how to create your own effects using the ones provided.

Sample code to run:

i2s_bclk, i2s_lclk, i2s_data = board.GP20, board.GP21, board.GP22
audio = audiobusio.I2SOut(bit_clock=i2s_bclk, word_select=i2s_lclk, data=i2s_data)
mixer = audiomixer.Mixer(voice_count=1, channel_count=1, sample_rate=44100, buffer_size=2048)
audio.play(mixer)

synth = synthio.Synthesizer(channel_count=CHANNELS, sample_rate=44100)
amp_env = synthio.Envelope(attack_time=0.02, attack_level=1, sustain_level=.9, release_time=0.1)
synth.envelope = amp_env
n1 = synthio.Note(261)

echo1 = audiodelays.Echo(delay_ms=500, decay=0.7, buffer_size=1024, channel_count=CHANNELS, sample_rate=44100)
echo1.play(synth)
mixer.voice[0].play(echo1)

synth.press(n1)
time.sleep(0.25)
synth.release(n1)

[relic-se]

I'm still digging into the code, but my first instinct is to have the delay_ms property just be a float in seconds as delay instead (ie: rather than 500 ms use 0.5 s). The calculation that is being used to allocate the buffer uses floating point regardless.

Speaking of buffer sizes, it may be nice to explore having a fixed buffer size and altering the rate of playback of that buffer depending on the delay setting. This would be similar to a bucket-brigade style delay pedal and require some more advanced interpolation, etc. I think it be worth exploring an "Analog" delay effect of this nature.

[tannewt]

Thanks for the initial PR! This is very exciting.

[gamblor21]

I'm still digging into the code, but my first instinct is to have the delay_ms property just be a float in seconds as delay instead (ie: rather than 500 ms use 0.5 s). The calculation that is being used to allocate the buffer uses floating point regardless.

I'm not too worried at this point as I'm using Echo basically to test out ideas/concepts. Is there a reason you would want time in a float of seconds vs more precise?

Same thought on the BlockInput for the buffer size. You are right if it could change a maximum size/current size would have to be done to prevent constant reallocation.

Speaking of buffer sizes, it may be nice to explore having a fixed buffer size and altering the rate of playback of that buffer depending on the delay setting. This would be similar to a bucket-brigade style delay pedal and require some more advanced interpolation, etc. I think it be worth exploring an "Analog" delay effect of this nature.

I think this would be good for another effect. I have tossed about the idea of a generic delay buffer that other effects could use.

[gamblor21]

Update for anyone following along:

• Mix has been added
• Mix/Delay/Decay are all BlockInput now. Unlike most synth uses these values have max/min values that have to be checked during the buffer computation time to ensure they are within the acceptable range.
• You can set the max delay and then alter the delay without memory reallocations.
• After looking at what I modeled the first code after in Mixer, I think re-writing it to be clearer (and more like synth) still has the performance. The Mixer code came from when the controllers were significantly slower and in fact the first version was only for M4 processors. That's next on my list.

Getting a lot closer.

[gamblor21]

Word to the wise... 44.1Khz dual channel audio with asyncio without enough sleep in a loop causes weird stuttering that you will try to debug all night long, and turns out just add a small delay to the sleep loop and you are good. Maybe a sign a 0 asyncio.sleep isn't running audio tasks fully?

But it did prompt me to add double buffering in.

[tannewt]

Maybe a sign a 0 asyncio.sleep isn't running audio tasks fully?

This is very weird! I'd expect audio stuff to run in between VM byte codes. I'd appreciate it if you could dig into why it isn't working.

[gamblor21]

Latest updates:

• Simplified the processing code, hopefully making it easier for others to follow
• Tested again 16-bit signed and 8-bit unsigned sources. So far Synth, RawSample and WaveFile. MP3 still is not working.
• Added in dynamic range compression like in Synth for 16 bit echos.
• Added a lot of comments to the code

Things are close now. I still want to see if I can tell why MP3s are not working (they should be don't think it is a processing limit).
I do have to clean up the documentation still.

[gamblor21]

This is very weird! I'd expect audio stuff to run in between VM byte codes. I'd appreciate it if you could dig into why it isn't working.

Took a look, seemed between two channels of 44.1Khz audio and constantly checking a rotary encoder I hit the processing limit as close as I can tell. Or the rotary encoder blocked long enough (and I was checking it often enough) to slow the audio down. So at this point seems to be a non-issue.

[gamblor21]

Things are close now. I still want to see if I can tell why MP3s are not working (they should be don't think it is a processing limit).

MP3Decoder as a source works now. Was a bug I just hadn't ran into. Also samples of unusual lengths work now (wasn't update the remaining sample buffer correctly, and most samples use values that just happened to work).

[relic-se]

I finally got around to testing out your audio_effects fork and playing around with audiodelays.Echo. I was hoping to recreate some form of chorus effect with this module, but I don't think that's possible with the current implementation. The decay parameter directly affects the level of the first repeat of the delay. Generally, I would assume that decay/feedback would only apply to additional repeats of the audio signal and the level of the initial copy would be controlled independently using the mix parameter.

For instance, if you wanted to create a strong "slapback" style delay, this is what I would expect to work in that scenario (I'm excluding buffer_size, channel_count, and sample_rate as they aren't relevant):

audiodelays.Echo(delay_ms=500, decay=0.0, mix=1.0)

The expected outcome would be a full volume repeat of the audio after 500ms and then no additional feedback/decay of that audio. Instead, there is no effect on the audio output because the decay parameter silences out all delayed audio including the first repeat.

The solution is relatively simple. Instead of applying decay as soon a sample is extracted from echo_buffer, only apply it when inserting the old sample back into echo_buffer. I have implemented this on my end by modifying /shared-module/audiodelays/Echo.c, but I'm a newbie when it comes to pull requests and I'm not certain how best to contribute here.

[relic-se]

I've created a pull request on your fork for your consideration, @gamblor21. gamblor21#1

[gamblor21]

I've created a pull request on your fork for your consideration, @gamblor21. gamblor21#1

I will take a look but probably not until tomorrow. What you were saying makes sense.

One thing I do plan to do is create more effects then just this echo. This really is a proof-of-concept still. I was looking a chorus briefly last night, and want to consider others (reverb) as well.

If you are playing with it another thing you can try is chaining echos (since that is my only effect ha). But basically echo1.play(echo2.play(echo3.play(echo4.play(synth))))

[relic-se]

Though the delay_ms parameter has been set up as a BlockInput object, it is not recalculating echo_buffer_len and subsequent read/write positions within audiodelays_echo_get_buffer. As a result, using a BlockInput parameter for delay_ms has no discernible affect beyond the initial value.

lfo1 = synthio.LFO(scale=250, offset=500, rate=0.5)
synth.blocks.append(lfo1)

echo1 = audiodelays.Echo(delay_ms=lfo1, decay=0.5, mix=1.0, buffer_size=4096, channel_count=CHANNELS, sample_rate=SAMPLES_RATE)
echo1.play(synth)
mixer.voice[0].play(echo1)

synth.press(60)
while True:
    print(lfo1.value)
    time.sleep(0.25)

In this example, it is clear that the value of the LFO is changing over time, but there is no audible change in the output.

[gamblor21]

Though the delay_ms parameter has been set up as a BlockInput object, it is not recalculating echo_buffer_len and subsequent read/write positions within audiodelays_echo_get_buffer. As a result, using a BlockInput parameter for delay_ms has no discernible affect beyond the initial value.

Oh good catch! Thanks, I'll have a fix for that soon I know what is wrong.

[gamblor21]

For instance, if you wanted to create a strong "slapback" style delay, this is what I would expect to work in that scenario (I'm excluding buffer_size, channel_count, and sample_rate as they aren't relevant):

The expected outcome would be a full volume repeat of the audio after 500ms and then no additional feedback/decay of that audio. Instead, there is no effect on the audio output because the decay parameter silences out all delayed audio including the first repeat.

First off, thank you very much for testing things for me!

Second I looked at the PR you added and what you are talking about here. I wrote echo to decay on the first playback to make it more of an echo. I think maybe you want more of a straight delay? So at T=0 you hear original sample, at T=500ms you hear the repeat of the original sample at full volume, and then nothing else after?

I think for that I either need to add a flag "don't decay first playback" or I'm thinking a delay effect may be a good idea. It is on my list of things to build.

I do want more effects as opposed to one audio effect to rule them all. (Chorus is also on that list)

[relic-se]

I will admit that the example I provided was an extreme use-case and not a good representation of the problem at hand. I still think this is an issue that should be addressed regardless of our interpretation of "echo" vs "delay". I will do my best to demonstrate the issue here.

echo = audiodelays.Echo(delay_ms=250, decay=x, mix=y)

Desired Outcome	T=0ms	T=250ms	T=500ms	Current decay (x)	Current mix (y)	Proposed decay (x)	Proposed mix (y)
Standard	50%	25%	12.5%	0.5	1.0	0.5	0.5
More Repeats	50%	37.5%	28.1%	0.75	0.67	0.75	0.5
Less Repeats	50%	12.5%	3.1%	N/A	N/A	0.25	0.5

The in proposed outcome, the level of the initial repeat (and essentially the most audible one) is controlled independently of the rate of the decay/feedback. In the third scenario where one might desire an audible delay/echo but have it trail off faster, it is not possible using the current technique (hence "N/A"). In order to achieve a rate of decay of 25%, you are limiting your maximum volume of that effect to 25%.

In my experience with musical equipment of many sorts (guitar effects pedals, rackmount effects, DSP plugins, manual tape machine + mixer effects, etc), the expected outcome is typically the proposed solution as it allows greater control of the effect.

Yes, it would technically be possible to achieve a single-shot delay with the proposed technique (decay=0.0, mix=1.0). However, I don't see why that is a problem. I feel that creating two different delay classes which mostly overlap each other may create more confusion than necessary and would take up more flash storage.

On the topic of other delay-based effects (chorus, phaser, etc), it is likely possible to simulate those effects with the current delay/echo effect. I plan on exploring that with the recent changes you've made to BlockInput support on delay_ms. Yet, there may still be room for specialized classes for those effects because it would likely not be very intuitive to the average user.

[tannewt]

Thanks for working on this! Just a few API comments.

[gamblor21]

@todbot A question came up about the mix parameter and how it should work. If we speed things up at value 0 and pass the sample straight through, with an effect like echo the sample will not be placed into the echo buffer so if mix is then increased the echo is lost. Is this what you intended?

Versus if mix is 0 you pass the sample straight through but still inject into into the echo buffer, but never hear that play, unless mix is increased?

Any thoughts? I also realize this functionality could change effect to effect as well.

[gamblor21]

@tannewt which board(s) should I enable this for? Right now I only have it for the Pico2. I could for an 2350 board. I haven't tried it against any others yet.

[tannewt]

@tannewt which board(s) should I enable this for? Right now I only have it for the Pico2. I could for an 2350 board. I haven't tried it against any others yet.

At least all RP2350. Then folks can enable it for other chips after they test it.

[relic-se]

@todbot A question came up about the mix parameter and how it should work. If we speed things up at value 0 and pass the sample straight through, with an effect like echo the sample will not be placed into the echo buffer so if mix is then increased the echo is lost. Is this what you intended?

Versus if mix is 0 you pass the sample straight through but still inject into into the echo buffer, but never hear that play, unless mix is increased?

Any thoughts? I also realize this functionality could change effect to effect as well.

Here's my demonstration of the issue. I've provided 2 examples, but the most distinct is likely the second where the effect is delayed 4 seconds before the mix value is turned back on.

import board, audiobusio, audiomixer, synthio, audiodelays, time

CHANNELS = 1
SAMPLE_RATE = 44100

i2s_bclk, i2s_lclk, i2s_data = board.GP19, board.GP20, board.GP21
audio = audiobusio.I2SOut(bit_clock=i2s_bclk, word_select=i2s_lclk, data=i2s_data)
mixer = audiomixer.Mixer(voice_count=1, channel_count=CHANNELS, sample_rate=SAMPLE_RATE, buffer_size=2048)
audio.play(mixer)

synth = synthio.Synthesizer(channel_count=CHANNELS, sample_rate=SAMPLE_RATE)
synth.envelope = synthio.Envelope(attack_time=0.02, attack_level=0.5, sustain_level=0.5, release_time=0.02)

echo1 = audiodelays.Echo(max_delay_ms=250, delay_ms=250, decay=0.5, sample_rate=SAMPLE_RATE, channel_count=CHANNELS)
echo1.play(synth)
mixer.voice[0].play(echo1)

# Demonstration of signal lost when mix=0.0
print("mix=0")
echo1.mix = 0.0
synth.press(60)
time.sleep(0.25)
synth.release(60)
time.sleep(0.1)
print("mix=1")
echo1.mix = 1.0
time.sleep(2)

# Demonstration of signal reserved when mix=0.0
print("mix=1")
echo1.mix = 1.0
synth.press(60)
time.sleep(0.25)
synth.release(60)
print("mix=0")
echo1.mix = 0.0
time.sleep(4)
print("mix=1")
echo1.mix = 1.0

The fix is very simple where it's just a matter of removing the branching logic for mix <= 0.01. For other effects, especially dynamics (compression, distortion, etc), it makes sense to keep this optimization.

[relic-se]

@tannewt which board(s) should I enable this for? Right now I only have it for the Pico2. I could for an 2350 board. I haven't tried it against any others yet.

At least all RP2350. Then folks can enable it for other chips after they test it.

I've tested this board on the original Raspberry Pi Pico (RP2040) with an I2S DAC (PCM5102). It can run through the examples at stereo 44.1k without any issue.

But when I tried adding audiodelays.Echo to a bigger audio project of mine, I quickly ran into memory allocation errors. I was only able to get things working by drastically reducing sample rate, buffer sizes, etc. That caused stuttering, but likely due to the small buffer sizes.

Since the actual processing is rather simple, I think the biggest hold back for many MCUs is going to be memory size.

[gamblor21]

I have 2 items unchecked but I think it is worth merging this now if it is ready. More PRs can add more features as we go.

I think the build failures are something with the build system but not 100% sure.

[tannewt]

Minor changes to have CIRCUITPY_AUDIOEFFECTS enable a bunch of future modules and have CIRCUITPY_AUDIODELAYS explicitly.

[gamblor21]

I think everything is good now but cannot see why the last readthedocs task is failing on a warning. None of the warnings in the file match any changes I did. Was there a change to the doc building system between my branch and PR maybe?

[jepler]

I think this is the warning you mean:

WARNING: Calling get_html_theme_path is deprecated. If you are calling it to define html_theme_path, you are safe to remove that code.

Yes, this is something that changed in the meantime, not due to your changes.

This warning should be resolved if you merge the Adafruit main branch into your PR branch, if you're comfortable doing that.

[tannewt]

Thank you! I'll override the failure since it is fixed on main.