Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create FX Functions #10

Merged
merged 79 commits into from
Aug 16, 2021
Merged
Show file tree
Hide file tree
Changes from 73 commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
076be18
Create filters.py
joshfeli Aug 2, 2021
0673d8c
Implemented skeleton for some filtering functions
joshfeli Aug 2, 2021
40178a8
First draft of `change_volume`
joshfeli Aug 2, 2021
2563a44
Second draft of `change_volume`
joshfeli Aug 2, 2021
20722e2
Delete comment
joshfeli Aug 2, 2021
87e6fd2
started to change parameters to reflect dict change
farooqashar Aug 3, 2021
3074560
started to change the volume function to reflect dict changes
farooqashar Aug 3, 2021
f17290e
started changing speed function
farooqashar Aug 3, 2021
d8a1161
removed sample rate parameter and change quality function
farooqashar Aug 3, 2021
e7fcdbf
added basic return dict
farooqashar Aug 3, 2021
50f30b6
added more basic dunder strings
farooqashar Aug 3, 2021
5d00717
format changes
farooqashar Aug 3, 2021
00c81db
changed tuple loop iteration of notes array
farooqashar Aug 3, 2021
53d4311
added note score in the loop for notes array
farooqashar Aug 3, 2021
f2d15ca
trying out adding notes
farooqashar Aug 3, 2021
6100bec
updated add_chords function to create chords for each note of the aud…
Watermelanie Aug 3, 2021
41509e5
PEP linting
joshfeli Aug 4, 2021
687f1cb
Did some work on filtering functions
joshfeli Aug 4, 2021
aa92b89
Merge branch 'master' into create_fx_functions
joshfeli Aug 4, 2021
a8b0bc8
Fixed `sample_rate` assignment
joshfeli Aug 4, 2021
3379556
Fixed error with numpy array concatenation
joshfeli Aug 4, 2021
2a3dd19
Continuing to debug `change_speed`
joshfeli Aug 4, 2021
a3c183b
Updated docstring for `text_to_sound`
joshfeli Aug 4, 2021
5f7d39a
Renamed `duration` variable
joshfeli Aug 4, 2021
9e3c0c7
Adding a second condition for slowing down audio
joshfeli Aug 4, 2021
74a02e1
Added temporary filters to views for testing
joshfeli Aug 4, 2021
1c68d93
Building on the filter function implementations
joshfeli Aug 5, 2021
26b9742
Outlining a new approach
joshfeli Aug 6, 2021
5f40fc9
Major refactoring on the filter functions
joshfeli Aug 6, 2021
7c86605
Added example `.wav` files to test filters
joshfeli Aug 9, 2021
cce15e2
Draft implementation of a function to get the note onsets in an audio…
joshfeli Aug 9, 2021
413b84d
Set up a `TestCase` to test the `filters.get_notes` function
joshfeli Aug 9, 2021
980b496
Updated `text_to_sound` and API endpoint to reflect new data structur…
joshfeli Aug 9, 2021
6945ffa
Create f4_sine.wav
joshfeli Aug 9, 2021
06937cd
More debugging `get_notes` using the `FiltersTestCase`
joshfeli Aug 9, 2021
8b4a2cb
Replaced test `.wav` files with mono sine waves
joshfeli Aug 10, 2021
af08e3d
Updated docstring/comments in `text_to_sound` function
joshfeli Aug 10, 2021
3518c5b
Changed cutoff frequency for `add_chords`
joshfeli Aug 10, 2021
18dd20b
Added (temporary) plotting functionality
joshfeli Aug 10, 2021
1fb908a
Added `good_bad_neutral.wav`
joshfeli Aug 10, 2021
a407dd7
Experimenting with more plotting, a phase deviation function, and inq…
joshfeli Aug 10, 2021
f78ce47
More debugging on filters from test case
joshfeli Aug 11, 2021
09d4bed
Starting a skeleton for `k_at_time` function
farooqashar Aug 11, 2021
993a07e
cleaning up `k_at_time`
farooqashar Aug 11, 2021
0442c55
adding skeleton for `k_for_note`
farooqashar Aug 11, 2021
5bf3d4b
handling mode initial
farooqashar Aug 11, 2021
d964b8b
implementing skeleton of `_get_frequency` and helpers
farooqashar Aug 11, 2021
5ae54ad
making frequency helpers private and formatting `change_pitch`
farooqashar Aug 11, 2021
3c3e4e4
Added comment for future reference
joshfeli Aug 11, 2021
eab3784
Clean up docstrings, return window indices _and_ sample indices in `g…
joshfeli Aug 11, 2021
ee21bb9
Jotting down some ideas for `overlap_notes`
joshfeli Aug 11, 2021
6aff27a
Changed threshold value for `_find_peaks`
irinazoccolini Aug 11, 2021
566ac6b
Working through the temporal/frequency resolution tradeoff
joshfeli Aug 11, 2021
6560dff
Commented out temporal resolution test
joshfeli Aug 11, 2021
a4263f0
Labeled tests and commented out to plot all results
joshfeli Aug 12, 2021
9dcceea
Experimenting with different thresholds and STFT hop sizes
joshfeli Aug 12, 2021
54c2fda
using freq in change pitch start
farooqashar Aug 12, 2021
75a6dab
adding skeleton test case
farooqashar Aug 12, 2021
3d0816e
adding skeleton basic test
farooqashar Aug 12, 2021
4bea742
adding basic change pitch without frequencies
farooqashar Aug 12, 2021
17ffe48
Changing pitch implementing
farooqashar Aug 12, 2021
040c2b7
formatting helper functions for changing pitch
farooqashar Aug 12, 2021
fac04b0
Adding test cases and revising add chords
farooqashar Aug 12, 2021
866b1f7
updating tests
farooqashar Aug 12, 2021
156af0e
adding more complicated sentences
farooqashar Aug 12, 2021
b8b574c
updating more sentences in `tests.py`
farooqashar Aug 12, 2021
f527ded
formatting changes in `filters.py`
farooqashar Aug 12, 2021
2620615
Resolved major frequency/temporal resolution conflicts
joshfeli Aug 13, 2021
f12712d
Merge branch 'create_fx_functions' of https://github.com/dhmit/sonifi…
joshfeli Aug 13, 2021
62dc50b
Some linting
joshfeli Aug 13, 2021
1223156
Reorganized filter functions/`TestCase`s, added exception checking fo…
joshfeli Aug 13, 2021
3d27e4f
Breakpoint cleanup
joshfeli Aug 13, 2021
db0a47a
Comment linting
joshfeli Aug 13, 2021
69d7501
More debugging and linting
joshfeli Aug 13, 2021
4d16e4c
Update
joshfeli Aug 13, 2021
e0a5c86
Some linting, added descriptions for failing test cases
joshfeli Aug 13, 2021
7ac4442
Merge branch 'master' into create_fx_functions
joshfeli Aug 13, 2021
08bd0f7
Delayed base64 encoding of audio signals to the API endpoint
joshfeli Aug 13, 2021
bc5cf72
Documenting more `TestCase` failures, resolving merge conflicts
joshfeli Aug 13, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added assets/f4_sine.wav
Binary file not shown.
Binary file added assets/good_bad_neutral.wav
Binary file not shown.
Binary file added assets/major_major.wav
Binary file not shown.
Binary file added assets/major_minor.wav
Binary file not shown.
Binary file added assets/minor_major.wav
Binary file not shown.
Binary file added assets/minor_minor.wav
Binary file not shown.
Binary file added assets/window_1024_sd.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/window_256_sd.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
470 changes: 470 additions & 0 deletions backend/app/analysis/filters.py

Large diffs are not rendered by default.

29 changes: 11 additions & 18 deletions backend/app/analysis/sentiment_analysis.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,44 +5,37 @@

def text_to_sound(user_text):
"""
Takes text, runs NLTK's sentiment analyzer, and then returns a sound object
Takes text, runs NLTK's sentiment analyzer, and then returns a sound object.
The formula used to generate a note frequency is based off of this article:
<https://towardsdatascience.com/music-in-python-2f054deb41f4>.

:param user_text: A str representing some input text
:return: A dictionary of audio metadata with the following keys:
audio_samples: An int16 Numpy array
sample_rate: An int, recorded in Hz
notes: A list of 2-element tuples (freq, duration) for each note's frequency (Hz) and duration (s) in the audio

:return: A tuple of audio data containing an int16 NumPy array of samples and a sample rate in Hz (int).
"""
A4 = 440
sample_rate = 44100
T = 1
t = np.linspace(0, T, T * sample_rate, False)
duration = 1
t = np.linspace(0, duration, duration * sample_rate, False)

nltk_analyzer = SentimentIntensityAnalyzer()
all_sentences = nltk.sent_tokenize(user_text)

sin_waves = []
all_notes = []

for each_sentence in all_sentences:
# the compound score ranges from -1 to 1, this shifts the score so that there is no negative score
# multiplying by 44 and rounding the score gives us a range of keys from 0 to 88
score = round((nltk_analyzer.polarity_scores(each_sentence)['compound'] + 1) * 44)

# formula calculates the frequency of a musical note using A4 as the base note
# the score is the nth key starting from key 0 being A0
# the score is the nth key starting from key 0 being A0 (hence the extra +1)
# the exact middle of the piano is between key 43 (E4) and 44 (F4) if A0 is the zeroth note
# thus, a neutral score should generate an F4 note
frequency = A4 * (2 ** ((score + 1 - 49) / 12))
sin_waves.append(np.sin(2 * np.pi * frequency * t))
all_notes.append((frequency, T, score))

audio = np.hstack(sin_waves)
audio *= 32767 / np.max(np.abs(audio))
audio = audio.astype(np.int16)

audio_metadata = {
'audio_samples': audio,
'sample_rate': sample_rate,
'notes': all_notes
}

return audio_metadata
return audio, sample_rate
111 changes: 111 additions & 0 deletions backend/app/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,117 @@
from scipy.io import wavfile
from textwrap import dedent

# Dictionary copied from the table in this web page: <https://pages.mtu.edu/~suits/notefreqs.html>
NOTE_FREQS = {
'C0': 16.35,
'C#0/Db0': 17.32,
'D0': 18.35,
'D#0/Eb0': 19.45,
'E0': 20.60,
'F0': 21.83,
'F#0/Gb0': 23.12,
'G0': 24.50,
'G#0/Ab0': 25.96,
'A0': 27.50,
'A#0/Bb0': 29.14,
'B0': 30.87,
'C1': 32.70,
'C#1/Db1': 34.65,
'D1': 36.71,
'D#1/Eb1': 38.89,
'E1': 41.20,
'F1': 43.65,
'F#1/Gb1': 46.25,
'G1': 49.00,
'G#1/Ab1': 51.91,
'A1': 55.00,
'A#1/Bb1': 58.27,
'B1': 61.74,
'C2': 65.41,
'C#2/Db2': 69.30,
'D2': 73.42,
'D#2/Eb2': 77.78,
'E2': 82.41,
'F2': 87.31,
'F#2/Gb2': 92.50,
'G2': 98.00,
'G#2/Ab2': 103.83,
'A2': 110.00,
'A#2/Bb2': 116.54,
'B2': 123.47,
'C3': 130.81,
'C#3/Db3': 138.59,
'D3': 146.83,
'D#3/Eb3': 155.56,
'E3': 164.81,
'F3': 174.61,
'F#3/Gb3': 185.00,
'G3': 196.00,
'G#3/Ab3': 207.65,
'A3': 220.00,
'A#3/Bb3': 233.08,
'B3': 246.94,
'C4': 261.63,
'C#4/Db4': 277.18,
'D4': 293.66,
'D#4/Eb4': 311.13,
'E4': 329.63,
'F4': 349.23,
'F#4/Gb4': 369.99,
'G4': 392.00,
'G#4/Ab4': 415.30,
'A4': 440.00,
'A#4/Bb4': 466.16,
'B4': 493.88,
'C5': 523.25,
'C#5/Db5': 554.37,
'D5': 587.33,
'D#5/Eb5': 622.25,
'E5': 659.25,
'F5': 698.46,
'F#5/Gb5': 739.99,
'G5': 783.99,
'G#5/Ab5': 830.61,
'A5': 880.00,
'A#5/Bb5': 932.33,
'B5': 987.77,
'C6': 1046.50,
'C#6/Db6': 1108.73,
'D6': 1174.66,
'D#6/Eb6': 1244.51,
'E6': 1318.51,
'F6': 1396.91,
'F#6/Gb6': 1479.98,
'G6': 1567.98,
'G#6/Ab6': 1661.22,
'A6': 1760.00,
'B6': 1975.53,
'C7': 2093.00,
'C#7/Db7': 2217.46,
'D7': 2349.32,
'D#7/Eb7': 2489.02,
'E7': 2637.02,
'F7': 2793.83,
'F#7/Gb7': 2959.96,
'G7': 3135.96,
'G#7/Ab7': 3322.44,
'A7': 3520.00,
'A#7/Bb7': 3729.31,
'B7': 3951.07,
'C8': 4186.01,
'C#8/Db8': 4434.92,
'D8': 4698.63,
'D#8/Eb8': 4978.03,
'E8': 5724.04,
'F8': 5587.65,
'F#8/Gb8': 5919.91,
'G8': 6271.93,
'G#8/Ab8': 6644.88,
'A8': 7040.00,
'A#8/Bb8': 7458.62,
'B8': 7902.13
}


def wav_to_base64(byte_array, sample_rate):
"""
Expand Down
Binary file removed backend/app/example.wav
Binary file not shown.
Loading