Processing a file - help wanted

[garrilla]

I'd like to build something a little like the LiveEffect sample.

However, I want to affect the audio selected from a file chosen by the user.

There's no example of how to operate on a file.

Does anyone have an example or guidance so I can let the user choose a file from their library, then (and this is the bit I'm missing) pass the audio across the JNI bridge to my oboe app?

I do need low-latency processing as I will be using motion data to parameterise my effect.

[philburk]

The DrumThumper example in Oboe reads drum sounds from WAV files and then plays them using Oboe.

drumthumper

[garrilla]

Hi @philburk - thanks for your reply.

Yes, I saw that drumthumper works like that, but it obviously has the wav files as resources in the build. I'm looking to let the user pick a file from local storage. Also drumthumper has both a single channel and single-shot approach. My users will be choosing tracks that may have up to 12 channels and be several minutes long.

My thoughts are to load the file into a FileInputStream then create a byte array from it. Then send a pointer to the byte array over JNI and parse it with ParseLib to make a make a FIFO out of the audio data. Any consideration I should bear in mind?

[garrilla]

For anyone passing by in search of a similar solution, here's how I solved this

1. On the Java/Kotlin side pick up the audio file (i used a WAV in this case) with an intent
2. Use a contentResolver on the audio file to create an inputStream
3. read the data from the inputStream into a byteArray
4. pass the byteArray over the JNI bridge to the native code
5. wrap the byteArray in a MemInputStream from the PARSELIB example
6. wrap the MemInputStream in a WavStreamReader, also from the PARSELIB example
7. create a SampleBuffer, from the IOLIB example, and load the WavStreamReader into it
8. create SampleSource, from the IOLIB example, and give it the SampleBuffer
9. give the SampleSource and SampleBuffer to the SimplePlayer from the IOLIB example
10. do the processing in the SampleSource's mixAudio() method bearing in mind all the rules for real-time processing in Oboe.

I also needed to do this on the block because I have a fixed window operation. to do this, I adapted the SampleBuffer class to add a method that would pull block data into mixAudio(), but that's only specific to some cases.

I hope that helps someone in the future.

[edenchisvo]

hie.
I am trying to do something similar this. I was wondering if you don't mind sharing examples of how you did it (I am new to all this), particularly steps 1,2,3,4

[garrilla]

On your Layout, have some clickable object - mine initially was a textview

    <TextView
        android:id="@+id/select_Audio"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:gravity="center"
        android:text="Click to Select an Audio File"
        android:textColor="#CAC6C6"
        android:textSize="16sp"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintHorizontal_bias="0.5"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        tools:ignore="MissingConstraints" />

Then, in your OnCreate, you will need to define the textview

select_Audio = findViewById(R.id.select_Audio);

and create a setOnClickListener for it, that will declare the intent

        select_Audio.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                Intent audio = new Intent();
                audio.setType("audio/x-wav"); // limit type to wav
                audio.setAction(Intent.ACTION_OPEN_DOCUMENT);
                startActivityForResult(Intent.createChooser(audio, "Select Audio"), PICK_AUDIO); // PICK_AUDIO is the code on the result

            }
        });

When you click on the Textview in the UI, it will cause it to open the file picker, and when you select a file (which are limited to wavs in my case) it will call a function onActivityResult that you will have to declare

public void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) {

        super.onActivityResult(requestCode, resultCode, data); 

        if (requestCode == PICK_AUDIO && resultCode == RESULT_OK) {

            // if one exists - teardown any old audio stream before creating a new one
            teardownAudioStreamNative(); // you can pick this up from the oboe example

            // Audio is Picked in format of URI
            AudioUri = data.getData();
            select_Audio.setText("Audio Selected");

            ContentResolver contentResolver = getContentResolver();
            AssetFileDescriptor audioFile;

            // try to resolve the file
            try {
                audioFile = contentResolver.openAssetFileDescriptor(AudioUri, "r");

               // next lines just used for logging
                DocumentFile documentFile = DocumentFile.fromSingleUri(this, AudioUri);
                Log.d(TAG, "File appears to have resolved! " + documentFile.getName() );
            } catch (FileNotFoundException err){
                Log.e(TAG, "onActivityResult: failed to resolve URI", null);
                return;
            }

             byte[] readData = null; // create a null byte array

            // try to create an inputStream from the file
            try {
                //create an input stream and get it's length
                FileInputStream fis = audioFile.createInputStream();
                int len = fis.available(); // there are some pitfalls with this method, read up on it

                //reassign the byte array and read the input stream into it
                readData = new byte[len];
                int i = fis.read(readData);

                Log.d(TAG, "Input Stream created! returns " + len);
            } catch (IOException io){
                Log.e(TAG, "onActivityResult: tried to resolve YRU", null);
                return;
            }

            setupAudioStreamNative(NUM_SAMPLE_CHANNELS, NUM_FRAMES_PER_BLOCK); //see Oboe sample
            startAudioStreamNative(); // see Oboe sample
            boolean sucess = loadWavAssetNative(readData); //see Oboe sample
            Log.d(TAG, success  ? "byte data transfered" : "something went wrong");
        }
    }

at this point, your sample data is now in JNI for you to use in C/C++

here's the JNI bridging code for the loadWavAssetNative()

JNIEXPORT jboolean JNICALL
    Java_com_mycompany_myproject_MainActivity_loadWavAssetNative(
            JNIEnv* env, jobject, jbyteArray bytearray) {
        int len = env->GetArrayLength (bytearray);

        // create a native byte array and popuate with the incoming array
        unsigned char* buf = new unsigned char[len];
        env->GetByteArrayRegion (bytearray, 0, len, reinterpret_cast<jbyte*>(buf));

        // wrap the byteArray in a MemInputStream
        MemInputStream stream(buf, len);

        // point the reader to the MemInputStream and parse it
        WavStreamReader reader(&stream);
        reader.parse();

        // ensure the buffer has has atleast 1 channel of audio
        jboolean isFormatValid = reader.getNumChannels() > 0;
        if(!isFormatValid) return isFormatValid;

        // create a sample buffer and point it at the reader
        AudioBuffer* sampleBuffer = new AudioBuffer();
        sampleBuffer->loadSampleData(&reader);

        // create a sample source with the buffer
        SampleSource* source = new SampleSource(sampleBuffer, 0.0);

        // add the sample source and buffer to to the player and start playing
        simpleAudioPlayer.addSampleSource(source, sampleBuffer);
        simpleAudioPlayer.play();

        delete[] buf;

        return isFormatValid;
    }

best of luck with the rest

[edenchisvo]

This is helpful, I am going to study it. Thank you very much.