any plans to integrate with LR-DTW?

[bendichter]

I think your repos might be just what I need! I am studying neurolinguistics and I would like to use dtw to do two things:

1. I am simultaneously recording microphone and neural data as people speak repetitions of the same phrase. One common issue is that we speak at slightly different timing every time. I would like to determine the warping from each individual trials to a template using the acoustics, and then apply this warp to the neural data. Could I use this repo for this?

2. I am also interested in trying to do dtw on the neural data alone, without the acoustics as a base modality, but I am worried that there is too much noise in the neural data for a good warp, and the path I get from naive dtw will be way off. I think I can solve this problem by regularizing the dtw path so that it prefers more direct paths. Could I use LR-DTW for that?

I suppose these use-cases don't necessarily require that your two repos be merged, but I could also see situations where I would want to warp a time series based on a base modality that is noisy and requires regularization.

Very interesting stuff! Thanks for making it public!

Ben

[rtavenar]

Hi,

First, it's always great to read that these repos might help for diverse applications, so thanks for the feedback. Concerning your first point, I believe this would definitely make sense to try to use dtw_resample for that. The only point I see is that up to now, dtw_resample is coded to have a single dimension as base modality, but I guess if you use speech, you would have multidimensional features (such as MFCC or CNN-extracted features, or ...) as base modality, so that would require a bit of code re-writing. Let me know about that so that I can maybe think of a code update that would allow that.

Second, LR-DTW might fit your needs. I have been thinking about providing a Cython-based version of it, so maybe it would be the occasion to try it on your data and see what it gives.

Finally, using both dtw_resample and LR-DTW together would require slight modification to the dtw_resample code base (as there is no single path for LR-DTW as for DTW) but it could be a nice application: in this case, I do not think I would merger both repos but rather have dtw_resample depend on LR-DTW.

Let me know about your speech experiments, I would be glad to help.

Romain

[bendichter]

Romain,

I'm glad I contacted you! Your are absolutely right that I need distance metrics that work on multimodal signals. That is true when I am warping based on audio and based on neural data (as I have multiple channels here as well).

For the LR-DTW, I'm not very familiar with that approach, but I thought the algorithm outputs a single path that is a compromise between the naive warp and the no warp. It sounds like it actually outputs a distribution over paths. Is there any way to get a maximum likelihood path? Is there a different algorithm out there that performs this? I am also looking into CTC, which I think might solve the problem more directly, but there as well I think I may need some type of warp regularization.

Thanks!
Ben

[bendichter]

oh, I also need to warp a time series with one sampling frequency and apply that warp to a time series with another sampling frequency.

[rtavenar]

After a while, I decided to start a new repo to answer your needs that would gather algorithms dedicated to the analysis of time series data. You can find it here, at the moment, it supports DTW resampling with multimodal data (documentation yet to come) and includes LR-DTW. Note however that the coupling between LR-DTW and dtw_resample has not been tested yet.

[bendichter]

Wow, this is great! I don't have time at the moment to look at this in-depth but I will certainly test this out, show it to my colleagues, and I'd be happy to help you as an alpha tester. Ben

…

On Fri, May 5, 2017 at 2:38 AM, Romain Tavenard ***@***.***> wrote: After a while, I decided to start a new repo to answer your needs that would gather algorithms dedicated to the analysis of time series data. You can find it here <https://github.com/rtavenar/tslearn>, at the moment, it supports DTW resampling with multimodal data (documentation yet to come) and includes LR-DTW. Note however that the coupling between LR-DTW and dtw_resample has not been tested yet. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAziEhN_ELeyF_Vejsappuj_aArTFKcJks5r2u4QgaJpZM4NJQv6> .

-- ---------- Ben Dichter PhD Candidate UC Berkeley and UCSF Joint Program in Bioengineering