Use accurate transcription to align

[RaulKite]

Hi,

I know that there is a method to just align words when I have an accurate transcription of the audio.

Even, I'm quite sure that I have seen anywhere the way to do that with Python but I'm not able to found it again.

Can someone point me the way to do that?

Thanks

[dgoryeo]

These projects (below) are all using a variation of the alignment approach. I think the first one is the one that does exactly what you need:

https://github.com/EtienneAb3d/WhisperTimeSync
https://github.com/m-bain/whisperX/blob/main/whisperx/alignment.py
https://github.com/dmarx/video-killed-the-radio-star

[RaulKite]

These projects (below) are all using a variation of the alignment approach. I think the first one is the one that does exactly what you need:

https://github.com/EtienneAb3d/WhisperTimeSync

https://github.com/m-bain/whisperX/blob/main/whisperx/alignment.py

https://github.com/dmarx/video-killed-the-radio-star

Thanks. I know that protects but them have some inconveniences for me.

I'm looking for the way of doing that with this project.

Thanks again

[Jeronymous]

Thank you @RaulKite for your loyalty :)

Indeed in theory it's possible to use the same approach as whisper-timestamped (i.e. Whisper models with their cross-attention weights) to align a given transcription of an audio (even if that transcription was not produced by whisper)
Such a suggestion came up recently in issue #40.

It requires to reorganize a bit the code, which is not a big deal.
But it also raises the question of the format of the "accurate" transcription.
Currently, whisper-timestamped is based on whisper transcription which includes segments (of maximum 30 sec) and hints about where do these segments start and end in the audio.
Having something that works when a transcription is given for a 1H audio, without any clue about where is what, requires more thinking / work.

So @RaulKite, what transcription format would you give for an audio?

EDIT: whisper transcription also include the (detected?) language. Is it an information that you would like to provide or want it to be automatic?

[dgoryeo]

To add to that, the alignment approach makes the solution more language dependent --less universal, for example Asian languages.

[RaulKite]

Thank you @RaulKite for your loyalty :)

Indeed in theory it's possible to use the same approach as whisper-timestamped (i.e. Whisper models with their cross-attention weights) to align a given transcription of an audio (even if that transcription was not produced by whisper) Such a suggestion came up recently in issue #40.

It requires to reorganize a bit the code, which is not a big deal. But it also raises the question of the format of the "accurate" transcription. Currently, whisper-timestamped is based on whisper transcription which includes segments (of maximum 30 sec) and hints about where do these segments start and end in the audio. Having something that works when a transcription is given for a 1H audio, without any clue about where is what, requires more thinking / work.

So @RaulKite, what transcription format would you give for an audio?

EDIT: whisper transcription also include the (detected?) language. Is it an information that you would like to provide or want it to be automatic?

My idea was to use WhisperHallu ( https://github.com/EtienneAb3d/WhisperHallu ) to minimize hallucinations and then whisper-timestamped to align word timestamps.

Regards

Raúl