how i fixed many errors in my png annotation import

[alecristia]

Prep work

before starting, I make sure I have the latest version & create a new branch. Png is special because it's shared between LAAC and EL1k. The idea is that EL1k users won't update png, it's going to be maintained by us and some of the updates will be shared with EL1k - therefore, we work on LAAC-LSCP's version and push some of the work to EL1k.

So I need to get the laac version. I download it:

source ~/ChildProjectVenv/bin/activate
datalad install [email protected]:/LAAC-LSCP/png2019.git
cd png2019
datalad run-procedure setup --private --confidential

And right now, given the state of affairs, I need to get on the eaf branch, and create a branch from there:

git checkout eaf
git checkout -b eaf-corrections

The first command threw:

Branch 'eaf' set up to track remote branch 'eaf' from 'origin'.
Switched to a new branch 'eaf'

so to be safe I did git log and compared the history with that online.

Mis-named annotation path

to begin with, i mis-named the path: right now it's mc, rather than eaf/mc.

I followed renaming annotations, which in this case meant:

child-project rename-annotations . --set mc --new-set eaf/mc
rm -r annotations/mc/
datalad save . -m "renamed path"
git push -u origin eaf-corrections 

this more complex version of push is only necessary because it's my first push since I created the branch

finding which one file is missing and why

There are 24 .eaf but only 23 imported:

This was because I had made a mistake and assigned the same section of the audio to one of the files. Fixed by fixing the assignment table.

contents of files from the same child are identical

This is clearly a bug in the import script.

purging duplicates from annotations.csv

The safest way to fix this is to remove all the annotations, fix all errors and import again.

source ~/ChildProjectVenv/bin/activate
child-project remove-annotations . --set eaf/mc

i did that and then looked at n of lines of annotations.csv (wc -l metadata/annotations.csv) and n of files in each of the converted folders (ls annotations/*/converted/ | wc -l).

There are 309 lines in annotations.csv but only 206 files in annotations/*/converted/. This could mean there's some extra stuff in annotations.csv. The extra lines are vtc or alice, so they should not be affecting my current analyses. Plus, there can be extra lines related to how vtc and alice are imported, which means that i shouldn't just look at the number of lines. Instead, i can do things like validating the project (which would detect duplicates if there are any). So this is NOT an issue at the end.

I then made edits to my import_eaf.py script and tried the import again:

source ~/ChildProjectVenv/bin/activate
python3 scripts/import_eaf.py

One bug that took me a while to find was:

an error occured while processing './annotations/eaf/mc/raw/c23_20190822_2015.eaf'
Traceback (most recent call last):
File "/Users/acristia/ChildProjectVenv/lib/python3.6/site-packages/ChildProject/annotations.py", line 185, in _import_annotation
df = import_function(path)
File "scripts/import_eaf.py", line 50, in converter
eaf = pympi.Elan.Eaf(filename)
File "/Users/acristia/ChildProjectVenv/lib/python3.6/site-packages/pympi/Elan.py", line 116, in init
parse_eaf(file_path, self)
File "/Users/acristia/ChildProjectVenv/lib/python3.6/site-packages/pympi/Elan.py", line 1433, in parse_eaf
tree_root = etree.parse(file_path).getroot()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xml/etree/ElementTree.py", line 1196, in parse
tree.parse(source, parser)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/xml/etree/ElementTree.py", line 586, in parse
source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: './annotations/eaf/mc/raw/c23_20190822_2015.eaf'

That was weird, since :
(ChildProjectVenv) (base) Mac-mini:png2019 acristia$ ls annotations/eaf/mc/raw/c23_20190822_2015.eaf

annotations/eaf/mc/raw/c23_20190822_2015.eaf

but this is just a pointer! So I needed to remember to do:

datalad get annotations/eaf/mc/raw/

and then try my script:

python3 scripts/import_eaf.py

This gave me some warnings like:

warning: annotation 'a24' not found in segments for './annotations/eaf/mc/raw/c08_20190820_1745.eaf'

I also opened a couple of the converted files, and they looked correct.

I then checked that annotations are functional within the package:

child-project validate . --check-annotations --ignore-files

This showed errors like:

warning: ./annotations/its/converted/e20191024_000253_008232_0_0.csv: 'SIL' is not a permitted value for column 'speaker_type' on line 31215, should be any of [FEM,MAL,CHI,OCH,NA]
validation failed, 499 error(s) occured

That means the .its need to be re-imported.

I tried

python scripts/import_its.py

which yielded

OSError: Error reading file './annotations/its/raw/e20191023_210456_013099.its': failed to load external entity "./annotations/its/raw/e20191023_210456_013099.its"

This is because I forgot to get the its raw, so

datalad get annotations/its/raw
python scripts/import_its.py

I got some errors and warnings that I think should be ignored and may be changed in the future (see #229).

In a nutshell, at this point, my annotations did not pass validation. I hesitated about pushing (which allowed others to see what I had done and help me fix mistakes) or not. @lucasgautheron said " In general, it is fine to push code that does not pass the tests in a feature branch as long as the master branch and tagged commits (i.e. releases) pass the tests". Since I'm working on my own branch, I pushed it with datalad save . -m "cleaning"followed by datalad push. I wish I hadn't done that, since that saves everything (including the annotations) and that takes a LONG time. Instead, I should have done:

datalad save metadata/ -m "cleaning"
datalad save scripts/ -m "cleaning"
datalad push

Notice that will update metadata and scripts, but not annotations/.

[lucasgautheron]

what about datalad get annotations/eaf/mc/raw/c23_20190822_2015.eaf ?

[alecristia]

of course! sorry for the silly error -- incorporated above

[alecristia]

at one point, I was not getting the same results as Lucas.

To get the same results, I needed to make sure I had everything the same, namely:

• update the childproject package: pip3 install git+https://github.com/LAAC-LSCP/ChildProject.git
• make sure that my analysis code upstream (I was working on reliability) was up to date: git pull
• make sure I had the most recent version of the dataset: git pull and then datalad update
• make sure I was in the right branch git checkout eaf-corrections
• make sure I had the most recent annotations: datalad get annotations/eaf/mc/converted annotations/its/converted/

[alecristia]

I've now received a new version of the annotation, and in this one the tier that needs to be imported is called "final". Otherwise, the names are the same as before, so I should be able to reuse the import scripts to over-write raw & converted, by just changing which tier is analyzed.

I found two scripts:

• prep_eaf_import.sh was one I made, but I think it's outdated. One thing that is good about it is that it keeps a record of the source of the eafs, and it renames the files. I added a note "this script is outdated! do not reuse". I made a copy called prep_eaf_import_recoding.sh

To execute this script, I had to unlock the files: datalad unlock annotations/eaf/mc

• import_eaf.py is for the actual import -- here is where I needed to change the tier name. But I'm worried that it's using a file that I composed by hand to attribute files to times/recordings, which I remember was wrong... Might this not be the latest version of the importation script?

[lucasgautheron]

The last version of import_eaf.py is all you need - you can remove the other files if you'd like. That might actually be better to prevent further confusion

[alecristia]

ok, turns out I was working on an outdated branch (!)

Switched to main, removed the confusing files, and created a new branch (recoding).

Then I unlocked and re-did the import, to have the new versions of files in raw.

I changed the tier name in import_eaf.py.

Then tried python scripts/import_eaf.py, but this failed:

OSError: Error reading file 'annotations/its/confidential/raw/e20190823_150227_009456.its': failed to load external entity "annotations/its/confidential/raw/e20190823_150227_009456.its"

This is because I also need the its:

datalad get annotations/its/confidential/raw/

After doing that, I could re-import. I pushed this commit to the new branch, and then waited to check something downstream (via #263) before merging into main.