[BUG] Semdedup Embedding Restart not working cleanly

[VibhuJawa]

Describe the bug

Currently our semdedup restart mechanism for embedding is not working cleanly.

This is because of following ( add_filename=False)

NeMo-Curator/nemo_curator/scripts/semdedup/compute_embeddings.py

Lines 62 to 64 in 3a31ab1

	ddf = read_data(
	input_files=input_files, file_type=args.input_file_type, add_filename=False
	)

And write to filename is False

NeMo-Curator/nemo_curator/scripts/semdedup/compute_embeddings.py

Line 78 in 3a31ab1

write_to_filename=False,

And get_remaining_files by default cant handle comparing files with different extensions.

NeMo-Curator/nemo_curator/utils/file_utils.py

Lines 66 to 80 in 3a31ab1

	def get_remaining_files(
	input_file_path, output_file_path, input_file_type, num_files=-1
	):
	"""
	This function returns a list of the files that still remain to be read.
	Args:
	input_file_path: The path of the input files.
	output_file_path: The path of the output files.
	input_file_type: The type of the input files.
	num_files: The max number of files to be returned. If -1, all files are returned.
	Returns:
	A list of files that still remain to be read.
	"""

[sarahyurick]

Happy to pair on this at some point; in general there are a couple of things I have been thinking should be refactored with DocumentDataset's read and write functions.

See: #50, #180, #293...

[VibhuJawa]

@sarahyurick , I think given your PRs , you should probably just take this on. Happy to provide input as needed. Let me know what you think.

[sarahyurick]

I'm not sure I can reproduce this. I ran:

python compute_embeddings.py \
    --input-data-dir "my_data" \
    --input-file-type "jsonl" \
    --input-file-extension "jsonl" \
    --config-file "semdedup_config.yaml"

where my_data is a directory with 2 JSONL files. In semdedup_config.yaml, I specified a different directory as the cache_dir where the 2 resulting Parquet files were written. When I rerun without changing anything, there are no errors.

LMK if there is anything else I should be setting or changing, otherwise we can close this issue.

[sarahyurick]

NVM, the issue is that it should not rerun if the embeddings are already present.

[sarahyurick]

Closed by #327.