Fix gpu_exact_dups due to deprecated flag

[davzoku]

Description

This PR is to fix gpu_exact_dups mention in this issue, #350

Referencing these 2 commits:

• 1c73ade#diff-67ff83fb9bb047c0cc82578699f04307c242ecb4249937c971dbf644494b56edR43
• 5e46cd8

I noticed the args.no_gpu is a deprecated argument, however during the refactoring process, one of this argument is being left out. This blocks gpu_exact_dups from executing correctly.

Usage

Prereqs:

add_id is executed on the jsonl to add id for dedup
existing .jsonl files in file directory eg.books_dedup/

gpu_exact_dups \
 --input-json-id-field="nemo_id" \
 --input-data-dir=books_dedup/ \
 --output-dir=exact_dedup \
 --device=gpu \
> logs/exact_dedup2.log 2>&1

Checklist

Results

Notice that after this commit, we can run gpu_exact_dups to completion.

[VibhuJawa]

Thanks for fixing this, the PR looks good to me, you will have to sign your commits though, using DCO though. A

NeMo-Curator/CONTRIBUTING.md

Lines 76 to 85 in 93b7922

	1) Make sure your PR does one thing. Have a clear answer to "What does this PR do?".
	2) Read General Principles and style guide above
	3) Make sure you sign your commits. E.g. use ``git commit -sS`` when committing.
	1) If you forget to do this, please follow the steps below to undo the commits and reapply the changes under a new (signed and signed-off) commit. Note: This will preserve your changes, but delete the git history of commits.
	```bash
	git reset --soft HEAD~N
	git add <insert all files you want to include>
	git commit -sS -m "My commit message"
	git push --force
	```

See below for learning how to fix this:
https://github.com/NVIDIA/NeMo-Curator/pull/351/checks?check_run_id=32854401493

[davzoku]

Hi @VibhuJawa, I have done the sign-off. lmk if there is any other issues. Thank you!

[VibhuJawa]

@davzoku , Its still missing them to signed, i think we missed the -S flag. Recommend trying below:

https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[davzoku]

@VibhuJawa, the commit should be signed now

[VibhuJawa]

@davzoku , Thanks a lot for your contribution, merged the fix in . Appreciate it.

[ms-leemina]

@VibhuJawa @davzoku Is this update applied to the docker image? With the docker image of nvcr.io/nvidia/nemo:24.09, the exact duplicate still not work