Skip to content

Commit

Permalink
update obsolete flag
Browse files Browse the repository at this point in the history
Signed-off-by: Walter Teng <[email protected]>
  • Loading branch information
davzoku committed Nov 12, 2024
1 parent 93b7922 commit 93f5210
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion nemo_curator/scripts/find_exact_duplicates.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def main(args):
df = read_data(
files[:num_files] if num_files else files,
file_type="jsonl",
backend="pandas" if args.no_gpu else "cudf",
backend="pandas" if args.device != "gpu" else "cudf",
files_per_partition=args.files_per_partition,
add_filename=False,
)[[id_field, text_field]]
Expand Down

0 comments on commit 93f5210

Please sign in to comment.