Skip reading files with incorrect extension

[sarahyurick]

Closes #214.

[ayushdg]

We might need to expand the list of extensions since some files are format like .json.gz.
I wonder if an alternative could be to expand get_all_files_paths_under to also filter on an extension. That way users can specify what extension they want to filter on.
I'm hoping that #50 will make things easier in this regard.

[sarahyurick]

Thanks @ayushdg ! I like your idea of having it in get_all_files_paths_under so I changed it to use that instead.

Also, I agree with you about .json.gz. I think it is outside the scope of this PR, but I have added it to #50 for now.

[sarahyurick]

An example of when we would expect this RuntimeError is for:

doc = DocumentDataset.read_json(in_files)

Where in_files is a string path to a directory with multiple JSONL files and a CRC file. Since the CRC file is not explicitly being filtered out, we raise the error.

[ayushdg]

We can leave this as is for now.
In theory there might be cases where a user filters by [.json, .jsonl] using the file filter, but will raise errors here. In practice I expect it to be unlikely so we can wait an see if there is any user feedback around this.

[sarahyurick]

All of these examples work:
(1)

input_files = get_all_files_paths_under(in_files, filter_by="jsonl")
input_dataset = DocumentDataset.read_json(input_files)

(2)

input_files = get_all_files_paths_under(in_files, filter_by=["jsonl"])
input_dataset = DocumentDataset.read_json(input_files)

(3)

# Returns a list containing only .jsonl, .parquet, and .csv files
input_files = get_all_files_paths_under(in_files, filter_by=["jsonl", "parquet", "csv"])

[ayushdg]

Thanks for the changes. Overall changes lgtm! Minor nits/comments.

As a followup it might make sense to track updating tutorials/notebooks to use this newer filter arg in the api but not required for this pr.

[ayushdg]

I wonder if this might get too noisy in some cases. I'm leaning towards warning once if we have to skip, but not for every file we skip.

[ayushdg]

We can leave this as is for now.
In theory there might be cases where a user filters by [.json, .jsonl] using the file filter, but will raise errors here. In practice I expect it to be unlikely so we can wait an see if there is any user feedback around this.

[sarahyurick]

Thanks @ayushdg ! Updated.

[sarahyurick]

Thank you @praateekmahajan ! I have addressed all your comments.

[praateekmahajan]

LGTM! Thanks for adding type hint as well. (left two small nits on comment / typehint)

Review has been addressed, thanks!