-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Writer adapter #77
Comments
Can you give me a specific use case? |
I'm using a different format of json as training data. The structure is a little different from the format used in datatrove. The output of the current writer cannot directly fit in my latter pipeline. For example, we may want to use the key 'meta' instead of 'metadata' and add a 'version' field to denote the version of the data. |
Makes sense, will add |
Please let me know if this is what you'd like @jordane95 #83 |
Do you think it is a good idea to also add a writer adapter for the jsonl writer?
The reader also has this functionality which greatly improves the flexibility when working with jsonl data with different keys.
The text was updated successfully, but these errors were encountered: