Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat]: packed reader and writer new api #171

Open
shaoting-huang opened this issue Feb 10, 2025 · 0 comments
Open

[feat]: packed reader and writer new api #171

shaoting-huang opened this issue Feb 10, 2025 · 0 comments

Comments

@shaoting-huang
Copy link
Collaborator

shaoting-huang commented Feb 10, 2025 •
edited
Loading

For the new packed writer API: Column groups and file paths are explicitly defined by users. The packed writer will not split column groups by memory. It follows the user-defined column groups to split arrow records and write into the corresponding file path.
For the new packed reader API: read paths are explicitly defined by users.
@shaoting-huang shaoting-huang mentioned this issue Feb 10, 2025
Merged
sre-ci-robot pushed a commit that referenced this issue Feb 17, 2025
@shaoting-huang
feat: packed reader and writer new api (#172)
58df0bb 
main feat for the patch:
* Column groups and file paths are explicitly defined by users. The
packed writer will not split column groups by memory. It follows the
user-defined column groups to split arrow records and write into the
corresponding file path.
* Change factory arrow filesystem to singleton arrow file system to
support Milvus init core.
* Separate arrow fs config from storage config.
related: #171

---------

Signed-off-by: shaoting-huang <[email protected]>
@shaoting-huang shaoting-huang mentioned this issue Feb 18, 2025
Merged
sre-ci-robot pushed a commit that referenced this issue Feb 20, 2025
@shaoting-huang
feat: arrow s3 file system support tls and iam (#173)
26992ec 
#171

---------

Signed-off-by: shaoting-huang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shaoting-huang