Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/335 parquet lf processing #338

Merged
merged 23 commits into from
Feb 6, 2025
Merged

Feature/335 parquet lf processing #338

merged 23 commits into from
Feb 6, 2025

Conversation

lchen-2101
Copy link
Collaborator

closes #335

lchen-2101 and others added 22 commits November 8, 2024 10:04
@lchen-2101
validate lazy frame
402064c
@lchen-2101
Merge branch 'main' into parquet_validator
74fdb2b
@lchen-2101
parquet cli
d98333f
@lchen-2101
Merge branch 'main' into parquet_validator
509c894
@jcadam14
Merge branch 'main' into parquet_validator
9d9d0c4
@jcadam14
Print statement for testing
4a2e845
@jcadam14
More test outputs
69c7fe6
@jcadam14
More test outputs
5f9dd3f
@lchen-2101
Merge branch 'main' into parquet_validator
9f843c5
@lchen-2101
test returning buffer instead of bytes
61aa497
@lchen-2101
revert buffer bytes
33384eb
@jcadam14
Fixed bug with index in chunks and process_errors
ed9c974
@jcadam14
Merge branch 'parquet_validator' of https://github.com/cfpb/regtech-d…
e6cd1fc 
…ata-validator into parquet_validator
@jcadam14
Added flush to print
64a1427
@jcadam14
Updated pandera to 0.22.0 to avoid dependency failure
dfc797c
@lchen-2101
fix: df.DataFrame -> pl.DataFrame
14b0c98
@lchen-2101
light processing on over threshold errors, and keep track of record c…
a341b4a 
…ount (#330)
@jcadam14
Forcing polars 1.19
054e765
@jcadam14
Pyarrow issue?
97453fb
@jcadam14
Issue with alpine image
704510b
@lchen-2101
feat: lf processing, refactors, dependencies clean up
1aa601d
@lchen-2101
linting, tests, deprecated method updates
74b89cd
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 5, 2025 •
edited
Loading

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)		Lines missing
  src/regtech_data_validator
  cli.py 102-109
  data_formatters.py
  validation_results.py
  validator.py 107, 109, 114, 129-130, 136, 210-211, 236-246, 268-275
Project Total  

This report was generated by python-coverage-comment-action

@jcadam14 jcadam14 merged commit b20b6e2 into main Feb 6, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jcadam14 jcadam14 jcadam14 approved these changes

@hkeeler hkeeler Awaiting requested review from hkeeler

@guffee23 guffee23 Awaiting requested review from guffee23

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add Parquet Processing
2 participants
@lchen-2101 @jcadam14