-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementation of metadata-based freshness #1060
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added this test because we're seeing odd behavior with the get_table
method (a BQ Client method). If we run get_table
during source freshness (line 726 in impl.py
) on a table that does not exist, it hangs without returning the expected NotFound
error, even when providing retry
. However when we do the same here, it seems to behave as expected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we understand this well enough to raise a bug with bigquery? https://github.com/googleapis/python-bigquery/issues
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's this issue: googleapis/python-bigquery#1674
Removing |
This PR reflects the changes that were merged directly into 1.7.latest via #1072. |
resolves #938
Problem
The current implementation of source freshness requires querying the data and requires the user provide a datetime field. This is slower and more expensive than it should be. It doesn't scale across multiple models. And some models do not have an appropriate datetime field.
Solution
Use source metadata where available.
Checklist