-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: improve watermark calculation by adding the option to cache it #6413
base: main
Are you sure you want to change the base?
Conversation
abab4ec
to
0cc9827
Compare
a6a60b0
to
313aea8
Compare
4ea09db
to
96b4732
Compare
a6d76f9
to
152b243
Compare
152b243
to
b21361c
Compare
var minTime, maxTime, watermark *time.Time | ||
err = rows.Scan(&minTime, &maxTime, &watermark) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This ends up sending **time.Time
to rows.Scan
, might not be well handled by all implementations. Also makes the pointers unsafe to access since they may still be nil. I think this would be safer:
var minTime, maxTime, watermark *time.Time | |
err = rows.Scan(&minTime, &maxTime, &watermark) | |
var minTime, maxTime, watermark time.Time | |
err = rows.Scan(&minTime, &maxTime, &watermark) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is needed for duckdb. It returns null
when there are no rows, so we need minTime, maxTime, watermark
to be pointers. Perhaps that is why we had different implementation for duckdb. We should probably go back to that. What do you think?
runtime/server/queries_metrics.go
Outdated
|
||
func decodeTimeRangeSummary(row map[string]any) (*runtimev1.TimeRangeSummary, error) { | ||
timeRangeSummary := decodedTimeRangeSummary{} // TODO: move this to a good place | ||
err := mapstructure.Decode(row, &timeRangeSummary) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you tried using WeakDecode
? Then I think it would be able to directly decode ISO strings into time.Time
types (but I may be wrong)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It wont work by itself. We need to create a decoder with DecoderConfig
and StringToTimeHookFunc
. I just reused the code from metricsview_aggregation
since we will also need for rilltime resolver.
This PR centralises the calculation of min, max and watermark to executors.
Executor.Timestamps
that does the calculation of min, max (previously in metrics_time_range resolver) and watermark.metrics_time_range
resolver to callExecutor.Timestamps
.Executor.BindQuery
that allows for setting the timestamps from outside. This allows for getting a cached timestamps by getting it frommetrics_time_range
resolver.MetricsViewTimeRange
to usemetrics_time_range
resolver.Executor.Query
to set cached timestamps.A future PR will refactor metrics_sql to move time range resolution to
Executor
. So it is not updated to use the new resolver.