DEVPROD-12696: Parallelize waterfall queries #8672
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sophstad
force-pushed
the
DEVPROD-12696-parallelize
branch
from
45bce22 to
eb1425a
Compare
khelif96
approved these changes
Jan 29, 2025
khelif96
reviewed
Jan 29, 2025
graphql/version_resolver.go
Outdated
Comment on lines
524
to
534
| // If activated is nil in the db we should resolve it and cache it for subsequent queries. There is a very low likely hood of this field being hit | ||
| if obj.Activated == nil { | ||
| version, err := model.VersionFindOne(model.VersionById(versionId)) | ||
| if err != nil { | ||
| return nil, InternalServerError.Send(ctx, fmt.Sprintf("fetching version '%s': %s", versionId, err.Error())) | ||
| } | ||
| if err = setVersionActivationStatus(ctx, version); err != nil { | ||
| return nil, InternalServerError.Send(ctx, fmt.Sprintf("setting version activation status: %s", err.Error())) | ||
| } | ||
| obj.Activated = version.Activated | ||
| } |
There was a problem hiding this comment.
Is this still necessary? We introduced the activated field a few years ago and have since introduced the TTL I don't think any versions that haven't been TTL'd should still have this field as nil.
There was a problem hiding this comment.
Thanks for the context, this was copy pasta. Removed!
khelif96
approved these changes
Jan 29, 2025
ablack12
reviewed
Jan 29, 2025
| ) | ||
|
|
||
| const ( | ||
| buildsKey = "builds" |
There was a problem hiding this comment.
nit: since this is only used as a private const in getVersionTasksPipeline, we can just define it as a const in there
There was a problem hiding this comment.
It's also used by GetWaterfallBuildVariants
ZackarySantana
added a commit
to ZackarySantana/evergreen
that referenced
this pull request
Jan 30, 2025
commit 4e3ed98 Author: Zackary Santana <[email protected]> Date: Thu Jan 30 13:34:26 2025 -0500 Revert "DEVPROD-13976 Add otel to presign file func (evergreen-ci#8673)" This reverts commit c856008. commit c856008 Author: Zackary Santana <[email protected]> Date: Thu Jan 30 09:30:17 2025 -0500 DEVPROD-13976 Add otel to presign file func (evergreen-ci#8673) commit 6d420a2 Author: Sophie Stadler <[email protected]> Date: Wed Jan 29 14:35:01 2025 -0500 DEVPROD-12696: Parallelize waterfall queries (evergreen-ci#8672)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DEVPROD-12696
Description
The waterfall schema currently returns versions separately from the page's build variants. This was a helpful schema for rendering the page easily, but limited how much we could leverage parallelization and Apollo caching.
I've added a field within the
Versiontype that returns all of the builds and tasks for that version. This means that instead of one large query, there are several that run in parallel. The new pipeline reuses a large chunk of the old one so it's pretty well tested. Though organizing the builds into build variants now needs to be done on the client side, performance overall should be faster.Testing
Added unit tests
On Honeycomb, you can see that
Version/waterfallBuildsruns a lot faster than the existingVersion/buildVariantsquery. I'm excited to see how this scales on prod data vs the current aggregation.