Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve schemeshard operations build time #10633

Open
12 of 28 tasks
ijon opened this issue Oct 18, 2024 · 0 comments
Open
12 of 28 tasks

Improve schemeshard operations build time #10633

ijon opened this issue Oct 18, 2024 · 0 comments
Assignees
@ijon

Comments

@ijon
Copy link
Collaborator

ijon commented Oct 18, 2024 •
edited
Loading

Compiling schemeshard and schemeshard operations takes significant amount of time.
(8394 seconds, which is 2.5 hours on a single core or 8.7 minutes on a 16 cores)

And while schemeshard consists of several parts including the core, schema operations support mechanics, split-merge subsystem, background compaction subsystem, long running schema processes ("meta-operations") like database backup/restore, and so on, the real concern this issue is trying to address (beyond the total schemeshard building time) is the build time of the single schema operation.

Schemeshard includes implementations of dozens of scheme operations (78 to be precise)...

TODO...

  • move TCompactionPolicy from flat_scheme_op.proto to some core .proto
  • and completely replace TCompactionPolicy with something else to use in public api -- current specification is unusable for that
  • break ydb/core/base dependency on flat_scheme_op.proto
  • localdb.h -- remove header dependency on flat_scheme_op.proto
  • schemeshard_impl.h -- split operations iface into schemeshard__operation_iface.h
  • schemeshard_impl.h -- remove use by operations
  • separate schemeshard core and operations "api"
    • make non intrusive separation
    • get rid of static_cast to TSchemeShard* in operations code
    • real api, may be with intrusive changes in operations
  • make anything coming from schemeshard__operation_iface.h forward declared
  • move static methods out of TSchemeShard (into some lib/utils module)
  • schemeshard_tx_infly.h -- remove dependency on flat_scheme_op.proto and flat_tx_scheme.proto
  • schemeshard_identifiers.h -- remove dependency on flat_scheme_op.proto
  • schemeshard__operation_memory_changes.h -- make clean interface without leaking specific types
  • schemeshard_info_types.h -- separate different types into separate files
  • schemeshard__operation_side_effects.h -- make clean interface without leaking specific types
  • schemeshard__operation_db_changes.h -- also clean from specific types
  • schemeshard.h, schemeshard_private.h -- separate event interface into client, core, operations etc interfaces and include relevant parts only where necessary
  • schemeshard__operation_common.h -- remove dependency on ydb/core/base/hive.h
  • schemeshard__operation_common.h -- remove dependency on schemeshard_private.h
  • schemeshard_utils.h
    • do not include from schemeshard_impl.h and other heavy headers
    • move TShardDeleter out of schemeshard_utils.h
    • move TSelfPinger out of schemeshard_utils.h
    • may be dismantle even
  • tests -- reduce influence of a single change on single ut_-module rebuild time
  • ...
@ijon ijon self-assigned this Oct 18, 2024
@ijon ijon mentioned this issue Oct 21, 2024
Merged
ijon added a commit that referenced this issue Oct 21, 2024
@ijon
schemeshard: split schemeshard__operation_common.cpp, move code from …
ff01ea7 
….h (#10631)

Move operations specific content of schemeshard__operation_common.cpp (bsv, cdc_stream, pq, subdomain) into separate files.
Move dependency rich methods definitions out of schemeshard__operation_common.h.

This is part of "improve schemeshard operations build time" effort (#10633).
@ijon ijon mentioned this issue Oct 21, 2024
Merged
san-kir-k pushed a commit to spaikus/ydb that referenced this issue Oct 21, 2024
@ijon @san-kir-k
schemeshard: split schemeshard__operation_common.cpp, move code from …
af71b20 
….h (ydb-platform#10631)

Move operations specific content of schemeshard__operation_common.cpp (bsv, cdc_stream, pq, subdomain) into separate files.
Move dependency rich methods definitions out of schemeshard__operation_common.h.

This is part of "improve schemeshard operations build time" effort (ydb-platform#10633).
This was referenced Oct 22, 2024
Merged
Merged
Open
ijon added a commit that referenced this issue Oct 29, 2024
@ijon
change some event scopes from struct to namespace (#10685)
6951a91 
Change event scopes from struct to namespace.

Only for those components, which events are used by schemeshard and in scheme operations.

When event classes are declared as inner classes of a `struct`, it is impossible to make forward declarations of those classes (without refactoring all those definitions in a very specific way). Changing event scope from `struct` to `namespace` enables easy forwarding of its event classes. And that is just a renaming and not a refactoring (no nontrivial changes).

Scopes changed:
* `TEvBlobDepot`
* `TEvBlockStore`
* `TEvColumnShard`
* `TEvConfigsDispatcher`
* `TEvConsole`
* `TEvDataShard`
* `TEvFileStore`
* `TEvHive`
* `TEvKesus`
* `TEvPersQueue`
* `TEvSchemeShard`
* `TEvSequenceShard`
* `TEvSubDomain`

This is part of "improve schemeshard operation build-time" effort (#10633).
This was referenced Nov 15, 2024
Merged
Draft
ijon added a commit that referenced this issue Nov 20, 2024
@ijon
schemeshard: forward declare events in a core header for operations (#…
2dd1cac 
…11655)

Switch `schemeshard_operation_part.h` to forward declarations of component events.
Move component specific header includes from `schemeshard_operation_part.h` to relevant suboperation modules.

Event definitions often use component specific types and definitions which require dragging along component specific headers. As a result, any schemeshard suboperation module (which have to include base header `schemeshard_operation_part.h`) transitively include headers from all components for all scheme suboperations. Which is a bit excessive.

Made possible by #10685.

This is part of "improve schemeshard operation build-time" effort (#10633).
@ijon ijon mentioned this issue Nov 22, 2024
Merged
ijon added a commit that referenced this issue Nov 26, 2024
@ijon
schemeshard: stop using schemeshard_utils.h as public header (#11895)
7c597c1 
Move `IsAllowedKeyType()` out from schemeshard (from `ydb/core/tx/schemeshard/schemeshard_utils.h` to `ydb/core/scheme/scheme_tablerefs.h`).

Stop using private header `schemeshard_utils.h` as public.

This is part of "improve schemeshard operation build-time" effort (#10633).
@ijon ijon mentioned this issue Nov 26, 2024
Merged
ijon added a commit that referenced this issue Nov 27, 2024
@ijon
schemeshard: remove schemeshard_utils.h from common dependencies (#12012
bb312c5 
)

Remove `schemeshard_utils.h` from `schemeshard_impl.h` includes.
Move `TShardDeleter` and `TSelfPinger` out of `schemeshard_utils.*`.

This is part of "improve schemeshard operation build-time" effort (#10633).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ijon ijon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ijon