Skip to content

Commit

Permalink
Avoid duplicate loading of component files
Browse files Browse the repository at this point in the history
As discovered in ComplianceAsCode#11190,
the `components.load()` function is called many times during a build
of product content. However, the component files need to be loaded only
once.

When the build systems resolves the content using `compile_all.py`, the
`compile_all.py` creates an instance of `ssg.builld_yaml.BuildLoader`
class.  Moreover, many other instances of `ssg.builld_yaml.BuildLoader`
class are created recursively as the loader recurses into
sub-directories of the given benchmark root directory.  When we iterate
over the directories, for each child directory the script creates a new
instance of the `ssg.build_yaml.BuildLoader` by the
`ssg.builld_yaml.BuildLoader._get_new_loader()` method. This method
should ensure that the component data and other data won't be
unnecessarily loaded again but will be just referenced from the parent
`ssg.builld_yaml.BuildLoader` that handles the parent directory.

The problem is the way how the
`ssg.builld_yaml.BuildLoader._load_components()` method is called in the
ctor of `ssg.builld_yaml.BuildLoader`. The ctor is called before
the initialization of the `rule_to_components` attribute. Also, there is
no code in this ctor or elsewhere assuring that it won't load the
components data if they were already loaded.

In this commit, we won't call `_load_components` in the ctor, which will
leave the opportunity to `_get_new_loader` to swap the data as the comment
said.

This should bring a speed up about 5 seconds.
  • Loading branch information
jan-cerny committed Oct 12, 2023
1 parent 5c06539 commit 354fdb4
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 4 deletions.
1 change: 1 addition & 0 deletions build-scripts/compile_all.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ def main():

loader = ssg.build_yaml.BuildLoader(
None, env_yaml, product_cpes, args.sce_metadata, args.stig_references)
loader.load_components()
load_benchmark_source_data_from_directory_tree(loader, env_yaml, product_yaml)

project_root_abspath = os.path.abspath(args.project_root)
Expand Down
7 changes: 3 additions & 4 deletions ssg/build_yaml.py
Original file line number Diff line number Diff line change
Expand Up @@ -1331,19 +1331,18 @@ def __init__(
if stig_reference_path:
self.stig_references = ssg.build_stig.map_versions_to_rule_ids(stig_reference_path)
self.components_dir = None
self.rule_to_components = self._load_components()
self.rule_to_components = None

def _load_components(self):
def load_components(self):
if "components_root" not in self.env_yaml:
return None
product_dir = self.env_yaml["product_dir"]
components_root = self.env_yaml["components_root"]
self.components_dir = os.path.abspath(
os.path.join(product_dir, components_root))
components = ssg.components.load(self.components_dir)
rule_to_components = ssg.components.rule_component_mapping(
self.rule_to_components = ssg.components.rule_component_mapping(
components)
return rule_to_components

def _process_values(self):
for value_yaml in self.value_files:
Expand Down

0 comments on commit 354fdb4

Please sign in to comment.