CA-394409: plugins: Fix multiple file patterns for the same directory

[bernhardkaindl]

CA-394409: Fix collecting logfiles of auto-cert-kit and tapdisk-logs:

• Fix multiple file patterns for the same directory:
  • Append the directory specifications to a list instead of overwriting the previous directory spec.

        if path in directory_specifications:
            directory_specifications[path].append((cap, pattern, negate))
        else:
            directory_specifications[path] = [(cap, pattern, negate)]

Then, when we loop over the directory_specifications, loop over the list of tree_output_entries:

   for directory, tree_output_entries in directory_specs.items():
        # Multiple tree_output calls may have appended multiple output entries:
        for directory_items in tree_output_entries:
            # Unpack the stored capability, pattern and negate flag of each row:
            capability, pattern, negate = directory_items

When ignoring shifting the indendation to loop over the tree_output_entries, the change is small:
The output of git log -w -p -n1 xen-bugtool of the commit is:

--- a/xen-bugtool
+++ b/xen-bugtool
@@ -602,7 +602,10 @@ def file_output(cap, path_list):

 def tree_output(cap, path, pattern = None, negate = False):
     if cap in entries:
-        directory_specifications[path] = (cap, pattern, negate)
+        if path in directory_specifications:
+            directory_specifications[path].append((cap, pattern, negate))
+        else:
+            directory_specifications[path] = [(cap, pattern, negate)]


 def traverse_directory_specifications(directory_specs, requested_capabilities):
@@ -611,7 +614,9 @@ def traverse_directory_specifications(directory_specs, requested_capabilities):
     :param directory_specs: Directories to lookup with cap, pattern, and negate.
     :param requested_capabilities: The list of requested capabilities.
     """
-    for directory, directory_items in directory_specs.items():
+    for directory, tree_output_entries in directory_specs.items():
+        # Multiple tree_output calls may have appended multiple output entries:
+        for directory_items in tree_output_entries:
             # Unpack the stored capability, pattern and negate flag of each row:
             capability, pattern, negate = directory_items
             # If the capability is in the requested inventory entries, check it:

• Ensure that this works as expected by extending the tests accordingly.

Change:

Fix this issue to collect logs from tapdisk and auto-cert-kit as expected.

Details:

• At least three plugins collect files in /var/log using different file patterns: Auto_Cert_Kit, tapdisk-logs and xcp-rrdd-plugins.
• After merging CP-41238, only one pattern or directory tree collection was supported: Previous patterns were overwritten by plugins loaded after plugins loaded earlier.
  • Directory tree patterns added by plugins loaded earlier were overwritten by plugins loaded later.
  • As auto-cert-kit is not installed by default, this issue could only be noticed by missing tapdisk logs.

Manual Verification Testing

curl -Lo xen-bugtool.CP-49944 https://github.com/xenserver-next/status-report/raw/CP-49944-multiple-patterns-for-one-directory/xen-bugtool
chmod +x xen-bugtool.CP-49944

DoD:

# mkdir /var/log/blktap
# touch /var/log/blktap/logfile.log
XENRT_BUGTOOL_BASENAME=bugtool ./xen-bugtool.CP-49944 -y --entries=tapdisk-logs,xcp-rrdd-plugins
# tar tf /var/opt/xen/bug-report/bugtool.tar.bz2|grep -e 'log$' -e blktap
bugtool/var/log/blktap/logfile.log
bugtool/var/log/xcp-rrdd-plugins.log

and likewise, for when auto-cert-kit is installed for:

# yum install auto-cert-kit 
# touch bugtool/var/log/auto-cert-kit{-plugin}.log
# XENRT_BUGTOOL_BASENAME=bugtool ./xen-bugtool.CP-49944 -y --entries=Auto_Cert_Kit
# tar tf /var/opt/xen/bug-report/bugtool.tar.bz2|grep -e 'log$'
bugtool/var/log/auto-cert-kit.log
bugtool/var/log/auto-cert-kit-plugin.log

[coveralls]

coverage: 93.137% (+1.3%) from 91.837%
when pulling 3a316bd on xenserver-next:CP-49944-multiple-patterns-for-one-directory
into 1c8d544 on xenserver:master.