[FEA] Support of multiple attempts

[amahussein]

Is your feature request related to a problem? Please describe.

PR #1324 (merged on Sep 6th 2024) which was released in v24.08.2, we implemented a workaround to handle multiple attempts within a single JVM.
This quick fix does not work well when the output of core-tools run is merged or compared since each JVM can process one attempt unware of the the existence of a different one.

It is not straightforward to tell whether an eventlog is sucessful or not taking into considertion DB eventlogs which are incomplete.
We should handle each attempt, then the attempt with highest number of attemptId should be the last one (maybe successful one).

This is a tricky change because there requires a change in the directory structure and the CSV files along with the QualX training/prediction code.

Impact:

• UUID will be <AppID, AttemptID> instead of just
• QualX joins will be affected
• New column attemptID to be added in Qual combined CSV files
• Folders for rawMetrics/tuning need to change to avoid overwriting same folder twice by multiple attempts

Better option:

• create a new internal key UUID that works as combined
• This will be the key used internally to join tables
• Folders or files will be renamed by the UUID
• Combined CSV files in the qual will have the new column
• Profiler should replace columnIndex by the UUID
• QualX should use the UUID for joins.

Other considerations

Assuming that the tools is processing 2 different file paths that represent the same AppID-AttemptID, then the tools should handle this duplication; or the upper layer that combine results together should check that records of the same UUID already exist.

Tasks

Preview Give feedback

1. 01-Add UUID to the AppInfo and AttemptID in the output file