O2-4592 - adding TOF dX and dZ to the track QA table with int8 precision #13787

miranov25 · 2024-12-09T21:41:03Z

O2-4592 - adding TOF dX and dZ to the track QA table with int8 precision

Description

This update adds key TOF information, such as dX and dZ, to the trackQA table in AO2D. These additions are critical for machine learning applications and conditional tracking post-corrections.

Information for Machine Learning

ΔX and ΔZ: + Expanding the search road - in this pull request
Integral Time: For hypotheses and Length (Δ).
Integral Material Budget: After the TPC.
Flags Information: To handle cases where more than one cluster is assigned to the track (e.g., pad overlap).

Usage of Information - Conditional Tracking as Post-Correction

ΔY and ΔZ:
- Correction for wrong mass hypotheses (tracking mass and TOF PID), improving matching efficiency and reducing fakes (primarily to recover TPC crossing points).
- Adjustment for electron bremsstrahlung.
- Compensation for imperfect drift velocity and distortion calibration.
Conditional Correction for Deep Secondaries:
- Based on decay radius and mother particle mass.
Background Probability Evaluation:
- Estimation of the probability of absorption or decay.

People involved:
@noferini, @ercolessi, @f3sch

github-actions · 2024-12-09T21:41:14Z

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1

alibuild · 2024-12-10T01:01:24Z

Error while checking build/O2/fullCI for 6b75f0f at 2024-12-10 02:01:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:385:23: error: 'struct o2::aodproducer::AODProducerWorkflowDPL::TrackQA' has no member named 'dTofdY'; did you mean 'dTofdX'?
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:2605:24: error: no match for 'operator[]' (operand types are 'o2::dataformats::GlobalTrackID' and 'o2::dataformats::GlobalTrackID::Source')
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:2607:25: error: 'gloCopy' was not declared in this scope
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:2609:30: error: 'safeInt8Clamp' was not declared in this scope
ninja: build stopped: subcommand failed.

Full log here.

alibuild · 2024-12-10T07:53:04Z

Error while checking build/O2/fullCI_slc9 for e654f33 at 2024-12-10 08:53:

## sw/BUILD/O2-latest/log
ninja: build stopped: subcommand failed.

Full log here.

alibuild · 2024-12-10T08:27:28Z

Error while checking build/O2/fullCI for e654f33 at 2024-12-10 09:27:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:385:23: error: 'struct o2::aodproducer::AODProducerWorkflowDPL::TrackQA' has no member named 'dTofdY'; did you mean 'dTofdX'?
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:2604:24: error: no match for 'operator[]' (operand types are 'o2::dataformats::GlobalTrackID' and 'o2::dataformats::GlobalTrackID::Source')
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:2606:25: error: 'gloCopy' was not declared in this scope
/sw/SOURCES/O2/13787-slc8_x86-64/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:2608:30: error: 'safeInt8Clamp' was not declared in this scope
ninja: build stopped: subcommand failed.

Full log here.

…gma precision nBins*2

Signed-off-by: Felix Schlepper <[email protected]>

miranov25 · 2024-12-10T11:23:17Z

Hello @f3sch,

Thank you for the patches! However, there is still an issue with the TOF source that I didn’t catch in the first iteration (the code wasn’t fully synced to the server, so first version was compiling well ).

2024-12-10@12:21:53:DEBUG:O2:O2:0: /scratch/alice/miranov/alicesw/sw/SOURCES/O2/dev/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx: In member function 'o2::aodproducer::AODProducerWorkflowDPL::TrackQA o2::aodproducer::AODProducerWorkflowDPL::processBarrelTrackQA(int, uint64_t, GIndex, const o2::globaltracking::RecoContainer&, const std::map<long unsigned int, int>&)':
2024-12-10@12:21:53:DEBUG:O2:O2:0: /scratch/alice/miranov/alicesw/sw/SOURCES/O2/dev/0/Detectors/AOD/src/AODProducerWorkflowSpec.cxx:2610:24: error: no match for 'operator[]' (operand types are 'o2::dataformats::GlobalTrackID' and 'o2::dataformats::GlobalTrackID::Source')
2024-12-10@12:21:53:DEBUG:O2:O2:0:  2610 |     if (contributorsGID[GIndex::Source::TOF].isIndexSet()) { // ITS-TPC-TRD-TOF, ITS-TPC-TOF, TPC-TRD-TOF, TPC-TOF
2024-12-10@12:21:53:DEBUG:O2:O2:0:       |                        ^
2024-12-10@12:21:53:DEBUG:O2:O2:0: At global scope:
2024-12-10@12:21:53:DEBUG:O2:O2:0: cc1plus: note: unrecognized command-line option '-Wno-unknown-warning-option' may have been intended to silence earlier diagnostics

I’ll revisit this topic later today after preparing the run-CTF list. If you’re able to address the second compilation problem in the meantime, I’d greatly appreciate it.

Best regards,
Marian

miranov25 · 2024-12-11T11:33:12Z

TOF Delta Clusters in the TrackQA Table

TOF information in the test sample production is highly compressible:

Most tracks do not have TOF information.
The entropy of the coded residuals (1 byte, compression factor ~15) is very low.

Input Set from Test Checks:
/alice/data/2023/LHC23zzh/544122/apass4_test3_debug_epn/0410/o2_ctf_run00544122_orbit0136011072_tf0000000001_epn311

Data compression:

*............................................................................*
*Br   23 :fDeltaTOFdX : fDeltaTOFdX/B                                        *
*Entries :    31089 : Total  Size=      31679 bytes  File Size  =       2073 *
*Baskets :        1 : Basket Size=      32113 bytes  Compression=  15.04     *
*............................................................................*
*Br   24 :fDeltaTOFdZ : fDeltaTOFdZ/B                                        *
*Entries :    31089 : Total  Size=      31683 bytes  File Size  =       2093 *
*Baskets :        1 : Basket Size=      32113 bytes  Compression=  14.90     *
*............................................................................*

Example track-TOF cluster delta

miranov25 · 2024-12-11T11:41:52Z

Hello @shahor02, @chiarazampolli, and Framework Admins,

The code is functioning well, and the TOF delta compression is highly efficient (see the comment above). We observe a 2-bit data volume increase with the inclusion of TOF. For most tracks, TOF information is missing, and for the remainder, entropy coding is very effective.

How should we proceed with the pull request? I would like to have this included in APass1.

Best regards,
Marian

shahor02 · 2024-12-11T13:36:12Z

Tagging @pzhristov and @ddobrigk

pzhristov · 2024-12-11T14:31:46Z

I suppose we will need also O2Physics/Common/TableProducer/Converters/tracksQaConverter.cxx to be able to read _001 version.

catalinristea · 2024-12-11T15:12:16Z

Hi Marian!

In case the TOF wasn't in data, could you please make sure the tpc-timeseries-workflow doesn't crash? Like

[INFO] Publishing M-shape correction map
[ERROR] invalid workflow in o2-tpc-time-series-workflow: No matching output found for TOF/MTC_ITSTPCTRD/0 as requested by data processor "tpc-time-series". Candidates:
[ERROR] -DPL/ENUM/2970000419
[ERROR] -CPV/CTFDATA/0
...

Thanks,
Catalin.

miranov25 · 2024-12-11T16:35:10Z

I suppose we will need also O2Physics/Common/TableProducer/Converters/tracksQaConverter.cxx to be able to read _001 version.

Hello Peter,

We have another pull request for that. I’m not entirely sure how it was implemented, as the code was written by Felix. I only created related pull request.

Here’s the link: AliceO2Group/O2Physics#8901

miranov25 · 2024-12-11T16:53:48Z

Hi Marian!

In case the TOF wasn't in data, could you please make sure the tpc-timeseries-workflow doesn't crash? Like

[INFO] Publishing M-shape correction map [ERROR] invalid workflow in o2-tpc-time-series-workflow: No matching output found for TOF/MTC_ITSTPCTRD/0 as requested by data processor "tpc-time-series". Candidates: [ERROR] -DPL/ENUM/2970000419 [ERROR] -CPV/CTFDATA/0 ...

Thanks, Catalin.

Hello Catalin,

I’m not entirely sure what you’re referring to here. The TPC time series should be independent of the AO2D conversion, so this seems like an unrelated issue. From the error message, it looks like this is part of the production script's output.

I found a similar error message in:

Detectors/TPC/workflow/src/TPCScalerSpec.cxx
I assume we might need to add a protection in another file used for the time series generation in this context. However, I could be missing something.

@matthias-kleiner, do you have any idea what could be causing the issue Catalin reported?

Best regards,
Marian

miranov25 · 2024-12-12T08:19:15Z

Hello AliceO2Group/framework-admins

Could you give me your approval? Not sure who the admin is.
@ddobrigk, @ktf can you approve it for the framework-admins?

miranov25 · 2024-12-12T10:22:56Z

Hi Marian!

In case the TOF wasn't in data, could you please make sure the tpc-timeseries-workflow doesn't crash? Like

[INFO] Publishing M-shape correction map [ERROR] invalid workflow in o2-tpc-time-series-workflow: No matching output found for TOF/MTC_ITSTPCTRD/0 as requested by data processor "tpc-time-series". Candidates: [ERROR] -DPL/ENUM/2970000419 [ERROR] -CPV/CTFDATA/0 ...

Thanks, Catalin.

For the record, I had a discussion with @mkleiner on Mattermost regarding the problem in the time-series device reported above:

Matthias Kleiner
[9:48 AM](https://mattermost.web.cern.ch/alitpccomact/pl/7x8syt6kqir8bye51a3sydp6sh)

Yes, I think there is some protection missing, but it looks like all the tests passed now. So it should not be a stopper for your PR.

Marian I Ivanov
[10:45 AM](https://mattermost.web.cern.ch/alitpccomact/pl/ccm9e941xp8jzq8qxw9ifp8fqy)

Could you implement that protection? In my pull request, I don't encounter this issue since I already test for the presence of TOF and only fill in the information if it's available—I don't require it unconditionally.

Alternatively, I can handle this as well, including adding TOF information to the time series, which we need for other purposes anyway. I can take care of it after completing the time series CTF tagging to ensure it’s included in the standard production.

Additionally, I’ll coordinate with Francesco to include other useful information, such as two-cluster tagging, which we currently miss but would be very valuable for fake removal.

miranov25 requested review from a team as code owners December 9, 2024 21:41

alibuild mentioned this pull request Dec 9, 2024

Please consider the following formatting changes to #13787 miranov25/AliceO2#5

Closed

miranov25 and others added 6 commits December 10, 2024 11:51

O2-4592 - adding TOF dX and dZ to the track QA table with relative si…

3a2d17d

…gma precision nBins*2

O2-4592 - typo fix

4599ee7

O2-4592 - clang-format

452c2ea

AOD: TrackQA_002

1c82aa5

Signed-off-by: Felix Schlepper <[email protected]>

AOD: Activate TrackQA_002

94a6073

Signed-off-by: Felix Schlepper <[email protected]>

O2-4592 - patch fix from Felix (not full fix yet)

35e0f64

miranov25 force-pushed the dev branch from e654f33 to 35e0f64 Compare December 10, 2024 11:12

O2-4592 - typ fix - getting TOF source

b84615f

miranov25 mentioned this pull request Dec 10, 2024

[Common] TrackQAConverter002- for TOF modification AliceO2Group/O2Physics#8901

Merged

pzhristov approved these changes Dec 11, 2024

View reviewed changes

pzhristov merged commit 4e92d0d into AliceO2Group:dev Dec 12, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

O2-4592 - adding TOF dX and dZ to the track QA table with int8 precision #13787

O2-4592 - adding TOF dX and dZ to the track QA table with int8 precision #13787

miranov25 commented Dec 9, 2024

github-actions bot commented Dec 9, 2024

alibuild commented Dec 10, 2024

alibuild commented Dec 10, 2024

alibuild commented Dec 10, 2024

miranov25 commented Dec 10, 2024

miranov25 commented Dec 11, 2024

miranov25 commented Dec 11, 2024

shahor02 commented Dec 11, 2024

pzhristov commented Dec 11, 2024

catalinristea commented Dec 11, 2024

miranov25 commented Dec 11, 2024

miranov25 commented Dec 11, 2024

miranov25 commented Dec 12, 2024

miranov25 commented Dec 12, 2024

O2-4592 - adding TOF dX and dZ to the track QA table with int8 precision #13787

O2-4592 - adding TOF dX and dZ to the track QA table with int8 precision #13787

Conversation

miranov25 commented Dec 9, 2024