TRACK: Sprint review 12.12.23

[cmarooney-stfc]

(a) Major items in progress or planned

ALEX:

• Filebacked: EPIC: modify all pixel operation which not change image shape to use apply method (THE filebacked epic) #1302 by 18.12.23 possibly earlier
• Tutorial validation: validate horace using Horace training course #1314 start 2.1.24 latest, end 8.1.24. Plus any tickets arising.
• Cylindrical/focussed projections (5d/5d) insert TICKETS 8.1.24 - 22.1.24
• Spherical projection with Tobyfit current resolution convolution and resolution plots are consistent with ortho_proj only #1040 23.1.24 to 23.2.24 (latest, could be 15.1.24) depending on scoping exercise TOBY ALEX (DUC?) TO DISCUSS 8.1.24

JACOB

• Reduce get_new_handle usage and disable shift_pixels #1420 CM TO REVIEW
• Toby/Multifit with file-backed untested #1423
• Add fitting file-backed tests for Toby/multifit #1424 CM TO REVIEW
• Tobyfit memory: Running out of memory on resolution calc for large npix #760 in progress, by 8.1.24
• Profiling: EPIC: Benchmarking PACE #676 BACKGROUND ONGOING 2.1.24-2.2.24 have fake data, need real data, want known time-hoggers JACOB TOBY TO DISCUSS
• Review documentation changes (how do we do this?) ask Excitations/David
• Radu Epic - TGP ADD TO JW tickets

CHRIS

Detector tickets

• detpar is not correctly wired to Experiment.detector_arrays #1432 and earlier (detector arrays are not restored to initial state after saving #1350, tobyfit_*_resconv_init to use IX_detector data from Experiment #1298, Settings for detector type parameters #1338, Debug test script for object_lookup for multiple detectors #1398)

Container tickets

• unique object container contains number of inefficiencies #1157
• bypass creation of duplicates when moving from unique containers to object lookup for tobyfit #1078
• Note on hashing issues, Toby to ticket

TOBY

• Talk to Duc about info in nxspe; DUC/TOBY Chase Mantid team detector info (script failing)
• Toby ticket hashing issues
• troubleshoot
• run challenging cases with H4

---

(b) Suggestions for next areas to do

• Parallelization TRACK: parallel issues #1253, includes:-
• Opening parallel workers fails to take place #1203 struggled to reproduce but needs to be confirmed & fixed, JW HIGH PRIORITY early Jan
• Performance - Identify large CPU time use in Tobyfit from hashifying, serializable and other sources, and implement quick-win solutions #1419 Tentative for JW 9.1.24
• Resolution function plot issues - TRACK: Resolution function plot issues #1408 Toby to deal with BACKGROUND, Jan 24
• ??? Refactor IX_mask and IX_map #922 Toby progressing; required for mgenie detector tools (???)Populate IX_detector_array with correct information from detector.dat #921

---

(c) Prioritization of new issues since last sprint review 28.11.23

All as above (a), no separate issues

---

(d) New issues prioritised up to 28.11.23

Process

No tickets

Filebacked

• External requests for additional filebacked combinations #1404
• remove internal write handle from sqw and pixels #1403
• rewrite join and parts of write_nsqw_to_sqw using page operations #1402

Performance

• efficiency of multifit_funceval #1372, apply_n algorithm  #1364 WAIT ON BENCHMARKING

Tobyfit/Detectors

• Resolution functions plots do not account for mosaic spread #1430
• Identify large CPU time use in Tobyfit from hashifying, serializable and other sources, and implement quick-win solutions #1419
• Debug test script for object_lookup for multiple detectors #1398

Parallel

• userpath in parallel system #1393 [POST-CONSTRUCTION, SCARF check of fixes to IDAaaS #12No tickets92, userpath provided to workers #1391]

Tests

No open tickets

Filebacked epic #1302

• remaining time estimated as 3 weeks = 19.12.23

---

(e) Previous issues still outstanding

Think these are largely dealt with above

Testing

• Alex validate horace using Horace training course #1314
• Alex mask_points have no unit tests and unclear location #1345
  Jacob created Document mask_points behaviour #1409 docs issue on mask points behaviour
  mask other issue is low priority - check number on Toby tickets

Fitting/filebacked/optimization
Both low priority to be done after validation

• Alex efficiency of multifit_funceval #1372 (both of these not in epic) - deferred as optimisation tasks - coupled to Jacob dqmat work? not scheduled (both)
• Alex apply_n algorithm  #1364

Filebacked
remaining filebacked tickets within 3 week estimate

• Alex filepath input versions of algorithms using PageOp #1352 part of EPIC: modify all pixel operation which not change image shape to use apply method (THE filebacked epic) #1302
• Alex modify replicate to support pageOp and sqw objects #1347 part of EPIC: modify all pixel operation which not change image shape to use apply method (THE filebacked epic) #1302
• Alex rewrite cat/split using page operations #1330 part of EPIC: modify all pixel operation which not change image shape to use apply method (THE filebacked epic) #1302 current task
• Alex save for filebacked sqw objects #1320 part of EPIC: modify all pixel operation which not change image shape to use apply method (THE filebacked epic) #1302
• mask_points have no unit tests and unclear location #1345 low priority
• New ticket to be added - remove old interface remove internal write handle from sqw and pixels #1403 ticket now added - part of filebacked 3 weeks

Documentation

• Jacob Dev. Documentation: Apply #1367 for apply and pageOp - scope may need clarifying given prioritization of pageOp
  awaiting completion of filebacked
• Filebacked workflow reliability #1341 Jacob to do as part of docs -
• Chris object_lookup.rand_in needs further documentation #1299 docs

Detectors/Tobyfit

• detector arrays are not restored to initial state after saving #1350
• Chris tobyfit_*_resconv_init to use IX_detector data from Experiment #1298 in progress, subject to Toby work above
• Chris Settings for detector type parameters #1338 awaits Duc population of nxspe
  Toby to progress this

Other

• Alex Inefficiencies in symmeterize_sqw related to filebacked operations #1354 performance issue; to be prioritized
• Alex mask_points have no unit tests and unclear location #1345 estimate 1d, after filebacked
• Chris Untested points in the code calling header_average #1025
• For Toby Incorrect initialisation of sqw object in resolution_plot #935 maybe fixed by test_tobyfit_resfun_2:test_projaxes_3 fails on aspect ratio #801 Toby to check

[cmarooney-stfc]

closed as superseded by #1470