TRACK: sprint review 6.2.24

[cmarooney-stfc]

Still uncovered

• alex: cylindrical projection - needs more input based on needs of spherical projection - scale: q is angstr only, need rlu
• detector information, waiting on Duc may not happen pre-construction
• pace-python (Duc) - finished 20.2 2 weeks to finish minor issues if available

prioritizing:

bugs (alex, end w/c 5/2/24) (jacob #1525 quick, #1524 lo-pr, #1519 longer, characterise for Duc, #1510 fixed) (joint #1505 to alex)
docs
spherical, cylindrical
(infrastructure working)

jacob profiling is at data assembly stage

profiling - slowdown due to IDAAAS issues, but same as 3.6.3
push on idaaas behaviour incl Toby
nb slurm setup may have multiple caches

use toby timing script

New issues since last sprint review 23.1.24

• by Jacob H4UE: Subtracting dnd from sqw results in sqw #1525
• by Jacob Writing multiple sqw from array seems to fail #1524
• by Jacob Parallel multifit failing for user #1519
• by Jacob Track Documentation feedback from Excitations users #1513
• by jacob EPIC: IDAaaS transform_sqw issue #1510
• by Alex H4UX: Image captions are invalid  #1511
• by Alex Describe spherical projection usage #1508
• by Alex H4UX: mf_leg_to_new is not very user friendly and does not converts binds properly #1505

NB H4UX epic is at

• EPIC: Horace 4 user experience #1464 which includes Alex' stuff above

New at last sprint review 23.1.24

Re: Toby's post-construction plans: action to review and record of storage at #1529. Ticket includes David Voneshen feedback noted previously in the last sprint review ticket.

From ALEX:

• H4UX: [a,b,c]=plot(1DObject) does not return correct c  #1501
• H4UX: cut without projection and with [] changes axis which should remain untouched #1499
• H4UX: update_file_format is not very user friendly #1498
• Implement Correct Magnetic Form_factor as PageOp and generic projection #1497
• H4UX: replicate_dnd should produce reasonable warning wrt replicate_sqw(___,'-set_pix') argument #1495
• H4UX: old head form generates very obscure error message on windows #1493
• H4UX: description of change_crystal_horace is outdated #1491
• H4UX: Helper class for cut, providing finer information on projection. #1487
• better data speed reporting while doing cuts #1483
• enable cut_dnd with projection and verify the robustness of the interpolation #1481
• 0-dimensions axes block is not correctly recovered from old structure  #1475
• H4UX: xycursor appears in inaccessable position and uncleas if works #1474
• H4UX: minor changes and additions to gen_sqw and cuts #1472

From JACOB

• Profiling Horace #1490
• Alternative distribution of SQw in parallel #1489
• Generate Data for benchmarking #1488 assigned to TOBY

From TOBY:

• String utilities in herbert_core need overhauling #1485
• Generate Data for benchmarking #1488 (see Jacob list above)

From last time:

(a) Major items in progress or planned

ALEX:

• Filebacked: EPIC: modify all pixel operation which not change image shape to use apply method (THE filebacked epic) #1302 by 18.12.23 possibly earlier - EPIC modified, most things finished/under review or minor side issues - separate tickets for remaining items filepath input versions of algorithms using PageOp #1352 Serializable equality method (eq) should be replaced by overloaded equal_to_tol #1147 (only if there is tim) OTHERWISE DONE
• Tutorial validation: validate horace using Horace training course #1314 start 2.1.24 latest, end 18.1.24. Plus any tickets arising. At 8.1.24 20% complete.
  (based on Alex local copy, check if current given multifit comments, check Sharepoint, circulate, cm to create sharepoint folder, by 10.1.24) DONE
• Spherical projection - done bar docs, Christian concerns - Christian has docs and example - ALEX to discuss with Duc on whether this meets user needs, soon (based on Christian's uncertainties, get representative dataset) - image of what Christian wants and what Christian got (+rb-no. if available)
• Fan IDAAAS problem - Alex to contact Fan re has she run script, what results
• Cylindrical projections (5d) after spherical and IDAAAS performance
• focussed projections (5d) insert TICKETS 8.1.24 - 22.1.24
• Horace help issue needs ticketing to continue having Fan work on this
• Spherical projection with Tobyfit (a) plot ellipsoid (b) resconv (needs 2 tickets) current resolution convolution and resolution plots are consistent with ortho_proj only #1040 23.1.24 to 23.2.24 (latest, could be 15.1.24) depending on scoping exercise TOBY - working, docs only, by 26.1.24 ALEX (DUC?) TO DISCUSS 8.1.24 do first to inform scope of next 2: post-construction

JACOB

• Tobyfit memory: Running out of memory on resolution calc for large npix #760 in progress, by 8.1.24 PR await submission, nb still not filebacked, may need pageop implementation for 4-way zip plus extra optimisations. Ensure this solves Duc's issue. PR awaits approval
• Profiling: EPIC: Benchmarking PACE #676 BACKGROUND ONGOING 2.1.24-2.2.24 have fake data, need real data, want known time-hoggers JACOB TOBY TO DISCUSS w/c 15.1.24 (profilng, parallelization) - JW plan emailed and in ticket - finish Feb w1
• Fix CI, need ticket, SL7 nodes retiring. (2d) Add additional Matlab versions to new R8 nodes.
• Jacob parallel_call failing on Jenkins for cut #1172 JENKINS ONLY -
• Review documentation changes (how do we do this?) still need to ask Excitations/David eg at Excitations awayday
• email from Boothroyd post-doc - need to investigate their problems
• pace-builder a/c needs 2fa - sort within week

JW notes: 17/1/24

• David harangued will schedule and arrange documentation review within Excitations. Plan akin to example script development assigning sections to individuals.
• Viviane's (Boothroyd's post-doc) IDAaaS workspace expired over New Year's break email sent requesting new one
• CI pipelines set up and "working" but lacking cppcheck, email sent to Alan Kyffin. Followup when time has passed to chase.

Update 18/1/24:

• Alan has installed more MATLAB versions and newest cppcheck

CHRIS

Detector tickets

• detpar is not correctly wired to Experiment.detector_arrays #1432 and earlier (detector arrays are not restored to initial state after saving #1350, tobyfit_*_resconv_init to use IX_detector data from Experiment #1298, Settings for detector type parameters #1338, Debug test script for object_lookup for multiple detectors #1398)

Container tickets

• unique object container contains number of inefficiencies #1157
• bypass creation of duplicates when moving from unique containers to object lookup for tobyfit #1078
• Note on hashing issues, Toby to ticket

TOBY

• Talk to Duc about info in nxspe; DUC/TOBY Chase Mantid team detector info (script failing): Duc says Merlin and Maps raw files have info - Toby to browse hdf5 output - need to get this into nxspe, raise Mantid epic - Toby and Jacob talk to Duc
• Toby ticket hashing issues, yes, Chris to add.
• troubleshoot, ongoing, background
• run challenging cases (generate them) with H4 e.g. cut-sym
• Radu Epic - TGP review and comment on JW tickets - this sprint

---

(b) Suggestions for next areas to do

• Parallelization TRACK: parallel issues #1253, includes:-
• Opening parallel workers fails to take place #1203 struggled to reproduce but needs to be confirmed & fixed, JW HIGH PRIORITY early Jan
• Performance - Identify large CPU time use in Tobyfit from hashifying, serializable and other sources, and implement quick-win solutions #1419 Tentative for JW 9.1.24
• Resolution function plot issues - TRACK: Resolution function plot issues #1408 Toby to deal with BACKGROUND, Jan 24
• ??? Refactor IX_mask and IX_map #922 Toby progressing; required for mgenie detector tools (???)Populate IX_detector_array with correct information from detector.dat #921

---

(c) New issues prioritised up to 28.11.23

Filebacked

• External requests for additional filebacked combinations #1404

Performance

• efficiency of multifit_funceval #1372, apply_n algorithm  #1364 WAIT ON BENCHMARKING

Tobyfit/Detectors

• Resolution functions plots do not account for mosaic spread #1430
• Identify large CPU time use in Tobyfit from hashifying, serializable and other sources, and implement quick-win solutions #1419
• Debug test script for object_lookup for multiple detectors #1398

Parallel

• userpath in parallel system #1393 [POST-CONSTRUCTION, SCARF check of fixes to IDAaaS #12No tickets92, userpath provided to workers #1391]

Tests

No open tickets

Filebacked epic #1302

• remaining time estimated as 3 weeks = 19.12.23

---

(e) Previous issues still outstanding

Think these are largely dealt with above
See previous sprint review 12.12.23 for details when the main issues are cleared

[cmarooney-stfc]

Closed as superseded by #1555