Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-43077: Convert all tasks to use CalibrateImageTask outputs #157

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

DM-43077: Convert all tasks to use CalibrateImageTask outputs #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
52dbbea
Switch to calibrateImage
parejkoj Jul 30, 2024
050a011
Add reprocessVisitImage
parejkoj Aug 9, 2024
e89ae97
Remove unnecessary useCiLimits
parejkoj Sep 11, 2024
a3c0d6a
Refactor ci_imsim to use reprocessVisitImage
parejkoj Sep 12, 2024
f42bd38
Drop analyzeCalibrateMetadata from HSC pipelines.
TallJimbo Oct 9, 2024
cd196af
Use 'pvi' instead of 'initial_pvi' in DRP diffim tasks.
TallJimbo Oct 9, 2024
40ae6fc
Move imSim updates to DRP-minimal-calibration
yalsayyad Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions bps/clustering/DRP-recalibrated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
visit_detector:
pipetasks: isr,inject_exposure,characterizeImage,calibrate,inject_visit,writePreSourceTable,transformPreSourceTable
pipetasks: isr,inject_exposure,calibrateImage,inject_visit,transformPreSourceTable
dimensions: visit,detector
equalDimensions: visit:exposure

Expand All @@ -37,7 +37,7 @@ cluster:
dimensions: tract,patch,band

finalizeSourceTable:
pipetasks: finalizeCharacterization,updateVisitSummary,writeRecalibratedSourceTable,transformSourceTable,consolidateSourceTable
pipetasks: finalizeCharacterization,updateVisitSummary,reprocessVisitImage,transformSourceTable,consolidateSourceTable
dimensions: visit
equalDimensions: visit:exposure

Expand Down
2 changes: 1 addition & 1 deletion bps/clustering/LSSTCam-imSim/DRP-DC2-clustering.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
visit_detector:
pipetasks: isr,inject_exposure,characterizeImage,calibrate,inject_visit,writeSourceTable,transformSourceTable
pipetasks: isr,inject_exposure,calibrateImage,inject_visit,transformSourceTable
dimensions: visit,detector
equalDimensions: visit:exposure

Expand Down
8 changes: 3 additions & 5 deletions pipelines/HSC/DRP-Prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@ subsets:
step1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
- analyzeAmpOffsetMetadata
description: |
Expand Down Expand Up @@ -74,15 +72,15 @@ subsets:
step2d:
subset:
- finalizeCharacterization
- writeRecalibratedSourceTable
- transformSourceTable
- consolidateSourceTable
- updateVisitSummary
- reprocessVisitImage
description: |
Visit-level tasks.
Allowed data query constraints: visit

writeRecalibratedSourceTable, transformSourceTable run per-detector
reprocessVisitImage, transformSourceTable run per-detector
consolidateSourceTable produces one data product per visit.
finalizeCharacterization will eventually model full focal plane PSFs.
step2e:
Expand Down
9 changes: 3 additions & 6 deletions pipelines/HSC/DRP-RC2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ tasks:
isolatedStarAssociation:
class: lsst.pipe.tasks.isolatedStarAssociation.IsolatedStarAssociationTask
config:
band_order: ['i', 'r', 'g', 'z', 'y', 'N921']
band_order: ["i", "r", "g", "z", "y", "N921"]
matchedVisitCoreWholeSkyPlot:
class: lsst.analysis.tools.tasks.WholeSkyAnalysisTask
config:
Expand All @@ -85,12 +85,9 @@ subsets:
step1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
- analyzeAmpOffsetMetadata
- analyzeCalibrateMetadata
description: |
Per-detector tasks that can be run together to start the DRP pipeline.

Expand Down Expand Up @@ -139,7 +136,7 @@ subsets:
- fgcmFitCycle
- fgcmOutputProducts
- updateVisitSummary
- writeRecalibratedSourceTable
- reprocessVisitImage
- transformSourceTable
- consolidateSourceTable
- catalogMatchVisit
Expand Down
15 changes: 5 additions & 10 deletions pipelines/HSC/DRP-RC2_subset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,8 +218,7 @@ subsets:
simpleSingleFrame:
subset:
- isr
- characterizeImage
- calibrate
- calibrateImage
description: |
A simplified single frame pipeline to make the quantum graph more readable.
multiVisit:
Expand All @@ -231,9 +230,7 @@ subsets:
singleFrame:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
- consolidatePreSourceTable
- consolidateVisitSummary
Expand All @@ -253,7 +250,7 @@ subsets:
- updateVisitSummary
- makeCcdVisitTable
- makeVisitTable
- writeRecalibratedSourceTable
- reprocessVisitImage
- transformSourceTable
- consolidateSourceTable
coadd_measurement:
Expand All @@ -273,9 +270,7 @@ subsets:
nightlyStep1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
description: |
Per-detector tasks that can be run together to start the DRP pipeline.
Expand Down Expand Up @@ -324,7 +319,7 @@ subsets:
- makeCcdVisitTable
- finalizeCharacterization
- updateVisitSummary
- writeRecalibratedSourceTable
- reprocessVisitImage
- transformSourceTable
- consolidateSourceTable
- nsrcMeasVisit
Expand Down
11 changes: 0 additions & 11 deletions pipelines/HSC/DRP-ci_hsc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,6 @@ tasks:
multibandDeblend.useCiLimits: true
# Do not run the deblender on isolated sources
multibandDeblend.processSingles: false
calibrate:
class: lsst.pipe.tasks.calibrate.CalibrateTask
# Significantly limit the number of sources to deblend, which saves time
# in both deblending and measurement
config:
# Only run the deblender on a small subset of blended parents
deblend.useCiLimits: true
writeRecalibratedSourceTable:
class: lsst.pipe.tasks.postprocess.WriteRecalibratedSourceTableTask
config:
connections.outputCatalog: source
isolatedStarAssociation:
class: lsst.pipe.tasks.isolatedStarAssociation.IsolatedStarAssociationTask
config:
Expand Down
11 changes: 7 additions & 4 deletions pipelines/LATISS/DRP.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ tasks:
photo_calib_provider: "global"
background_provider: "input_summary"

reprocessVisitImage:
class: lsst.drp.tasks.reprocess_visit_image.ReprocessVisitImageTask
config:
do_use_sky_corr: false

selectDeepCoaddVisits:
class: lsst.pipe.tasks.selectImages.BestSeeingSelectVisitsTask
config:
Expand Down Expand Up @@ -111,9 +116,7 @@ subsets:
step1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
description: |
Per-detector tasks that can be run together to start the DRP pipeline.
Expand Down Expand Up @@ -147,7 +150,7 @@ subsets:
- fgcmBuildFromIsolatedStars
- fgcmFitCycle
- fgcmOutputProducts
- writeRecalibratedSourceTable
- reprocessVisitImage
- transformSourceTable
- consolidateSourceTable
- updateVisitSummary
Expand Down
9 changes: 4 additions & 5 deletions pipelines/LSSTCam-imSim/DRP-ci_imsim.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,11 @@ tasks:
multibandDeblend.useCiLimits: true
# Do not run the deblender on isolated sources
multibandDeblend.processSingles: false
calibrate:
class: lsst.pipe.tasks.calibrate.CalibrateTask
# Significantly limit the number of sources to deblend, which saves time
# in both deblending and measurement
reprocessVisitImage:
class: lsst.drp.tasks.reprocess_visit_image.ReprocessVisitImageTask
config:
# Only run the deblender on a small subset of blended parents
# Significantly limit the number of sources to deblend, which saves time
# in both deblending and measurement.
deblend.useCiLimits: true
catalogMatchVisit:
class: lsst.analysis.tools.tasks.astrometricCatalogMatch.AstrometricCatalogMatchVisitTask
Expand Down
8 changes: 3 additions & 5 deletions pipelines/LSSTCam/DRP.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,7 @@ subsets:
step1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
description: |
Per-detector tasks that can be run together to start the DRP pipeline.
Expand Down Expand Up @@ -52,15 +50,15 @@ subsets:
step2d:
subset:
- finalizeCharacterization
- writeRecalibratedSourceTable
- transformSourceTable
- consolidateSourceTable
- updateVisitSummary
- reprocessVisitImage
description: |
Visit-level tasks.
Allowed data query constraints: visit

writeRecalibratedSourceTable, transformSourceTable run per-detector
transformSourceTable runs per-detector
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why reprocessVisitImage is not included here like many others below?

consolidateSourceTable produces one data product per visit.
finalizeCharacterization will eventually model full focal plane PSFs.
step2e:
Expand Down
8 changes: 3 additions & 5 deletions pipelines/LSSTCam/nightly-validation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,7 @@ subsets:
step1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
description: |
Detector level tasks. TO DO: requests include synthetic source injection,
Expand Down Expand Up @@ -79,15 +77,15 @@ subsets:
step2d:
subset:
- finalizeCharacterization
- writeRecalibratedSourceTable
- transformSourceTable
- consolidateSourceTable
- updateVisitSummary
- reprocessVisitImage
description: |
Visit-level tasks.
Allowed data query constraints: visit

writeRecalibratedSourceTable, transformSourceTable run per-detector
transformSourceTable runs per-detector
consolidateSourceTable produces one data product per visit.
finalizeCharacterization will eventually model full focal plane PSFs.
step2e:
Expand Down
36 changes: 2 additions & 34 deletions pipelines/LSSTCam/quickLook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,41 +22,9 @@
doDefect: false
doDark: false
doFlat: false

# TO DO: Replace with calibrateImage in a way that we can use the
# preSources downstream.
characterizeImage:
class: lsst.pipe.tasks.characterizeImage.CharacterizeImageTask
config:
doApCorr: false
doDeblend: false
calibrate:
class: lsst.pipe.tasks.calibrate.CalibrateTask
config:
python: |
config.measurement.plugins.names = ['base_CircularApertureFlux',
'base_PsfFlux',
'base_LocalBackground',
'base_SdssCentroid',
'base_SdssShape',
'base_Variance',
'base_Jacobian',
'base_PixelFlags',
'base_GaussianFlux',
'base_SkyCoord',
'base_FPPosition',
'base_ClassificationSizeExtendedness',
'base_CompensatedTophatFlux',
]
config.measurement.slots.shape = "base_SdssShape"
config.measurement.slots.psfShape = "base_SdssShape_psf"
config.measurement.plugins['base_PixelFlags'].masksFpAnywhere.remove('STREAK')
config.measurement.plugins['base_PixelFlags'].masksFpCenter.remove('STREAK')
config.doApCorr = False
config.doDeblend = False
config.astrometry.sourceSelector["science"].doRequirePrimary = False
config.astrometry.sourceSelector["science"].doIsolated = False
calibrateImage:
class: lsst.pipe.tasks.calibrate.CalibrateImageTask
# TO DO: DM-41189 Get pipeline that will attach an atool here from lauren

Check warning on line 27 in pipelines/LSSTCam/quickLook.yaml

View workflow job for this annotation

GitHub Actions / call-workflow / yamllint 

27:7 [comments-indentation] comment not indented like content
subsets:
step1:
subset:
Expand Down
7 changes: 2 additions & 5 deletions pipelines/LSSTComCam/DRP.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,7 @@ subsets:
step1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
description: |
Per-detector tasks that can be run together to start the DRP pipeline.
Expand Down Expand Up @@ -52,15 +50,14 @@ subsets:
step2d:
subset:
- finalizeCharacterization
- writeRecalibratedSourceTable
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is reprocessVisitImage not included here on purpose?

- transformSourceTable
- consolidateSourceTable
- updateVisitSummary
description: |
Visit-level tasks.
Allowed data query constraints: visit

writeRecalibratedSourceTable, transformSourceTable run per-detector
transformSourceTable run per-detector
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
transformSourceTable run per-detector
transformSourceTable runs per-detector

consolidateSourceTable produces one data product per visit.
finalizeCharacterization will eventually model full focal plane PSFs.
step2e:
Expand Down
8 changes: 3 additions & 5 deletions pipelines/LSSTComCam/nightly-validation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,7 @@ subsets:
step1:
subset:
- isr
- characterizeImage
- calibrate
- writePreSourceTable
- calibrateImage
- transformPreSourceTable
description: |
Detector level tasks. TO DO: requests include synthetic source injection,
Expand Down Expand Up @@ -79,15 +77,15 @@ subsets:
step2d:
subset:
- finalizeCharacterization
- writeRecalibratedSourceTable
- transformSourceTable
- consolidateSourceTable
- updateVisitSummary
- reprocessVisitImage
description: |
Visit-level tasks.
Allowed data query constraints: visit

writeRecalibratedSourceTable, transformSourceTable run per-detector
transformSourceTable run per-detector
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
transformSourceTable run per-detector
transformSourceTable runs per-detector

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or include reprocessVisitImage.

consolidateSourceTable produces one data product per visit.
finalizeCharacterization will eventually model full focal plane PSFs.
step2e:
Expand Down
Loading
Loading