Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-48279: APDB simulation #263

Draft
wants to merge 25 commits into
base: main
Choose a base branch
from
Draft

DM-48279: APDB simulation #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
8c64e3a
Make testDataFrameIndex a staticmethod
isullivan Jan 8, 2025
9d1a69a
Make method static in diaPipe
isullivan Jan 4, 2025
fd06e6a
Initial working version of an APDB simulator
isullivan Jan 2, 2025
11b6a78
Add final table conversions
isullivan Jan 8, 2025
1d87fa5
Fix angle from area calculation
isullivan Jan 9, 2025
53d5dab
Fix reported number of real sources
isullivan Jan 9, 2025
b199f65
Write testApdb outputs
isullivan Jan 9, 2025
15a4c8d
Add apdbTest_marker output
isullivan Jan 9, 2025
8f0b089
Add option to limit the number of fake sources by max database capacity
isullivan Jan 9, 2025
8d2bede
Write to the APDB in chunks for large tables
isullivan Jan 10, 2025
b571ebe
Enlarge the region to load diaObjects from
isullivan Jan 13, 2025
b60e8a6
Restore forced source table formatting
isullivan Jan 13, 2025
9352f5c
Skip writing forced source table if empty
isullivan Jan 13, 2025
1cd40d2
ForcedSource 'band' must be a string
isullivan Jan 13, 2025
a43d838
band needs to be a single character for the APDB schema
isullivan Jan 14, 2025
4bd0d57
Only update nDiaSources once
isullivan Jan 14, 2025
ec9635b
Only write new and updated diaObjects to APDB
isullivan Jan 14, 2025
998e6fb
Add timing log messages
isullivan Jan 14, 2025
c1e142e
Temporarily delete failing tests not relevant to the APDB simulation
isullivan Jan 24, 2025
9c5c874
Change default false positive rate to 2-1
isullivan Jan 26, 2025
481b302
Fill diaSource table
isullivan Jan 26, 2025
e98d0ee
Fill diaObject table
isullivan Jan 26, 2025
53cad48
Ensure new random-filled dataframes have the right schema
isullivan Jan 26, 2025
2ca6906
Add correct format of band to DiaSource table
isullivan Jan 26, 2025
3ffec1a
Fix diaObject format for nearbyLowzGal column
isullivan Jan 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions python/lsst/ap/association/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,4 @@
from .version import *
from .mpSkyEphemerisQuery import *
from .ssSingleFrameAssociation import *
from .testApdb import *
6 changes: 4 additions & 2 deletions python/lsst/ap/association/diaPipe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -788,7 +788,8 @@ def writeToApdb(self, updatedDiaObjects, associatedDiaSources, diaForcedSources)
diaForcedSourceStore)
self.log.info("APDB updated.")

def testDataFrameIndex(self, df):
@staticmethod
def testDataFrameIndex(df):
"""Test the sorted DataFrame index for duplicates.

Wrapped as a separate function to allow for mocking of the this task
Expand Down Expand Up @@ -921,7 +922,8 @@ def mergeCatalogs(self, originalCatalog, newCatalog, catalogName):
mergedCatalog = pd.concat([originalCatalog], sort=True)
return mergedCatalog.loc[:, originalCatalog.columns]

def updateObjectTable(self, diaObjects, diaSources):
@staticmethod
def updateObjectTable(diaObjects, diaSources):
"""Update the diaObject table with the new diaSource records.

Parameters
Expand Down
Loading
Loading