HISTORY

TAG: snpcacheload-6-0-24-1
DATE: 06/05/2024
STAFF: lec
fl2e/fl2-869/Reduce Max within distance of SNP-to-marker associations from 10kb to 2kb
change: MARKER_PAD  from 10000 to 2000, on this line:
MARKER_PAD      = 2000
#       modified:   snpmrkwithin.py

TAG: snpcacheload-6-0-22-6
TAG: snpcacheload-6-0-22-5
DATE: 11/07/2023
STAFF: lec
wts2-1323/fl3-619/other lib_py_postgres produts/db.setReturnAsSybase → db.setReturnAsMGI
remove: SNP_BACKUP_LOCALPATH from Configuration.default

TAG: snpcacheload-6-0-22-4
DATE: 04/18/2023
STAFF: sc
6-0-22-1 has current correct code - master is not correct, maybe
a push was missed when merging wts2-837 branch to trunk
This tag restores master to 6-0-22-1 code

TAG: snpcacheload-6-0-22-1
DATE: 10/03/2022
STAFF: sc
wts2-837 remap snps merge to trunk

TAG: snpcacheload-wts2-837-1
DATE: 09/08/2022
STAFF: sc
CHANGES: WTS2-837 remap snps

TAG: snpcacheload-6-0-16-4
TAG: snpcacheload-6-0-16-3
TAG: snpcacheload-6-0-16-2
TAG: snpcacheload-6-0-16-1
DATE: 09/01/2020
STAFF: lec
TR13204/Infrastructure Release

TAG: snpcacheload-6-0-7-1
DATE: 01/05/2017
STAFF: lec 
CHANGES:
1) TR12069/obsolete db.setAutoTranslate()
nothing to do here, except remove some db.() calls that were already commented out

TAG: snpcacheload-6-0-5-1
DATE: 09/01/2016
STAFF: lec 
CHANGES:
TR11083/Retire/Archive Nomenclature Table (NOM_Marker)
 _Marker_Status_key != 2  changed to "= 1"

TAG: snpcacheload-6-0-4-1
DATE: 06/16/2016
STAFF: lec
CHANGES:
1) convert to git
                          calls snpmarker.sh directly
TAG: snpcacheload-6-0-3-2
DATE: 03/21/2016
STAFF: dbm
CHANGES:
1) snpmarker_wrapper.sh - Removed because loadadmin/bin/genReleaseData.csh now
                          calls snpmarker.sh directly
TAG: snpcacheload-6-0-3-1
DATE: 02/03/2016
STAFF: sc
CHANGES: Merge tr12142 to trunk

TAG: snpcacheload-tr12142-6
DATE: 12/21/2015
STAFF: sc
CHANGES: 
1) snpmarker.py - exclude withdrawn markers 
2) snpmrkwithin.py - exclude withdrawn markers, cytogenetic markers and heritible phenotypic feature type.

TAG: snpcacheload-tr12142-5
DATE: 11/24/2015
STAFF: sc
CHANGES: US106
1) load new SNP_ConsensusSnp_Marker._Transcript_Protein_key
   from new table SNP_Transcript_Marker (script to load this table added to 
        dbsnpload)
2) Pull out all changes from another US which created and loaded 
    new table SNP_RefSeq_Accession table

TAG: snpcacheload-tr12142-4
DATE: 11/19/2015
STAFF: sc
CHANGES: US39
1) snpmarker.sh - uncomment call to snpmrklocus.py

TAG: snpcacheload-tr12142-3
DATE: 11/19/2015
STAFF: sc
CHANGES: US39
1) snpmrklocus.py - update locus-region calculation store as distance_direction
    rather than updating fxn class 
2) remove SNP_Accession_drop.object

TAG: snpcacheload-tr12142-2
DATE: 11/15/2015
STAFF: sc
CHANGES: Using new SNP_RefSeq_Accession
1) Configuration.default - update: ACC_TABLE=SNP_RefSeq_Accession
2) snpmarker.sh 
* Change references to SNP_Accession to SNP_RefSeq_Accession
* Add dropping of keys as well as indexes
* Stop using internal SNP_Accession_drop.object as we are now using and truncating
    SNP_RefSeq_Accession
3) snpmarker.py
* remove deleteAccessions  and getMaxAccessionKey methods
* update all Accession comments as appropriate
* bug fix to calculation if distance for markers on the '-' strand
* addition of 'proximal' or 'distal' rather than 'not applicable' for markers with null strand

TAG: snpcacheload-tr12142-1
DATE: 11/09/2015
STAFF: sc
CHANGES: 
1) snpmarker.py:
* removed db.setReturnAsSybase(False) so that queries return in sybase db.sql
   format
* removed db.setTrace(True), password = db.get_sqlPassword(), db.set_sqlLogin(user, password, server, database)
* created a second temp table to
    * create a lookup of csKeys to process
    * query in batches of csKeys
* Added code to process in batches
* factored out a new method writeBCP for batch processing
* Added two new columns to bcp file, distance_from and distance_direction
2) snpmarker.sh - commented out call to snpmrklocus.py - we are no longer
    updating locus-region function class, not yet removing the script from cvs
3) snpmrkwithin.py
* added calculation of exact distance from marker to populate distance_from
* populate distance direction (upstream, downstream, or not applicable)
* removed assigning of 'within * KB' fxn class, only assign fxn classes;
** 'within coordinates of'
** 'within distance of'
* updated to 10KB from 1000KB 
i* removed db.setReturnAsSybase(False) so that queries return in sybase db.sql
   format
* removed db.setTrace(True), password = db.get_sqlPassword(), db.set_sqlLogin(user, password, server, database)

TAG: snpcacheload-tr12142-BP
DATE: 11/03/2015
STAFF: sc
CHANGES: Project Florida branch

TAG: snpcacheload-6-0-0-1
DATE: 07/22/2015
STAFF: lec 
CHANGES:
1) TR11750/postgres

TAG: snpcacheload-tr11750-BP
DATE: 06/15/2015
STAFF: dbm
CHANGES: Branch point tag

TAG: snpcacheload-5-2-1-2
DATE: 01/14/2015
STAFF: lec
CHANGES:
1) SNP_Accession_drop.object : add 'if exists'

TAG: snpcacheload-5-2-1-1
DATE: 12/17/2014
STAFF: kstone
CHANGES:
1) modified bcpin.csh calls to comply with other postgres conversion changes

TAG: snpcacheload-5-1-5-1
DATE: 12/02/2013
STAFF: dbm
CHANGES:
1) TR11475: shell "-x" cleanup

TAG: snpcacheload-5-1-4-5
DATE: 6/27/2013
STAFF: dbm
CHANGES:
1) snpmarker_wrapper.sh: logging updates

TAG: snpcacheload-5-1-4-4
TAG: snpcacheload-5-1-4-3
DATE: 6/26/2013
STAFF: dbm
CHANGES:
1) snpmarker_wrapper.sh: bug fix sourcing config file

TAG: snpcacheload-5-1-4-2
TAG: snpcacheload-5-1-4-1
DATE: 05/09/2013, 06/13/2013
STAFF: lec
CHANGES:
1) merged branch 'tr11248' to trunk (GXD/Cre/SNP)

TAG: snpcacheload-5-1-1-2
DATE: 12/03/2012
STAFF: sc
CHANGES:
1) IN_SYNC=no

TAG: snpcacheload-5-1-1-1
DATE: 11/02/2012
STAFF: dbm
CHANGES:
1) Config changes for TR11163

TAG: snpcacheload-postgres-5-1-4-1
DATE: 03/06/2013
STAFF: lec
CHANGES:
1) TR11248/TR10788
postgres conversion

TAG: snpcacheload-postgres-3
TAG: snpcacheload-postgres-2
TAG: snpcacheload-postgres-1
DATE: 05/21/2012
STAFF: sc
CHANGES:
1) updated to postgres

TAG: snpcacheload-postgres-BP
DATE: 05/18/2012
STAFF: sc
CHANGES: TR10961 - postgres conversion

TAG: snpcacheload-4-4-2-3
DATE: 11/29/2011
STAFF: sc
CHANGES: TR10916
1) snpmarker.py - delete accessions before getting max(_Accession_key)

TAG: snpcacheload-4-4-2-2
TAG: snpcacheload-4-4-2-1
DATE: 09/01/2011
STAFF: lec
CHANGES:
1) TR10805/snpmarker.sh/MRK_Location_Cache
2) snpmrklocus.py/snpmrkwithin.py; added _Organism_key = 1

TAG: snpcacheload-4-3-5-1
DATE: 06/11/2010
STAFF: dbm
CHANGES:
1) Configuration.default: Moving backend databases to hobbiton. Changed
   comment on SNP_BACKUP_LOCALPATH to reflect new path.

TAG: snpcacheload-4-3-0-2
DATE: 09/08/2009
STAFF: dbm
CHANGES:
1) snpmarker_weekly.sh: Set SCRIPT_NAME variable for process control so it
   doesn't use the setting from the calling script (sundaytasks.csh)

TAG: snpcacheload-4-3-0-1
DATE: 08/31/2009
STAFF: dbm
CHANGES:
1) snpmarker_weekly.sh: Set process control flag after the SNP DB load
   is done (TR 9779)

TAG: snpcacheload-4-1-0-2
DATE: 06/05/2008
STAFF: sc
CHANGES: B37, dbSNP Build 128
1) removed rowcount limitation which was set in deleteAccessions
   (Ugh - pretty bad bug)

TAG: snpcacheload-4-1-0-1
DATE: 06/04/2008
STAFF: sc
CHANGES: B37, dbSNP Build 128
1) added indexes to #snpmkr1

TAG: snpcacheload-4-0-2-1
DATE: 05/22/2008
STAFF: sc
CHANGES: B37, dbSNP Build 128
1) dump transaction then do deletes of SNP_Accession in loop

TAG: snpcacheload-3-5-4-2
TAG: snpcacheload-3-5-4-1
DATE: 10/29/2006
STAFF: sc
CHANGES: 
1) snpmarker_weekly.sh - added mailx of logs
2) snpmarker.sh - fixed so creates new log file at start (was always
   appending)

TAG: snpcacheload-3-5-1-6
TAG: snpcacheload-3-5-1-5
DATE: 
STAFF: sc
RELEASE:
CHANGES: load doc changes
1) removed all *.csh scripts (didn't happen in 3-5-1-3 tag for some reason)
2) Configuration.default - added log file configuration
3) *.sh - use new logging configuration andchange some "| tee -a" to ">> file 2>&1"
   to clean up the stdouterr.log

TAG: snpcacheload-3-5-1-4
DATE: 11/06/2006
STAFF: sc
RELEASE: 
CHANGES: Configuration.default - removed DLAJOBSTREAMFUNC as it is defined
         in mgiconfig

TAG: snpcacheload-3-5-1-3
TAG: snpcacheload-3-5-1-2
DATE: 10/06/2006
STAFF: lec
RELEASE: MGI3.51, build 36
CHANGES: changed CACHEINSTALLDIR to SNPCACHELOAD for TR7894
10/12 - removed all *.csh scripts

TAG: snpcacheload-3-5-1-1
DATE: 09/06/2006
STAFF: sc
RELEASE: MGI3.51, build 36
CHANGES: merge build36 branch to trunK (note this is a second merge
         of this branch to the trunk

TAG: snpcacheload-mgiconfig-7
DATE: 09/06/2006
STAFF: sc
RELEASE: MGI3.51, build 36
CHANGES:
1) added system_docs

TAG: snpcacheload-mgiconfig-6
TAG: snpcacheload-mgiconfig-5
DATE: 08/24/2006
STAFF: sc
RELEASE: MGI3.51, build 36
CHANGES:
1) corrected mgidbutilities config var name
2) added sys.stdout.flush

TAG: snpcacheload-mgiconfig-4
DATE: 08/10/2006
STAFF: sc
RELEASE: MGI3.51, build 36
CHANGES:
1) remove snplocation.py
2) Configuration - removed SNP_BACKUP_REMOTEPATH to mgiconfig
                 - updated SNP_BACKUP to SNP_BACKUP_LOCALPATH
		 - added MRKR_QTLTYPE_KEY
3) snpmarker_weekly.sh - updated to use SNP_BACKUP_REMOTEPATH and 
                 SNP_BACKUP_LOCALPATH
4) snpmrkwithin.py - updated to not load distance relationships, only
                   - update locus_region and determine 'within coordinates of'

TAG: snpcacheload-mgiconfig-3
DATE: 08/01/2006
STAFF: sc
RELEASE: MGI3.51, build 36
CHANGES:
1) converted to bourne shell

TAG: snpcacheload-3-5-0-4
DATE: 07/01/2006
STAFF: lec
CHANGES:
1) mgiconfig merge

TAG: snpcacheload-mgiconfig-2
TAG: snpcacheload-mgiconfig-1
DATE: 06/30/2006
STAFF: lec
CHANGES:
1) mgiconfig: all products updated for new mgiconfig env variables
(forgot to make branch point tag before creating branch "mgiconfig")

TAG: snpcacheload-3-5-0-3
TAG: snpcacheload-3-5-0-2
DATE: 05/31/2006
STAFF: sc
RELEASE: MGI3.5
CHANGES:
1) Configuration.defaults - updated to reflect we are loading a dump to
  production snp database rather than bcp in
2) snpmarker_weekly.csh - updated to reflect we are loading a dump to
  production snp database rather than bcp in

TAG: snpcacheload-3-5-0-1
DATE: 05/19/2006
STAFF: sc
RELEASE: MGI3.5
CHANGES:
1) snpmrkwithin.py - new version that joins between markers
   snps and the exclude list in memory.
2) snpmrkwithin.py -  now handles markers with null strand
   in particular the MIT marker which have coordinates loaded
   by unistsload, part of the MGI3.5 release

TAG: snpcacheload-3-4-4-3
DATE: 04/27/2006
STAFF: sc
RELEASE: MGI3.44
CHANGES:
1) snpmarker_weekly.csh added more logging
2) Configuration.defaults - removed MAX_MARKER_QUERY_BATCH

TAG: snpcacheload-3-4-4-2
DATE: 04/13/2006
STAFF: sc
RELEASE: MGI3.44
CHANGES:
1) added snpmarker_weekly.csh
2) Configuration.default - added PRODSNP_DBSCHEMADIR

TAG: snpcacheload-3-4-4-1
DATE: 04/06/2006
STAFF: sc
RELEASE: MGI3.44
CHANGES: merge tr7203 branch to trunk

TAG: snpcacheload-3-4-3-2
TAG: snpcacheload-3-4-3-1
DATE: 02/10/2006
STAFF: sc
RELEASE: 3.43
CHANGES: merge tr7392 branch to trunk

TAG: snpcacheload-tr7392-2
DATE: 2/07/2006
STAFF: sc
RELEASE: 3.43 alpha
CHANGES: snpmrkwithin.py - added query time reporting

TAG: snpcacheload-tr7392-1
TAG: snpcacheload-tr7392-BP
DATE: 1/19/2006
STAFF: sc
RELEASE:
CHANGES: Branch point tag for SNP Shoe horn tr7392
     branched off of tag snpcacheload-tr7203-1

TAG: snpcacheload-tr7203-1
DATE: 1/19/2006
STAFF: sc
RELEASE:
CHANGES:
1) Configuration.defaults - added MAX_BCP_LINES
  and MAX_QUERY_BATCH
2) snpcoord.py, snpmarker.py, snpmrkwithin.py - 
   removed created/modified by/date from bcp file
3) snpmarker.csh - processes multiple snpmrkwithin.py bcp files
4) snpmrkwithin.py - now creates multiple  bcp files based on 
   a configurable number of rows. This is because bcp executed
   from a shell script doesn't recognize files > 2Gb
   - now processes in batches (configurable size) of 
   _ConsensusSnp_keys's instead of by chromosome, this is
   because of python memory errors when query is too large
   

TAG: snpcacheload-tr7203-3
DATE: 04/03/2006
STAFF: sc
RELEASE: 3.44 - Dev Build 2
CHANGES:
1) Configuration.default - 
   a) fixed typo in MRKLOC_CACHEFILE value' 
   b) added SNP_MRK_WITHIN_FILE, ACC_FILE
   c) set PRODSNP_DBSERVER to PROD1_MGI
2) updated scripts to use the above configuration

TAG: snpcacheload-tr7203-2
DATE: 3/27/2006
STAFF: sc
RELEASE: 3.44
CHANGES: tr7203
1) copied snpmrkwithin.py from trunk to get query time reporting that
   was added after tr7203 branch was created
2) snpmrklocation.py - load snp..MRK_Location_Cache
3) snpmarker.csh - updated to load to SNP database, and run snpmrklocation.py
4) snpmarker.py - updated to talk to SNP database, removed
   created/modified, updated _Feature_key to _Coord_Cache_key, 
   get mgd keys from configuration (ldb, mgitype)
5) Configuration.default - added ldb, mgityp keys, added
   SNP_DBSCHEMA, MRKLOC_CACHETABLE, MRKLOC_CACHEFILE
6) snpmrkwithin.py - updated to use binary search to find markers by
   chromosome and only process a configurable set of consensus snps
   on that chromosome at a time.
7) snpmrklocus.py - updated to talk to snp database, new locus-region
   upstream/downstream algorithm
8) Install - added creation of output directories

TAG: snpcacheload-tr7392-BP
DATE: 1/19/2006
STAFF: sc
RELEASE:
CHANGES: Branch point tag for SNP Shoe horn tr7392
     branched off of tag snpcacheload-tr7203-1

TAG: snpcacheload-tr7203-1
DATE: 1/19/2006
STAFF: sc
RELEASE:
CHANGES:
1) Configuration.defaults - added MAX_BCP_LINES
  and MAX_QUERY_BATCH
2) snpcoord.py, snpmarker.py, snpmrkwithin.py - 
   removed created/modified by/date from bcp file
3) snpmarker.csh - processes multiple snpmrkwithin.py bcp files
4) snpmrkwithin.py - now creates multiple  bcp files based on 
   a configurable number of rows. This is because bcp executed
   from a shell script doesn't recognize files > 2Gb
   - now processes in batches (configurable size) of 
   _ConsensusSnp_keys's instead of by chromosome, this is
   because of python memory errors when query is too large
   
TAG: snpcacheload-tr7203-BP
DATE: 1/10/2006
STAFF: sc
RELEASE: 
CHANGES: branch point tag for tr7203 dbsnp build 125
         changes 

TAG: snpcacheload-3-4-0-4
DATE: 10/17/2005
STAFF: sc
RELEASE: dev
CHANGES:
1) added accessionlib import

TAG: snpcacheload-3-4-0-3
DATE: 10/15/2005
STAFF: sc
RELEASE: dev
CHANGES:
1) snpmarker.py - added numericpart and prefixpart to ACC_Accession

TAG: snpcacheload-3-4-0-2
DATE: 10/11/2005
STAFF: sc
RELEASE: dev
CHANGES:
1) snpmarker.py - updated vocab key and now do substring(mgd..ACC_Accession.accid, 3, 10)
   before joining with radar..MGI_SNP_MArker.accid because we prepended with 'rs' and 'ss'

TAG: snpcacheload-3-4-0-1
DATE: 10/11/2005
STAFF: dbm
RELEASE: dev
CHANGES:
 1) Fixed upper/lower case problem with locus region constants in
    snpmrklocus.py

TAG: snpcacheload-1-0-2
DATE: 09/29/2005
STAFF: dbm
RELEASE: dev
CHANGES:
 1) Added snpmrklocus.py and snpmrkwithin.py to handle the
    association of RefSNPs to MGI markers

DATE: 09/28/2005
STAFF: sc
RELEASE: dev
CHANGES:
1) snpmarker.py - added union of VOC_term and
   MGI_Translation to resolve all fxn classes

DATE: 09/25/2005
STAFF: sc
RELEASE: dev
CHANGES:
1) snpcoord.py - updated alleleSummary column name
   wrapped db.sql in try/except
2) snpmarker.py - wrapped db.sql in try/except

DATE: 09/23/2005
STAFF: sc
RELEASE: dev
CHANGES:
1) snpcoord.py - updated keys since OMIM release, 
     updated to handle new inMultiCoord attribute
2) snpmarker.py - updated keys since OMIM release

TAG: snpcacheload-1-0-1
DATE: 09/01/2005
STAFF: sc
RELEASE: dev
CHANGES: added commenting and pulled environ var
   creation out of wrappers and into Configuration 

TAG: snpcacheload-1-0-0
DATE: 08/22/2005
STAFF: sc
RELEASE: dev
CHANGES: initial dev tag - still to do: 
 1) delete accessions