Version 1.44.0

Version 1.44.0 introduces the following changes:

• New database indexes have been added to improve performance (thanks @braffes).
• Some rarely or never used indexes have been dropped.
• The option to set an allele designation with a status of 'ignore' has been removed. This was not really used but added an unnecessary filtering contraint on multiple queries.
• The number of isolates that match the current isolate at different LIN code thresholds are now displayed on an isolate record page (if LIN code has been assigned). The number is hyperlinked and takes you to a query that returns the matching isolates.

An administrator can also decide whether or not to include classification scheme or LIN code scheme matches on the isolate information page. These are enabled by default, if defined, but can be disabled with the following options in config.xml:

show_lincode_matches="no"
show_classification_schemes="no"

In addition, the number of LIN code thresholds to display can be set with the following option (replace the number with the required number of thresholds):

show_lincode_thresholds="5"

Full Changelog: v_1.43.3...v_1.44.0

Version 1.43.3

Version 1.43.3 brings the following changes:

• An optional message can be added on the allele and profile download pages to indicate that users should use the API for scripted data downloads. It is much easier to rate-limit the API than the web interface and it generally uses less resources to serve data. This message can be enabled by setting the URL for the API in bigsdb.conf, e.g. rest_url=https://rest.pubmlst.org

• The JQuery.UI.multiselect widget is now used for dropdown boxes with large numbers of options, e.g. locus lists on the isolate query page, or where multiple options can be selected, such as the the 'includes' field list on various plugins. Where appropriate, the list is also searchable to make finding and selecting values quicker and easier.

• Javascript performance has been improved for background loading of isolate query form elements.
• Limits can now be set for the number of offline jobs that any single user can run for individual plugins. See the updated format of the job_limits.conf file found in the conf directory. If the limit is reached then further jobs for that particular plugin and user will be queued.

Full Changelog: v_1.43.2...v_1.43.3

Version 1.43.2

Version 1.43.2 brings the following changes:

• Additional logging information added. This includes the particular locus or scheme selected in data downloads, or the isolate record being shown.
• The cgST with fewest missing loci is now highlighted on the isolate info page when the isolate has multiple cgSTs.
• The query dashboard is now shown for bookmarked and publication queries.
• There is now an option to force users to log-in if they want to run offline jobs (see jobs_require_login=1 in bigsdb.conf).
• There is now an option to automatically check the box to tag incomplete loci at specified identity thresholds (rather than just at 100% identity)
• 'Ambiguous read' flags are now automatically set on tags if sequence contains non-A,G,T,C characters.
• JQuery.UI.multiselect is now used for the flag dropdown box on the tag scan page. Previously we used an unformatted scrolling list box where an allele already had a flag but this was 4-5 lines long and made the table ugly and multiple flags difficult to select. For new tags there was a standard dropdown box which meant only one flag could be selected. As scan tables can be many thousands of rows long, the dropbox is only rendered when it is first visible in the current viewport.

Full Changelog: v_1.43.1...v_1.43.2

Version 1.43.1

Hotfix release to fix the isolate query modification trigger not appearing on the isolate query page (broken in commit d4b5439).

Full Changelog: v_1.43.0...v_1.43.1

Version 1.43.0

This version adds optional web logging to record user access. A new table log has been added to the bigsdb_auth database that records IP address, username, and page called with a timestamp. In addition, the REST API logging to the bigsdb_rest database has been improved by the addition of client and username information for when authenticated access is used.

To enable logging, you need to set the following in bigsdb.conf:

web_log_to_db=1
rest_log_to_db=1

Logging requires writing to the database on each page access so there is a very small performance penalty to enabling this. The tables are however unlogged (i.e. data are not written to the PosgreSQL write-ahead log) which makes them considerably faster than ordinary tables but data in them will be lost in the event of a database crash or unclean shutdown.

As every page access is recorded the log tables will grow in size over time. It is recommended that they are pruned regularly to remove records older than a specified period of time - this may also be required by GDPR! The easiest way to do this is to set up a scheduled CRON job by adding the following to /etc/crontab:

0  *  *  *  *  postgres psql -c "DELETE FROM log WHERE timestamp < NOW() - INTERVAL '7 days'" bigsdb_rest > /dev/null
10 *  *  *  *  postgres psql -c "DELETE FROM log WHERE timestamp < NOW() - INTERVAL '7 days'" bigsdb_auth > /dev/null

Additionally, there is a fix to the locus stats function in the sequence typing database necessitated by the introduction of schemes that can include locus presence/absence in their definitions. You therefore need to update the bigsdb_rest, bigsdb_auth, and any sequence definition database using the rest_v1.43.sql, auth_v1.43.sql, and seqdefdb_v1.43.sql scripts respectively. See https://github.com/kjolley/BIGSdb/blob/develop/Upgrading_README.txt.

Note that in the unlikely event that you have defined schemes that utilize locus presence/absence (where profiles can include the 'P' designation to indicate 'presence'), rather than just sequence variation, you should update the locus stats within the typing database with the following commands, otherwise the reported minimum allele length may be wrong.

DELETE FROM locus_stats;
INSERT INTO locus_stats(locus,datestamp,allele_count,min_length,max_length) 
SELECT loci.id,MAX(sequences.datestamp),COUNT(sequences.allele_id),MIN(LENGTH(sequence)),MAX(LENGTH(sequence)) 
FROM loci LEFT JOIN sequences ON loci.id=sequences.locus 
WHERE allele_id NOT IN ('N','0','P') OR allele_id IS NULL 
GROUP BY loci.id;

Full Changelog: v_1.42.7...v_1.43.0

Version 1.42.7

This version:

• Adds a new Reports plugin - this is used to generate customized genome reports. See https://github.com/kjolley/Klebsiella_reports for
  examples of how to initiate and modify report templates.
• Adds an option to silently reject alleles that fail similarity check when batch uploading.
• Adds an option to make some fields expected for genome assemblies - set required="genome_expected" in the field attribute in config.xml.
• Makes the 'expand width' view persistent and allow its use for the front-end dashboard. This expands the width of page to that of the web browser and is useful if your results tables are very wide and you have a large monitor.
• Adds an option to return partial sequence matches for scheme queries using the API.

Full Changelog: v_1.42.6...v_1.42.7

Version 1.42.6

This release adds the following functionality and improvements:

• All tables can now be searched by a list of attributes. This has been possible for isolates, profiles and alleles (when using the specific-locus query) for a long time now, but it is now possible for example, to search for allele designations or tags based on a list of isolates or a list of loci.
• N50 gridlines have been added to the sequence bin cumulative length chart (ff62aca).
• Multiple optimizations have been made to improve page load performance. This has largely focussed on the following:
  • Reducing the number of database calls required by taking calls out of loops and instead retrieving batched results from the database in a single call and then caching these results (b5998a4, 6131878, e1ee2b6, 96cc321).
  • A more efficient SQL query has been made when generating the database view that filters the isolate table to include only records accessible by the current user, depending on their status and access to private records (797a45a).
  • Removed the plugin manager isolate count check used to decide whether to list the plugin on the contents page (e399d07)
• Selected fields are now included in the ReporTree partitions summary. Any date field defined in the isolate table can now also be selected and used for the temporal component of cluster definitions.

Full Changelog: v_1.42.5...v_1.42.6

Version 1.42.5

This release introduces a new plugin that provides a wrapper for ReporTree (Mixão et al. 2023 Genome Med 15:43).

See https://bigsdb.readthedocs.io/en/latest/data_analysis/reportree.html for details.

Version 1.42.4

This release improves web-based sequence scanning by:

• showing the position of the first in-frame stop codon for loci defined as 'complete CDS'. The position has a coloured background that changes from red to green based on the position (red when it is at the 5' end, green when at the 3' end). This makes it easier for a curator to identify which matches are likely to be assignable as complete CDS alleles and/or whether they have internal stop codons other than near the 3' end of the sequence.

• adds an option to separate the search for start and stop codons. You can now choose to just look for in-frame stop codons that are within a specified percentage of the length of the best matching allele - either shorter or longer.

Version 1.42.3

This version fixes an issue with exporting LINcode fields with isolate records.