Skip to content

v3.0.0
Compare
Choose a tag to compare
@elrayle elrayle released this 09 Oct 17:39
v3.0.0
26e7025
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Overview Changes in Release 3.0.0

This release updates authorities and modifies the processing of performance statistics. The performance statistics changes are breaking which drives the major release.

  • If you modified the performance calculations or display, or use that data in another context, see PR #217 to understand the refactor that was done to that code and data structures.

  • Previously, there were two sub-tasks for each request (e.g. :load, :normalization). To get a better insight into performance issues, :load was split into :retrieve and :graph_load. This means that older data for :load is no longer used in performance calculations or graph generation. The data was not removed from the database, but the column holding the :load data was renamed from load_time_ms to retrieve_plus_graph_load_time_ms to avoid confusion.

Actions Required to Upgrade

Update gem

Update to qa_server v3.0.0

bundle update qa_server

Update database

Pull in one new migration to add the performance_history table. Should add migration:

  • _update_performance_history_table.qa_server.rb
rake qa_server:install:migrations

Run migrations

rake db:migrate

Add Prepends

A prepend is added to record the time to read data from the source authority and the time to load the graph.

Update /config/application.rb to include the following prepend registrations. Prepends were added for release 2.2.0. If you don't have any prepends yet, see the release notes for 2.2.0. Otherwise, just add the registration for RDF::Graph with your existing prepends.

config.to_prepare do
  # There should be existing prepends for FindTerm and SearchQuery.  See release 2.2.0.
  RDF::Graph.prepend PrependedRdf::RdfGraph
end

Update configurations (optional)

All configuration changes are optional. See details for configuration changes under Description of Changes below.

Description of Changes

Changes to authorities:

  • adds pagination start_record parameter for LD4P cached data
  • adds RDA Registry authority
  • splits LCNAF into 2 authorities for LD4P cached data focused locnames and real world object focused locnames_rwo
  • adds wikidata authority - NOTE: The first pass on fetching through this authority is non-performant.

Changes to performance data processing:

A significant refactor was made on the calculation and display of performance data. If you override any of the performance calculations or display, you will want to take a close look at the changes in PR #217.

Performance graphs are created for all combinations of...

  • authority: each authority + all authorities
  • time period: :day, :month, :year
  • action: :all_actions, :search, :fetch (NEW)

This produces 9 graphs per authority plus 9 more that include data from all authorities.

The performance data table splits details into combinations for...

  • authority: each authority + all authorities
  • 10th percentile (fastest), average, 90th percentile (slowest)
  • action: :all_actions, :search, :fetch (NEW)
  • task: :retrieve, :graph_load, :normalization, :full_action (labeled TOTAL) (MODIFIED - separated :load into :retrieve and :graph_load)

where...

  • :retrieve includes network costs + cache/authority processing for retrieving data from the cache or directly from an authority
  • :graph_load includes the time to load the retrieved data into an RDF::Graph
  • :normalization includes any time spent by QA to normalize the data including filtering out results, extracting extended context, and writing the normalized json
  • :full_action is the length of time spent in the search method or find method. There is a small amount of overhead for Rails routing and controller processing that is not included in this time.

Changes to Configuration Options

All new configurations are optional.

The performance reporting has new configurations to control the output in the Monitor Status page:

  • config.performance_y_axis_maximum = 4000 (ms)
  • config. performance_retrieve_color = '#ABC3C9'
  • config. performance_graph_load_color = '#E8DCD3'

Removed configuration:

  • config. performance_load_color

It is recommended that you copy the new configurations to keep your local config file in sync with the configurations.

Copy only the new configs...

FROM: lib/generators/qa_server/templates/config/initializers/qa_server.rb
TO: config/initializers/qa_server.rb

Migrations

A migration (e.g. update_performance_history_table) was added to modify the performance_history table.

  • changes action column from a string to an integer. This column is an ActiveRecord enum and should never have been set to string type. ActiveRecord was ignoring that type and treating it like an integer at least for some databases. This is a bug fix.
  • rename :load_time_ms to :retrieve_plus_graph_load_time_ms (to avoid confusion with new :graph_load_time_ms)
  • add columns :retreive_time_ms, :graph_load_time_ms, and :action_time_ms
  • index on

Change Log

  • refactor performance data and graphs to include stats by action as well as by authority and time period
  • adds pagination start_record parameter for LD4P cached data
  • adds RDA Registry authority
  • splits LCNAF into 2 authorities for LD4P cached data focused locnames and real world object focused locnames_rwo
  • adds wikidata authority - NOTE: The first pass on fetching through this authority is non-performant.
Assets 2
Loading