v21.9.6

WARNING: This release contains schema migrations.
Make sure to run $ make migrate after upgrading the code if you run ScanCode.io with the "local development" setup.
https://scancodeio.readthedocs.io/en/latest/installation.html#upgrading
This is not needed if you run with Docker.

• Add ability to "archive" projects, from the Web UI, API and command line interface.
  Data cleanup of the project's input, codebase, and output directories is available
  during the archive operation.
  Archived projects cannot be modified anymore and are hidden by default from the
  project list.
  A project cannot be archived if one of its related run is queued or already running. #312

• Remove the run_extractcode pipe in favor of extractcode API. #312

• The scancode.run_scancode pipe now uses an optimal number of available CPUs for
  multiprocessing by default.
  The exact number of parallel processes available to ScanCode.io can be defined
  using the SCANCODEIO_PROCESSES setting. #302

• Renamed the SCANCODE_DEFAULT_OPTIONS setting to SCANCODE_TOOLKIT_CLI_OPTIONS. #302

• Log the outputs of run_scancode as progress indication. #300

v21.8.2

• Upgrade ScanCode-toolkit to version 21.7.30

• Add new documentation chapters and tutorials on the usage of the Web User Interface. #241

• Add ability to register custom pipelines through a new SCANCODEIO_PIPELINES_DIRS
  setting. #237

• Add a pipeline scan_package.ScanPackage to scan a single package archive with
  ScanCode-toolkit. #25

• Detected Package dependencies are not created as Package instance anymore but stored
  on the Package model itself in a new dependencies field. #228

• Add the extra_data field on the DiscoveredPackage model. https://github.com/nexB/scancode.io/issues/191

• Improve XLSX creation. We now check that the content is correctly added before
  calling XlsxWriter and report and error if the truncated can be truncated. #206

• Add support for VMWare Photon-based Docker images and rootfs. This is an RPM-based
  Linux distribution

v21.6.10

• Add support for VM image formats extraction such as VMDK, VDI and QCOW.
  See https://github.com/nexB/extractcode#archive-format-kind-file_system for the full
  list of supported extensions.
  The new extraction feature requires the installation of libguestfs-tools,
  see https://github.com/nexB/extractcode#adding-support-for-vm-images-extraction for
  installation details. #132

• Add the ability to disable multiprocessing and threading entirely through the
  SCANCODEIO_PROCESSES setting. Use 0 to disable multiprocessing and use -1 to also
  disable threading. #185

• Missing project workspace are restored on reports (xlsx, json) creation. This allow
  to download reports even if the project workspace (input, codebase) was deleted. #154

• Add ability to search on all list views. #184

• Add the is_binary, is_text, and is_archive fields to the CodebaseResource model. #75

v21.5.12

• Adds a new way to fetch docker images using skopeo provided as a
  plugin using docker:// reference URL-like pointers to a docker image.
  The syntax is docker:// where is the string
  that would be used in a "docker pull " command.
  Also rename scanpipe.pipes.fetch.download() to fetch_http() #174

• Pipeline status modals are now loaded asynchronously and available from the
  project list view.

• Fix an issue accessing codebase resource content using the scan_codebase and
  load_inventory pipelines. #147

v21.4.28

• The installation local timezone can be configured using the TIME_ZONE setting.
  The current timezone in now included in the dates representation in the web UI.
  #142

• Fix pipeline failure issue related to the assignment of un-saved (not valid) packages.
  #162

• Add a new QUEUED status to differentiate a pipeline that is in the queue for execution
  from a pipeline execution not requested yet.
  #130

• Refactor the multiprocessing code for file and package scanning.
  All database related operation are now executed in the main process as forking the
  existing database connection in sub-processes is a source of issues.
  Add progress logging for scan_for_files and scan_for_application_packages pipes. #145

• Links from the charts to the resources list are now also filtered by
  in_package/not_in_package if enabled on the project details view. #124

• Add ability to filter on codebase resource detected values such as licenses,
  copyrights, holders, authors, emails, and urls. #153

• Filtered list views from a click on chart sections can now be opened in a new tab
  using ctrl/meta + click. #125

• Add links to codebase resource and to discovered packages in list views.

v21.4.14

• Implement timeout on the scan functions, default to 120 seconds per resources. #135

• Fix issue with closing modal buttons in the web UI. #116 #141

v21.4.5

• Add support for Docker and VM images using RPMs such as Fedora, CentOS, RHEL,
  and openSUSE linux distributions. #6

• Add a compliance alert system based on license policies provided through a
  policies.yml file. #90 The compliance alerts are computed from the license_expression and
  stored on the codebase resource. When the policy feature is enabled, the compliance
  alert values are displayed in the UI and returned in all the downloadable results.
  The enable and setup the policy feature, refer to https://scancodeio.readthedocs.io/en/latest/scancodeio-settings.html#scancodeio-policies-file

• Add a new codebase resource detail view including the file content.
  Detected value can be displayed as annotation in the file source. #102

• Download URLs can be provided as inputs on the project form.
  Each URL is fetched and added to the project input directory. #100

• Run celery worker with the "threads" pool implementation.
  Implement parallelization with ProcessPoolExecutor for file and package scans.
  Add a SCANCODEIO_PROCESSES settings to control the multiprocessing CPUs count. #70

• Optimize "tag" type pipes using the update() API in place of save() on the QuerySet
  iteration. #70

• Use the extractcode API for the Docker pipeline.
  This change helps with performance and results consistency between pipelines. #70

• Create the virtualenv using the virtualenv.pyz app in place of the bundled "venv". #104

• Consistent ordering for the pipelines, now sorted alphabetically.

1.1.0 (Feb 16, 2021)

• Display project extra data in the project details view. #88

• Add a @Profile decorator for profiling pipeline step execution. #73

• Support inputs as tarballs in root_filesystem pipelines.
  The input archives are now extracted with extractcode to the codebase/ directory. #96

• Improve support for unknown distros in docker and root_filesystem pipelines.
  The pipeline logs the distro errors on the project instead of failing. #97

• Implement Pipeline registration through distribution entry points.
  Pipeline can now be installed as part of external libraries.
  With this change pipelines are no longer referenced by the
  Python script path, but by their registered name.
  This is a breaking command line API change. #91

• Add a "Run Pipeline" button in the Pipeline modal of the Project details view.
  Pipelines can now be added from the Project details view. #84

• Upgrade scancode-toolkit to version 21.2.9

• Allow to start the pipeline run immediately on addition in the add_pipeline action
  of the Project API endpoint. #92

• Rename the pipes.outputs module to pipes.output for consistency.

• Remove the dependency on Metaflow. #82

---

WARNING: The new Pipelines syntax is not backward compatible with v1.0.x, the default built-in Pipeline were migrated.
Refer to the migration section bellow if you are using custom Pipelines.

Migrate Pipeline to 1.1.x:

• Remove django.setup()
• Remove @step decorators
• Remove start() and end() methods
• Remove all the self.next() calls
• Remove the if __name__ == "__main__" section
• Add a steps argument after the methods:

steps = (
    step1,
    step2,
)

See https://github.com/nexB/scancode.io/tree/1.1.0/scanpipe/pipelines for more Pipeline examples.

1.0.7 (Feb 2, 2021)

• Add user interface to manage Projects from a web browser. All the command-line features are available #24
• Log messages from Pipeline execution on a new Run instance log field #66
• Add support for scancode pipes and Project name with whitespaces
• Add a profile() method on the Run model for profiling pipeline execution #73

1.0.6 (Dec 12, 2020)

• Add a management command to delete a Project and its related work directories #65
• Add CSV and XLSX support for the output management command #46
• Add a to_xlsx output pipe returning XLSX compatible content #46
• Add a "status" management command to display Project status information #66
• Fix the env_file location to run commands from outside the root dir #64
• Add utilities to save project error in the database during Pipeline execution #64
• Install psycopg2-binary instead of psycopg2 on non-Linux platforms #64