Usage

Table of Contents:

• Using Docker for Song
• Using Java SDK for Song
• Using Java song-client
• Using Python SDK for Song
• Song GO Client

Using Docker for Song

The score-client and song-client are command line tools for local communication with the score-server and song-server, respectively.

Debugging with IntelliJ

To enable remote debugging of a running docker container in IntelliJ, use the song-client container, which exposes a JVM debug port. Create a remote debug profile with the following configuration:

Host: localhost
Port: 5005
Use module classpath: song-client

To debug the song-server, create a remote debug profile in Intellij with the following configuration:

Host: localhost
Port: 5006
Use module classpath: song-server

Testing running dependencies

Song provides several make targets for locally testing the running dependencies in Docker.

Ping Score server

Command:

make _ping_score_server

Expected response:

http://object-storage:9000/oicr.icgc.test/data/heliograph?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230328T182314Z&X-Amz-SignedHeaders=host&X-Amz-Expires=299&X-Amz-Credential=minio%2F20230328%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=efa7dac7efe9f6120ebfb6c143067e057ff3500443ff5c732e4709cad74f0cd2

Ping Song server

Command:

make _ping_song_server

Expected response:

true

Test Song Database

Command:

make _ping_song_db

Expected response:

 ?column?
----------
        1
(1 row)

Test Object Storage

Command:

make _setup-object-storage

Expected response:

upload: ../score-data/heliograph to s3://oicr.icgc.test/data/heliograph

Testing Submissions

Song also provides make targets for testing submissions and publication states.

Upload data files to cloud storage

Command:

make test-score-upload


# for Mac M1 use the following
DOCKER_BUILDKIT=0 make test-score-upload

Expected response:

Uploading object: '/data/submit/example.vcf.gz.idx' using the object id 2a54c4f5-9f37-511c-bb31-de4e6c9e7926
100% [##################################################]  Parts: 1/1, Checksum: 100%, Write/sec: 0B/s, Read/sec: 0B/s
Finalizing...
Total execution time:         3.277 s
Total bytes read    :               0
Total bytes written :              25
Upload completed
Uploading object: '/data/submit/example.vcf.gz' using the object id 8d4d29ae-e9b8-521e-b531-c51930024b0e
100% [##################################################]  Parts: 1/1, Checksum: 100%, Write/sec: 3.9K/s, Read/sec: 0B/s
Finalizing...
Total execution time:         3.148 s
Total bytes read    :               0
Total bytes written :              52
Upload completed

Publish the analysis

Command:

make test-publish

Expected response:

AnalysisId 7b70d598-ec9b-4019-b0d5-98ec9b50199f successfully published

Unpublish the analysis

Command:

make test-unpublish

Expected response:

AnalysisId 7b70d598-ec9b-4019-b0d5-98ec9b50199f successfully unpublished

Using Java SDK for Song

The Song Java SDK is a tool that makes it easier for developers interact with a Song server in their own applications. This tool let you perform various operations, such as submitting a payload, publishing and unpublishing analysis.

To get started with the Song Java SDK, you can import the song-java-sdk.jar distribution file, or add it as a dependency in your Java application using Maven.

If you are adding it as a Maven dependency, use the following code in your project's pom.xml file:

<dependency>
  <groupId>bio.overture</groupId>
  <artifactId>song-java-sdk</artifactId>
  <version>SONG_JAVA_SDK_VERSION</version>
</dependency>

Once the dependency is imported, you can start using all the classes and interfaces provided by Song SDK in your project.

You may start providing the Song Server details and Access token using following Rest Client configuration and creating a Toolbox object.

 RestClientConfig restClientConfig =
     DefaultRestClientConfig.builder()
         .serverUrl(SONG_SERVER_URL)
         .accessToken(ACCESS_TOKEN)
         .build();

 Toolbox toolbox = Toolbox.createToolbox(restClientConfig);

After providing above configuration you can initialize a Song API object.

SongApi songApi = toolbox.getSongApi();

SongApi exposes all the methods to interact with Song server.

To check the status of Song server you can use:

System.out.println("isAlive:" + songApi.isAlive());

To retrieve an Analysis by ID use:

System.out.println("Analysis data:" + songApi.getAnalysis(STUDY_ID, ANALYSIS_ID));

Using Java song-client

The Java song-client is a Java CLI tool that communicates with a song-server using the song-java-sdk.

To compile the song-client go to the song-client/ folder and run mvn clean package, this will generate a -dist.tar.gz file located in the target/ folder.

After extracting the distribution file, song-client can be configured via the ./conf/application.yml file.

Alternatively, the client can be configured through environment variables, which take precedence over the application.yml config. For example, to run the song-client config command using environment variables could be done via:

CLIENT_SERVER_URL=http://localhost:8080 \
CLIENT_STUDY_ID=ABC123 \
CLIENT_PROGRAM_NAME=sing \
CLIENT_DEBUG=true \
CLIENT_ACCESS_TOKEN=myAccessToken \
./bin/sing config

Run ./bin/sing help to find more commands.

Using Python SDK for Song

The Song REST client is a Python module that allows you to interact with a Song server through Python with minimal coding effort. It lets you submit payloads, publish, unpublish analyses, and more. From there, you can use the power of Python to process and analyze the data within those objects as you see fit.

The official SONG Python SDK is publically hosted on PyPi

Python 3.6 or higher is required.

1. To get started, create your Python project and install song using pip:

pip install overture-song

2. Create an ApiConfig object. This object contains the configuration needed to interact with the Song API.

from overture_song.model import ApiConfig
api_config = ApiConfig(SONG_SERVER_URL, STUDY_ID, ACCESS_TOKEN)

Please replace SONG_SERVER_URL with the SONG server you want to connect to, STUDY_ID with the Study ID which you want to read/modify data to, and ACCESS_TOKEN with a valid access token to prove to SONG server that you're allowed to read or make changes.

For more information about access token refer to: https://song-docs.readthedocs.io/en/develop/sdk/old_tutorial.html#getting-daco-access

3. Next, import and configure the API client:

from overture_song.client import Api
api = Api(api_config)

4. As a sanity check, ensure that the server is running:

api.isAlive()

For more info go to: https://song-docs.readthedocs.io/en/develop/sdk/python/python_sdk.html#song-python-sdk-ref

Song GO client

Getting the program

If your computer runs Linux or MacOS, download a pre-built binary from here.

Choose the version that matches your operating system, and download it. To verify the installation, type song-go-client help. If it gives you a screen full of help text, the program is working, and your installation is complete.

If you are running an operating system that isn't Mac or Linux or prefer to compile the client yourself, you can compile it using go build. It will package the client into a song-go-client executable file.

Configuration

To connect to the Song server, upload data, and make changes to a study, you need to configure your song-go-client providing the following information:

• The URL of the Song server that you want to upload to.
• The name of the study that you're uploading data for.
• An access token from the authentication service that your Song server is using. This is essentially a password that the system uses to know that you're authorized to modify the study. Keep it private and don't share it!

From the command line, run ./song-go-client configure. It will display the current configuration settings and the name of the file in which they are stored.

Showing configuration for config file '/Users/lrivera/.song.yaml'
accessToken: <mySecretAccessToken>
songURL: http://localhost:8080/
study: ABC123

You can edit that file with any text editor, or type ./song-go-client configure --edit and fill in the values when prompted.

Using the GO Client

Check Song Server status

To check the Song server status run the following command:

./song-go-client status

Expected Response:

true

Submit a metadata payload

First, a metadata payload must be prepared. The payload must conform to an analysis_type that has been registered as a schema. For help with creating or updating schemas please see the Dynamic Schemas documentation.

Example files can be found in the github repository.

Once you have formatted the payload correctly, use the song-go-client submit command to upload the payload.

Command:

./song-go-client submit exampleVariantCall.json

The response will contain the analysisId auto generated:

{"analysisId":"469f92a5-a78d-46a7-9f92-a5a78d36a7f8","status":"OK"}

Search analysis by ID

To search for the submitted analysis and observe that the field analysisState is set to UNPUBLISHED:

./song-go-client search -a <analysisId>

The response will contain the analysis if found.

For more information, go to: https://github.com/overture-stack/SONG/tree/develop/song-go-client