Cyz2Json - Convert flow cytometry CYZ files to JSON format.

Introduction

This program converts flow cytometry data stored in a CytoBuoy CYZ format file to JSON format.

Although CYZ files work with the Cytobuoy supplied CytoClus program, we want to access the data from a wider set of data science tooling including Python and R. We therefore convert to JSON.

This program uses the CytoBuoy CyzFile API and targets the Microsoft .NET 8 framework on Windows, macOS or Linux.

Build instructions

Build with GitHub actions

Commit your code and push it to GitHub. then add a tag to the commit and push it to GitHub. like the sample below

git tag v0.0.50 && git push origin v0.0.50

the action will build the binary and push it to the release

You will find result in : https://github.com/ecotaxa/cyz2json/actions and release in : https://github.com/ecotaxa/cyz2json/releases

Build with Docker

Only build cyz2json for Linux

First build a container to build the program:

docker build -t cyz2json-builder .     

Then run the container to build the cyz2json tool:

docker run -v $(pwd):/src cyz2json-builder bash -c "dotnet publish -c Release --self-contained true -r linux-x64"     

Build on the PC

Installation

The first step is to install the Microsoft dotnet runtime version 8. Microsoft provides detailed instructions at https://learn.microsoft.com/en-us/dotnet/core/install/.

For example, installation on Linux Ubuntu 22.04 is as follows:

sudo apt-get update && sudo apt-get install -y dotnet-sdk-8.0

You can test that dotnet is installed on your system by typing:

dotnet --info

The Cyz2Json program can then be compiled by cloning the project from GitHub and typing dotnet build -o bin.

Running the program

The program can be run, on Linux, for example, as follows:

dotnet bin/Cyz2Json.dll input.cyz --output output.json

Running with executable loaded from github

export cyzversion=v0.0.49
rm -rf cyz2json/* && curl -L -o cyz2json/cyz2json.zip https://github.com/ecotaxa/cyz2json/releases/download/$cyzversion/cyz2json-ubuntu-latest.zip && pushd cyz2json && unzip cyz2json.zip && popd && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:. && cyz2json/Cyz2Json ./Deployment\ 1\ 2024-07-18\ 21h12.cyz --metadatagreedy=false --output ./Deployment\ 1\ 2024-07-18\ 21h12.json

On Windows

Cyz2Json.EXE input.cyz --output output.json

On MacOSX

bypass security with xattr command

xattr -d com.apple.quarantine ./cyz2json-macos-latest/*

2 methods to run

dotnet bin/Cyz2Json.dll input.cyz --output output.json

or

Cyz2Json.exe input.cyz --output output.json

Bulk file conversion

We often find ourselves needing to undertake a bulk conversion of a large set of files. The following shell script can help:

#!/usr/bin/env bash
for file in *.cyz
do
    dotnet ~/git/Cyz2Json/bin/Cyz2Json.dll "$file" --output "$file".json
done

Processing CYZ JSON files using Python

To load a JSON flow cytometry data file into Python:

import json

data = json.load(open("pond.json", encoding="utf-8-sig"))
print(data)

Note the need to explicitly specify the encoding to deal with Microsoft’s and Python’s differences in interpretation of the standards regarding byte order marks in UTF-8 files.

R example

To load a JSON flow cytometry data file into R:

library(jsonlite)

json_data <- fromJSON("pond.json")
print(json_data)

## from Release

Download the release for you platfrom, and unpack on your folder choice.

### On MacOSX You will need to bypass security You have 2 possibilities : using xattr command. for exmaple: “` xattr -d com.apple.quarantine /opt/cytosense_to_ecotaxa_pipeline/bin/* “` or if you do not want use xattr you need to go in Systems Settings > Security & Privacy > General and allow the app to be opened for the 10+ libraries. You need to relauch sevaral time the cyz2json binary to acheive all neccessary permissions.

Notes

If a CYZ file contains images, we currently base64 encode them and include them inline in the JSON. This is costly in terms of disk and only time will tell if it is a sensible design decision. A future enhancement would be to include a flag that writes the files out as JPEG images.

Note that the images are un-cropped. This is an expedience to allow cross platform operation (The CyzFile-API only supports cropping on Windows platforms).

Acknowledgements

My thanks to the following organisations for supporting this work:

The Alan Turing Institute.

The Finnish Marine Research Infrastructure consortium (FINMARI)

The Centre for Environment, Fisheries and Aquaculture Science (Cefas)

CytoBuoy b.v.

I am grateful to Rob Lievaart at Cytobuoy for providing the libraries, code and examples on which this software is based. The CyzFile-API is licensed under the terms described in CyzFile-API_LICENSE.TXT.

Thanks to Eric Payne at Cefas for Visual Studio wizardry.

Disclaimers

The OBAMA-NEXT project has been approved under HORIZON-CL6-2022-BIODIV-01-01: Observing and mapping biodiversity and ecosystems, with particular focus on coastal and marine ecosystems (Grant Agreement 101081642). Funded by the European Union and UK Research and Innovation. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or UK Research and Innovation. Neither the European Union nor the granting authority can be held responsible for them.