diff --git a/contributing/index.html b/contributing/index.html index dd2e3ce3..4719a33c 100644 --- a/contributing/index.html +++ b/contributing/index.html @@ -666,27 +666,27 @@
  • - + - Get Started + Modifying Code
  • - + - Pull Request Guidelines + Automated Checks
  • - + - Documentation style + Submitting a Pull Request @@ -783,7 +783,7 @@

    Submit FeedbackGet Started

    +

    Modifying Code

    Ready to contribute? Here's how to set up caveclient for local development.

    -

    Pull Request Guidelines

    Before you submit a pull request, check that it meets these guidelines:

    • The pull request should include tests if adding a new feature.
      • -
    • The docs should be updated with whatever changes you have made. Put +
    • +

      The docs should be updated with whatever changes you have made. Put your new functionality into a function with a docstring, and make sure the new - functionality is documented after building the documentation.

      • + functionality is documented after building the documentation (described above).

      + +
    • +

      Once you submit a pull request, automated checks will run. You may require administrator approval before running these checks if this is your first time contributing to the repo.

      +
    -

    Documentation style

    -

    We use mkdocs to build the documentation. In particular, we -use the mkdocs-material theme, and a -variety of other extensions.

    -
    -

    Note

    -

    More information codifying our documentation style and principles coming soon. For -now, just try to follow the style of the existing documentation.

    -
    diff --git a/search/search_index.json b/search/search_index.json index 5ab121ef..285b2a6e 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"

    CAVE is short for Connectome Annotation Versioning Engine. CAVE is a set of microservices that provide a framework for storing and versioning connectomics data and large sets of dynamic annotations, metadata, and segmentations. This repository supplies client-side code to easily interact with the microservices in CAVE.

    "},{"location":"#documentation-layout","title":"Documentation Layout","text":"

    To learn how to install caveclient, visit Installation.

    To see hands-on examples of using caveclient in a notebook, visit the Tutorials.

    To see API reference documentation for interacting with a caveclient.CAVEclient object, visit the Client API.

    To see a glossary of terms used in the documentation, visit the Glossary (work in progress).

    For information about how to contribute to the documentation or the package, visit the Contributing page. Feedback on the documentation is welcome! Please open an issue or use the \"Edit this page\" button at the top right of any page to suggest changes.

    To read a full description of the Connectome Annotation Versioning Engine, including a description of the various services please read this paper.

    "},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#740-december-12-2024","title":"7.4.0 (December 12, 2024)","text":"

    annotation module: Added support for \"None\", \"none\", etc. to remove notice_text and improved documentation on method

    "},{"location":"changelog/#732-december-10-2024","title":"7.3.2 (December 10, 2024)","text":"

    Dropped oldest supported numpy requirements and declared python 3.8 support

    "},{"location":"changelog/#731-december-5-2024","title":"7.3.1 (December 5, 2024)","text":"

    Improved documentation to annotation.update_table_metadata

    "},{"location":"changelog/#730-december-5-2024","title":"7.3.0 (December 5, 2024)","text":"

    Added get_minimal_covering_nodes to chunkedgraph (#274)

    "},{"location":"changelog/#721-december-3-2024","title":"7.2.1 (December 3, 2024)","text":"

    Added raw strings to avoid deprecation warning (#275)

    "},{"location":"changelog/#720-december-3-2024","title":"7.2.0 (December 3, 2024)","text":"

    Allow get_state_json to accept direct URLs (#271)

    "},{"location":"changelog/#710-december-3-2024","title":"7.1.0 (December 3, 2024)","text":"

    Added get_leaves_multiple to chunkedgraph module (#273)

    "},{"location":"changelog/#700-november-20-2024","title":"7.0.0 (November 20, 2024)","text":"
    • Simplified skeleton client interface to only accept formats of 'dict' and 'swc' Structure of the dictionary returned was also changed compared to previous \"json\" format, where the dict is flat with only one level of keys. Corresponding change made to MeshParty 1.18.0 allows for hydration of MeshParty Skeleton object from this dict format.
    "},{"location":"changelog/#650-november-15-2024","title":"6.5.0 (November 15, 2024)","text":"
    • Added endpoints for bulk skeleton retrieval and querying whether a skeleton exists or not.
    "},{"location":"changelog/#641","title":"6.4.1","text":"
    • Fix networkx deprecation warning.
    "},{"location":"changelog/#640-october-31-2024","title":"6.4.0 (October 31, 2024)","text":"
    • Added a CAVEclientMock function to ease the use of fully mocked CAVEclients in testing.
    • Fix a bug about use of global server.
    • Fix writing of local server secret to be more secure.
    "},{"location":"changelog/#630","title":"6.3.0","text":"
    • Add a convenience function to get a complete L2 feature dataframe.
    "},{"location":"changelog/#620","title":"6.2.0","text":"
    • Fix things so that auth will not error if the secrets directory is read-only.
    "},{"location":"changelog/#612","title":"6.1.2","text":"
    • Fix query filter bug in tables interface.
    "},{"location":"changelog/#611","title":"6.1.1","text":"
    • Fix warnings.
    "},{"location":"changelog/#610","title":"6.1.0","text":"
    • Added support for inequality filters for numerical columns in materialization tables interface.
    "},{"location":"changelog/#600-october-14-2024","title":"6.0.0 (October 14, 2024)","text":"
    • Refactored CAVEclient internals away from a factory syntax. Most users should not notice a difference, but code that relied on specific subclient type logic may be affected.
    "},{"location":"changelog/#5250","title":"5.25.0","text":"
    • Added ability to suppress table warnings in client.materialize.query_table and similar methods by setting log_warnings=False
    "},{"location":"changelog/#5200-april-8-2024","title":"5.20.0 (April 8, 2024)","text":"
    • Added generalized support for detecting server versions to provide timely exceptions to users
    • Used new support to check that chunkegraph has updated version before using spatial bounds kwarg on client.chunkedgraph.level2_chunk_graph
    • Added support for postign and getting segment properties files to client.state
    "},{"location":"changelog/#5180","title":"5.18.0","text":"
    • Added serialization support for pandas.index
    "},{"location":"changelog/#5173","title":"5.17.3","text":"
    • Minor documentation typo fix
    "},{"location":"changelog/#5172","title":"5.17.2","text":"
    • Bug fixes related to table_manager interface
    "},{"location":"changelog/#5171","title":"5.17.1","text":"
    • Bug fixes related to table_manager interface
    "},{"location":"changelog/#5170","title":"5.17.0","text":"
    • Fix attrs in dataframe attributes of client.materialize results to remove numpy arrays to allow concatenation of dataframes
    • Added getting multiple schemas in one call to improve initialization of table_manager interface of materialization
    "},{"location":"changelog/#5161","title":"5.16.1","text":"
    • Bugfix on client.chunkedgrpah.level2_chunk_graph
    "},{"location":"changelog/#5160","title":"5.16.0","text":"
    • Added bounding box query to client.chunkedgraph.level2_chunk_graph
    • Fix default materialization version client when server not advertising correctly
    "},{"location":"changelog/#5151-jan-18-2024","title":"5.15.1 (Jan 18, 2024)","text":"
    • minor improvements to release process
    "},{"location":"changelog/#5150-jan-18-2024","title":"5.15.0 (Jan 18, 2024)","text":"
    • Improved documentation with types
    • Improved testing on more python versions
    • Bugfixes for pyton 3.12 compatability
    "},{"location":"changelog/#5140-november-24-2023","title":"5.14.0 (November 24, 2023)","text":"
    • Made automatic detection of neuroglancer versioning when constructing link shortener links
    "},{"location":"changelog/#5130-october-26-2023","title":"5.13.0 (October 26, 2023)","text":"
    • Add option to get expired versions to client.materialize.get_versions
    "},{"location":"changelog/#5121-october-16-2023","title":"5.12.1 (October 16, 2023)","text":"
    • Bugfixes for client.chunkedgraph.get_latest_roots
    "},{"location":"changelog/#5120-october-16-2023","title":"5.12.0 (October 16, 2023)","text":"
    • Improved logic for client.chunkedgraph.get_latest_roots to work forward or backwards in time
    "},{"location":"changelog/#5110-september-19-2023","title":"5.11.0 (September 19, 2023)","text":"
    • Added filter_regex_dict options to client.materialize.query_table interface
    "},{"location":"changelog/#5102-august-162023","title":"5.10.2 (August 16,2023)","text":"
    • Fixed pyarrow support for live_live query
    "},{"location":"changelog/#5101-august-142023","title":"5.10.1 (August 14,2023)","text":"
    • Changed random_sample argument to be an integer number of annotations rather than a floating fraction of table
    • Added option to live_query
    "},{"location":"changelog/#590-august-14-2023","title":"5.9.0 (August 14, 2023)","text":"
    • Added support for native pyarrow deserialization, allowing upgrade to pyarrow version
    "},{"location":"changelog/#580","title":"5.8.0","text":"
    • Allowed int64 root ids to serialize properly
    • Added warning that client.materialize.tables interface is in beta
    "},{"location":"changelog/#570","title":"5.7.0","text":"
    • Fix to ensure stop_layer is at least 1
    • Added client.chunkedgraph.suggest_latest_roots
    "},{"location":"changelog/#560","title":"5.6.0","text":"
    • Added views to client.materialize.tables interface
    • Added optional argument to allow invalid root ids when querying live live, versus creating an exception
    "},{"location":"changelog/#551","title":"5.5.1","text":"
    • documentation fixes on client.materialize.join_query
    "},{"location":"changelog/#550","title":"5.5.0","text":"
    • added methods for different neuroglancer state formats to client.state.
    "},{"location":"changelog/#543","title":"5.4.3","text":"
    • Added 'view' querying options to materialization
    • Added client.materialize.tables interface
    • Added client.materialize.get_tables_metadata to get all metadata in one call
    "},{"location":"changelog/#520","title":"5.2.0","text":"
    • Added local caching of datastack names > server_address to simplify initialization of clients with servers other than global.daf-apis.com.

    Cache is saved on a local file ~/.cloudvolume/secrets/cave_datastack_to_server_map.json

    Cache will populate the first time caveclient.CAVEclient('my_datastack', server_address=\"https://my_server.com\") is called. Subsequent calls can then just be caveclient.CAVEclient('my_datastack').

    "},{"location":"changelog/#510","title":"5.1.0","text":"
    • Added get_oldest_timestamp call to chunkedgraph
    "},{"location":"changelog/#501","title":"5.0.1","text":"
    • Fixed bug with desired_resolution being set at the client level was being ignored in >5.0.0
    "},{"location":"changelog/#500","title":"5.0.0","text":"
    • Added support for the new CAVE Materialization 3.0 API Includes support for the new materialization API, which allows for server side conversion of the units of position, and ensures that all positions are returned with the same units, even after joins.
    • Added support for querying databases that were materialized without merging tables together. This will allow for faster materializations.
    • Removed support for LiveLive query from the Materialization 2.0 API client. Note.. <5.0.0 clients interacting with MaterializationEngine >4.7.0 servers will use live live query but will doubly convert the units of position if you ask for a desired resolution, as the old client will also do a conversion server side.
    • Fixed interaction with api version querying of servers from individual clients to work with verify=False. (useful for testing)
    • Stored infromation from client about mapping between dataframe and table names and original column names.
    • Added support for suffixes and select columns to be passed by dictionary rather than list making the selection an application of suffixes more explicit when there are collisions between column names in joined tables.
    "},{"location":"changelog/#older-upgrade-notes","title":"Older Upgrade Notes","text":"

    Change all select_column calls to pass dictionaries rather than lists. Change all suffix calls to pass dictionaries rather than lists. Advocate for your server administrator to upgrade to MaterializationEngine 4.7.0 or later, so you can use the new MaterializationEngine 3.0 API and client.

    "},{"location":"contributing/","title":"Contributing","text":"

    Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

    You can contribute in many ways:

    "},{"location":"contributing/#types-of-contributions","title":"Types of Contributions","text":""},{"location":"contributing/#report-bugs","title":"Report Bugs","text":"

    Report bugs to our issues page.

    If you are reporting a bug, please include:

    • Your operating system name and version.
    • Any details about your local setup that might be helpful in troubleshooting.
    • Detailed steps to reproduce the bug, in the form of a minimal reproducible example.
    "},{"location":"contributing/#fix-bugs","title":"Fix Bugs","text":"

    Look through the GitHub issues for bugs. Anything tagged with \"bug\" and \"help wanted\" is open to whoever wants to implement it.

    "},{"location":"contributing/#implement-features","title":"Implement Features","text":"

    Look through the GitHub issues for features. Anything tagged with \"enhancement\" and \"help wanted\" is open to whoever wants to implement it.

    "},{"location":"contributing/#write-documentation","title":"Write Documentation","text":"

    caveclient could always use more documentation, whether as part of the official caveclient docs, in docstrings, or even on the web in blog posts, articles, and such.

    "},{"location":"contributing/#submit-feedback","title":"Submit Feedback","text":"

    The best way to send feedback is to create an issue on GitHub.

    If you are proposing a feature:

    • Explain in detail how it would work.
    • Keep the scope as narrow as possible, to make it easier to implement.
    • Remember that while contributions are welcome, developer/maintainer time is limited.
    "},{"location":"contributing/#get-started","title":"Get Started","text":"

    Ready to contribute? Here's how to set up caveclient for local development.

    • Fork the repo on GitHub.
    • Clone your fork locally
    git clone git@github.com:your_name_here/CAVEclient.git\n
    • Ensure pip is installed.
    • Create a virtual environment (here we use venv):
    python3 -m venv .venv\n
    • Start your virtualenv:
    source .venv/bin/activate\n
    • Create a branch for local development:
    git checkout -b name-of-your-bugfix-or-feature\n
    • Make your changes locally
    • Install development requirements:
    pip install -r test_requirements.txt\npip install -e .\n
    • When you're done making changes, check that your changes pass the tests by running pytest:
    pytest tests\n

    Note that once you submit your pull request, GitHub Actions will run the tests also, including on multiple operating systems and Python versions. Your pull request will have to pass on all of these before it can be merged.

    • Ensure your contribution meets style guidelines. First, install ruff:
    pip install ruff\n
    • Fix linting and formatting. From the root of the repository, run the following commands:
    ruff check . --extend-select I --fix\nruff format .\n
    • Commit your changes and push your branch to GitHub:
    git add .\ngit commit -m \"Your detailed description of your changes.\"\ngit push origin name-of-your-bugfix-or-feature\n
    • Submit a pull request through the GitHub website.
    "},{"location":"contributing/#pull-request-guidelines","title":"Pull Request Guidelines","text":"

    Before you submit a pull request, check that it meets these guidelines:

    • The pull request should include tests if adding a new feature.
    • The docs should be updated with whatever changes you have made. Put your new functionality into a function with a docstring, and make sure the new functionality is documented after building the documentation.
    "},{"location":"contributing/#documentation-style","title":"Documentation style","text":"

    We use mkdocs to build the documentation. In particular, we use the mkdocs-material theme, and a variety of other extensions.

    Note

    More information codifying our documentation style and principles coming soon. For now, just try to follow the style of the existing documentation.

    "},{"location":"glossary/","title":"Glossary","text":"

    Warning

    This glossary is a work in progress; for now we are documenting the commonly used terms that we need to define. Please feel free to contribute definitions or additional terms.

    • Datastack
    • Voxel resolution
    • Segmentation resolution
    • MIP
    • Segmentation
    • View
    "},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#stable-release","title":"Stable release","text":"

    To install caveclient, run this command in your terminal:

    pip install caveclient\n

    This is the preferred method to install caveclient, as it will always install the most recent stable release.

    You can also specify a particular version, e.g.

    pip install caveclient==5.0.0\n

    If you don't have pip installed, this Python installation guide can guide you through the process.

    "},{"location":"installation/#from-source","title":"From source","text":"

    The source for caveclient can be downloaded from the Github repo.

    You can either clone the public repository:

    git clone git://github.com/CAVEconnectome/CAVEclient\n

    Or download the tarball:

    curl -OJL https://github.com/CAVEconnectome/CAVEclient/tarball/master\n

    Once you have a copy of the source, you can install it with:

    pip install .\n

    Or in editable mode, it can be installed with:

    pip install -e .\n
    "},{"location":"api/","title":"Overview","text":"

    The most common method of interacting with the CAVE Framework is by instantiating a client (caveclient.CAVEclient) and then using that client to interact with various services. Under the hood, the CAVEclient is a collection of individual clients, which can be accessed via properties. For example, to access the materialization client, you can use client.materialize, which (up to the exact version) will actually return a MaterializationClient object.

    These pages describe the functionality of each of the individual clients, but note that some features may not be available if the server you are accessing is using older versions of particular services.

    "},{"location":"api/annotation/","title":"client.annotation","text":"

    Client for interacting with the annotation engine.

    Methods:

    Name Description create_table

    Creates a new data table based on an existing schema

    delete_annotation

    Delete one or more annotations in a table. Annotations that are

    delete_table

    Marks a table for deletion requires super admin privileges

    get_annotation

    Retrieve an annotation or annotations by id(s) and table name.

    get_annotation_count

    Get number of annotations in a table

    get_table_metadata

    Get metadata about a table

    get_tables

    Gets a list of table names for a aligned_volume_name

    post_annotation

    Post one or more new annotations to a table in the AnnotationEngine.

    post_annotation_df

    Post one or more new annotations to a table in the AnnotationEngine.

    process_position_columns

    Process a dataframe into a list of dictionaries

    raise_for_status

    Raises requests.HTTPError, if one occurred.

    stage_annotations

    Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema.

    update_annotation

    Update one or more new annotations to a table in the AnnotationEngine.

    update_annotation_df

    Update one or more annotations to a table in the AnnotationEngine using a

    update_metadata

    Update the metadata on an existing table

    upload_staged_annotations

    Upload annotations directly from an Annotation Guide object.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.create_table","title":"create_table(table_name, schema_name, description, voxel_resolution, reference_table=None, track_target_id_updates=None, flat_segmentation_source=None, user_id=None, aligned_volume_name=None, write_permission='PRIVATE', read_permission='PUBLIC', notice_text=None)","text":"

    Creates a new data table based on an existing schema

    Parameters:

    Name Type Description Default table_name str

    Name of the new table. Cannot be the same as an existing table

    required schema_name str

    Name of the schema for the new table.

    required description str

    Human readable description for what is in the table. Should include information about who generated the table What data it covers, and how it should be interpreted. And who should you talk to if you want to use it. An Example: a manual synapse table to detect chandelier synapses on 81 PyC cells with complete AISs [created by Agnes - agnesb@alleninstitute.org, uploaded by Forrest]

    required voxel_resolution List[float]

    voxel resolution points will be uploaded in, typically nm, i.e [1,1,1] means nanometers [4,4,40] would be 4nm, 4nm, 40nm voxels

    required reference_table str

    If the schema you are using is a reference schema Meaning it is an annotation of another annotation. Then you need to specify what the target table those annotations are in.

    None track_target_id_updates bool

    Indicates whether to automatically update reference table's foreign key if target annotation table row is updated.

    None flat_segmentation_source str

    the source to a flat segmentation that corresponds to this table i.e. precomputed:\\gs:\\mybucket his_tables_annotation

    None user_id int

    If you are uploading this schema on someone else's behalf and you want to link this table with their ID, you can specify it here Otherwise, the table will be created with your userID in the user_id column.

    None aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None write_permission str

    What permissions to give the table for writing. One of PRIVATE: only you can write to this table (DEFAULT) GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable

    'PRIVATE' read_permission str

    What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data (DEFAULT)

    'PUBLIC' notice_text str

    Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None, no text. If you want to remove text, send empty string.

    None

    Returns:

    Type Description json

    Response JSON

    Examples:

    Basic annotation table:

    description = \"Some description about the table\"\nvoxel_res = [4,4,40]\nclient.create_table(\"some_synapse_table\", \"synapse\", description, voxel_res)\n
    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_annotation","title":"delete_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"

    Delete one or more annotations in a table. Annotations that are deleted are recorded as 'non-valid' but are not physically removed from the table.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required annotation_ids (dict or list)

    A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON: a list of new annotation IDs.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_table","title":"delete_table(table_name, aligned_volume_name=None)","text":"

    Marks a table for deletion requires super admin privileges

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation","title":"get_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"

    Retrieve an annotation or annotations by id(s) and table name.

    Parameters:

    Name Type Description Default table_name str

    Name of the table

    required annotation_ids int or iterable

    ID or IDS of the annotation to retreive

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description list

    Annotation data

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation_count","title":"get_annotation_count(table_name, aligned_volume_name=None)","text":"

    Get number of annotations in a table

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description int

    number of annotations

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_table_metadata","title":"get_table_metadata(table_name, aligned_volume_name=None)","text":"

    Get metadata about a table

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    metadata about table

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_tables","title":"get_tables(aligned_volume_name=None)","text":"

    Gets a list of table names for a aligned_volume_name

    Parameters:

    Name Type Description Default aligned_volume_name str or None

    Name of the aligned_volume, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client

    None

    Returns:

    Type Description list

    List of table names

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation","title":"post_annotation(table_name, data, aligned_volume_name=None)","text":"

    Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations refer to 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required data (dict or list)

    A list of (or a single) dict of schematized annotation data matching the target table.

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation_df","title":"post_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"

    Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations see 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required df DataFrame

    A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.

    required position_columns Optional[Union[Iterable[str], Mapping[str, str]]]

    if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.process_position_columns","title":"process_position_columns(df, position_columns) staticmethod","text":"

    Process a dataframe into a list of dictionaries

    Parameters:

    Name Type Description Default df DataFrame

    Dataframe to process

    required position_columns Optional[Union[Iterable[str], Mapping[str, str]]]

    See .post_annotation_df

    required

    Returns:

    Type Description dict

    Annotations ready for posting

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.raise_for_status","title":"raise_for_status(r, log_warning=True) staticmethod","text":"

    Raises requests.HTTPError, if one occurred.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.stage_annotations","title":"stage_annotations(table_name=None, schema_name=None, update=False, id_field=False, table_resolution=None, annotation_resolution=None)","text":"

    Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema. StagedAnnotation objects can be uploaded directly with upload_staged_annotations.

    Parameters:

    Name Type Description Default table_name str

    Table name to stage annotations for, by default None.

    None schema_name str

    Schema name to use to make annotations. Only needed if the table_name is not set, by default None

    None update bool

    Set to True if individual annotations are going to be updated, by default False.

    False id_field bool

    Set to True if id fields are to be specified. Not needed if update is True, which always needs id fields. Optional, by default False

    False table_resolution list - like or None

    Voxel resolution of spatial points in the table in nanometers. This is found automatically from the info service if a table name is provided, by default None. If annotation_resolution is also set, this allows points to be scaled correctly for the table.

    None annotation_resolution list - like

    Voxel resolution of spatial points provided by the user when creating annotations. If the table resolution is also available (manually or from the info service), annotations are correctly rescaled for the volume. By default, None.

    None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation","title":"update_annotation(table_name, data, aligned_volume_name=None)","text":"

    Update one or more new annotations to a table in the AnnotationEngine. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.

    Notes

    If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required data (dict or list)

    A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON: a list of new annotation IDs.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation_df","title":"update_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"

    Update one or more annotations to a table in the AnnotationEngine using a dataframe as format. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.

    Notes

    If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required df DataFrame

    A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.

    required position_columns Iterable[str] or Mapping[str, str] or None

    if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_metadata","title":"update_metadata(table_name, description=None, flat_segmentation_source=None, read_permission=None, write_permission=None, user_id=None, notice_text=None, aligned_volume_name=None)","text":"

    Update the metadata on an existing table

    Parameters:

    Name Type Description Default table_name str required description str

    Defaults to the None type, in which case no change will be made to this metadata field.

    None flat_segmentation_source str

    Defaults to the None type, in which case no change will be made to this metadata field.

    None read_permission str

    What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data Defaults to the None type, in which case no change will be made to this metadata field.

    None write_permission str

    What permissions to give the table for writing. One of PRIVATE: only you can write to this table GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable Defaults to the None type, in which case no change will be made to this metadata field.

    None user_id int

    Note, if you use this you will not be able to update the metadata on this table any longer and depending on permissions may not be able to read or write to it Defaults to the None type, in which case no change will be made to this metadata field.

    None notice_text str

    Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. If you wish to remove the notice_text pass an empty string, or \"none\", \"None\", \"NONE\", or any other capitaliztion of the word \"none\". Defaults to the None type, in which case no change will be made to this metadata field.

    None aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.upload_staged_annotations","title":"upload_staged_annotations(staged_annos, aligned_volume_name=None)","text":"

    Upload annotations directly from an Annotation Guide object. This method uses the options specified in the object, including table name and if the annotation is an update or not.

    Parameters:

    Name Type Description Default staged_annos AnnotationGuide

    AnnotationGuide object with a specified table name and a collection of annotations already filled in.

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description List or dict

    If new annotations are posted, a list of ids. If annotations are being updated, a dictionary with the mapping from old ids to new ids.

    "},{"location":"api/auth/","title":"client.auth","text":"

    Client to find and use auth tokens to access the other services.

    Methods:

    Name Description get_group_users

    Get users in a group

    get_new_token

    Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.

    get_token

    Load a token with a given key the specified token file

    get_tokens

    Get the tokens setup for this users

    get_user_information

    Get user data.

    save_token

    Conveniently save a token in the correct format.

    setup_token

    Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment.

    Attributes:

    Name Type Description request_header

    Formatted request header with the specified token

    token

    Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.

    "},{"location":"api/auth/#caveclient.auth.AuthClient.request_header","title":"request_header property","text":"

    Formatted request header with the specified token

    "},{"location":"api/auth/#caveclient.auth.AuthClient.token","title":"token property writable","text":"

    Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.

    "},{"location":"api/auth/#caveclient.auth.AuthClient.get_group_users","title":"get_group_users(group_id)","text":"

    Get users in a group

    Parameters:

    Name Type Description Default group_id int

    ID value for a given group

    required

    Returns:

    Type Description list

    List of dicts of user ids. Returns empty list if group does not exist.

    "},{"location":"api/auth/#caveclient.auth.AuthClient.get_new_token","title":"get_new_token(open=False, no_text=False)","text":"

    Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.

    Parameters:

    Name Type Description Default open bool

    If True, opens a web browser to the web page where you can generate a new token.

    False"},{"location":"api/auth/#caveclient.auth.AuthClient.get_token","title":"get_token(token_key=None)","text":"

    Load a token with a given key the specified token file

    Parameters:

    Name Type Description Default token_key str or None

    key in the token file JSON, by default None. If None, uses 'token'.

    None"},{"location":"api/auth/#caveclient.auth.AuthClient.get_tokens","title":"get_tokens()","text":"

    Get the tokens setup for this users

    Returns:

    Type Description list[dict]:

    a list of dictionary of tokens, each with the keys \"id\": the id of this token \"token\": the token (str) \"user_id\": the users id (should be your ID)

    "},{"location":"api/auth/#caveclient.auth.AuthClient.get_user_information","title":"get_user_information(user_ids)","text":"

    Get user data.

    Parameters:

    Name Type Description Default user_ids list of int

    user_ids to look up

    required"},{"location":"api/auth/#caveclient.auth.AuthClient.save_token","title":"save_token(token=None, token_key=default_token_key, overwrite=False, token_file=None, switch_token=True, write_to_server_file=True, ignore_readonly=True, local_server=True)","text":"

    Conveniently save a token in the correct format.

    After getting a new token by following the instructions in authclient.get_new_token(), you can save it with a fully default configuration by running:

    token = 'my_shiny_new_token'

    authclient.save_token(token=token)

    Now on next load, authclient=AuthClient() will make an authclient instance using this token. If you would like to specify more information about the json file where the token will be stored, see the parameters below.

    Parameters:

    Name Type Description Default token str

    New token to save, by default None

    None token_key str

    Key for the token in the token_file json, by default \"token\"

    default_token_key overwrite bool

    Allow an existing token to be changed, by default False

    False token_file str

    Path to the token file, by default None. If None, uses the default file location specified above.

    None switch_token bool

    If True, switch the auth client over into using the new token, by default True

    True write_to_server_file bool

    If True, will write token to a server specific file to support this machine interacting with multiple auth servers.

    True ignore_readonly bool

    If True, will only attempt to save a token if the directory is writeable.

    True local_server bool

    If True, saves the token to the local server file as well.

    True"},{"location":"api/auth/#caveclient.auth.AuthClient.setup_token","title":"setup_token(make_new=True, open=True)","text":"

    Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.

    Parameters:

    Name Type Description Default make_new bool

    If True, will make a new token, else prompt you to open a page to retrieve an existing token.

    True open bool

    If True, opens a web browser to the web page where you can retrieve a token.

    True"},{"location":"api/chunkedgraph/","title":"client.chunkedgraph","text":"

    Client for interacting with the chunkedgraph.

    Methods:

    Name Description get_roots

    Get the root ID for a list of supervoxels.

    get_root_id

    Get the root ID for a specified supervoxel.

    get_minimal_covering_nodes

    Get the minimal covering nodes for a list of root IDs.

    get_merge_log

    Get the merge log (splits and merges) for an object.

    get_change_log

    Get the change log (splits and merges) for an object.

    get_user_operations

    Get operation details for a user ID. Currently, this is only available to

    get_tabular_change_log

    Get a detailed changelog for neurons.

    get_leaves_many

    Get all supervoxels for a list of root IDs.

    get_leaves

    Get all supervoxels for a root ID.

    do_merge

    Perform a merge on the chunked graph.

    undo_operation

    Undo an operation.

    execute_split

    Execute a multicut split based on points or supervoxels.

    preview_split

    Get supervoxel connected components from a preview multicut split.

    get_children

    Get the children of a node in the chunked graph hierarchy.

    get_contact_sites

    Get contacts for a root ID.

    find_path

    Find a path between two locations on a root ID using the level 2 chunked

    get_subgraph

    Get subgraph of root id within a bounding box.

    level2_chunk_graph

    Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.

    remesh_level2_chunks

    Submit specific level 2 chunks to be remeshed in case of a problem.

    get_operation_details

    Get the details of a list of operations.

    get_lineage_graph

    Returns the lineage graph for a root ID, optionally cut off in the past or

    get_latest_roots

    Returns root IDs that are related to the given root_id at a given

    get_original_roots

    Returns root IDs that are the latest successors of a given root ID.

    is_latest_roots

    Check whether these root IDs are still a root at this timestamp.

    suggest_latest_roots

    Suggest latest roots for a given root id, based on overlap of component

    is_valid_nodes

    Check whether nodes are valid for given timestamp range.

    get_root_timestamps

    Retrieves timestamps when roots where created.

    get_past_ids

    For a set of root IDs, get the list of IDs at a past or future time point

    get_delta_roots

    Get the list of roots that have changed between timetamp_past and

    get_oldest_timestamp

    Get the oldest timestamp in the database.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    timestamp Optional[datetime]

    The default timestamp for queries which expect a timestamp. If None, uses the

    segmentation_info

    Complete segmentation metadata

    base_resolution

    MIP 0 resolution for voxels assumed by the ChunkedGraph

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.timestamp","title":"timestamp: Optional[datetime.datetime] property writable","text":"

    The default timestamp for queries which expect a timestamp. If None, uses the current time.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.segmentation_info","title":"segmentation_info property","text":"

    Complete segmentation metadata

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.base_resolution","title":"base_resolution property","text":"

    MIP 0 resolution for voxels assumed by the ChunkedGraph

    Returns:

    Type Description list

    3-long list of x/y/z voxel dimensions in nm

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_roots","title":"get_roots(supervoxel_ids, timestamp=None, stop_layer=None)","text":"

    Get the root ID for a list of supervoxels.

    Parameters:

    Name Type Description Default supervoxel_ids list or np.array of int

    Supervoxel IDs to look up.

    required timestamp datetime

    UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.

    None stop_layer int or None

    If True, looks up IDs only up to a given stop layer. Default is None.

    None

    Returns:

    Type Description np.array of np.uint64

    Root IDs containing each supervoxel.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_id","title":"get_root_id(supervoxel_id, timestamp=None, level2=False)","text":"

    Get the root ID for a specified supervoxel.

    Parameters:

    Name Type Description Default supervoxel_id int

    Supervoxel id value

    required timestamp datetime

    UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description int64

    Root ID containing the supervoxel.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_minimal_covering_nodes","title":"get_minimal_covering_nodes(node_ids)","text":"

    Get the minimal covering nodes for a list of root IDs.

    Parameters:

    Name Type Description Default nodes_ids Iterable of int or np.int64

    List of root IDs to query.

    required

    Returns:

    Type Description np.array of np.int64:

    List of PCG node_ids that minimally and exactly cover the input nodes

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_merge_log","title":"get_merge_log(root_id)","text":"

    Get the merge log (splits and merges) for an object.

    Parameters:

    Name Type Description Default root_id int

    Object root ID to look up.

    required

    Returns:

    Type Description list

    List of merge events in the history of the object.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_change_log","title":"get_change_log(root_id, filtered=True)","text":"

    Get the change log (splits and merges) for an object.

    Parameters:

    Name Type Description Default root_id int

    Object root ID to look up.

    required filtered bool

    Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query object root_id (filtered=False). Defaults to True.

    True

    Returns:

    Type Description dict

    Dictionary summarizing split and merge events in the object history, containing the following keys:

    • \"n_merges\": int
      • Number of merges
    • \"n_splits\": int
      • Number of splits
    • \"operations_ids\": list of int
      • Identifiers for each operation
    • \"past_ids\": list of int
      • Previous root ids for this object
    • \"user_info\": dict of dict
      • Dictionary keyed by user (string) to a dictionary specifying how many merges and splits that user performed on this object
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_user_operations","title":"get_user_operations(user_id, timestamp_start, include_undo=True, timestamp_end=None)","text":"

    Get operation details for a user ID. Currently, this is only available to admins.

    Parameters:

    Name Type Description Default user_id int

    User ID to query (use 0 for all users (admin only)).

    required timestamp_start datetime

    Timestamp to start filter (UTC).

    required include_undo bool

    Whether to include undos. Defaults to True.

    True timestamp_end datetime

    Timestamp to end filter (UTC). Defaults to now.

    None

    Returns:

    Type Description DataFrame

    DataFrame including the following columns:

    • \"operation_id\": int
      • Identifier for the operation.
    • \"timestamp\": datetime.datetime
      • Timestamp of the operation.
    • \"user_id\": int
      • User who performed the operation.
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_tabular_change_log","title":"get_tabular_change_log(root_ids, filtered=True)","text":"

    Get a detailed changelog for neurons.

    Parameters:

    Name Type Description Default root_ids list of int

    Object root IDs to look up.

    required filtered bool

    Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query objects in root_ids (filtered=False). Defaults to True.

    True

    Returns:

    Type Description dict of pd.DataFrame

    The keys are the root IDs, and the values are DataFrames with the following columns and datatypes:

    • \"operation_id\": int
      • Identifier for the operation.
    • \"timestamp\": int
      • Timestamp of the operation, provided in milliseconds. To convert to datetime, use datetime.datetime.utcfromtimestamp(timestamp/1000).
    • \"user_id\": int
      • User who performed the operation.
    • \"before_root_ids: list of int
      • Root IDs of objects that existed before the operation.
    • \"after_root_ids: list of int
      • Root IDs of objects created by the operation. Note that this only records the root id that was kept as part of the query object, so there will only be one in this list.
    • \"is_merge\": bool
      • Whether the operation was a merge.
    • \"user_name\": str
      • Name of the user who performed the operation.
    • \"user_affiliation\": str
      • Affiliation of the user who performed the operation.
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves_many","title":"get_leaves_many(root_ids, bounds=None, stop_layer=None)","text":"

    Get all supervoxels for a list of root IDs.

    Parameters:

    Name Type Description Default root_ids Iterable

    Root IDs to query.

    required bounds

    If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.

    None stop_layer int

    If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).

    None

    Returns:

    Type Description dict

    Dict relating ids to contacts

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves","title":"get_leaves(root_id, bounds=None, stop_layer=None)","text":"

    Get all supervoxels for a root ID.

    Parameters:

    Name Type Description Default root_id int

    Root ID to query.

    required bounds

    If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.

    None stop_layer int

    If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).

    None

    Returns:

    Type Description np.array of np.int64

    Array of supervoxel IDs (or node ids if stop_layer>1).

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.do_merge","title":"do_merge(supervoxels, coords, resolution=(4, 4, 40))","text":"

    Perform a merge on the chunked graph.

    Parameters:

    Name Type Description Default supervoxels iterable

    An N-long list of supervoxels to merge.

    required coords array

    An Nx3 array of coordinates of the supervoxels in units of resolution.

    required resolution tuple

    What to multiply coords by to get nanometers. Defaults to (4,4,40).

    (4, 4, 40)"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.undo_operation","title":"undo_operation(operation_id)","text":"

    Undo an operation.

    Parameters:

    Name Type Description Default operation_id int

    Operation ID to undo.

    required

    Returns:

    Type Description dict"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.execute_split","title":"execute_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None)","text":"

    Execute a multicut split based on points or supervoxels.

    Parameters:

    Name Type Description Default source_points array or list

    Nx3 list or array of 3d points in nm coordinates for source points (red).

    required sink_points array or list

    Mx3 list or array of 3d points in nm coordinates for sink points (blue).

    required root_id int

    Root ID of object to do split preview.

    required source_supervoxels (array, list or None)

    If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None sink_supervoxels (array, list or None)

    If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None

    Returns:

    Name Type Description operation_id int

    Unique ID of the split operation

    new_root_ids list of int

    List of new root IDs resulting from the split operation.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.preview_split","title":"preview_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None, return_additional_ccs=False)","text":"

    Get supervoxel connected components from a preview multicut split.

    Parameters:

    Name Type Description Default source_points array or list

    Nx3 list or array of 3d points in nm coordinates for source points (red).

    required sink_points array or list

    Mx3 list or array of 3d points in nm coordinates for sink points (blue).

    required root_id int

    Root ID of object to do split preview.

    required source_supervoxels (array, list or None)

    If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None sink_supervoxels (array, list or None)

    If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None return_additional_ccs bool

    If True, returns any additional connected components beyond the ones with source and sink points. In most situations, this can be ignored. By default, False.

    False

    Returns:

    Name Type Description source_connected_component list

    Supervoxel IDs in the component with the most source points.

    sink_connected_component list

    Supervoxel IDs in the component with the most sink points.

    successful_split bool

    True if the split worked.

    other_connected_components (optional) : list of lists of int

    List of lists of supervoxel IDs for any other resulting connected components. Only returned if return_additional_ccs is True.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_children","title":"get_children(node_id)","text":"

    Get the children of a node in the chunked graph hierarchy.

    Parameters:

    Name Type Description Default node_id int

    Node ID to query.

    required

    Returns:

    Type Description np.array of np.int64

    IDs of child nodes.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_contact_sites","title":"get_contact_sites(root_id, bounds, calc_partners=False)","text":"

    Get contacts for a root ID.

    Parameters:

    Name Type Description Default root_id int

    Root ID to query.

    required bounds

    Bounds within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]] for which to find contacts. Running this query without bounds is too slow.

    required calc_partners bool

    If True, get partner root IDs. By default, False.

    False

    Returns:

    Type Description dict

    Dict relating ids to contacts

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.find_path","title":"find_path(root_id, src_pt, dst_pt, precision_mode=False)","text":"

    Find a path between two locations on a root ID using the level 2 chunked graph.

    Parameters:

    Name Type Description Default root_id int

    Root ID to query.

    required src_pt array

    3-element array of xyz coordinates in nm for the source point.

    required dst_pt array

    3-element array of xyz coordinates in nm for the destination point.

    required precision_mode bool

    Whether to perform the search in precision mode. Defaults to False.

    False

    Returns:

    Name Type Description centroids_list array

    Array of centroids along the path.

    l2_path np.array of int

    Array of level 2 chunk IDs along the path.

    failed_l2_ids np.array of int

    Array of level 2 chunk IDs that failed to find a path.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_subgraph","title":"get_subgraph(root_id, bounds)","text":"

    Get subgraph of root id within a bounding box.

    Parameters:

    Name Type Description Default root_id int

    Root (or any node ID) of chunked graph to query.

    required bounds array

    3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates.

    required

    Returns:

    Type Description np.array of np.int64

    Node IDs in the subgraph.

    np.array of np.double

    Affinities of edges in the subgraph.

    np.array of np.int32

    Areas of nodes in the subgraph.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.level2_chunk_graph","title":"level2_chunk_graph(root_id, bounds=None)","text":"

    Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.

    Parameters:

    Name Type Description Default root_id int

    Root id of object

    required bounds array

    3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates (use client.chunkedgraph.base_resolution to view this default resolution for your chunkedgraph client). Note that the result will include any level 2 nodes which have chunk boundaries within some part of this bounding box, meaning that the representative point for a given level 2 node could still be slightly outside of these bounds. If None, returns all level 2 chunks for the root ID.

    None

    Returns:

    Type Description list of list

    Edge list for level 2 chunked graph. Each element of the list is an edge, and each edge is a list of two node IDs (source and target).

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.remesh_level2_chunks","title":"remesh_level2_chunks(chunk_ids)","text":"

    Submit specific level 2 chunks to be remeshed in case of a problem.

    Parameters:

    Name Type Description Default chunk_ids list

    List of level 2 chunk IDs.

    required"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_operation_details","title":"get_operation_details(operation_ids)","text":"

    Get the details of a list of operations.

    Parameters:

    Name Type Description Default operation_ids Iterable[int]

    List/array of operation IDs.

    required

    Returns:

    Type Description dict of str to dict

    A dict of dicts of operation info, keys are operation IDs (as strings), values are a dictionary of operation info for the operation. These dictionaries contain the following keys:

    • \"added_edges\"/\"removed_edges\": list of list of int
      • List of edges added (if a merge) or removed (if a split) by this operation. Each edge is a list of two supervoxel IDs (source and target).
    • \"roots\": list of int
      • List of root IDs that were created by this operation.
    • \"sink_coords\": list of list of int
      • List of sink coordinates for this operation. The sink is one of the points placed by the user when specifying the operation. Each sink coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
    • \"source_coords\": list of list of int
      • List of source coordinates for this operation. The source is one of the points placed by the user when specifying the operation. Each source coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
    • \"timestamp\": str
      • Timestamp of the operation.
    • \"user\": str
      • User ID number who performed the operation (as a string).
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_lineage_graph","title":"get_lineage_graph(root_id, timestamp_past=None, timestamp_future=None, as_nx_graph=False, exclude_links_to_future=False, exclude_links_to_past=False)","text":"

    Returns the lineage graph for a root ID, optionally cut off in the past or the future.

    Each change in the chunked graph creates a new root ID for the object after that change. This function returns a graph of all root IDs for a given object, tracing the history of the object in terms of merges and splits.

    Parameters:

    Name Type Description Default root_id int

    Object root ID.

    required timestamp_past datetime or None

    Cutoff for the lineage graph backwards in time. By default, None.

    None timestamp_future datetime or None

    Cutoff for the lineage graph going forwards in time. By default, uses the timestamp property for this client, which defaults to the current time.

    None as_nx_graph

    If True, a NetworkX graph is returned.

    False exclude_links_to_future

    If True, links from nodes before timestamp_future to after timestamp_future are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.

    False exclude_links_to_past

    If True, links from nodes before timestamp_past to after timestamp_past are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.

    False

    Returns:

    Type Description dict

    Dictionary describing the lineage graph and operations for the root ID. Not returned if as_nx_graph is True. The dictionary contains the following keys:

    • \"directed\" : bool
      • Whether the graph is directed.
    • \"graph\" : dict
      • Dictionary of graph attributes.
    • \"links\" : list of dict
      • Each element of the list is a dictionary describing an edge in the lineage graph as \"source\" and \"target\" keys.
    • \"multigraph\" : bool
      • Whether the graph is a multigraph.
    • \"nodes\" : list of dict
      • Each element of the list is a dictionary describing a node in the lineage graph, usually with \"id\", \"timestamp\", and \"operation_id\" keys.
    DiGraph

    NetworkX directed graph of the lineage graph. Only returned if as_nx_graph is True.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_latest_roots","title":"get_latest_roots(root_id, timestamp=None, timestamp_future=None)","text":"

    Returns root IDs that are related to the given root_id at a given timestamp. Can be used to find the \"latest\" root IDs associated with an object.

    Parameters:

    Name Type Description Default root_id int

    Object root ID.

    required timestamp datetime or None

    Timestamp of where to query IDs from. If None, uses the timestamp property for this client, which defaults to the current time.

    None timestamp_future datetime or None

    DEPRECATED name, use timestamp instead. Timestamp to suggest IDs from (note can be in the past relative to the root). By default, None.

    None

    Returns:

    Type Description ndarray

    1d array with all latest successors.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_original_roots","title":"get_original_roots(root_id, timestamp_past=None)","text":"

    Returns root IDs that are the latest successors of a given root ID.

    Parameters:

    Name Type Description Default root_id int

    Object root ID.

    required timestamp_past datetime or None

    Cutoff for the search going backwards in time. By default, None.

    None

    Returns:

    Type Description ndarray

    1d array with all latest successors.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_latest_roots","title":"is_latest_roots(root_ids, timestamp=None)","text":"

    Check whether these root IDs are still a root at this timestamp.

    Parameters:

    Name Type Description Default root_ids array-like of int

    Root IDs to check.

    required timestamp datetime

    Timestamp to check whether these IDs are valid root IDs in the chunked graph. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description np.array of bool

    Array of whether these are valid root IDs.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.suggest_latest_roots","title":"suggest_latest_roots(root_id, timestamp=None, stop_layer=None, return_all=False, return_fraction_overlap=False)","text":"

    Suggest latest roots for a given root id, based on overlap of component chunk IDs. Note that edits change chunk IDs, and so this effectively measures the fraction of unchanged chunks at a given chunk layer, which sets the size scale of chunks. Higher layers are coarser.

    Parameters:

    Name Type Description Default root_id int

    Root ID of the potentially outdated object.

    required timestamp datetime

    Datetime at which \"latest\" roots are being computed, by default None. If None, uses the timestamp property for this client, which defaults to the current time. Note that this has to be a timestamp after the creation of the root_id.

    None stop_layer int

    Chunk level at which to compute overlap, by default None. No value will take the 4th from the top layer, which emphasizes speed and works well for larger objects. Lower values are slower but more fine-grained. Values under 2 (i.e. supervoxels) are not recommended except in extremely fine grained scenarios.

    None return_all bool

    If True, return all current IDs sorted from most overlap to least, by default False. If False, only the top is returned.

    False return_fraction_overlap bool

    If True, return all fractions sorted by most overlap to least, by default False. If False, only the top value is returned.

    False"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_valid_nodes","title":"is_valid_nodes(node_ids, start_timestamp=None, end_timestamp=None)","text":"

    Check whether nodes are valid for given timestamp range.

    Valid is defined as existing in the chunked graph. This makes no statement about these IDs being roots, supervoxel or anything in-between. It also does not take into account whether a root ID has since been edited.

    Parameters:

    Name Type Description Default node_ids array-like of int

    Node IDs to check.

    required start_timestamp datetime

    Timestamp to check whether these IDs were valid after this timestamp. Defaults to None (assumes now).

    None end_timestamp datetime

    Timestamp to check whether these IDs were valid before this timestamp. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description np.array of bool

    Array of whether these are valid IDs.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_timestamps","title":"get_root_timestamps(root_ids, latest=False, timestamp=None)","text":"

    Retrieves timestamps when roots where created.

    Parameters:

    Name Type Description Default root_ids

    Iterable of root IDs to query.

    required latest bool

    If False, returns the first timestamp that the root_id was valid for each root ID. If True, returns the newest/latest timestamp for each root ID. Note, this will return the timestamp at which the query was run when the root is currently valid. This means that you will get a different answer if you make this same query at a later time if you don't specify a timestamp parameter.

    False timestamp datetime

    Timestamp to query when using latest=True. Use this to provide consistent results for a particular timestamp. If an ID is still valid at a point in the future past this timestamp, the query will still return this timestamp as the latest moment in time. An error will occur if you provide a timestamp for which the root ID is not valid. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description np.array of datetime.datetime

    Array of timestamps when root_ids were created.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_past_ids","title":"get_past_ids(root_ids, timestamp_past=None, timestamp_future=None)","text":"

    For a set of root IDs, get the list of IDs at a past or future time point that could contain parts of the same object.

    Parameters:

    Name Type Description Default root_ids Iterable of int

    Iterable of root IDs to query.

    required timestamp_past datetime or None

    Time of a point in the past for which to look up root ids. Default is None.

    None timestamp_future datetime or None

    Time of a point in the future for which to look up root ids. Not implemented on the server currently. Default is None.

    None

    Returns:

    Type Description dict

    Dict with keys \"future_id_map\" and \"past_id_map\". Each is a dict whose keys are the supplied root_ids and whose values are the list of related root IDs at timestamp_past/timestamp_future.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_delta_roots","title":"get_delta_roots(timestamp_past, timestamp_future=datetime.datetime.now(datetime.timezone.utc))","text":"

    Get the list of roots that have changed between timetamp_past and timestamp_future.

    Parameters:

    Name Type Description Default timestamp_past datetime

    Past timepoint to query

    required timestamp_future datetime

    Future timepoint to query. Defaults to datetime.datetime.now(datetime.timezone.utc).

    now(utc)

    Returns:

    Name Type Description old_roots np.ndarray of np.int64

    Roots that have expired in that interval.

    new_roots np.ndarray of np.int64

    Roots that are new in that interval.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_oldest_timestamp","title":"get_oldest_timestamp()","text":"

    Get the oldest timestamp in the database.

    Returns:

    Type Description datetime

    Oldest timestamp in the database.

    "},{"location":"api/client/","title":"caveclient.CAVEclient","text":"

    A manager for all clients sharing common datastack and authentication information.

    This client wraps all the other clients and keeps track of the things that need to be consistent across them. To instantiate a client:

    from caveclient import CAVEclient\n\nclient = CAVEclient(datastack_name='my_datastack',\n                    server_address='www.myserver.com',\n                    auth_token_file='~/.mysecrets/secrets.json')\n

    Then

    • client.annotation is an AnnotationClient (see client.annotation)
    • client.auth is an AuthClient (see client.auth)
    • client.chunkedgraph is a ChunkedGraphClient (see client.chunkedgraph)
    • client.info is an InfoServiceClient (see client.info)
    • client.l2cache is an L2CacheClient (see client.l2cache)
    • client.materialize is a MaterializationClient (see client.materialize)
    • client.skeleton is a SkeletonClient (see client.skeleton)
    • client.schema is a SchemaClient (see client.schema)
    • client.state is a neuroglancer JSONService (see client.state)

    All subclients are loaded lazily and share the same datastack name, server address, and auth tokens where used.

    Parameters:

    Name Type Description Default datastack_name str

    Datastack name for the services. Almost all services need this and will not work if it is not passed.

    None server_address str or None

    URL of the framework server. If None, chooses the default server global.daf-apis.com. Optional, defaults to None.

    None auth_token_file str or None

    Path to a json file containing the auth token. If None, uses the default location. See Auth client documentation. Optional, defaults to None.

    default_token_file auth_token_key str

    Dictionary key for the token in the the JSON file. Optional, default is 'token'.

    'token' auth_token str or None

    Direct entry of an auth token. If None, uses the file arguments to find the token. Optional, default is None.

    None max_retries int or None

    Sets the default number of retries on failed requests. Optional, by default 2.

    None pool_maxsize int or None

    Sets the max number of threads in a requests pool, although this value will be exceeded if pool_block is set to False. Optional, uses requests defaults if None.

    None pool_block

    If True, prevents the number of threads in a requests pool from exceeding the max size. Optional, uses requests defaults (False) if None.

    None desired_resolution Iterable[float] or None

    If given, should be a list or array of the desired resolution you want queries returned in useful for materialization queries.

    None info_cache

    Pre-computed info cache, bypassing the lookup of datastack info from the info service. Should only be used in cases where this information is cached and thus repetitive lookups can be avoided.

    None version Optional[int]

    The default materialization version of the datastack to use. If None, the latest version is used. Optional, defaults to None.

    None See Also

    set_session_defaults

    get_session_defaults

    Methods:

    Name Description change_auth

    Change the authentication token and reset services.

    Attributes:

    Name Type Description annotation AnnotationClient

    A client for the annotation service. See client.annotation

    auth AuthClient

    A client for the auth service. See client.auth for more information.

    chunkedgraph ChunkedGraphClient

    A client for the chunkedgraph service. See client.chunkedgraph

    datastack_name str

    The name of the datastack for the client.

    info InfoServiceClient

    A client for the info service. See client.info for more information.

    l2cache L2CacheClient

    A client for the L2 cache service. See client.l2cache

    materialize MaterializationClient

    A client for the materialization service. See client.materialize

    schema SchemaClient

    A client for the EM Annotation Schemas service. See client.schema

    server_address

    The server address for the client.

    skeleton SkeletonClient

    A client for the skeleton service. See client.skeleton

    state JSONService

    A client for the neuroglancer state service. See client.state

    timestamp Optional[datetime]

    The default timestamp to use for queries which rely on a timestamp.

    version Optional[int]

    The default materialization version of the datastack to use for queries which

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.annotation","title":"annotation: AnnotationClient property","text":"

    A client for the annotation service. See client.annotation for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.auth","title":"auth: AuthClient property","text":"

    A client for the auth service. See client.auth for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.chunkedgraph","title":"chunkedgraph: ChunkedGraphClient property","text":"

    A client for the chunkedgraph service. See client.chunkedgraph for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.datastack_name","title":"datastack_name: str property","text":"

    The name of the datastack for the client.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.info","title":"info: InfoServiceClient property","text":"

    A client for the info service. See client.info for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.l2cache","title":"l2cache: L2CacheClient property","text":"

    A client for the L2 cache service. See client.l2cache for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.materialize","title":"materialize: MaterializationClient property","text":"

    A client for the materialization service. See client.materialize for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.schema","title":"schema: SchemaClient property","text":"

    A client for the EM Annotation Schemas service. See client.schema for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.server_address","title":"server_address property","text":"

    The server address for the client.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.skeleton","title":"skeleton: SkeletonClient property","text":"

    A client for the skeleton service. See client.skeleton for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.state","title":"state: JSONService property","text":"

    A client for the neuroglancer state service. See client.state for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.timestamp","title":"timestamp: Optional[datetime] property","text":"

    The default timestamp to use for queries which rely on a timestamp.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.version","title":"version: Optional[int] property writable","text":"

    The default materialization version of the datastack to use for queries which expect a version. Also sets the timestamp to the corresponding timestamp of the version for queries which rely on a timestamp.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.change_auth","title":"change_auth(auth_token_file=None, auth_token_key=None, auth_token=None)","text":"

    Change the authentication token and reset services.

    Parameters:

    Name Type Description Default auth_token_file str

    New auth token json file path, by default None, which defaults to the existing state.

    None auth_token_key str

    New dictionary key under which the token is stored in the json file, by default None, which defaults to the existing state.

    None auth_token str

    Direct entry of a new token, by default None.

    None"},{"location":"api/config/","title":"Configuration","text":""},{"location":"api/config/#caveclient.set_session_defaults","title":"caveclient.set_session_defaults(max_retries=3, pool_block=False, pool_maxsize=10, backoff_factor=0.1, backoff_max=120, status_forcelist=(502, 503, 504))","text":"

    Set global default values to configure how all clients will communicate with servers. Should be done prior to initializing a client.

    Note that these values are only used when not set at the client level.

    Parameters:

    Name Type Description Default max_retries int

    The maximum number of retries each connection should attempt. Set to 0 to fail on the first retry.

    3 pool_block bool

    Whether the connection pool should block for connections.

    False pool_maxsize int

    The maximum number of connections to save in the pool.

    10 backoff_factor Union[float, int]

    A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). The query will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. For example, if the backoff_factor is 0.1, then will sleep for [0.0s, 0.2s, 0.4s, 0.8s, \u2026] between retries. No backoff will ever be longer than backoff_max.

    0.1 backoff_max Union[float, int]

    The maximum backoff time.

    120 status_forcelist Optional[Collection]

    A set of integer HTTP status codes that we should force a retry on.

    (502, 503, 504) Usage 
    from caveclient import set_session_defaults\n\nset_session_defaults(\n    max_retries=5, # would increase the default number of retries\n    backoff_factor=0.5, # would increase the default backoff factor between retries\n    backoff_max=240, # would increase the default maximum backoff time\n    status_forcelist=(502, 503, 504, 505), # would add 505 to the default list\n)\n\nset_session_defaults() # would revert all defaults to their original values\n
    Notes

    Calling this function will set the default values for all clients created after the call.

    Calling this function with any arguments missing will reset that value to the default value.

    See Also:

    urllib3.util.Retry

    requests.adapters.HTTPAdapter

    "},{"location":"api/config/#caveclient.get_session_defaults","title":"caveclient.get_session_defaults()","text":"

    Get the current default values for session configuration.

    Returns:

    Type Description dict

    Dictionary of current default values for session configuration.

    "},{"location":"api/datastack_lookup/","title":"Datastack lookup","text":""},{"location":"api/datastack_lookup/#caveclient.datastack_lookup.reset_server_address_cache","title":"caveclient.datastack_lookup.reset_server_address_cache(datastack, filename=None)","text":"

    Remove one or more datastacks from the datastack-to-server cache.

    Parameters:

    Name Type Description Default datastack str or list of str

    Datastack names to remove from the cache, by default None

    required filename str

    Name of the cache file, by default None

    None"},{"location":"api/info/","title":"client.info","text":"

    Client for interacting with the info service.

    Methods:

    Name Description annotation_endpoint

    AnnotationEngine endpoint for a dataset.

    get_aligned_volume_info

    Gets the info record for a aligned_volume

    get_datastack_info

    Gets the info record for a datastack

    get_datastacks

    Query which datastacks are available at the info service

    get_datastacks_by_aligned_volume

    Lookup what datastacks are associated with this aligned volume

    image_cloudvolume

    Generate a cloudvolume instance based on the image source, using authentication if needed and

    image_source

    Cloud path to the imagery for the dataset

    refresh_stored_data

    Reload the stored info values from the server.

    segmentation_cloudvolume

    Generate a cloudvolume instance based on the segmentation source, using authentication if needed and

    segmentation_source

    Cloud path to the chunkgraph-backed Graphene segmentation for a dataset

    synapse_segmentation_source

    Cloud path to the synapse segmentation for a dataset

    viewer_resolution

    Get the viewer resolution metadata for this datastack

    viewer_site

    Get the base Neuroglancer URL for the dataset

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.annotation_endpoint","title":"annotation_endpoint(datastack_name=None, use_stored=True)","text":"

    AnnotationEngine endpoint for a dataset.

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True

    Returns:

    Type Description str

    Location of the AnnotationEngine

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_aligned_volume_info","title":"get_aligned_volume_info(datastack_name=None, use_stored=True)","text":"

    Gets the info record for a aligned_volume

    Parameters:

    Name Type Description Default datastack_name str

    datastack_name to look up. If None, uses the one specified by the client. By default None

    None use_stored bool

    If True and the information has already been queried for that dataset, then uses the cached version. If False, re-queries the infromation. By default True

    True

    Returns:

    Type Description dict or None

    The complete info record for the aligned_volume

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastack_info","title":"get_datastack_info(datastack_name=None, use_stored=True)","text":"

    Gets the info record for a datastack

    Parameters:

    Name Type Description Default datastack_name str

    datastack to look up. If None, uses the one specified by the client. By default None

    None use_stored bool

    If True and the information has already been queried for that datastack, then uses the cached version. If False, re-queries the infromation. By default True

    True

    Returns:

    Type Description dict or None

    The complete info record for the datastack

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks","title":"get_datastacks()","text":"

    Query which datastacks are available at the info service

    Returns:

    Type Description list

    List of datastack names

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks_by_aligned_volume","title":"get_datastacks_by_aligned_volume(aligned_volume=None)","text":"

    Lookup what datastacks are associated with this aligned volume

    Args: aligned_volume (str, optional): aligned volume to lookup. Defaults to None.

    Raises: ValueError: if no aligned volume is specified

    Returns: list: a list of datastack string

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_cloudvolume","title":"image_cloudvolume(**kwargs)","text":"

    Generate a cloudvolume instance based on the image source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.

    Requires cloudvolume to be installed, which is not included by default.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_source","title":"image_source(datastack_name=None, use_stored=True, format_for='raw')","text":"

    Cloud path to the imagery for the dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True format_for 'raw', 'cloudvolume', or 'neuroglancer'

    Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.

    'raw'

    Returns:

    Type Description str

    Formatted cloud path to the flat segmentation

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.refresh_stored_data","title":"refresh_stored_data()","text":"

    Reload the stored info values from the server.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_cloudvolume","title":"segmentation_cloudvolume(use_client_secret=True, **kwargs)","text":"

    Generate a cloudvolume instance based on the segmentation source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.

    Requires cloudvolume to be installed, which is not included by default.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_source","title":"segmentation_source(datastack_name=None, format_for='raw', use_stored=True)","text":"

    Cloud path to the chunkgraph-backed Graphene segmentation for a dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True format_for 'raw', 'cloudvolume', or 'neuroglancer'

    Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"graphene://https://\" type path is used If 'neuroglancer', a \"graphene://https://\" type path is used, as needed by Neuroglancer.

    'raw'

    Returns:

    Type Description str

    Formatted cloud path to the Graphene segmentation

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.synapse_segmentation_source","title":"synapse_segmentation_source(datastack_name=None, use_stored=True, format_for='raw')","text":"

    Cloud path to the synapse segmentation for a dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the dataset to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True format_for 'raw', 'cloudvolume', or 'neuroglancer'

    Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.

    'raw'

    Returns:

    Type Description str

    Formatted cloud path to the synapse segmentation

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_resolution","title":"viewer_resolution(datastack_name=None, use_stored=True)","text":"

    Get the viewer resolution metadata for this datastack

    Parameters:

    Name Type Description Default datastack_name

    If None use the default one configured in the client

    None use_stored

    Use the cached value, if False go get a new value from server

    True

    Returns:

    Type Description array

    Voxel resolution as a len(3) np.array

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_site","title":"viewer_site(datastack_name=None, use_stored=True)","text":"

    Get the base Neuroglancer URL for the dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True

    Returns:

    Type Description str

    Base URL for the Neuroglancer viewer

    "},{"location":"api/l2cache/","title":"client.l2cache","text":"

    Client for interacting with the level2 cache service.

    Methods:

    Name Description cache_metadata

    Retrieves the meta data for the cache

    get_l2data

    Gets the attributed statistics data for L2 ids.

    get_l2data_table

    Gets the attributed statistics data for L2 ids, returned as a dataframe.

    has_cache

    Checks if the l2 cache is available for the dataset

    table_mapping

    Retrieves table mappings for l2 cache.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.cache_metadata","title":"cache_metadata()","text":"

    Retrieves the meta data for the cache

    Returns:

    Type Description dict

    keys are attribute names, values are datatypes

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data","title":"get_l2data(l2_ids, attributes=None)","text":"

    Gets the attributed statistics data for L2 ids.

    Parameters:

    Name Type Description Default l2_ids list or ndarray

    A list of level 2 ids.

    required attributes list

    A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:

    • area_nm2
    • chunk_intersect_count
    • max_dt_nm
    • mean_dt_nm
    • pca
    • pca_val
    • rep_coord_nm
    • size_nm3

    See the tutorial for more description.

    None

    Returns:

    Type Description dict

    Keys are l2 ids, values are data.

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data_table","title":"get_l2data_table(l2_ids, attributes=None, split_columns=True)","text":"

    Gets the attributed statistics data for L2 ids, returned as a dataframe.

    Parameters:

    Name Type Description Default l2_ids list or ndarray

    A list of level 2 ids.

    required attributes list

    A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:

    • area_nm2
    • chunk_intersect_count
    • max_dt_nm
    • mean_dt_nm
    • pca
    • pca_val
    • rep_coord_nm
    • size_nm3

    See the tutorial for more description.

    None split_columns bool

    Whether to split columns with multiple values into separate columns.

    True

    Returns:

    Type Description DataFrame

    A pandas dataframe with the requested attributes as columns and indexed by l2_id.

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.has_cache","title":"has_cache(datastack_name=None)","text":"

    Checks if the l2 cache is available for the dataset

    Parameters:

    Name Type Description Default datastack_name str

    The name of the datastack to check, by default None (if None, uses the client's datastack)

    None

    Returns:

    Type Description bool

    True if the l2 cache is available, False otherwise

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.table_mapping","title":"table_mapping()","text":"

    Retrieves table mappings for l2 cache.

    Returns:

    Type Description dict

    keys are pcg table names, values are dicts with fields l2cache_id and cv_path.

    "},{"location":"api/materialize/","title":"client.materialize","text":"

    Client for interacting with the materialization engine.

    Methods:

    Name Description get_table_metadata

    Get metadata about a table

    get_tables

    Gets a list of table names for a datastack

    get_tables_metadata

    Get metadata about tables

    get_timestamp

    Get datetime.datetime timestamp for a materialization version.

    get_unique_string_values

    Get unique string values for a table

    get_version_metadata

    Get metadata about a version

    get_versions

    Get the versions available

    get_versions_metadata

    Get the metadata for all the versions that are presently available and valid

    get_view_metadata

    Get metadata for a view

    get_view_schema

    Get schema for a view

    get_view_schemas

    Get schema for a view

    get_views

    Get all available views for a version

    ingest_annotation_table

    Trigger supervoxel lookup and root ID lookup of new annotations in a table.

    join_query

    Generic query on materialization tables

    live_live_query

    Beta method for querying cave annotation tables with root IDs and annotations

    live_query

    Generic query on materialization tables

    lookup_supervoxel_ids

    Trigger supervoxel lookups of new annotations in a table.

    map_filters

    Translate a list of filter dictionaries from a point in the

    most_recent_version

    Get the most recent version of materialization for this datastack name

    query_table

    Generic query on materialization tables

    query_view

    Generic query on a view

    synapse_query

    Convenience method for querying synapses.

    Attributes:

    Name Type Description cg_client

    The chunked graph client.

    datastack_name

    The name of the datastack.

    homepage HTML

    The homepage for the materialization engine.

    server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    tables TableManager

    The table manager for the materialization engine.

    version int

    The version of the materialization. Can be used to set up the

    views ViewManager

    The view manager for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.cg_client","title":"cg_client property","text":"

    The chunked graph client.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.datastack_name","title":"datastack_name property","text":"

    The name of the datastack.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.homepage","title":"homepage: HTML property","text":"

    The homepage for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.tables","title":"tables: TableManager property","text":"

    The table manager for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.version","title":"version: int property writable","text":"

    The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.

    Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.views","title":"views: ViewManager property","text":"

    The view manager for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_table_metadata","title":"get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)","text":"

    Get metadata about a table

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required datastack_name str or None

    Name of the datastack_name. If None, uses the one specified in the client.

    None version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to print out warnings to the logger. Defaults to True.

    True

    Returns:

    Type Description dict

    Metadata dictionary for table

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables","title":"get_tables(datastack_name=None, version=None)","text":"

    Gets a list of table names for a datastack

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client

    None version int or None

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None

    Returns:

    Type Description list

    List of table names

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables_metadata","title":"get_tables_metadata(datastack_name=None, version=None, log_warning=True)","text":"

    Get metadata about tables

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack_name. If None, uses the one specified in the client.

    None version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to print out warnings to the logger. Defaults to True.

    True

    Returns:

    Type Description dict

    Metadata dictionary for table

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_timestamp","title":"get_timestamp(version=None, datastack_name=None)","text":"

    Get datetime.datetime timestamp for a materialization version.

    Parameters:

    Name Type Description Default version int or None

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None datastack_name str or None

    Datastack name, by default None. If None, defaults to the value set in the client.

    None

    Returns:

    Type Description datetime

    Datetime when the materialization version was frozen.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_unique_string_values","title":"get_unique_string_values(table, datastack_name=None)","text":"

    Get unique string values for a table

    Parameters:

    Name Type Description Default table str

    Table to query

    required datastack_name Optional[str]

    Datastack to query. If None, uses the one specified in the client.

    None

    Returns:

    Type Description dict[str]

    A dictionary of column names and their unique values

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_version_metadata","title":"get_version_metadata(version=None, datastack_name=None)","text":"

    Get metadata about a version

    Parameters:

    Name Type Description Default version int or None

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None datastack_name str or None

    Datastack name, by default None. If None, defaults to the value set in the client.

    None

    Returns:

    Type Description dict

    Dictionary of metadata about the version

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions","title":"get_versions(datastack_name=None, expired=False)","text":"

    Get the versions available

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack, by default None. If None, uses the one specified in the client.

    None expired bool

    Whether to include expired versions, by default False.

    False

    Returns:

    Type Description dict

    Dictionary of versions available

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions_metadata","title":"get_versions_metadata(datastack_name=None, expired=False)","text":"

    Get the metadata for all the versions that are presently available and valid

    Parameters:

    Name Type Description Default datastack_name str or None

    Datastack name, by default None. If None, defaults to the value set in the client.

    None expired bool

    Whether to include expired versions, by default False.

    False

    Returns:

    Type Description list[dict]

    List of metadata dictionaries

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_metadata","title":"get_view_metadata(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"

    Get metadata for a view

    Parameters:

    Name Type Description Default view_name str

    Name of view to query.

    required materialization_version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description dict

    Metadata of view

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schema","title":"get_view_schema(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"

    Get schema for a view

    Parameters:

    Name Type Description Default view_name str

    Name of view to query.

    required materialization_version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description dict

    Schema of view.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schemas","title":"get_view_schemas(materialization_version=None, datastack_name=None, log_warning=True)","text":"

    Get schema for a view

    Parameters:

    Name Type Description Default materialization_version Optional[int]

    Version to query. If None, will use version set by client.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description dict

    Schema of view.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_views","title":"get_views(version=None, datastack_name=None)","text":"

    Get all available views for a version

    Parameters:

    Name Type Description Default version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None datastack_name str

    Datastack to query. If None, uses the one specified in the client.

    None

    Returns:

    Type Description list

    List of views

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.ingest_annotation_table","title":"ingest_annotation_table(table_name, datastack_name=None)","text":"

    Trigger supervoxel lookup and root ID lookup of new annotations in a table.

    Parameters:

    Name Type Description Default table_name str

    Table to trigger

    required datastack_name str

    Datastack to trigger it. Defaults to what is set in client.

    None

    Returns:

    Type Description dict

    Status code of response from server

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.join_query","title":"join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"

    Generic query on materialization tables

    Parameters:

    Name Type Description Default tables list of lists with length 2 or 'str'

    list of two lists: first entries are table names, second entries are the columns used for the join.

    required filter_in_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None

    None filter_out_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None

    None filter_equal_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None

    None filter_greater_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are exclusive upper-bound, by default None

    None filter_less_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are exclusive lower-bound, by default None

    None filter_greater_equal_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are inclusive upper-bound, by default None

    None filter_less_equal_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are inclusive lower-bound, by default None

    None filter_spatial_dict dict of dicts

    outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None.

    None filter_regex_dict dict of dicts

    outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None

    None select_columns dict of lists of str

    keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated.

    None offset int

    result offset to use. Defaults to None. Will only return top K results.

    None limit int

    maximum results to return (server will set upper limit, see get_server_config)

    None suffixes dict

    suffixes to use for duplicate columns, keys are table names, values are the suffix

    None datastack_name str

    datastack to query. If None defaults to one specified in client.

    None return_df bool

    whether to return as a dataframe default True, if False, data is returned as json (slower)

    True split_positions bool

    whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower)

    False materialization_version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None metadata bool

    toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True desired_resolution Iterable

    What resolution to convert position columns to. Defaults to None will use defaults.

    None random_sample int

    if given, will do a tablesample of the table to return that many annotations

    None log_warning bool

    Whether to log warnings, by default True

    True

    Returns:

    Type Description DataFrame

    a pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_live_query","title":"live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, allow_invalid_root_ids=False, random_sample=None, log_warning=True)","text":"

    Beta method for querying cave annotation tables with root IDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.

    Parameters:

    Name Type Description Default table str

    Principle table to query

    required timestamp datetime

    Timestamp to query

    required joins

    List of joins, where each join is a list of [table1,column1, table2, column2]

    None filter_in_dict

    A dictionary with tables as keys, values are dicts with column keys and list values to accept.

    None filter_out_dict

    A dictionary with tables as keys, values are dicts with column keys and list values to reject.

    None filter_equal_dict

    A dictionary with tables as keys, values are dicts with column keys and values to equate.

    None filter_greater_dict

    A dictionary with tables as keys, values are dicts with column keys and values as exclusive upper-bound.

    None filter_less_dict

    A dictionary with tables as keys, values are dicts with column keys and values as exclusive lower-bound.

    None filter_greater_equal_dict

    A dictionary with tables as keys, values are dicts with column keys and values as inclusive upper-bound.

    None filter_less_equal_dict

    A dictionary with tables as keys, values are dicts with column keys and values as inclusive lower-bound.

    None filter_spatial_dict

    A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds.

    None filter_regex_dict

    A dictionary with tables as keys, values are dicts with column keys and values of regex strings.

    None select_columns

    A dictionary with tables as keys, values are lists of columns to select.

    None offset int

    Value to offset query by.

    None limit int

    Limit of query.

    None datastack_name str

    Datastack to query. Defaults to set by client.

    None split_positions bool

    Whether to split positions into separate columns, True is faster.

    False metadata bool

    Whether to attach metadata to dataframe.

    True suffixes dict

    What suffixes to use on joins, keys are table_names, values are suffixes.

    None desired_resolution Iterable

    What resolution to convert position columns to.

    None allow_missing_lookups bool

    If there are annotations without supervoxels and root IDs yet, allow results.

    False allow_invalid_root_ids bool

    If True, ignore root ids not valid at the given timestamp, otherwise raise an error.

    False random_sample int

    If given, will do a table sample of the table to return that many annotations.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description

    Results of query

    Examples:

    >>> from caveclient import CAVEclient\n>>> client = CAVEclient('minnie65_public_v117')\n>>> live_live_query(\"table_name\", datetime.datetime.now(datetime.timezone.utc),\n>>>    joins=[[table_name, table_column, joined_table, joined_column],\n>>>             [joined_table, joincol2, third_table, joincol_third]]\n>>>    suffixes={\n>>>        \"table_name\":\"suffix1\",\n>>>        \"joined_table\":\"suffix2\",\n>>>        \"third_table\":\"suffix3\"\n>>>    },\n>>>    select_columns= {\n>>>        \"table_name\":[ \"column\",\"names\"],\n>>>        \"joined_table\":[\"joined_colum\"]\n>>>    },\n>>>    filter_in_dict= {\n>>>        \"table_name\":{\n>>>            \"column_name\":[included,values]\n>>>        }\n>>>    },\n>>>    filter_out_dict= {\n>>>        \"table_name\":{\n>>>            \"column_name\":[excluded,values]\n>>>        }\n>>>    },\n>>>    filter_equal_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_greater_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_less_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_greater_equal_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_less_equal_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_spatial_dict\"= {\n>>>        \"table_name\": {\n>>>        \"column_name\": [[min_x, min_y, min_z], [max_x, max_y, max_z]]\n>>>    }\n>>>    filter_regex_dict\"= {\n>>>        \"table_name\": {\n>>>        \"column_name\": \"regex_string\"\n>>>     }\n
    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_query","title":"live_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"

    Generic query on materialization tables

    Parameters:

    Name Type Description Default table str

    Table to query

    required timestamp datetime

    Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time.

    required filter_in_dict dict

    Keys are column names, values are allowed entries.

    None filter_out_dict dict

    Keys are column names, values are not allowed entries.

    None filter_equal_dict dict

    Keys are column names, values are specified entry.

    None filter_greater_dict dict

    Keys are column names, values are exclusive upper-bounds.

    None filter_less_dict dict

    Keys are column names, values are exclusive lower-bounds.

    None filter_greater_equal_dict dict

    Keys are column names, values are inclusive upper-bounds.

    None filter_less_equal_dict dict

    Keys are column names, values are inclusive lower-bounds.

    None filter_spatial_dict dict

    Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]].

    None filter_regex_dict dict

    Keys are column names, values are regex strings.

    None select_columns list of str

    Columns to select.

    None offset int

    Offset in query result.

    None limit int

    Maximum results to return (server will set upper limit, see get_server_config).

    None datastack_name str

    Datastack to query. If None, defaults to one specified in client.

    None split_positions bool

    Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower).

    False post_filter bool

    Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True.

    True metadata bool

    Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True merge_reference bool

    Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table.

    True desired_resolution Iterable

    Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata.

    None random_sample int

    If given, will do a tablesample of the table to return that many annotations.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description DataFrame

    A pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.lookup_supervoxel_ids","title":"lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)","text":"

    Trigger supervoxel lookups of new annotations in a table.

    Parameters:

    Name Type Description Default table_name str

    Table to trigger

    required annotation_ids list

    List of annotation ids to lookup. Default is None, which will trigger lookup of entire table.

    None datastack_name str

    Datastack to trigger it. Defaults to what is set in client.

    None

    Returns:

    Type Description dict

    Status code of response from server

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.map_filters","title":"map_filters(filters, timestamp, timestamp_past)","text":"

    Translate a list of filter dictionaries from a point in the future to a point in the past

    Parameters:

    Name Type Description Default filters list[dict]

    filter dictionaries with root_ids

    required timestamp datetime

    timestamp to query

    required timestamp_past datetime

    timestamp to query from

    required

    Returns:

    Type Description list[dict]

    filter dictionaries with past root_ids

    dict

    mapping of future root_ids to past root_ids

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.most_recent_version","title":"most_recent_version(datastack_name=None)","text":"

    Get the most recent version of materialization for this datastack name

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client

    None

    Returns:

    Type Description int

    Most recent version of materialization for this datastack name

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_table","title":"query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)","text":"

    Generic query on materialization tables

    Parameters:

    Name Type Description Default table str

    Table to query

    required filter_in_dict dict

    Keys are column names, values are allowed entries, by default None

    None filter_out_dict dict

    Keys are column names, values are not allowed entries, by default None

    None filter_equal_dict dict

    Keys are column names, values are specified entry, by default None

    None filter_greater_dict dict

    Keys are column names, values are exclusive upper-bound value, by default None

    None filter_less_dict dict

    Keys are column names, values are exclusive lower-bound value, by default None

    None filter_greater_equal_dict dict

    Keys are column names, values are inclusive upper-bound value, by default None

    None filter_less_equal_dict dict

    Keys are column names, values are inclusive lower-bound value, by default None

    None filter_spatial_dict dict

    Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None

    None filter_regex_dict dict

    Keys are column names, values are regex strings, by default None

    None select_columns list of str

    Columns to select, by default None

    None offset int

    Result offset to use, by default None. Will only return top K results.

    None limit int

    Maximum results to return (server will set upper limit, see get_server_config), by default None

    None datastack_name str

    Datastack to query, by default None. If None, defaults to one specified in client.

    None return_df bool

    Whether to return as a dataframe, by default True. If False, data is returned as json (slower).

    True split_positions bool

    Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)

    False materialization_version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None timestamp datetime

    Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version

    None metadata bool

    Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True merge_reference bool

    Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that

    True desired_resolution Iterable[float]

    Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata

    None get_counts bool

    Whether to get counts of the query, by default False

    False random_sample int

    If given, will do a tablesample of the of the table to return that many annotations

    None log_warning bool

    Whether to log warnings, by default True

    True

    Returns:

    Type Description DataFrame

    A pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_view","title":"query_view(view_name, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None)","text":"

    Generic query on a view

    Parameters:

    Name Type Description Default view_name str

    View to query

    required filter_in_dict dict

    Keys are column names, values are allowed entries, by default None

    None filter_out_dict dict

    Keys are column names, values are not allowed entries, by default None

    None filter_equal_dict dict

    Keys are column names, values are specified entry, by default None

    None filter_greater_dict dict

    Keys are column names, values are exclusive upper-bound, by default None

    None filter_less_dict dict

    Keys are column names, values are exclusive lower-bound, by default None

    None filter_greater_equal_dict dict

    Keys are column names, values are inclusive upper-bound, by default None

    None filter_less_equal_dict dict

    Keys are column names, values are inclusive lower-bound, by default None

    None filter_spatial_dict dict

    Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None

    None filter_regex_dict dict

    Keys are column names, values are regex strings, by default None

    None select_columns list of str

    Columns to select, by default None

    None offset int

    Result offset to use, by default None. Will only return top K results.

    None limit int

    Maximum results to return (server will set upper limit, see get_server_config), by default None

    None datastack_name str

    Datastack to query, by default None. If None, defaults to one specified in client.

    None return_df bool

    Whether to return as a dataframe, by default True. If False, data is returned as json (slower).

    True split_positions bool

    Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)

    False materialization_version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None metadata bool

    Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True merge_reference bool

    Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that

    True desired_resolution Iterable[float]

    Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata

    None get_counts bool

    Whether to get counts of the query, by default False

    False random_sample int

    If given, will do a tablesample of the of the table to return that many annotations

    None

    Returns:

    Type Description DataFrame

    A pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.synapse_query","title":"synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)","text":"

    Convenience method for querying synapses.

    Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.

    Parameters:

    Name Type Description Default pre_ids Union[int, Iterable, ndarray]

    Pre-synaptic cell(s) to query.

    None post_ids Union[int, Iterable, ndarray]

    Post-synaptic cell(s) to query.

    None bounding_box Optional[Union[list, ndarray]]

    [[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset.

    None bounding_box_column str

    Which synapse location column to filter by.

    'post_pt_position' timestamp datetime

    Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version.

    None remove_autapses bool

    Whether to remove autapses from query results.

    True include_zeros bool

    Whether to include synapses to/from id=0 (out of segmentation).

    True limit int

    Number of synapses to limit. Server-side limit still applies.

    None offset int

    Number of synapses to offset query.

    None split_positions bool

    Whether to split positions into separate columns, True is faster.

    False desired_resolution Iterable[float]

    List or array of the desired resolution you want queries returned in useful for materialization queries.

    None materialization_version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None metadata bool

    Whether to attach metadata to dataframe in the df.attr dictionary.

    True

    Returns:

    Type Description DataFrame

    Results of query.

    "},{"location":"api/schema/","title":"client.schema","text":"

    Client for interacting with the schema service.

    Methods:

    Name Description get_schemas

    Get the available schema types

    schema_definition

    Get the definition of a specified schema_type

    schema_definition_all

    Get the definition of all schema_types

    schema_definition_multi

    Get the definition of multiple schema_types

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.get_schemas","title":"get_schemas()","text":"

    Get the available schema types

    Returns:

    Type Description list

    List of schema types available on the Schema service.

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition","title":"schema_definition(schema_type)","text":"

    Get the definition of a specified schema_type

    Parameters:

    Name Type Description Default schema_type str

    Name of a schema_type

    required

    Returns:

    Type Description json

    Schema definition

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_all","title":"schema_definition_all()","text":"

    Get the definition of all schema_types

    Returns:

    Type Description dict

    Dictionary of schema definitions. Keys are schema names, values are definitions.

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_multi","title":"schema_definition_multi(schema_types)","text":"

    Get the definition of multiple schema_types

    Parameters:

    Name Type Description Default schema_types list

    List of schema names

    required

    Returns:

    Type Description dict

    Dictionary of schema definitions. Keys are schema names, values are definitions.

    "},{"location":"api/skeleton/","title":"client.skeleton","text":"

    Client for interacting with the skeleton service.

    Methods:

    Name Description get_skeleton

    Gets basic skeleton information for a datastack

    get_cache_contents

    Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.

    skeletons_exist

    Confirm or deny that a set of root ids have H5 skeletons in the cache.

    get_bulk_skeletons

    Generates skeletons for a list of root ids without retrieving them.

    generate_bulk_skeletons_async

    Generates skeletons for a list of root ids without retrieving them.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_skeleton","title":"get_skeleton(root_id, datastack_name=None, skeleton_version=3, output_format='dict', log_warning=True, verbose_level=0)","text":"

    Gets basic skeleton information for a datastack

    Parameters:

    Name Type Description Default root_id int

    The root id of the skeleton to retrieve

    required datastack_name str

    The name of the datastack to check

    None skeleton_version int

    The skeleton version to generate and retrieve. Options are documented in SkeletonService. Use 0 for Neuroglancer-compatibility. Use -1 for latest.

    3 output_format string

    The format to retrieve. Options are: - 'dict': A dictionary - 'swc': A pandas DataFrame

    'dict'

    Returns:

    Type Description

    Skeleton of the requested type. See output_format for details.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_cache_contents","title":"get_cache_contents(datastack_name=None, skeleton_version=3, root_id_prefixes=0, limit=0, log_warning=True)","text":"

    Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.skeletons_exist","title":"skeletons_exist(datastack_name=None, skeleton_version=3, root_ids=0, log_warning=True)","text":"

    Confirm or deny that a set of root ids have H5 skeletons in the cache.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_bulk_skeletons","title":"get_bulk_skeletons(root_ids, datastack_name=None, skeleton_version=3, output_format='dict', generate_missing_skeletons=False, log_warning=True, verbose_level=0)","text":"

    Generates skeletons for a list of root ids without retrieving them.

    Parameters:

    Name Type Description Default root_ids List

    A list of root ids of the skeletons to generate

    required datastack_name str

    The name of the datastack to check

    None skeleton_version int

    The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.

    3"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.generate_bulk_skeletons_async","title":"generate_bulk_skeletons_async(root_ids, datastack_name=None, skeleton_version=None, log_warning=True, verbose_level=0)","text":"

    Generates skeletons for a list of root ids without retrieving them.

    Parameters:

    Name Type Description Default root_ids List

    A list of root ids of the skeletons to generate

    required datastack_name str

    The name of the datastack to check

    None skeleton_version int

    The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.

    None"},{"location":"api/state/","title":"client.state","text":"

    Client to interface with the JSON state service.

    Methods:

    Name Description build_neuroglancer_url

    Build a URL for a Neuroglancer deployment that will automatically retrieve specified state.

    get_neuroglancer_info

    Get the info field from a Neuroglancer deployment

    get_property_json

    Download a Neuroglancer JSON state

    get_state_json

    Download a Neuroglancer JSON state

    save_state_json_local

    Save a Neuroglancer JSON state to a JSON file locally.

    upload_property_json

    Upload a Neuroglancer JSON state

    upload_state_json

    Upload a Neuroglancer JSON state

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    state_service_endpoint str

    Endpoint URL for posting JSON state

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.state_service_endpoint","title":"state_service_endpoint: str property","text":"

    Endpoint URL for posting JSON state

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.build_neuroglancer_url","title":"build_neuroglancer_url(state_id, ngl_url=None, target_site=None, static_url=False, format_properties=False)","text":"

    Build a URL for a Neuroglancer deployment that will automatically retrieve specified state. If the datastack is specified, this is prepopulated from the info file field \"viewer_site\". If no ngl_url is specified in either the function or the client, a fallback neuroglancer deployment is used.

    Parameters:

    Name Type Description Default state_id int

    State id to retrieve

    required ngl_url str

    Base url of a neuroglancer deployment. If None, defaults to the value for the datastack or the client. As a fallback, a default deployment is used.

    None target_site seunglab or cave - explorer or mainline or None

    Set this to 'seunglab' for a seunglab deployment, or either 'cave-explorer'/'mainline' for a google main branch deployment. If None, checks the info field of the neuroglancer endpoint to determine which to use. Default is None.

    None static_url bool

    If True, treats \"state_id\" as a static URL directly to the JSON and does not use the state service.

    False format_properties bool

    If True, formats the url as a segment_properties info file

    False

    Returns:

    Type Description str

    The full URL requested

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.get_neuroglancer_info","title":"get_neuroglancer_info(ngl_url=None)","text":"

    Get the info field from a Neuroglancer deployment

    Parameters:

    Name Type Description Default ngl_url str(optional)

    URL to a Neuroglancer deployment. If None, defaults to the value for the datastack or the client.

    None

    Returns:

    Type Description dict

    JSON-formatted info field from the Neuroglancer deployment

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.get_property_json","title":"get_property_json(state_id)","text":"

    Download a Neuroglancer JSON state

    Parameters:

    Name Type Description Default state_id int

    ID of a JSON state uploaded to the state service.

    required

    Returns:

    Type Description dict

    JSON specifying a Neuroglancer state.

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.get_state_json","title":"get_state_json(state_id)","text":"

    Download a Neuroglancer JSON state

    Parameters:

    Name Type Description Default state_id int or string

    ID of a JSON state uploaded to the state service. If a string, treats this as the url to the json state.

    required

    Returns:

    Type Description dict

    JSON specifying a Neuroglancer state.

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.save_state_json_local","title":"save_state_json_local(json_state, filename, overwrite=False)","text":"

    Save a Neuroglancer JSON state to a JSON file locally.

    Parameters:

    Name Type Description Default json_state dict

    Dict representation of a neuroglancer state

    required filename str

    Filename to save the state to

    required overwrite bool

    Whether to overwrite the file if it exists. Default False.

    False

    Returns:

    Type Description None"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_property_json","title":"upload_property_json(property_json, state_id=None, max_size=2500000)","text":"

    Upload a Neuroglancer JSON state

    Parameters:

    Name Type Description Default propery_json dict

    Dict representation of a neuroglancer segment properties json

    required state_id int

    ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.

    None max_size Optional[int]

    Maximum size in bytes for the data to upload. Default is 2.5MB. Set to None for no limit.

    2500000

    Returns:

    Type Description int

    state_id of the uploaded JSON state

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_state_json","title":"upload_state_json(json_state, state_id=None, timestamp=None)","text":"

    Upload a Neuroglancer JSON state

    Parameters:

    Name Type Description Default json_state dict

    Dict representation of a neuroglancer state

    required state_id int

    ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.

    None timestamp Optional[time]

    Timestamp for json state date. Requires state_id.

    None

    Returns:

    Type Description int

    state_id of the uploaded JSON state

    "},{"location":"api/testing/","title":"caveclient.tools.testing","text":"

    Functions:

    Name Description CAVEclientMock

    Created a mocked CAVEclient function for testing using the responses library to mock

    get_materialiation_info

    Get the materialization versions and version metadata for the materialization service.

    get_server_versions

    Get the server versions for the services used in testing.

    get_server_information

    Generate the datastack name and server locations used in testing.

    get_api_versions

    Get the API versions for the services used in testing.

    default_info

    Generate a info service info file for testing

    "},{"location":"api/testing/#caveclient.tools.testing.CAVEclientMock","title":"CAVEclientMock(datastack_name=None, global_server=None, local_server=None, info_file=None, chunkedgraph=False, chunkedgraph_server_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, chunkedgraph_api_versions=None, materialization=False, materialization_server_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, materialization_api_versions=None, available_materialization_versions=None, set_version=None, set_version_metadata=None, json_service=False, json_service_server_version=DEFAULT_JSON_SERVICE_SERVER_VERSION, skeleton_service=False, skeleton_service_server_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, schema_api_versions=None, l2cache=False, l2cache_disabled=False, global_only=False)","text":"

    Created a mocked CAVEclient function for testing using the responses library to mock the server responses. This function returns a drop-in replacement for the CAVEclient function that will be able to initialize itself and selected individual service clients with the selected options.

    Note that the test configuration is intended to be purely for pytest purposes and should not actually result in calls to active endpoints.

    Parameters:

    Name Type Description Default datastack_name str

    Name of the test datastack, by default None

    None global_server str

    Test global server address, by default None

    None local_server str

    Test local server address, by default None

    None info_file dictionary

    Info service dictionary for the datastack, by default None

    None chunkedgraph bool

    If True, configures the client to initialize a chunkedgraph subclient, by default False

    False chunkedgraph_server_version str

    Sets the value of the chunkedgraph server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.

    DEFAULT_CHUNKEDGRAPH_SERVER_VERSION chunkedgraph_api_versions list

    List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default None. If None, returns the value in CHUNKEDGRAPH_API_VERSIONS.

    None materialization bool

    If True, configures the client to initalize a materialization subclient, by default False Note that materialization being set to True will also configure the chunkedgraph client.

    False materialization_server_version str

    Sets the value of the materialization server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_MATERIALIZATION_SERVER_VERSON.

    DEFAULT_MATERIALIZATION_SERVER_VERSON available_materialization_versions list

    List of materialization database versions that the materialization client thinks exists, by default None. If None, returns the value in DEFAULT_MATERIALIZATION_VERSONS.

    None materialization_api_versions list

    List of materialization API versions that the materialization client thinks exists, by default None. If None, returns the value in MATERIALIZATION_API_VERSIONS.

    None set_version Optional[int]

    If set, will set the version of the materialization server to the value of set_version, by default None. To work, this version must be in the list of available materialization versions.

    None set_version_metadata Optional[dict]

    If set, will set the version metadata of the materialization server to the value of set_version_metadata. Default value is in DEFAULT_MATERIALIZATION_VERSION_METADATA.

    None json_service bool

    If True, configures the client to initalize a materialization subclient, by default False

    False json_service_server_version _type_

    Sets the value of the json state server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_JSON_SERVICE_SERVER_VERSION.

    DEFAULT_JSON_SERVICE_SERVER_VERSION skeleton_service bool

    If True, configures the client to initalize a skeleton service subclient, by default False

    False skeleton_service_server_version _type_

    Sets the value of the skeleton service version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_SKELETON_SERVICE_SERVER_VERSION.

    DEFAULT_SKELETON_SERVICE_SERVER_VERSION l2cache bool

    If True, configures the client to initialize an l2cache subclient, by default False Note that l2cache being set to True will also configure the chunkedgraph client.

    False l2cache_disabled bool

    If True, allows a subclient to be initialized, but emulates a situation without an L2 cache, by default False Only used if l2cache is True.

    False global_only bool

    If True, only initializes the global services and does not use a datastack, by default False.

    False

    Returns:

    Type Description CAVEclient

    A mocked and initialized CAVEclient object for testing

    Examples:

    To make a basic pytest fixture to test chunkedgraph features with an initialized CAVEclient object in your pytest conftest.py file:

    import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\ntest_datastack = \"test_stack\"\ntest_global_server = \"https://test.cave.com\"\ntest_local_server = \"https://local.cave.com\"\n\n@pytest.fixture()\ndef test_client():\n    return CAVEclientMock(\n        datastack_name=test_datastack,\n        global_server=test_global_server,\n        local_server=test_local_server,\n        chunkedgraph=True,\n    )\n\nYou can also create more complex fixtures with multiple services initialized and specific server versions:\n\n```python\n@pytest.fixture()\ndef fancier_test_client():\n    return CAVEclientMock(\n        datastack_name=test_datastack,\n        global_server=test_global_server,\n        local_server=test_local_server,\n        chunkedgraph=True,\n        chunkedgraph_server_version=\"3.0.2\",\n        materialization=True,\n        materialization_server_version=\"4.21.4\",\n        l2cache=True,\n    )\n
    "},{"location":"api/testing/#caveclient.tools.testing.get_materialiation_info","title":"get_materialiation_info(materialization_versions=DEFAULT_MATERIALIZATION_VERSONS, version_metadata=DEFAULT_MATERIALIZATION_VERSION_METADATA)","text":"

    Get the materialization versions and version metadata for the materialization service.

    Parameters:

    Name Type Description Default materialization_versions list

    List of materialization database versions that the materialization client thinks exists, by default DEFAULT_MATERIALIZATION_VERSONS.

    DEFAULT_MATERIALIZATION_VERSONS version_metadata dict

    Version metadata for the materialization service, by default DEFAULT_MATERIALIZATION_VERSION_METADATA.

    DEFAULT_MATERIALIZATION_VERSION_METADATA

    Returns:

    Type Description dict

    Dictionary with keys: \"materialization_versions\", \"version_metadata\".

    "},{"location":"api/testing/#caveclient.tools.testing.get_server_versions","title":"get_server_versions(chunkedgraph_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, materialization_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, skeleton_service_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, json_service_version=DEFAULT_JSON_SERVICE_SERVER_VERSION)","text":"

    Get the server versions for the services used in testing.

    Parameters:

    Name Type Description Default chunkedgraph_version str

    Version of the chunkedgraph server, by default DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.

    DEFAULT_CHUNKEDGRAPH_SERVER_VERSION materialization_version str

    Version of the materialization server, by default DEFAULT_MATERIALIZATION_SERVER_VERSON.

    DEFAULT_MATERIALIZATION_SERVER_VERSON skeleton_service_version str

    Version of the skeleton service server, by default DEFAULT_SKELETON_SERVICE_SERVER_VERSION.

    DEFAULT_SKELETON_SERVICE_SERVER_VERSION json_service_version str

    Version of the json service server, by default DEFAULT_JSON_SERVICE_SERVER_VERSION.

    DEFAULT_JSON_SERVICE_SERVER_VERSION

    Returns:

    Type Description dict

    Dictionary with keys: \"chunkedgraph_version\", \"materialization_version\", \"skeleton_service_version\", \"json_service_version\". Values are Version objects from packaging.versions.

    "},{"location":"api/testing/#caveclient.tools.testing.get_server_information","title":"get_server_information(datastack_name=TEST_DATASTACK, global_server=TEST_GLOBAL_SERVER, local_server=TEST_LOCAL_SERVER)","text":"

    Generate the datastack name and server locations used in testing.

    Parameters:

    Name Type Description Default datastack_name str

    Datastack value, by default the value in TEST_DATASTACK.

    TEST_DATASTACK global_server str

    Server for global services, by default TEST_GLOBAL_SERVER.

    TEST_GLOBAL_SERVER local_server str

    Server for local services, by default TEST_LOCAL_SERVER.

    TEST_LOCAL_SERVER

    Returns:

    Type Description dict

    Dictionary with keys: \"datastack_name\", \"local_server\", \"global_server\".

    "},{"location":"api/testing/#caveclient.tools.testing.get_api_versions","title":"get_api_versions(chunkedgraph_api_versions=CHUNKEDGRAPH_API_VERSIONS, materialization_api_versions=MATERIALIZATION_API_VERSIONS, schema_api_versions=SCHEMA_API_VERSIONS)","text":"

    Get the API versions for the services used in testing.

    Parameters:

    Name Type Description Default chunkedgraph_api_versions list

    List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default CHUNKEDGRAPH_API_VERSIONS.

    CHUNKEDGRAPH_API_VERSIONS materialization_api_versions list

    List of materialization API versions that the materialization client thinks exists, by default MATERIALIZATION_API_VERSIONS.

    MATERIALIZATION_API_VERSIONS schema_api_versions list

    List of schema API versions that the schema client thinks exists, by default SCHEMA_API_VERSIONS.

    SCHEMA_API_VERSIONS

    Returns:

    Type Description dict

    Dictionary with keys: \"chunkedgraph_api_versions\", \"materialization_api_versions\", \"schema_api_versions\".

    "},{"location":"api/testing/#caveclient.tools.testing.default_info","title":"default_info(local_server=TEST_LOCAL_SERVER)","text":"

    Generate a info service info file for testing

    Parameters:

    Name Type Description Default local_server str

    Name of the local service, by default the value in TEST_LOCAL_SERVER.

    TEST_LOCAL_SERVER

    Returns:

    Type Description dict

    Info file for the datastack.

    "},{"location":"tutorials/","title":"Getting Started","text":"

    AnnotationFramework client is a package for simplifying interactions with HTML services associated with the CAVE (Connectome Annotation Versioning Engine).

    For a larger introduction to CAVE and its services, see the main GitHub organization site: https://github.com/CAVEconnectome

    "},{"location":"tutorials/#installation","title":"Installation","text":"

    The CAVEclient can be installed with pip:

    $ pip install caveclient\n
    "},{"location":"tutorials/#assumptions","title":"Assumptions","text":"

    The code is setup to work flexibly with any deployment of these services, but you need to specify the server_address if that address is not https://globalv1.daf-apis.com/ for each client when initializing it. Similarly, the clients can query the info service for metadata to simplify the interaction with a datastack, but you have to specify a datastack name.

    "},{"location":"tutorials/advanced/","title":"Advanced Usage","text":""},{"location":"tutorials/advanced/#changing-session-configurations","title":"Changing session configurations","text":"

    It is possible to change details of how a client talks to the various servers it needs to interface with. For instance, the CAVEclient will attempt to retry specific failed requests to the server, but will only try a specific number of times, and will only wait specific amounts of time between retries. These values can be changed via the set_session_defaults method. For instance, to change the number of retries to 5, and to increase the delay between subsequent retries, you could do:

    from caveclient import set_session_defaults\n\nset_session_defaults(max_retries=5, backoff_factor=0.5)\n

    Note that this needs to happen before initializing the client for this to work properly. Some of these parameters are also adjustable at the client level.

    To view the current session defaults, you can use the get_session_defaults method:

    from caveclient import get_session_defaults\n\nclient.get_session_defaults()\n

    More information on the available parameters can be found in the API documentation.

    "},{"location":"tutorials/advanced/#writing-unit-tests-using-caveclient","title":"Writing unit tests using caveclient","text":"

    If you have a module that relies on caveclient and you want to write unit tests for it, you can use the caveclient.testing module to help you create an initialized, mocked client with user-specified options. This helps deal with the complex conversation that caveclient has with various services when initializing itself so that you get the appropriate functionality. The function CAVEclientMock offers a number of options for how to build a client, including server versions for various services, available API versions, and more. Importantly, the module provides sensible defaults but also allows you to override them as needed. Note that using testing requires the responses to be installed, which is not a dependency of caveclient.

    The CAVEclientMock function requires you to specify which services you expect to call on in order to ensure that your code is working as expected. For example, to create a pytest fixture that would utilize the chunkedgraph module, but otherwise use default nonsense test values, you could include this in your conftest.py:

    import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\n@pytest.fixture()\ndef test_client():\n    return CAVEclientMock(\n        chunkedgraph=True,\n    )\n

    Then, in your test module, you can use the test_client fixture to get a client. Note that after mocked initialization, the client will attempt normal network requests and thus you should mock the responses. If you only care to get a specific value back from a given function, you can use pytest-mock to mock the response to a given function call. For example, if you have a function do_something_with_roots that takes a caveclient that uses the get_roots function, you could mock the get_roots function so that it return a specific value:

    from pytest_mock import mocker\nfrom conftest import test_client\nfrom my_module import do_something_with_roots\n\ndef test_get_roots(mocker, test_client):\n    mocker.patch.object(test_client.chunkedgraph, 'get_roots', return_value=[1, 2, 3])\n    test_data = # Something appropriate\n    test_output = do_something_with_roots(test_client, test_data)\n    assert test_output == # The expected answer\n

    Note that if you your own datastack info_file that has a different local_server address than the default value (TEST_LOCAL_SERVER variable, which defaults to https://local.cave.com) you will also need to specify a local_server with the same value as in the local_server field of your info file.

    "},{"location":"tutorials/advanced/#specifying-conditions","title":"Specifying conditions","text":"

    While sensible defaults are provided, you can also specify things like server versions to make sure your code works with the versions of the services you expect. For example, let's make a richer mock client that specifies the server versions for the chunkedgraph, materailization, and l2cache services:

    @pytest.fixture()\ndef version_specified_client():\n    return CAVEclientMock(\n        chunkedgraph=True,\n        chunkedgraph_server_version='3.0.1',\n        materialization=True,\n        materialization_server_version='2.0.0',\n        l2cache=True,\n    )\n

    Note that some services like l2cache do not currently use a server version to offer different functionality, and this value is not exposed for them currently. See the API documentation for more information.

    You can also override default values like server_address or datastack_name: 

    @pytest.fixture()\ndef server_specified_client():\n    return CAVEclientMock(\n        datastack_name='my_datastack',\n        server_address='http://my.server.com',\n        chunkedgraph=True,\n        materialization=True,\n    )\n

    If you want to get access to the various default values of server version, datastack name, datastack info file, and api versions, you can use functions get_server_versions, get_server_information, default_info, get_api_versions respectively. Each of these functions will return a dictionary of the values that can be used as a kwargs input into CAVEclientMock. If you specify your own override values, it will take precedence over the default value and you can just use the dictionary in your tests. See the caveclient tests for an example of how to use these functions.

    "},{"location":"tutorials/annotation/","title":"Annotation","text":"

    The AnnotationClient is used to interact with the AnnotationEngine service to create tables from existing schema, upload new data, and download existing annotations. Note that annotations in the AnnotationEngine are not linked to any particular segmentation, and thus do not include any root ids. An annotation client is accessed with client.annotation.

    "},{"location":"tutorials/annotation/#getting-existing-tables","title":"Getting existing tables","text":"

    A list of the existing tables for the datastack can be found with get_tables().

    all_tables = client.annotation.get_tables()\nall_tables[0]\n

    Each table has three main properties that can be useful to know:

    • table_name : The table name, used to refer to it when uploading or downloading annotations. This is also passed through to the table in the Materialized database.
    • schema_name : The name of the table's schema from EMAnnotationSchemas (see below).
    • max_annotation_id : An upper limit on the number of annotations already contained in the table.
    "},{"location":"tutorials/annotation/#downloading-annotations","title":"Downloading annotations","text":"

    You can download the JSON representation of a data point through the get_annotation() method. This can be useful if you need to look up information on unmaterialized data, or to see what a properly templated annotation looks like.

    table_name = all_tables[0]['table_name']      # 'ais_analysis_soma'\nannotation_id = 100\nclient.annotation.get_annotation(annotation_ids=annotation_id, table_name=table_name)\n
    "},{"location":"tutorials/annotation/#create-a-new-table","title":"Create a new table","text":"

    One can create a new table with a specified schema with the create_table() method:

    client.annotation.create_table(table_name='test_table',\n                               schema_name='microns_func_coreg',\n                               voxel_resolution = [1,1,1],\n                               description=\"some text to describe your table\")\n

    The voxel resolution is the units your position columns will be uploaded in [1,1,1] would imply a nm location, where as [4,4,40] would correspond to voxels of that size. If you are uploading points from a neuroglancer session, you want this to match the units of that neuroglancer view.

    Note there are some optional metadata parameters to create_table()

    • notice_text : This is text that will show up to users who access this data as a warning. This could be used to warn users that the data is not complete or checked yet, or to advertise that a particular publication should be cited when using this table.
    • read_permission : one of \"PRIVATE\" which means only you can read data in this table. \"PUBLIC\" (default) which means anyone can read this table that has read permissions to this dataset. So if and only if you can read the segmentation results of this data, you can read this table. \"GROUP\" which means that you must share a common group with this user for them to be able to read. We need to make a way to discover what groups you are in and who you share groups with.
    • write_permission: one of \"PRIVATE\" (default), which means only you can write to this table. \"PUBLIC\" which means anyone can write to this table that has write permissions to this dataset. Note although this means anyone can add data, no annotations are ever truly overwritten. \"GROUP\" which means that you must share a common group with this user for them to be able to write. We need to make a way to discover what groups you are in and who you share groups with.

    If you change your mind about what you want for metadata, some but not all fields can be updated with update_metadata(). This includes the description, the notice_text, and the permissions, but not the name, schema or voxel resolution.

    # to update description\nclient.annotation.update_metadata(table_name='test_table',\n                                  description=\"a new description for my table\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n                                  notice_text=\"This table isn't done yet, don't trust it. Contact me\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n                                  read_permisison=\"PUBLIC\")\n

    New data can be generated as a dict or list of dicts following the schema and uploaded with post_annotation. For example, a microns_func_coreg point needs to have:

    • type set to microns_func_coreg
    • pt set to a dict with position as a key and the xyz location as a value.
    • func_id set to an integer.

    The following could would create a new annotation and then upload it to the service. Note that you get back the annotation id(s) of what you uploaded.

    new_data = {'type': 'microns_func_coreg',\n            'pt': {'position': [1,2,3]},\n            'func_id': 0}\nclient.annotation.post_annotation(table_name='test_table', data=[new_data])\n

    There are methods to simplify annotation uploads if you have a pandas dataframe whose structure mirrors the struction of the annotation schema you want to upload

    import pandas as pd\n\ndf = pd.DataFrame([{'id':0,\n         'type': 'microns_func_coreg',\n         'pt_position': [1,2,3]},\n         'func_id': 0},\n        {'id':1,\n        'type': 'microns_func_coreg',\n        'pt_position': [3,2,1]},\n        'func_id': 2}])\nclient.annotation.post_annotation_df('test_table', df)\n

    Note that here I specified the IDs of my annotations, which you can do, but then its up to you to assure that the IDs don't collide with other IDs. If you leave them blank then the service will assign the IDs for you.

    There is a similar method for updating update_annotation_df()

    "},{"location":"tutorials/annotation/#staged-annotations","title":"Staged Annotations","text":"

    Staged annotations help ensure that the annotations you post follow the appropriate schema, both by providing guides to the field names and locally validating against a schema before uploading. The most common use case for staged annotations is to create a StagedAnnotation object for a given table, then add annotations to it individually or as a group, and finally upload to the annotation table.

    To get a StagedAnnotation object, you can start with either a table name or a schema name. Here, we'll assume that there's already a table called \"my_table\" that is running a \"cell_type_local\" schema. If we want to add new annotations to the table, we simply use the table name with stage_annotations().

    stage = client.annotation.stage_annotations(\"my_table\")\n

    This stage object retrieves the schema for the table and hosts a local collection of annotations. Every time you add an annotation, it is immediately validated against the schema. To add an annotation, use the add method:

    stage.add(\n    cell_type = \"pyramidal_cell\",\n    classification_system=\"excitatory\",\n    pt_position=[100,100,10],\n)\n

    The argument names derive from fields in the schema and you must provide all required fields. Any number of annotations can be added to the stage. A dataframe of annotations can also be added with stage.add_dataframe, and requires an exact match between column names and schema fields. The key difference between this and posting a dataframe directly is that annotations added to a StagedAnnotations are validated locally, allowing any issues to be caught before uploading.

    You can see the annotations as a list of dictionary records with stage.annotation_list or as a Pandas dataframe with stage.annotation_dataframe. Finally, if you initialized the stage with a table name, this information is stored in the stage and you can simply upload it from the client.

    client.annotation.upload_staged_annotations(stage)\n

    Updating annotations requires knowing the annotation id of the annotation you are updating, which is not required in the schema otherwise. In order to stage updated annotations, set the update parameter to True when creating the stage.

    update_stage = client.annotation.stage_annotations(\"my_table\", update=True)\nupdate_stage.add(\n    id=1,\n    cell_type = \"stellate_cell\",\n    classification_system=\"excitatory\",\n    pt_position=[100,100,10],\n)\n

    The update also informs the framework client to treat the annotations as an update and it will use the appropriate methods automatically when uploading client.annotation.upload_staged_annotations.

    If you want to specify ids when posting new annotations, id_field can be set to True when creating the StagedAnnotation object. This will enforce an id column but still post the data as new annotations.

    If you might be adding spatial data in coordinates that might be different than the resolution for the table, you can also set the annotation_resolution when creating the stage. The stage will convert between the resolution you specify for your own annotations and the resolution that the table expects.

    stage = client.annotation.stage_annotations(\"my_table\", annotation_resolution=[8,8,40])\nstage.add(\n    cell_type='pyramidal_cell',\n    classification_system=\"excitatory\",\n    pt_position=[50,50,10],\n)\n
    "},{"location":"tutorials/authentication/","title":"Authentication","text":"

    Authentication tokens are generally needed for programmatic access to our services. The AuthClient handles storing and loading your token or tokens and inserting it into requests in other clients.

    We can access the auth client from client.auth. Once you have saved a token, you probably won't interact with this client very often, however it has some convenient features for saving new tokens the first time. Let's see if you have a token already. Probably not.

    client = CAVEclient()\nauth = client.auth\nprint(f\"My current token is: {auth.token}\")\n
    "},{"location":"tutorials/authentication/#getting-a-new-token","title":"Getting a new token","text":"

    To get a new token, you will need to manually acquire it. For convenience, the function client.auth.get_new_token() provides instructions for how to get and save the token.

    By default, the token is saved to ~/.cloudvolume/secrets/cave-secret.json as a string under the key token. This makes it compatible by default with Cloudvolume projects, which can come in handy. The following steps will save a token to the default location.

    auth.get_new_token()\n
    new_token = 'abcdef1234567890' #This is the text you see after you visit the website.\nauth.save_token(token=new_token)\nprint(f\"My token is now: {auth.token}\")\n

    Note that requesting a new token will invalidate your previous token on the same project. If you want to use the same token across different computers, you will need to share the same token information.

    "},{"location":"tutorials/authentication/#loading-saved-tokens","title":"Loading saved tokens","text":"

    Try opening ~/.cloudvolume/secrets/cave-secret.json to see what we just created.

    If we had wanted to use a different file or a different json key, we could have specified that in auth.save_token.

    Because we used the default values, this token is used automatically when we initialize a new CAVEclient. If we wanted to use a different token file, token key, or even directly specify a token we could do so here.

    client = CAVEclient(datastack_name)\nprint(f\"Now my basic token is: {client.auth.token}\")\n\nclient_direct = CAVEclient(datastack_name, auth_token='another_fake_token_678')\nprint(f\"A directly specified token is: {client_direct.auth.token}\")\n

    If you use a CAVEclient, the AuthClient and its token will be automatically applied to any other services without further use.

    "},{"location":"tutorials/chunkedgraph/","title":"Chunked Graph","text":"

    The chunkedgraph is a dynamic oct-tree connected components supervoxel graph.

    A visual representation of an oct-tree (Wikipedia (WhiteTimberwolf) CC BY-SA 3.0)

    As with any oct-tree, it is organized in hierarchical levels, with the bottom level 1 corresponding to the supervoxels of the segmentations, and the top level being the unique connected components of the supervoxel graph.

    A figure illustrating the spatial chunking and editing of a the chunkedgraph. From Dorkenwald et. al 2021

    The ChunkedGraph client allows one to interact with the ChunkedGraph service, which stores and updates the supervoxel agglomeration graph. This is most often useful for looking up an object root id of a supervoxel or looking up supervoxels belonging to a root id. The ChunkedGraph client is at client.chunkedgraph.

    "},{"location":"tutorials/chunkedgraph/#look-up-a-supervoxel","title":"Look up a supervoxel","text":"

    Usually in Neuroglancer, one never notices supervoxel ids, but they are important for programmatic work. In order to look up the root id for a location in space, one needs to use the supervoxel segmentation to get the associated supervoxel id. The ChunkedGraph client makes this easy using the client.chunkedgraph.get_root_id() method.

    sv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id)\n

    However, as proofreading occurs, the root id that a supervoxel belongs to can change. By default, this function returns the current state, however one can also provide a UTC timestamp to get the root id at a particular moment in history. This can be useful for reproducible analysis. Note below that the root id for the same supervoxel is different than it is now.

    import datetime\n\n# I looked up the UTC POSIX timestamp from a day in early 2019.\ntimestamp = datetime.datetime.utcfromtimestamp(1546595253)\n\nsv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id, timestamp=timestamp)\n

    If you are doing this across lots of supervoxels (or any nodes) then you can do it more efficiently in one request with client.chunkedgraph.get_roots()

    node_ids = [104200755619042523, 104200755619042524,104200755619042525]\nroot_ids = client.chunkedgraph.get_roots(node_ids)\n
    "},{"location":"tutorials/chunkedgraph/#getting-supervoxels-for-a-root-id","title":"Getting supervoxels for a root id","text":"

    A root id is associated with a particular agglomeration of supervoxels, which can be found with the client.chunkedgraph.get_leaves() method. A new root id is generated for every new change in the chunkedgraph, so time stamps do not apply.

    root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id)\n

    You can also query the chunkedgraph not all the way to the bottom, using the stop_layer option

    root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id,stop_layer=2)\n

    This will get all the level 2 IDs for this root, which correspond to the lowest chunk of the hierarchy. An analogous option exists for client.chunkedgraph.get_roots(). This is useful to help find nodes to query within the l2cache, amongst other things.

    "},{"location":"tutorials/chunkedgraph/#other-functions","title":"Other functions","text":"

    There are a variety of other interesting functions to explore in client.chunkedgraph.

    "},{"location":"tutorials/framework/","title":"Introduction","text":""},{"location":"tutorials/framework/#caveclient-one-client-for-all-services","title":"CAVEclient: one client for all services","text":"

    The CAVE Framework consists of a number of different services, each with a specific set of tasks that it can perform through REST endpoints. The CAVEclient is designed to ease programmatic interaction with all of the various endpoints. In addition, most programmatic access requires the use of authentication tokens. In order to collect a given server, datastack name, and user token together into a coherent package that can be used on multiple endpoints, the CAVEclient builds appropriately configured clients for each of the specific services. Each of the individual services has their own specific documentation as well.

    "},{"location":"tutorials/framework/#global-and-local-services","title":"Global and Local Services","text":"

    There are two categories of data in CAVE: Global and local. Local services are associated with a single so-called datastack, which refers to a precise collection of imagery and segmentation data that function together. For example, EM imagery and a specific pychunkedgraph segmentation would be one datastack, while the same EM imagery but an initial static segmentation would be another. Datastacks are referred to by a short name, for instance pinky100_public_flat_v185.

    Global services are those that are potentially shared across multiple different specific datastacks. These include the info service, which can describe the properties of all available datastacks, the authentication service, and the state service that hosts neuroglancer states. Global services are associated with a particular URL (by default http://globalv1.daf-apis.com), but not a single datastack.

    "},{"location":"tutorials/framework/#initializing-a-caveclient","title":"Initializing a CAVEclient","text":"

    Assuming that the services are on http://globalv1.daf-apis.com and authentication tokens are either not being used or set up with default values (see Authentication), a simple CAVEclient that can only access global services can be initialized:

    from caveclient import CAVEclient\n\nclient = CAVEclient()\n

    Just to confirm that this works, let's see if we can get the EM image source from the InfoService. If you get a list of names of datastacks, all is good. If you have not yet set up an authentication token or you get an authentication error, look at Getting a new token for information about how to set up your auth token.

    client.info.get_datastacks()\n

    If you have a specific datastack you want to use, you can initialize your CAVEclient with it. This gives you access to the full range of client functions.

    client = CAVEclient(datastack_name='my_datastack')\n
    "},{"location":"tutorials/framework/#using-other-server-addresses","title":"Using Other Server Addresses","text":"

    If your data is hosted by a different global server, you specify its address when initializing the client.

    client = CAVEclient(datastack_name='my_datastack', server_address='http://global.myserver.com')\n

    By default, if you pass both a server address and a datastack, the client will store the mapping from datastack to server address in the same location as the default for authentication tokens. Once stored, the client will automatically use the correct server address for the datastack if none is provided. You can override storing the server address by passing write_server_address=False. Datastacks can be removed from the cache using

    caveclient.datastack_lookup.reset_server_address_cache(datastack_name).

    "},{"location":"tutorials/framework/#accessing-specific-clients","title":"Accessing specific clients","text":"

    Each client can be accessed as a property of the main client. See the documentation at left for the capabilities of each. Assuming your client is named client, the subclients for each service are:

    • Authentication Service : client.auth
    • AnnotationEngine : client.annotation
    • PyChunkedGraph : client.chunkedgraph
    • InfoService : client.info
    • EM Annotation Schemas : client.schemas
    • JSON Neuroglancer State Service : client.state
    • Skeleton Service : client.skeletonservice
    "},{"location":"tutorials/info/","title":"Info Service","text":"

    A datastack has a number of complex paths to various data sources that together comprise a datastack. Rather than hardcode these paths, the InfoService allows one to query the location of each data source. This is also convenient in case data sources change.

    An InfoClient is accessed at client.info.

    client = CAVEclient(datastack_name)\nprint(f\"This is an info client for {client.info.datastack_name} on {client.info.server_address}\")\n
    "},{"location":"tutorials/info/#accessing-datastack-information","title":"Accessing datastack information","text":"

    All of the information accessible for the datastack can be seen as a dict using get_datastack_info().

    info.get_datastack_info()\n

    Individual entries can be found as well. Use tab autocomplete to see the various possibilities.

    info.graphene_source()\n
    "},{"location":"tutorials/info/#adjusting-formatting","title":"Adjusting formatting","text":"

    Because of the way neuroglancer looks up data versus cloudvolume, sometimes one needs to convert between gs:// style paths to https://storage.googleapis.com/ stype paths. All of the path sources in the info client accept a format_for argument that can handle this, and correctly adapts to graphene vs precomputed data sources.

    neuroglancer_style_source = info.image_source(format_for='neuroglancer')\nprint(f\"With gs-style: { neuroglancer_style_source }\")\n\ncloudvolume_style_source = info.image_source(format_for='cloudvolume')\nprint(f\"With https-style: { cloudvolume_style_source }\")\n
    "},{"location":"tutorials/l2cache/","title":"Level 2 Cache","text":"

    To understand the level 2 cache, you must understand the structure of the chunkedgraph so see the chunkedgraph tutorial.

    Nodes on the second level or layer of the graph, corresponds to all the supervoxels that are locally connected to one another within a single level 2 spatial \"chunk\" of the data. The Level 2 Cache, is a service whose job it is to track and update relevant statistics about every level 2 node within the a chunkedgraph. The source code of this service can be found here.

    "},{"location":"tutorials/l2cache/#finding-level-2-nodes","title":"Finding Level 2 Nodes","text":"

    The chunkedgraph can be used to find the level2 nodes of a rootID using a stop_layer=2 keyword argument on the client.chunkedgraph.get_leaves(). Conversely the level 2 node of a supervoxel can be found using the same keyword argument of client.chunkedgraph.get_roots(). Note if you don't specify a timestamp it will give you the level2 node that is presently associated with the object.

    "},{"location":"tutorials/l2cache/#statistics","title":"Statistics","text":"

    The statistics that are available are:

    • area_nm2: The surface area of the object in square nanometers. Does not include border touching voxels
    • size_nm3: The volume of the object in cubic nanometers, based on counting voxels in the object.
    • max_dt_nm: The maximum edge distance transform of that object in nanometers. Meant to capture the maximum \"thickness\" of the voxels in the node.
    • mean_dt_nm: The average edge distance transform of that object in nanometers. Meant to capture the average \"thickness\" of voxels in that node.
    • rep_coord_nm: A list of x,y,z coordinates in nanometers that represent a point within the object that is designed to be close to the \"center\" of the object. This is the location of the max_dt_nm value.
    • chunk_intersect_count: A 2 x 3 matrix representing the 6 sides of the chunk, and whose values represent how many voxels border that side of the chunk. Meant to help understand significant the borders with other chunks are. Ordering is the [[x_bottom, y_bottom, z_bottom],[x_top, y_top, z_top]] where {xyz}_bottom refers to the face which has the smallest values for that dimension, and {xyz}_top refers to the face which has the largest.
    • pca A 3x3 matrix representing the principal components of the xyz point cloud of voxels for this object. Ordering is NxD where N is the components and D are the xyz dimensions. Meant to help desribe the orientation of the level 2 chunk. Note that this is not calculated for very small objects and so might not be present for all level 2 nodes. You will see that its availability correlates strongly with size_nm3.
    • pca_val The 3 principal component values for the PCA components.
    "},{"location":"tutorials/l2cache/#retrieving-level-2-statistics","title":"Retrieving Level 2 Statistics","text":"

    Level 2 stats about nodes can be retreived using the client.l2cache.get_l2data() method. It simply takes a list of level 2 nodes you want to retrieve. Optionally you can specify only the attributes that you are interested in retrieving which will speed up the request.

    "},{"location":"tutorials/l2cache/#missing-data","title":"Missing Data","text":"

    The service is constantly watching for changes made to objects and recalculating stats on new level2 nodes that are created, in order to keep its database of statistics current. This however takes some time, and is subject to sporadic rare failures. If you request stats on a level 2 node which are not in the database, you will receive an empty dictionary for that node. This will immediately trigger the system to recalculate the statistics of that missing data, and so it should be available shortly (on the order of seconds) if systems are operational. Please note that PCA is not calculated for very small objects because it is not meaningful. So if you are interested in differentiating whether PCA is not available because it hasn't been calculated, vs when its not available because it is not possible to calculate, you should ask for at least one other non PCA statistic as well. You will see that its availability correlates strongly with size_nm3.

    "},{"location":"tutorials/l2cache/#use-cases","title":"Use Cases","text":""},{"location":"tutorials/l2cache/#calculate-total-area-and-volume-of-cells","title":"Calculate Total Area and Volume of Cells","text":"

    Say you want to calculate the total surface area and volume of a object in the dataset. The areas and volume of each component can simply be added together to do this.

    import pandas as pd\nroot_id = 648518346349541252\nlvl2nodes = client.chunkedgraph.get_leaves(root_id,stop_layer=2)\nl2stats = client.l2cache.get_l2data(lvl2nodes, attributes=['size_nm3','area_nm2'])\nl2df = pd.DataFrame(l2stats).T\ntotal_area_um2=l2df.area_nm2.sum()/(1000*1000)\ntotal_volume_um3 = l2df.size_nm3.sum()/(1000*1000*1000)\n

    By utilizing the bounds argument of get_leaves, you can also do simple spatially restricted analysis of objects. In fact, because you have data on each level2 node individually, you can segregate the neuron using any labelling of its topology.

    "},{"location":"tutorials/l2cache/#skeletonization","title":"Skeletonization","text":"

    Level 2 nodes have \"cross chunk\" edges within the chunkedgraph which represent what level 2 nodes that object is locally connected to. This forms a graph between the level 2 nodes of the object that can be retrieved using the chunkedgraph function client.chunkedgraph. This graph represents a topological representation of the neuron at the resolution of individual chunks, and is guaranteed to be fully connected, unlike a voxel or mesh representation of the neuron which can have gaps where there are defects in the segmentation volume or incorrectly inferred edges at self contact locations.

    The level 2 graph can be turned into a skeleton representation of a neuron using a graph based TEASAR like algorithm as described for skeletonizing meshes in this MeshParty Documentation. There is an implementation of this approach that utilizes the chunkedgraph and the L2cache if available here and on pypi as pcg-skel. In this implementation the l2cache is used to more accurately place the level 2 nodes in space using the rep_coord_nm value.

    Note that there is detailed documentation on the Skeleton Client interface at the skeletonization tutorial.

    "},{"location":"tutorials/l2cache/#trajectory-distributions","title":"Trajectory Distributions","text":"

    If one is interested in the bulk direction of processes in a region of the brain, one can start with supervoxels in a region, find level 2 nodes that correspond to them, filter out components based on size, (or other criteria such as whether they are part of objects that have components in some other brain area) and look at the distribution of PCA components to understand the directions that those processes are moving within that region of space.

    "},{"location":"tutorials/materialization/","title":"Materialization","text":"

    The Materialization client allows one to interact with the materialized annotation tables, that were posted to the annotation service (the annotations tutorial).

    To see the entire class visit the API doc.

    The service regularly looks up all annotations and the segids underneath all the boundspatialpoints. You can then query these tables to find out the IDs that underlie the annotations, or the annotations that now intersect with certain IDs.

    For example, one common pattern is that you have identified a cell based on the location of its cell body, and you have an annotation there.

    You want to know what are the inputs onto the cell, so you first query the annotation table with your soma annotation, asking for the current ID underneath that soma. Then you query a synapse table for all synapse annotations that have a post-synaptic ID equal to the ID from your soma annotation.

    In this way your code stays the same, as the proofreading changes and you can track the connectivity of your cell over time.

    "},{"location":"tutorials/materialization/#initializing-the-client","title":"Initializing the client","text":"

    By default when you initialize the overall client, it will choose the most recent materialization version available. This may or may not be desirable depending on your use case. If your code involves using specific IDs then you should be using a specific version that is tied to a timepoint where those IDs are valid.

    To see what versions are available, use the client.materialize.get_versions() function.

    client.materialize.get_versions()\n

    Each version has a timestamp it was run on as well as a date when it will expire. You can query all this metadata for a specific version using client.materialize.get_version_metadata() or all versions using client.materialize.get_versions_metadata().

    To change the default version, alter the .version property of the client. This will change the version for all subsequent calls which expect one, unless you specify a different version in the method call. Note that this also sets the timestamp property of the client to the timestamp of the version for methods which expect a timestamp.

    client.version = 9\n

    You can also specify the version when you initialize the client, e.g.:

    client = CAVEclient('minnie65_public', version=661)\n

    Or, you can specify the version when making a particular method call.

    "},{"location":"tutorials/materialization/#browsing-versions","title":"Browsing versions","text":"

    To see what tables are available in a version you can use client.materialize.get_tables().

    If you want to read about the description of what that table is, use the annotationengine client client.materialize.get_table_metadata().

    If you want to read more about the schema for the annotation table use the schema service client.schema.schema_definition().

    Note, the materialization service has a human readable webpage that links to the other services that might be more convenient for you to browse, to get a link there in ipython display client.materialize.homepage

    for some important tables, the info service has a pointer to which table you should use in the metadata for the datastack. client.info.get_datastack_info()['synapse_table'] and client.info.get_datastack_info()['soma_table'].

    To see how many annotations are in a particular table use

    nannotations=client.materialize.get_annotation_count('my_table')\n
    "},{"location":"tutorials/materialization/#querying-tables","title":"Querying tables","text":"

    To query a small table, you can just download the whole thing using client.materialize.query_table() which will return a dataframe of the table.

    Note however, some tables, such as the synapse table might be very large 200-300 million rows and the service will only return the first 200,000 results, and not in a deterministic manner. NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.

    To just get a preview, use the limit argument (but note again that this won't be a reproducible set)

    df=client.materialize.query_table('my_table', limit=10)\n

    For most applications, you will want to filter the query in some way.

    We offer seven kinds of filters you can apply: filter_in_dict filter_out_dict, filter_equal_dict, filter_greater_dict, filter_less_dict, filter_greater_equal_dict, and filter_less_equal_dict. For query_table each is specified as a dictionary where the keys are column names, and the values are a list of values (or single value in the case of filter_equal).

    So for example to query a synapse table for all synapses onto a neuron in flywire you would use

    synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n                                  filter_equal_dict = {'post_pt_root_id': MYID})\n

    The speed of querying is affected by a number of factors, including the size of the data. To improve the performance of results, you can reduce the number of columns returned using select_columns.

    So for example, if you are only interested in the root_ids and locations of pre_synaptic terminals you might limit the query with select_columns. Also, it is convenient to return the with positions as a column of np.array([x,y,z]) coordinates for many purposes. However, sometimes you might prefer to have them split out as separate x, y, z columns. To enable this option use split_columns=True. split_columns=True is faster, as combining them is an extra step.

    synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n                                  filter_equal_dict = {'post_pt_root_id': MYID},\n                                  select_columns=['id','pre_pt_root_id', 'pre_pt_position'],\n                                  split_columns=True)\n
    "},{"location":"tutorials/materialization/#desired-resolution","title":"Desired Resolution","text":"

    Often you want to have position information in different units. For example, to consider synapse locations or soma locations, you might want to have positions in nanometers or microns.

    To create neuroglancer views, you might want positions in integer voxels of a size that aligns with the resolution you are used to using Neuroglancer at.

    Annotation tables can be created and uploaded in varying resolutions according to whatever the user of the table felt was natural. This information is available in the metadata for that table. In addition, you may pass desired_resolution as a keyword argument which will automatically convert all spatial positions into voxels of that size in nanometers.

    So if you want positions in nanometers, you would pass desired_resolution=[1,1,1]. If you want positions in microns you would pass desired_resolution=[1000,1000,1000]. If you want positions in 4,4,40nm voxel coordinates to use with cloud-volume or neuroglancer you would pass desired_resolution=[4,4,40].

    "},{"location":"tutorials/materialization/#spatial-filters","title":"Spatial Filters","text":"

    You can also filter columns that are associated with spatial locations based upon being within a 3d bounding box.

    This is done by adding a filter_spatial_dict argument to query_table. The units of the bounding box should be in the units of the voxel_resolution of the table (which can be obtained from client.materialize.get_table_metadata()).

    bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nsynapse_table = client.info.get_datastack_info('synapse_table')\ndf=client.materialize.query_table(synapse_table,\n                                  filter_equal_dict = {'post_pt_root_id': MYID},\n                                  filter_spatial_dict = {'post_pt_position': bounding_box})\n
    "},{"location":"tutorials/materialization/#synapse-query","title":"Synapse Query","text":"

    For synapses in particular, we have a simplified method for querying them with a reduced syntax. client.materialize.synapse_query() lets you specify pre and post synaptic partners as keyword arguments and bounding boxes. The defaults make reasonable assumptions about what you want to query, namely that the synapse_table is the table that the info service advertises, and that if you specify a bounding box, that you want the post_pt_position. These can be overridden of course, but the above bounding box query is simplified to.

    NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.

    bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\ndf=client.materialize.query_table(post_ids = MYID,\n                                  bounding_box=bounding_box)\n
    "},{"location":"tutorials/materialization/#live-query","title":"Live Query","text":"

    In order to query the materialized tables above you can only use IDs that were present at the timestamp of the materialization. If you query the tables with an ID that is not valid during the time of the materialization you will get empty results.

    To check if root_ids are valid at your materialization's timestamp, you can use client.chunkedgraph.is_latest_roots()

    import numpy as np\nmat_time = client.materialize.get_timestamp()\nis_latest = client.chunkedgraph.is_latest_roots([MYID], timestamp=mat_time)\nassert(np.all(is_latest))\n

    If you need to lookup what happened to that ID, you can use the chunkedgraph lineage tree, to look into the future or the past, depending on your application you can use client.chunkedgraph.get_lineage_graph().

    Again, the ideal situation is that you have an annotation in the database which refers to your objects of interest, and querying that table by the id column will return the object in the most recent materialization.

    However, sometimes you might be browsing and proofreadding the data and get an ID that is more recent that the most recent version available. For convenience, you can use client.materialize.live_query().

    to automatically update the results of your query to a time in the future, such as now. For example, to pass now, use datetime.datetime.now(datetime.timezone.utc). Note all timestamps are in UTC throughout the codebase.

    import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.live_query(synapse_table,\n                                  datetime.datetime.now(datetime.timezone.utc),\n                                  filter_equal_dict = {'post_pt_root_id': MYID})\n

    This will raise an ValueError exception if the IDs passed in your filters are not valid at the timestamp given

    You can also pass a timestamp directly to query_table and it will call live_query automatically.

    import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n                                  timestamp=datetime.datetime.now(datetime.timezone.utc),\n                                  filter_equal_dict = {'post_pt_root_id': MYID})\n

    Also, keep in mind if you run multiple queries and at each time pass datetime.datetime.now(datetime.timezone.utc), there is no guarantee that the IDs will be consistent from query to query, as proofreading might be happening at any time. For larger scale analysis constraining oneself to a materialized version will ensure consistent results.

    Versions have varying expiration times in order to support the tradeoff between recency and consistency, so before undertaking an analysis project consider what version you want to query and what your plan will be to update your analysis to future versions.

    "},{"location":"tutorials/materialization/#content-aware-interface-experimental","title":"Content-aware Interface (Experimental)","text":"

    As of version 5.8.0, we have introduced a new interface to query tables and views. This interface might have small but breaking changes in the near future. :::

    In order to make the querying interface more consistent across tables, we have introduced an additional alternative interface to filtering and querying data via the client.materialize.tables object. When you instantiate this object, this object finds all of the existing tables and the list of their columns and lets you filter the tables as arguments in the function with suggestions. Moreover, the filtering arguments and the querying arguments are separated into two.

    Let's see how this works with a simplest example --- downloading a table called nucleus_detection_v0. First, we reference the table as a function and then we run the query --- this is exactly the same as client.materialize.query_table('nucleus_detection_v0').

    client = CAVEclient('minnie65_public')\nnuc_df = client.materialize.tables.nucleus_detection_v0().query()\n

    Where things differ is when we add filters. If we want to query based on a set of values for the field \"id\", for example, we add that as an argument:

    my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(id=my_ids).query()\n

    Where in this example the id= queries the column id based on the schema. These values can be either individual elements (i.e. an integer or a string) or a list/array of elements, and any field can be used. The tooling will automatically sort out how to format the filtering appropriately when running the query. Importantly, the filtering is identical between querying all types of tables and queries. To see the complete list of fields that can be queried, you can tab-autocomplete or in Jupyter or IPython glance at the docstring with client.materialize.tables.nucleus_detection_v0?.

    In addition to filtering by one or many values, you can also do spatial queries. To find only annotations within a particular bounding box, you need to specify the spatial point (e.g. pt_position). For each such spatial point, there will be an argument {spatial_point_position}_bbox (e.g. pt_position_bbox) that accepts a 2x3 list of coordinates specifying the upper and lower bounds for the query. For example, to find all nucleus centroids within a particular bounding box, it would be:

    bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n    pt_position_bbox=bounding_box\n).query()\n

    If you are not using any filters, you can omit the parenthesis and use the get_all or get_all_live functions directly, which act similarly to the query and live_query functions respectively. The first example could be rewritten as:

    nuc_df = client.materialize.tables.nucleus_detection_v0.get_all()\n

    If you want to list all available fields, you can use the .fields attribute. Similarly, you can get all numeric fields with the .numeric_fields attribute and all spatial fields (allowing bounding box queries) with .spatial_fields.

    nuc_df = client.materialize.tables.nucleus_detection_v0.spatial_fields\n

    If you need to specify the table programmatically, you can also use a dictionary-style approach to getting the table filtering function. For example, an equivalent version of the above line would be:

    my_ids = [373879, 111162]\nmy_table = 'nucleus_detection_v0'\nnuc_df = client.materialize.tables[my_table](id=my_ids).query()\n

    The query function can also take arguments relating to timestamps or formatting where they act just like in the other query method. In particular, the arguments that apply to query are: select_columns, offset, limit, split_positions, materialization_version, timestamp, metadata, desired_resolution, and get_counts. For example, to add a desired resolution and split positions in the above query, it would look like:

    my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n    id=my_ids\n).query(\n    split_positions=True,\n    desired_resolution=[1,1,1],\n)\n

    Inequalities can also be used in filtering numeric columns. Here, you can pass a dictionary instead of a list of values, with the keys being inequality operators (\">\", \">=\", \"<\", and \"<=\") and the values being the comparison. For example, to query for all nuclei with a volume greater than 1000:

    client.materialize.tables.nucleus_detection_v0(\n    volume={\">\": 1000}\n).query()\n

    You can also use multiple inequalities in the same dictionary to filter within a range. For example, to query for all nuclei with a volume between 500 and 750:

    client.materialize.tables.nucleus_detection_v0(\n    volume={\">\": 500, \"<\": 750}\n).query()\n

    If you want to do a live query instead of a materialized query, the filtering remains identical but we use the live_query function instead. The one required argument for live_query is the timestamp.

    my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n    id=my_ids\n).live_query(\n    timestamp=datetime.datetime.now(datetime.timezone.utc),\n)\n

    The live query functions have similar but slightly different arguments: timestamp (required), offset, limit, split_positions, metadata, desired_resolution, and allow_missing_lookups.

    Note that way that IPython handles docstrings means that while you can use ? to get the docstring of the filtering part of the function, you can't simply do something like client.materialize.tables.nucleus_detection_v0().query?. It will tell you the function can't be found, because technically the query function does not yet exist until the table filtering function is called.

    Instead, if you want to glimpse the docstring of the query or live_query functions, you need to split it into two lines:

    qry_func = client.materialize.tables.nucleus_detection_v0().query\nqry_func?\n

    Finally, if the project you are working with has views, a similar interface is available to them via client.materialize.views. Currently views are not compatible with live query, and so only the .query function is available.

    "},{"location":"tutorials/schemas/","title":"EM Annotation Schemas","text":"

    The EMAnnotationSchemas client lets one look up the available schemas and how they are defined. This is mostly used for programmatic interactions between services, but can be useful when looking up schema definitions for new tables.

    "},{"location":"tutorials/schemas/#get-the-list-of-schema","title":"Get the list of schema","text":"

    One can get the list of all available schema with the schema method. Currently, new schema have to be generated on the server side, although we aim to have a generic set available to use.

    client.schema.get_schemas()\n
    "},{"location":"tutorials/schemas/#view-a-specific-schema","title":"View a specific schema","text":"

    The details of each schema can be viewed with the schema_definition method, formatted as per JSONSchema.

    example_schema = client.schema.schema_definition('microns_func_coreg')\nexample_schema\n

    This is mostly useful for programmatic interaction between services at the moment, but can also be used to inspect the expected form of an annotation by digging into the format.

    example_schema['definitions']['FunctionalCoregistration']\n
    "},{"location":"tutorials/skeletonization/","title":"Skeletonization","text":"

    The skeleton service enables you to generate and retrieve skeletons from the server. This saves you the trouble of performing skeletonization routines on your local machine, while also saving you the trouble of understanding many of the details inside of skeletonization. Furthermore, the server offers the ability to generate multiple skeletons simultaneously, in parallel. And lastly, the server caches each skeleton that is generated so that it only ever needs to be generated once. All subsequent requests for the same skeleton will simply return the cached skeleton with notable turn-around-time improvements.

    "},{"location":"tutorials/skeletonization/#initializing-the-client","title":"Initializing the client","text":"

    The simplest way to initialize the CAVEclient is by merely providing the datastack of interest:

    import caveclient as cc\n\nclient = cc.CAVEclient(<datastack_name>)\n

    With a CAVEclient built, you can now investigate the current build version of the SkeletonClient:

    client.skeleton.get_version()\n

    And you can see a list of available skeleton versions. In most cases you will want to use the highest (most recent) version provided.

    client.skeleton.get_versions()\n
    "},{"location":"tutorials/skeletonization/#retrieving-a-skeleton","title":"Retrieving a skeleton","text":"

    Retrieve a skeleton using get_skeleton(). The simplest usage is:

    sk = client.skeleton.get_skeleton(\n    <root_id>,\n    output_format=<output_format>,\n)\n

    where the availale output_formats (described below) are:

    • 'dict' (default if unspecified)
    • 'swc' (a Pandas Dataframe)

    If the skeleton doesn't exist in the server cache, it may take 20-60 seconds to generate the skeleton before it is returned. This function will block during that time. Any subsequent retrieval of the same skeleton should go very quickly however.

    To specify a nondefault skeleton version:

    sk = client.skeleton.get_skeleton(\n    <root_id>,\n    skeleton_version=<sk_version>,\n    output_format=<output_format>,\n)\n

    To specify a specific datastack:

    sk = client.skeleton.get_skeleton(\n    <root_id>,\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n    output_format=<output_format>,\n)\n
    "},{"location":"tutorials/skeletonization/#peering-into-the-contents-of-the-cache","title":"Peering into the contents of the cache","text":"

    Most end-users shouldn't need to use the following function very much, but to see the contents of the cache for a given root id, set of root ids, root id prefix, or set of prefixes:

    get_cache_contents(\n    root_id_prefixes=<root_id>,\n)\n

    You can also add additional parameters as needed:

    get_cache_contents(\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n    root_id_prefixes=<root_id>,\n)\n

    The primary parameter, root_id_prefixes, can be a list of root ids:

    get_cache_contents(\n    root_id_prefixes=[<root_id>, <root_id>, ...],\n)\n

    The primary parameter can also be a root id prefix, which will match any associated root ids. Since this could potentially return a large list of results, there is an optional limit parameter so you don't overwhelm the memory of your processing environment, e.g., a Jupyter notebook or some other Python script running on your local machine:

    get_cache_contents(\n    root_id_prefixes=<root_id_prefix>,\n    limit=<limit>,\n)\n

    Note that the limit only constraints the size of the return value. The internal operation of the function will still receive the full list when it passes the prefix on to CloudFiles. Consequently, calling this function for a short prefix may block for a long time.

    And of course you can also pass in a list of prefixes (or a mixture of full ids and partial prefixes):

    get_cache_contents(\n    root_id_prefixes=[<root_id_prefix>, <root_id_prefix>, ...],\n    limit=<limit>,\n)\n
    "},{"location":"tutorials/skeletonization/#querying-the-presence-of-a-skeleton-in-the-cache","title":"Querying the presence of a skeleton in the cache","text":"

    The function shown above isn't necessarily the most direct way to simply inquire whether a skeleton exists in the cache for a given root id. For that purpose, the following function is better suited:

    skeletons_exist(\n    root_ids=<root_id>,\n)\n

    Or:

    skeletons_exist(\n    root_ids=[<root_id>, <root_id>, ...],\n)\n

    Note that this function doesn't accept prefixes, as supported by cache_query_contents(). Only full root ides are supported. When querying with as single root id, the return value will be a boolean. When querying with a list of ids, the return value will be a Python dictionary mapping from each id to a boolean.

    This function also takes the same optional parameters described above:

    skeletons_exist(\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n    root_ids=<root_id>,  # Or [<root_id>, <root_id>, ...],\n)\n
    "},{"location":"tutorials/skeletonization/#retrieving-multiple-skeletons","title":"Retrieving multiple skeletons","text":"

    You can retrieve a large set of skeletons in a single function call:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n)\n

    If any skeletons are not generated yet, the default behavior is to skip those root ids and only return skeletons that are already available. But you can override this default behavior:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    generate_missing_skeletons=[False|True],\n)\n

    Any root ids for which skeletonization is required will be generated one at a time, at a cost of 20-60 seconds each. Consequently, there is a hard-coded limit of 10, after which all subsequent missing skeletons will not be returned.

    By default, skeletons are returned in JSON format. However SWC is also supported, thusly:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    output_format=<\"json\"|\"swc\">\n)\n

    And the usual defaults can be overridden again:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n)\n
    "},{"location":"tutorials/skeletonization/#generating-multiple-skeletons-in-parallel","title":"Generating multiple skeletons in parallel","text":"

    get_bulk_skeletons() is not an effective way to produce a large number of skeletons since it operates synchronously, generating one skeleton at a time. In order to generate a large number of skeletons it is better to do so in parallel. The following function dispatches numerous root ids for skeletonization without returning anything immediately. The root ids are then distributed on the server for parallel skeletonization and eventual caching. Once they are in the cache, you can retrieve them. Of course, it can be tricky to know when they are available. That is addressed further below. Here's how to dispatch asynchronous bulk skeletonization:

    generate_bulk_skeletons_async(\n    root_ids=[<root_id>, <root_id>, ...],\n)\n

    And with the usual overrides:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n)\n
    "},{"location":"tutorials/skeletonization/#retrieving-asynchronously-generated-skeletons","title":"Retrieving asynchronously generated skeletons","text":"

    In order to retrieve asynchronously generated skeletons, it is necessary to poll the cache for the availability of the skeletons and then eventually retrieve them. Here's an example of such a workflow:

    # Dispatch multiple asynchronous, parallel skeletonization and caching processes\ngenerate_bulk_skeletons_async(root_ids)\n\n# Repeatedly query the cache for the existence of the skeletons until they are all available\nwhile True:\n    skeletons_that_exist = client.skeleton.skeletons_exist(root_ids=rids)\n    num_skeletons_found = sum(skeletons_that_exist.values())\n    if num_skeletons_found == len(rids):\n        break\n    sleep(10)  # Pause for ten seconds and check again\n\n# Retrieve the skeletons (remember, SWC is also offered)\nskeletons_json = get_bulk_skeletons(root_ids)\n
    "},{"location":"tutorials/state/","title":"JSON Neuroglancer State Service","text":"

    We store the JSON description of a Neuroglancer state in a simple database at the JSON Service. This is a convenient way to build states to distribute to people, or pull states to parse work by individuals. The JSON Client is at client.state

    client.state\n
    "},{"location":"tutorials/state/#retrieving-a-state","title":"Retrieving a state","text":"

    JSON states are found simply by their ID, which you get when uploading a state. You can download a state with get_state_json.

    example_id = 4845531975188480\nexample_state = client.state.get_state_json(test_id)\nexample_state['layers'][0]\n
    "},{"location":"tutorials/state/#uploading-a-state","title":"Uploading a state","text":"

    You can also upload states with upload_state_json. If you do this, the state id is returned by the function. Note that there is no easy way to query what you uploaded later, so be VERY CAREFUL with this state id if you wish to see it again.

    Note: If you are working with a Neuroglancer Viewer object or similar, in order to upload, use viewer.state.to_json() to generate this representation.

    example_state['layers'][0]['name'] = 'example_name'\nnew_id = client.state.upload_state_json(example_state)\n
    test_state = client.state.get_state_json(new_id)\ntest_state['layers'][0]['name']\n
    "},{"location":"tutorials/state/#generating-a-neuroglancer-url","title":"Generating a Neuroglancer URL","text":"

    Once you have a state ID, you want to turn it into a well-formatted link. So you don\\'t have to remember all the endpoints, we can do this from the state client.

    ngl_base = 'neuromancer-seung-import.appspot.com'\nclient.state.build_neuroglancer_url(new_id, ngl_base)\n

    Note that the neuroglancer base can be found in the info service under client.info.viewer_site().

    "}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"

    CAVE is short for Connectome Annotation Versioning Engine. CAVE is a set of microservices that provide a framework for storing and versioning connectomics data and large sets of dynamic annotations, metadata, and segmentations. This repository supplies client-side code to easily interact with the microservices in CAVE.

    "},{"location":"#documentation-layout","title":"Documentation Layout","text":"

    To learn how to install caveclient, visit Installation.

    To see hands-on examples of using caveclient in a notebook, visit the Tutorials.

    To see API reference documentation for interacting with a caveclient.CAVEclient object, visit the Client API.

    To see a glossary of terms used in the documentation, visit the Glossary (work in progress).

    For information about how to contribute to the documentation or the package, visit the Contributing page. Feedback on the documentation is welcome! Please open an issue or use the \"Edit this page\" button at the top right of any page to suggest changes.

    To read a full description of the Connectome Annotation Versioning Engine, including a description of the various services please read this paper.

    "},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#740-december-12-2024","title":"7.4.0 (December 12, 2024)","text":"

    annotation module: Added support for \"None\", \"none\", etc. to remove notice_text and improved documentation on method

    "},{"location":"changelog/#732-december-10-2024","title":"7.3.2 (December 10, 2024)","text":"

    Dropped oldest supported numpy requirements and declared python 3.8 support

    "},{"location":"changelog/#731-december-5-2024","title":"7.3.1 (December 5, 2024)","text":"

    Improved documentation to annotation.update_table_metadata

    "},{"location":"changelog/#730-december-5-2024","title":"7.3.0 (December 5, 2024)","text":"

    Added get_minimal_covering_nodes to chunkedgraph (#274)

    "},{"location":"changelog/#721-december-3-2024","title":"7.2.1 (December 3, 2024)","text":"

    Added raw strings to avoid deprecation warning (#275)

    "},{"location":"changelog/#720-december-3-2024","title":"7.2.0 (December 3, 2024)","text":"

    Allow get_state_json to accept direct URLs (#271)

    "},{"location":"changelog/#710-december-3-2024","title":"7.1.0 (December 3, 2024)","text":"

    Added get_leaves_multiple to chunkedgraph module (#273)

    "},{"location":"changelog/#700-november-20-2024","title":"7.0.0 (November 20, 2024)","text":"
    • Simplified skeleton client interface to only accept formats of 'dict' and 'swc' Structure of the dictionary returned was also changed compared to previous \"json\" format, where the dict is flat with only one level of keys. Corresponding change made to MeshParty 1.18.0 allows for hydration of MeshParty Skeleton object from this dict format.
    "},{"location":"changelog/#650-november-15-2024","title":"6.5.0 (November 15, 2024)","text":"
    • Added endpoints for bulk skeleton retrieval and querying whether a skeleton exists or not.
    "},{"location":"changelog/#641","title":"6.4.1","text":"
    • Fix networkx deprecation warning.
    "},{"location":"changelog/#640-october-31-2024","title":"6.4.0 (October 31, 2024)","text":"
    • Added a CAVEclientMock function to ease the use of fully mocked CAVEclients in testing.
    • Fix a bug about use of global server.
    • Fix writing of local server secret to be more secure.
    "},{"location":"changelog/#630","title":"6.3.0","text":"
    • Add a convenience function to get a complete L2 feature dataframe.
    "},{"location":"changelog/#620","title":"6.2.0","text":"
    • Fix things so that auth will not error if the secrets directory is read-only.
    "},{"location":"changelog/#612","title":"6.1.2","text":"
    • Fix query filter bug in tables interface.
    "},{"location":"changelog/#611","title":"6.1.1","text":"
    • Fix warnings.
    "},{"location":"changelog/#610","title":"6.1.0","text":"
    • Added support for inequality filters for numerical columns in materialization tables interface.
    "},{"location":"changelog/#600-october-14-2024","title":"6.0.0 (October 14, 2024)","text":"
    • Refactored CAVEclient internals away from a factory syntax. Most users should not notice a difference, but code that relied on specific subclient type logic may be affected.
    "},{"location":"changelog/#5250","title":"5.25.0","text":"
    • Added ability to suppress table warnings in client.materialize.query_table and similar methods by setting log_warnings=False
    "},{"location":"changelog/#5200-april-8-2024","title":"5.20.0 (April 8, 2024)","text":"
    • Added generalized support for detecting server versions to provide timely exceptions to users
    • Used new support to check that chunkegraph has updated version before using spatial bounds kwarg on client.chunkedgraph.level2_chunk_graph
    • Added support for postign and getting segment properties files to client.state
    "},{"location":"changelog/#5180","title":"5.18.0","text":"
    • Added serialization support for pandas.index
    "},{"location":"changelog/#5173","title":"5.17.3","text":"
    • Minor documentation typo fix
    "},{"location":"changelog/#5172","title":"5.17.2","text":"
    • Bug fixes related to table_manager interface
    "},{"location":"changelog/#5171","title":"5.17.1","text":"
    • Bug fixes related to table_manager interface
    "},{"location":"changelog/#5170","title":"5.17.0","text":"
    • Fix attrs in dataframe attributes of client.materialize results to remove numpy arrays to allow concatenation of dataframes
    • Added getting multiple schemas in one call to improve initialization of table_manager interface of materialization
    "},{"location":"changelog/#5161","title":"5.16.1","text":"
    • Bugfix on client.chunkedgrpah.level2_chunk_graph
    "},{"location":"changelog/#5160","title":"5.16.0","text":"
    • Added bounding box query to client.chunkedgraph.level2_chunk_graph
    • Fix default materialization version client when server not advertising correctly
    "},{"location":"changelog/#5151-jan-18-2024","title":"5.15.1 (Jan 18, 2024)","text":"
    • minor improvements to release process
    "},{"location":"changelog/#5150-jan-18-2024","title":"5.15.0 (Jan 18, 2024)","text":"
    • Improved documentation with types
    • Improved testing on more python versions
    • Bugfixes for pyton 3.12 compatability
    "},{"location":"changelog/#5140-november-24-2023","title":"5.14.0 (November 24, 2023)","text":"
    • Made automatic detection of neuroglancer versioning when constructing link shortener links
    "},{"location":"changelog/#5130-october-26-2023","title":"5.13.0 (October 26, 2023)","text":"
    • Add option to get expired versions to client.materialize.get_versions
    "},{"location":"changelog/#5121-october-16-2023","title":"5.12.1 (October 16, 2023)","text":"
    • Bugfixes for client.chunkedgraph.get_latest_roots
    "},{"location":"changelog/#5120-october-16-2023","title":"5.12.0 (October 16, 2023)","text":"
    • Improved logic for client.chunkedgraph.get_latest_roots to work forward or backwards in time
    "},{"location":"changelog/#5110-september-19-2023","title":"5.11.0 (September 19, 2023)","text":"
    • Added filter_regex_dict options to client.materialize.query_table interface
    "},{"location":"changelog/#5102-august-162023","title":"5.10.2 (August 16,2023)","text":"
    • Fixed pyarrow support for live_live query
    "},{"location":"changelog/#5101-august-142023","title":"5.10.1 (August 14,2023)","text":"
    • Changed random_sample argument to be an integer number of annotations rather than a floating fraction of table
    • Added option to live_query
    "},{"location":"changelog/#590-august-14-2023","title":"5.9.0 (August 14, 2023)","text":"
    • Added support for native pyarrow deserialization, allowing upgrade to pyarrow version
    "},{"location":"changelog/#580","title":"5.8.0","text":"
    • Allowed int64 root ids to serialize properly
    • Added warning that client.materialize.tables interface is in beta
    "},{"location":"changelog/#570","title":"5.7.0","text":"
    • Fix to ensure stop_layer is at least 1
    • Added client.chunkedgraph.suggest_latest_roots
    "},{"location":"changelog/#560","title":"5.6.0","text":"
    • Added views to client.materialize.tables interface
    • Added optional argument to allow invalid root ids when querying live live, versus creating an exception
    "},{"location":"changelog/#551","title":"5.5.1","text":"
    • documentation fixes on client.materialize.join_query
    "},{"location":"changelog/#550","title":"5.5.0","text":"
    • added methods for different neuroglancer state formats to client.state.
    "},{"location":"changelog/#543","title":"5.4.3","text":"
    • Added 'view' querying options to materialization
    • Added client.materialize.tables interface
    • Added client.materialize.get_tables_metadata to get all metadata in one call
    "},{"location":"changelog/#520","title":"5.2.0","text":"
    • Added local caching of datastack names > server_address to simplify initialization of clients with servers other than global.daf-apis.com.

    Cache is saved on a local file ~/.cloudvolume/secrets/cave_datastack_to_server_map.json

    Cache will populate the first time caveclient.CAVEclient('my_datastack', server_address=\"https://my_server.com\") is called. Subsequent calls can then just be caveclient.CAVEclient('my_datastack').

    "},{"location":"changelog/#510","title":"5.1.0","text":"
    • Added get_oldest_timestamp call to chunkedgraph
    "},{"location":"changelog/#501","title":"5.0.1","text":"
    • Fixed bug with desired_resolution being set at the client level was being ignored in >5.0.0
    "},{"location":"changelog/#500","title":"5.0.0","text":"
    • Added support for the new CAVE Materialization 3.0 API Includes support for the new materialization API, which allows for server side conversion of the units of position, and ensures that all positions are returned with the same units, even after joins.
    • Added support for querying databases that were materialized without merging tables together. This will allow for faster materializations.
    • Removed support for LiveLive query from the Materialization 2.0 API client. Note.. <5.0.0 clients interacting with MaterializationEngine >4.7.0 servers will use live live query but will doubly convert the units of position if you ask for a desired resolution, as the old client will also do a conversion server side.
    • Fixed interaction with api version querying of servers from individual clients to work with verify=False. (useful for testing)
    • Stored infromation from client about mapping between dataframe and table names and original column names.
    • Added support for suffixes and select columns to be passed by dictionary rather than list making the selection an application of suffixes more explicit when there are collisions between column names in joined tables.
    "},{"location":"changelog/#older-upgrade-notes","title":"Older Upgrade Notes","text":"

    Change all select_column calls to pass dictionaries rather than lists. Change all suffix calls to pass dictionaries rather than lists. Advocate for your server administrator to upgrade to MaterializationEngine 4.7.0 or later, so you can use the new MaterializationEngine 3.0 API and client.

    "},{"location":"contributing/","title":"Contributing","text":"

    Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

    You can contribute in many ways:

    "},{"location":"contributing/#types-of-contributions","title":"Types of Contributions","text":""},{"location":"contributing/#report-bugs","title":"Report Bugs","text":"

    Report bugs to our issues page.

    If you are reporting a bug, please include:

    • Your operating system name and version.
    • Any details about your local setup that might be helpful in troubleshooting.
    • Detailed steps to reproduce the bug, in the form of a minimal reproducible example.
    "},{"location":"contributing/#fix-bugs","title":"Fix Bugs","text":"

    Look through the GitHub issues for bugs. Anything tagged with \"bug\" and \"help wanted\" is open to whoever wants to implement it.

    "},{"location":"contributing/#implement-features","title":"Implement Features","text":"

    Look through the GitHub issues for features. Anything tagged with \"enhancement\" and \"help wanted\" is open to whoever wants to implement it.

    "},{"location":"contributing/#write-documentation","title":"Write Documentation","text":"

    caveclient could always use more documentation, whether as part of the official caveclient docs, in docstrings, or even on the web in blog posts, articles, and such.

    "},{"location":"contributing/#submit-feedback","title":"Submit Feedback","text":"

    The best way to send feedback is to create an issue on GitHub.

    If you are proposing a feature:

    • Explain in detail how it would work.
    • Keep the scope as narrow as possible, to make it easier to implement.
    • Remember that while contributions are welcome, developer/maintainer time is limited.
    "},{"location":"contributing/#modifying-code","title":"Modifying Code","text":"

    Ready to contribute? Here's how to set up caveclient for local development.

    • Fork the repo on GitHub.
    • Clone your fork locally
    git clone git@github.com:your_name_here/CAVEclient.git\n
    • We use uv for various developer tasks. Ensure you have uv installed according to the installation instructions.

    Note

    While we recommend using uv as described here, these tasks could also be achieved using pip to install and run the various required tools. You can view the development requirements and build/check commands in our pyproject.toml, so we avoid duplicating them here.

    • Navigate to the newly cloned directory, e.g.:
    cd CAVEclient\n
    • Create a synced virtual environment, optionally specifying a Python version:
    uv sync --python 3.12\n
    • Create a branch for local development:
    git checkout -b name-of-your-bugfix-or-feature\n

    • Make your changes locally

    • If you have added code that should be tested, add tests.

    • If you have modified dependencies in any way, make sure to run

    uv sync\n
    • Make sure you have added documentation for any additions or modifications to public functions or classes. You can build the documentation locally to make sure it renders correctly with:
    uvx --from poethepoet poe doc-build\n
    "},{"location":"contributing/#automated-checks","title":"Automated Checks","text":"
    • Run the autoformatter:
    uvx --from poethepoet poe lint-fix\n
    • Ensure that your changes pass the checks that will be run on Github Actions, including building the documentation, checking the formatting of the code, and running the tests. To run all at once, do:
    uvx --from poethepoet poe checks\n

    • You may be interested in running some of these checks individually, such as:

      • To run the tests:
      uvx --from poethepoet poe test\n
      • To build the documentation:
      uvx --from poethepoet poe doc-build\n
      • To run the linter
      uvx --from poethepoet poe lint\n
    "},{"location":"contributing/#submitting-a-pull-request","title":"Submitting a Pull Request","text":"
    • Ensure your code has passed all of the tests described above.
    • Commit your changes and push your branch to GitHub:
    git add .\ngit commit -m \"Your detailed description of your changes.\"\ngit push origin name-of-your-bugfix-or-feature\n
    • Submit a pull request through the GitHub website.

    Before you submit a pull request, check that it meets these guidelines:

    • The pull request should include tests if adding a new feature.

    • The docs should be updated with whatever changes you have made. Put your new functionality into a function with a docstring, and make sure the new functionality is documented after building the documentation (described above).

    • Once you submit a pull request, automated checks will run. You may require administrator approval before running these checks if this is your first time contributing to the repo.

    "},{"location":"glossary/","title":"Glossary","text":"

    Warning

    This glossary is a work in progress; for now we are documenting the commonly used terms that we need to define. Please feel free to contribute definitions or additional terms.

    • Datastack
    • Voxel resolution
    • Segmentation resolution
    • MIP
    • Segmentation
    • View
    "},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#stable-release","title":"Stable release","text":"

    To install caveclient, run this command in your terminal:

    pip install caveclient\n

    This is the preferred method to install caveclient, as it will always install the most recent stable release.

    You can also specify a particular version, e.g.

    pip install caveclient==5.0.0\n

    If you don't have pip installed, this Python installation guide can guide you through the process.

    "},{"location":"installation/#from-source","title":"From source","text":"

    The source for caveclient can be downloaded from the Github repo.

    You can either clone the public repository:

    git clone git://github.com/CAVEconnectome/CAVEclient\n

    Or download the tarball:

    curl -OJL https://github.com/CAVEconnectome/CAVEclient/tarball/master\n

    Once you have a copy of the source, you can install it with:

    pip install .\n

    Or in editable mode, it can be installed with:

    pip install -e .\n
    "},{"location":"api/","title":"Overview","text":"

    The most common method of interacting with the CAVE Framework is by instantiating a client (caveclient.CAVEclient) and then using that client to interact with various services. Under the hood, the CAVEclient is a collection of individual clients, which can be accessed via properties. For example, to access the materialization client, you can use client.materialize, which (up to the exact version) will actually return a MaterializationClient object.

    These pages describe the functionality of each of the individual clients, but note that some features may not be available if the server you are accessing is using older versions of particular services.

    "},{"location":"api/annotation/","title":"client.annotation","text":"

    Client for interacting with the annotation engine.

    Methods:

    Name Description create_table

    Creates a new data table based on an existing schema

    delete_annotation

    Delete one or more annotations in a table. Annotations that are

    delete_table

    Marks a table for deletion requires super admin privileges

    get_annotation

    Retrieve an annotation or annotations by id(s) and table name.

    get_annotation_count

    Get number of annotations in a table

    get_table_metadata

    Get metadata about a table

    get_tables

    Gets a list of table names for a aligned_volume_name

    post_annotation

    Post one or more new annotations to a table in the AnnotationEngine.

    post_annotation_df

    Post one or more new annotations to a table in the AnnotationEngine.

    process_position_columns

    Process a dataframe into a list of dictionaries

    raise_for_status

    Raises requests.HTTPError, if one occurred.

    stage_annotations

    Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema.

    update_annotation

    Update one or more new annotations to a table in the AnnotationEngine.

    update_annotation_df

    Update one or more annotations to a table in the AnnotationEngine using a

    update_metadata

    Update the metadata on an existing table

    upload_staged_annotations

    Upload annotations directly from an Annotation Guide object.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.create_table","title":"create_table(table_name, schema_name, description, voxel_resolution, reference_table=None, track_target_id_updates=None, flat_segmentation_source=None, user_id=None, aligned_volume_name=None, write_permission='PRIVATE', read_permission='PUBLIC', notice_text=None)","text":"

    Creates a new data table based on an existing schema

    Parameters:

    Name Type Description Default table_name str

    Name of the new table. Cannot be the same as an existing table

    required schema_name str

    Name of the schema for the new table.

    required description str

    Human readable description for what is in the table. Should include information about who generated the table What data it covers, and how it should be interpreted. And who should you talk to if you want to use it. An Example: a manual synapse table to detect chandelier synapses on 81 PyC cells with complete AISs [created by Agnes - agnesb@alleninstitute.org, uploaded by Forrest]

    required voxel_resolution List[float]

    voxel resolution points will be uploaded in, typically nm, i.e [1,1,1] means nanometers [4,4,40] would be 4nm, 4nm, 40nm voxels

    required reference_table str

    If the schema you are using is a reference schema Meaning it is an annotation of another annotation. Then you need to specify what the target table those annotations are in.

    None track_target_id_updates bool

    Indicates whether to automatically update reference table's foreign key if target annotation table row is updated.

    None flat_segmentation_source str

    the source to a flat segmentation that corresponds to this table i.e. precomputed:\\gs:\\mybucket his_tables_annotation

    None user_id int

    If you are uploading this schema on someone else's behalf and you want to link this table with their ID, you can specify it here Otherwise, the table will be created with your userID in the user_id column.

    None aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None write_permission str

    What permissions to give the table for writing. One of PRIVATE: only you can write to this table (DEFAULT) GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable

    'PRIVATE' read_permission str

    What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data (DEFAULT)

    'PUBLIC' notice_text str

    Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None, no text. If you want to remove text, send empty string.

    None

    Returns:

    Type Description json

    Response JSON

    Examples:

    Basic annotation table:

    description = \"Some description about the table\"\nvoxel_res = [4,4,40]\nclient.create_table(\"some_synapse_table\", \"synapse\", description, voxel_res)\n
    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_annotation","title":"delete_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"

    Delete one or more annotations in a table. Annotations that are deleted are recorded as 'non-valid' but are not physically removed from the table.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required annotation_ids (dict or list)

    A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON: a list of new annotation IDs.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_table","title":"delete_table(table_name, aligned_volume_name=None)","text":"

    Marks a table for deletion requires super admin privileges

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation","title":"get_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"

    Retrieve an annotation or annotations by id(s) and table name.

    Parameters:

    Name Type Description Default table_name str

    Name of the table

    required annotation_ids int or iterable

    ID or IDS of the annotation to retreive

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description list

    Annotation data

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation_count","title":"get_annotation_count(table_name, aligned_volume_name=None)","text":"

    Get number of annotations in a table

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description int

    number of annotations

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_table_metadata","title":"get_table_metadata(table_name, aligned_volume_name=None)","text":"

    Get metadata about a table

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required aligned_volume_name str

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    metadata about table

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_tables","title":"get_tables(aligned_volume_name=None)","text":"

    Gets a list of table names for a aligned_volume_name

    Parameters:

    Name Type Description Default aligned_volume_name str or None

    Name of the aligned_volume, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client

    None

    Returns:

    Type Description list

    List of table names

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation","title":"post_annotation(table_name, data, aligned_volume_name=None)","text":"

    Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations refer to 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required data (dict or list)

    A list of (or a single) dict of schematized annotation data matching the target table.

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation_df","title":"post_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"

    Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations see 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required df DataFrame

    A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.

    required position_columns Optional[Union[Iterable[str], Mapping[str, str]]]

    if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.process_position_columns","title":"process_position_columns(df, position_columns) staticmethod","text":"

    Process a dataframe into a list of dictionaries

    Parameters:

    Name Type Description Default df DataFrame

    Dataframe to process

    required position_columns Optional[Union[Iterable[str], Mapping[str, str]]]

    See .post_annotation_df

    required

    Returns:

    Type Description dict

    Annotations ready for posting

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.raise_for_status","title":"raise_for_status(r, log_warning=True) staticmethod","text":"

    Raises requests.HTTPError, if one occurred.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.stage_annotations","title":"stage_annotations(table_name=None, schema_name=None, update=False, id_field=False, table_resolution=None, annotation_resolution=None)","text":"

    Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema. StagedAnnotation objects can be uploaded directly with upload_staged_annotations.

    Parameters:

    Name Type Description Default table_name str

    Table name to stage annotations for, by default None.

    None schema_name str

    Schema name to use to make annotations. Only needed if the table_name is not set, by default None

    None update bool

    Set to True if individual annotations are going to be updated, by default False.

    False id_field bool

    Set to True if id fields are to be specified. Not needed if update is True, which always needs id fields. Optional, by default False

    False table_resolution list - like or None

    Voxel resolution of spatial points in the table in nanometers. This is found automatically from the info service if a table name is provided, by default None. If annotation_resolution is also set, this allows points to be scaled correctly for the table.

    None annotation_resolution list - like

    Voxel resolution of spatial points provided by the user when creating annotations. If the table resolution is also available (manually or from the info service), annotations are correctly rescaled for the volume. By default, None.

    None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation","title":"update_annotation(table_name, data, aligned_volume_name=None)","text":"

    Update one or more new annotations to a table in the AnnotationEngine. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.

    Notes

    If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required data (dict or list)

    A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON: a list of new annotation IDs.

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation_df","title":"update_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"

    Update one or more annotations to a table in the AnnotationEngine using a dataframe as format. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.

    Notes

    If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.

    Parameters:

    Name Type Description Default table_name str

    Name of the table where annotations will be added

    required df DataFrame

    A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.

    required position_columns Iterable[str] or Mapping[str, str] or None

    if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description json

    Response JSON

    "},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_metadata","title":"update_metadata(table_name, description=None, flat_segmentation_source=None, read_permission=None, write_permission=None, user_id=None, notice_text=None, aligned_volume_name=None)","text":"

    Update the metadata on an existing table

    Parameters:

    Name Type Description Default table_name str required description str

    Defaults to the None type, in which case no change will be made to this metadata field.

    None flat_segmentation_source str

    Defaults to the None type, in which case no change will be made to this metadata field.

    None read_permission str

    What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data Defaults to the None type, in which case no change will be made to this metadata field.

    None write_permission str

    What permissions to give the table for writing. One of PRIVATE: only you can write to this table GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable Defaults to the None type, in which case no change will be made to this metadata field.

    None user_id int

    Note, if you use this you will not be able to update the metadata on this table any longer and depending on permissions may not be able to read or write to it Defaults to the None type, in which case no change will be made to this metadata field.

    None notice_text str

    Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. If you wish to remove the notice_text pass an empty string, or \"none\", \"None\", \"NONE\", or any other capitaliztion of the word \"none\". Defaults to the None type, in which case no change will be made to this metadata field.

    None aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.upload_staged_annotations","title":"upload_staged_annotations(staged_annos, aligned_volume_name=None)","text":"

    Upload annotations directly from an Annotation Guide object. This method uses the options specified in the object, including table name and if the annotation is an update or not.

    Parameters:

    Name Type Description Default staged_annos AnnotationGuide

    AnnotationGuide object with a specified table name and a collection of annotations already filled in.

    required aligned_volume_name str or None

    Name of the aligned_volume. If None, uses the one specified in the client.

    None

    Returns:

    Type Description List or dict

    If new annotations are posted, a list of ids. If annotations are being updated, a dictionary with the mapping from old ids to new ids.

    "},{"location":"api/auth/","title":"client.auth","text":"

    Client to find and use auth tokens to access the other services.

    Methods:

    Name Description get_group_users

    Get users in a group

    get_new_token

    Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.

    get_token

    Load a token with a given key the specified token file

    get_tokens

    Get the tokens setup for this users

    get_user_information

    Get user data.

    save_token

    Conveniently save a token in the correct format.

    setup_token

    Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment.

    Attributes:

    Name Type Description request_header

    Formatted request header with the specified token

    token

    Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.

    "},{"location":"api/auth/#caveclient.auth.AuthClient.request_header","title":"request_header property","text":"

    Formatted request header with the specified token

    "},{"location":"api/auth/#caveclient.auth.AuthClient.token","title":"token property writable","text":"

    Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.

    "},{"location":"api/auth/#caveclient.auth.AuthClient.get_group_users","title":"get_group_users(group_id)","text":"

    Get users in a group

    Parameters:

    Name Type Description Default group_id int

    ID value for a given group

    required

    Returns:

    Type Description list

    List of dicts of user ids. Returns empty list if group does not exist.

    "},{"location":"api/auth/#caveclient.auth.AuthClient.get_new_token","title":"get_new_token(open=False, no_text=False)","text":"

    Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.

    Parameters:

    Name Type Description Default open bool

    If True, opens a web browser to the web page where you can generate a new token.

    False"},{"location":"api/auth/#caveclient.auth.AuthClient.get_token","title":"get_token(token_key=None)","text":"

    Load a token with a given key the specified token file

    Parameters:

    Name Type Description Default token_key str or None

    key in the token file JSON, by default None. If None, uses 'token'.

    None"},{"location":"api/auth/#caveclient.auth.AuthClient.get_tokens","title":"get_tokens()","text":"

    Get the tokens setup for this users

    Returns:

    Type Description list[dict]:

    a list of dictionary of tokens, each with the keys \"id\": the id of this token \"token\": the token (str) \"user_id\": the users id (should be your ID)

    "},{"location":"api/auth/#caveclient.auth.AuthClient.get_user_information","title":"get_user_information(user_ids)","text":"

    Get user data.

    Parameters:

    Name Type Description Default user_ids list of int

    user_ids to look up

    required"},{"location":"api/auth/#caveclient.auth.AuthClient.save_token","title":"save_token(token=None, token_key=default_token_key, overwrite=False, token_file=None, switch_token=True, write_to_server_file=True, ignore_readonly=True, local_server=True)","text":"

    Conveniently save a token in the correct format.

    After getting a new token by following the instructions in authclient.get_new_token(), you can save it with a fully default configuration by running:

    token = 'my_shiny_new_token'

    authclient.save_token(token=token)

    Now on next load, authclient=AuthClient() will make an authclient instance using this token. If you would like to specify more information about the json file where the token will be stored, see the parameters below.

    Parameters:

    Name Type Description Default token str

    New token to save, by default None

    None token_key str

    Key for the token in the token_file json, by default \"token\"

    default_token_key overwrite bool

    Allow an existing token to be changed, by default False

    False token_file str

    Path to the token file, by default None. If None, uses the default file location specified above.

    None switch_token bool

    If True, switch the auth client over into using the new token, by default True

    True write_to_server_file bool

    If True, will write token to a server specific file to support this machine interacting with multiple auth servers.

    True ignore_readonly bool

    If True, will only attempt to save a token if the directory is writeable.

    True local_server bool

    If True, saves the token to the local server file as well.

    True"},{"location":"api/auth/#caveclient.auth.AuthClient.setup_token","title":"setup_token(make_new=True, open=True)","text":"

    Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.

    Parameters:

    Name Type Description Default make_new bool

    If True, will make a new token, else prompt you to open a page to retrieve an existing token.

    True open bool

    If True, opens a web browser to the web page where you can retrieve a token.

    True"},{"location":"api/chunkedgraph/","title":"client.chunkedgraph","text":"

    Client for interacting with the chunkedgraph.

    Methods:

    Name Description get_roots

    Get the root ID for a list of supervoxels.

    get_root_id

    Get the root ID for a specified supervoxel.

    get_minimal_covering_nodes

    Get the minimal covering nodes for a list of root IDs.

    get_merge_log

    Get the merge log (splits and merges) for an object.

    get_change_log

    Get the change log (splits and merges) for an object.

    get_user_operations

    Get operation details for a user ID. Currently, this is only available to

    get_tabular_change_log

    Get a detailed changelog for neurons.

    get_leaves_many

    Get all supervoxels for a list of root IDs.

    get_leaves

    Get all supervoxels for a root ID.

    do_merge

    Perform a merge on the chunked graph.

    undo_operation

    Undo an operation.

    execute_split

    Execute a multicut split based on points or supervoxels.

    preview_split

    Get supervoxel connected components from a preview multicut split.

    get_children

    Get the children of a node in the chunked graph hierarchy.

    get_contact_sites

    Get contacts for a root ID.

    find_path

    Find a path between two locations on a root ID using the level 2 chunked

    get_subgraph

    Get subgraph of root id within a bounding box.

    level2_chunk_graph

    Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.

    remesh_level2_chunks

    Submit specific level 2 chunks to be remeshed in case of a problem.

    get_operation_details

    Get the details of a list of operations.

    get_lineage_graph

    Returns the lineage graph for a root ID, optionally cut off in the past or

    get_latest_roots

    Returns root IDs that are related to the given root_id at a given

    get_original_roots

    Returns root IDs that are the latest successors of a given root ID.

    is_latest_roots

    Check whether these root IDs are still a root at this timestamp.

    suggest_latest_roots

    Suggest latest roots for a given root id, based on overlap of component

    is_valid_nodes

    Check whether nodes are valid for given timestamp range.

    get_root_timestamps

    Retrieves timestamps when roots where created.

    get_past_ids

    For a set of root IDs, get the list of IDs at a past or future time point

    get_delta_roots

    Get the list of roots that have changed between timetamp_past and

    get_oldest_timestamp

    Get the oldest timestamp in the database.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    timestamp Optional[datetime]

    The default timestamp for queries which expect a timestamp. If None, uses the

    segmentation_info

    Complete segmentation metadata

    base_resolution

    MIP 0 resolution for voxels assumed by the ChunkedGraph

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.timestamp","title":"timestamp: Optional[datetime.datetime] property writable","text":"

    The default timestamp for queries which expect a timestamp. If None, uses the current time.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.segmentation_info","title":"segmentation_info property","text":"

    Complete segmentation metadata

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.base_resolution","title":"base_resolution property","text":"

    MIP 0 resolution for voxels assumed by the ChunkedGraph

    Returns:

    Type Description list

    3-long list of x/y/z voxel dimensions in nm

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_roots","title":"get_roots(supervoxel_ids, timestamp=None, stop_layer=None)","text":"

    Get the root ID for a list of supervoxels.

    Parameters:

    Name Type Description Default supervoxel_ids list or np.array of int

    Supervoxel IDs to look up.

    required timestamp datetime

    UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.

    None stop_layer int or None

    If True, looks up IDs only up to a given stop layer. Default is None.

    None

    Returns:

    Type Description np.array of np.uint64

    Root IDs containing each supervoxel.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_id","title":"get_root_id(supervoxel_id, timestamp=None, level2=False)","text":"

    Get the root ID for a specified supervoxel.

    Parameters:

    Name Type Description Default supervoxel_id int

    Supervoxel id value

    required timestamp datetime

    UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description int64

    Root ID containing the supervoxel.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_minimal_covering_nodes","title":"get_minimal_covering_nodes(node_ids)","text":"

    Get the minimal covering nodes for a list of root IDs.

    Parameters:

    Name Type Description Default nodes_ids Iterable of int or np.int64

    List of root IDs to query.

    required

    Returns:

    Type Description np.array of np.int64:

    List of PCG node_ids that minimally and exactly cover the input nodes

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_merge_log","title":"get_merge_log(root_id)","text":"

    Get the merge log (splits and merges) for an object.

    Parameters:

    Name Type Description Default root_id int

    Object root ID to look up.

    required

    Returns:

    Type Description list

    List of merge events in the history of the object.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_change_log","title":"get_change_log(root_id, filtered=True)","text":"

    Get the change log (splits and merges) for an object.

    Parameters:

    Name Type Description Default root_id int

    Object root ID to look up.

    required filtered bool

    Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query object root_id (filtered=False). Defaults to True.

    True

    Returns:

    Type Description dict

    Dictionary summarizing split and merge events in the object history, containing the following keys:

    • \"n_merges\": int
      • Number of merges
    • \"n_splits\": int
      • Number of splits
    • \"operations_ids\": list of int
      • Identifiers for each operation
    • \"past_ids\": list of int
      • Previous root ids for this object
    • \"user_info\": dict of dict
      • Dictionary keyed by user (string) to a dictionary specifying how many merges and splits that user performed on this object
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_user_operations","title":"get_user_operations(user_id, timestamp_start, include_undo=True, timestamp_end=None)","text":"

    Get operation details for a user ID. Currently, this is only available to admins.

    Parameters:

    Name Type Description Default user_id int

    User ID to query (use 0 for all users (admin only)).

    required timestamp_start datetime

    Timestamp to start filter (UTC).

    required include_undo bool

    Whether to include undos. Defaults to True.

    True timestamp_end datetime

    Timestamp to end filter (UTC). Defaults to now.

    None

    Returns:

    Type Description DataFrame

    DataFrame including the following columns:

    • \"operation_id\": int
      • Identifier for the operation.
    • \"timestamp\": datetime.datetime
      • Timestamp of the operation.
    • \"user_id\": int
      • User who performed the operation.
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_tabular_change_log","title":"get_tabular_change_log(root_ids, filtered=True)","text":"

    Get a detailed changelog for neurons.

    Parameters:

    Name Type Description Default root_ids list of int

    Object root IDs to look up.

    required filtered bool

    Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query objects in root_ids (filtered=False). Defaults to True.

    True

    Returns:

    Type Description dict of pd.DataFrame

    The keys are the root IDs, and the values are DataFrames with the following columns and datatypes:

    • \"operation_id\": int
      • Identifier for the operation.
    • \"timestamp\": int
      • Timestamp of the operation, provided in milliseconds. To convert to datetime, use datetime.datetime.utcfromtimestamp(timestamp/1000).
    • \"user_id\": int
      • User who performed the operation.
    • \"before_root_ids: list of int
      • Root IDs of objects that existed before the operation.
    • \"after_root_ids: list of int
      • Root IDs of objects created by the operation. Note that this only records the root id that was kept as part of the query object, so there will only be one in this list.
    • \"is_merge\": bool
      • Whether the operation was a merge.
    • \"user_name\": str
      • Name of the user who performed the operation.
    • \"user_affiliation\": str
      • Affiliation of the user who performed the operation.
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves_many","title":"get_leaves_many(root_ids, bounds=None, stop_layer=None)","text":"

    Get all supervoxels for a list of root IDs.

    Parameters:

    Name Type Description Default root_ids Iterable

    Root IDs to query.

    required bounds

    If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.

    None stop_layer int

    If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).

    None

    Returns:

    Type Description dict

    Dict relating ids to contacts

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves","title":"get_leaves(root_id, bounds=None, stop_layer=None)","text":"

    Get all supervoxels for a root ID.

    Parameters:

    Name Type Description Default root_id int

    Root ID to query.

    required bounds

    If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.

    None stop_layer int

    If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).

    None

    Returns:

    Type Description np.array of np.int64

    Array of supervoxel IDs (or node ids if stop_layer>1).

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.do_merge","title":"do_merge(supervoxels, coords, resolution=(4, 4, 40))","text":"

    Perform a merge on the chunked graph.

    Parameters:

    Name Type Description Default supervoxels iterable

    An N-long list of supervoxels to merge.

    required coords array

    An Nx3 array of coordinates of the supervoxels in units of resolution.

    required resolution tuple

    What to multiply coords by to get nanometers. Defaults to (4,4,40).

    (4, 4, 40)"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.undo_operation","title":"undo_operation(operation_id)","text":"

    Undo an operation.

    Parameters:

    Name Type Description Default operation_id int

    Operation ID to undo.

    required

    Returns:

    Type Description dict"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.execute_split","title":"execute_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None)","text":"

    Execute a multicut split based on points or supervoxels.

    Parameters:

    Name Type Description Default source_points array or list

    Nx3 list or array of 3d points in nm coordinates for source points (red).

    required sink_points array or list

    Mx3 list or array of 3d points in nm coordinates for sink points (blue).

    required root_id int

    Root ID of object to do split preview.

    required source_supervoxels (array, list or None)

    If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None sink_supervoxels (array, list or None)

    If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None

    Returns:

    Name Type Description operation_id int

    Unique ID of the split operation

    new_root_ids list of int

    List of new root IDs resulting from the split operation.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.preview_split","title":"preview_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None, return_additional_ccs=False)","text":"

    Get supervoxel connected components from a preview multicut split.

    Parameters:

    Name Type Description Default source_points array or list

    Nx3 list or array of 3d points in nm coordinates for source points (red).

    required sink_points array or list

    Mx3 list or array of 3d points in nm coordinates for sink points (blue).

    required root_id int

    Root ID of object to do split preview.

    required source_supervoxels (array, list or None)

    If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None sink_supervoxels (array, list or None)

    If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.

    None return_additional_ccs bool

    If True, returns any additional connected components beyond the ones with source and sink points. In most situations, this can be ignored. By default, False.

    False

    Returns:

    Name Type Description source_connected_component list

    Supervoxel IDs in the component with the most source points.

    sink_connected_component list

    Supervoxel IDs in the component with the most sink points.

    successful_split bool

    True if the split worked.

    other_connected_components (optional) : list of lists of int

    List of lists of supervoxel IDs for any other resulting connected components. Only returned if return_additional_ccs is True.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_children","title":"get_children(node_id)","text":"

    Get the children of a node in the chunked graph hierarchy.

    Parameters:

    Name Type Description Default node_id int

    Node ID to query.

    required

    Returns:

    Type Description np.array of np.int64

    IDs of child nodes.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_contact_sites","title":"get_contact_sites(root_id, bounds, calc_partners=False)","text":"

    Get contacts for a root ID.

    Parameters:

    Name Type Description Default root_id int

    Root ID to query.

    required bounds

    Bounds within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]] for which to find contacts. Running this query without bounds is too slow.

    required calc_partners bool

    If True, get partner root IDs. By default, False.

    False

    Returns:

    Type Description dict

    Dict relating ids to contacts

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.find_path","title":"find_path(root_id, src_pt, dst_pt, precision_mode=False)","text":"

    Find a path between two locations on a root ID using the level 2 chunked graph.

    Parameters:

    Name Type Description Default root_id int

    Root ID to query.

    required src_pt array

    3-element array of xyz coordinates in nm for the source point.

    required dst_pt array

    3-element array of xyz coordinates in nm for the destination point.

    required precision_mode bool

    Whether to perform the search in precision mode. Defaults to False.

    False

    Returns:

    Name Type Description centroids_list array

    Array of centroids along the path.

    l2_path np.array of int

    Array of level 2 chunk IDs along the path.

    failed_l2_ids np.array of int

    Array of level 2 chunk IDs that failed to find a path.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_subgraph","title":"get_subgraph(root_id, bounds)","text":"

    Get subgraph of root id within a bounding box.

    Parameters:

    Name Type Description Default root_id int

    Root (or any node ID) of chunked graph to query.

    required bounds array

    3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates.

    required

    Returns:

    Type Description np.array of np.int64

    Node IDs in the subgraph.

    np.array of np.double

    Affinities of edges in the subgraph.

    np.array of np.int32

    Areas of nodes in the subgraph.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.level2_chunk_graph","title":"level2_chunk_graph(root_id, bounds=None)","text":"

    Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.

    Parameters:

    Name Type Description Default root_id int

    Root id of object

    required bounds array

    3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates (use client.chunkedgraph.base_resolution to view this default resolution for your chunkedgraph client). Note that the result will include any level 2 nodes which have chunk boundaries within some part of this bounding box, meaning that the representative point for a given level 2 node could still be slightly outside of these bounds. If None, returns all level 2 chunks for the root ID.

    None

    Returns:

    Type Description list of list

    Edge list for level 2 chunked graph. Each element of the list is an edge, and each edge is a list of two node IDs (source and target).

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.remesh_level2_chunks","title":"remesh_level2_chunks(chunk_ids)","text":"

    Submit specific level 2 chunks to be remeshed in case of a problem.

    Parameters:

    Name Type Description Default chunk_ids list

    List of level 2 chunk IDs.

    required"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_operation_details","title":"get_operation_details(operation_ids)","text":"

    Get the details of a list of operations.

    Parameters:

    Name Type Description Default operation_ids Iterable[int]

    List/array of operation IDs.

    required

    Returns:

    Type Description dict of str to dict

    A dict of dicts of operation info, keys are operation IDs (as strings), values are a dictionary of operation info for the operation. These dictionaries contain the following keys:

    • \"added_edges\"/\"removed_edges\": list of list of int
      • List of edges added (if a merge) or removed (if a split) by this operation. Each edge is a list of two supervoxel IDs (source and target).
    • \"roots\": list of int
      • List of root IDs that were created by this operation.
    • \"sink_coords\": list of list of int
      • List of sink coordinates for this operation. The sink is one of the points placed by the user when specifying the operation. Each sink coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
    • \"source_coords\": list of list of int
      • List of source coordinates for this operation. The source is one of the points placed by the user when specifying the operation. Each source coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
    • \"timestamp\": str
      • Timestamp of the operation.
    • \"user\": str
      • User ID number who performed the operation (as a string).
    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_lineage_graph","title":"get_lineage_graph(root_id, timestamp_past=None, timestamp_future=None, as_nx_graph=False, exclude_links_to_future=False, exclude_links_to_past=False)","text":"

    Returns the lineage graph for a root ID, optionally cut off in the past or the future.

    Each change in the chunked graph creates a new root ID for the object after that change. This function returns a graph of all root IDs for a given object, tracing the history of the object in terms of merges and splits.

    Parameters:

    Name Type Description Default root_id int

    Object root ID.

    required timestamp_past datetime or None

    Cutoff for the lineage graph backwards in time. By default, None.

    None timestamp_future datetime or None

    Cutoff for the lineage graph going forwards in time. By default, uses the timestamp property for this client, which defaults to the current time.

    None as_nx_graph

    If True, a NetworkX graph is returned.

    False exclude_links_to_future

    If True, links from nodes before timestamp_future to after timestamp_future are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.

    False exclude_links_to_past

    If True, links from nodes before timestamp_past to after timestamp_past are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.

    False

    Returns:

    Type Description dict

    Dictionary describing the lineage graph and operations for the root ID. Not returned if as_nx_graph is True. The dictionary contains the following keys:

    • \"directed\" : bool
      • Whether the graph is directed.
    • \"graph\" : dict
      • Dictionary of graph attributes.
    • \"links\" : list of dict
      • Each element of the list is a dictionary describing an edge in the lineage graph as \"source\" and \"target\" keys.
    • \"multigraph\" : bool
      • Whether the graph is a multigraph.
    • \"nodes\" : list of dict
      • Each element of the list is a dictionary describing a node in the lineage graph, usually with \"id\", \"timestamp\", and \"operation_id\" keys.
    DiGraph

    NetworkX directed graph of the lineage graph. Only returned if as_nx_graph is True.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_latest_roots","title":"get_latest_roots(root_id, timestamp=None, timestamp_future=None)","text":"

    Returns root IDs that are related to the given root_id at a given timestamp. Can be used to find the \"latest\" root IDs associated with an object.

    Parameters:

    Name Type Description Default root_id int

    Object root ID.

    required timestamp datetime or None

    Timestamp of where to query IDs from. If None, uses the timestamp property for this client, which defaults to the current time.

    None timestamp_future datetime or None

    DEPRECATED name, use timestamp instead. Timestamp to suggest IDs from (note can be in the past relative to the root). By default, None.

    None

    Returns:

    Type Description ndarray

    1d array with all latest successors.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_original_roots","title":"get_original_roots(root_id, timestamp_past=None)","text":"

    Returns root IDs that are the latest successors of a given root ID.

    Parameters:

    Name Type Description Default root_id int

    Object root ID.

    required timestamp_past datetime or None

    Cutoff for the search going backwards in time. By default, None.

    None

    Returns:

    Type Description ndarray

    1d array with all latest successors.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_latest_roots","title":"is_latest_roots(root_ids, timestamp=None)","text":"

    Check whether these root IDs are still a root at this timestamp.

    Parameters:

    Name Type Description Default root_ids array-like of int

    Root IDs to check.

    required timestamp datetime

    Timestamp to check whether these IDs are valid root IDs in the chunked graph. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description np.array of bool

    Array of whether these are valid root IDs.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.suggest_latest_roots","title":"suggest_latest_roots(root_id, timestamp=None, stop_layer=None, return_all=False, return_fraction_overlap=False)","text":"

    Suggest latest roots for a given root id, based on overlap of component chunk IDs. Note that edits change chunk IDs, and so this effectively measures the fraction of unchanged chunks at a given chunk layer, which sets the size scale of chunks. Higher layers are coarser.

    Parameters:

    Name Type Description Default root_id int

    Root ID of the potentially outdated object.

    required timestamp datetime

    Datetime at which \"latest\" roots are being computed, by default None. If None, uses the timestamp property for this client, which defaults to the current time. Note that this has to be a timestamp after the creation of the root_id.

    None stop_layer int

    Chunk level at which to compute overlap, by default None. No value will take the 4th from the top layer, which emphasizes speed and works well for larger objects. Lower values are slower but more fine-grained. Values under 2 (i.e. supervoxels) are not recommended except in extremely fine grained scenarios.

    None return_all bool

    If True, return all current IDs sorted from most overlap to least, by default False. If False, only the top is returned.

    False return_fraction_overlap bool

    If True, return all fractions sorted by most overlap to least, by default False. If False, only the top value is returned.

    False"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_valid_nodes","title":"is_valid_nodes(node_ids, start_timestamp=None, end_timestamp=None)","text":"

    Check whether nodes are valid for given timestamp range.

    Valid is defined as existing in the chunked graph. This makes no statement about these IDs being roots, supervoxel or anything in-between. It also does not take into account whether a root ID has since been edited.

    Parameters:

    Name Type Description Default node_ids array-like of int

    Node IDs to check.

    required start_timestamp datetime

    Timestamp to check whether these IDs were valid after this timestamp. Defaults to None (assumes now).

    None end_timestamp datetime

    Timestamp to check whether these IDs were valid before this timestamp. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description np.array of bool

    Array of whether these are valid IDs.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_timestamps","title":"get_root_timestamps(root_ids, latest=False, timestamp=None)","text":"

    Retrieves timestamps when roots where created.

    Parameters:

    Name Type Description Default root_ids

    Iterable of root IDs to query.

    required latest bool

    If False, returns the first timestamp that the root_id was valid for each root ID. If True, returns the newest/latest timestamp for each root ID. Note, this will return the timestamp at which the query was run when the root is currently valid. This means that you will get a different answer if you make this same query at a later time if you don't specify a timestamp parameter.

    False timestamp datetime

    Timestamp to query when using latest=True. Use this to provide consistent results for a particular timestamp. If an ID is still valid at a point in the future past this timestamp, the query will still return this timestamp as the latest moment in time. An error will occur if you provide a timestamp for which the root ID is not valid. If None, uses the timestamp property for this client, which defaults to the current time.

    None

    Returns:

    Type Description np.array of datetime.datetime

    Array of timestamps when root_ids were created.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_past_ids","title":"get_past_ids(root_ids, timestamp_past=None, timestamp_future=None)","text":"

    For a set of root IDs, get the list of IDs at a past or future time point that could contain parts of the same object.

    Parameters:

    Name Type Description Default root_ids Iterable of int

    Iterable of root IDs to query.

    required timestamp_past datetime or None

    Time of a point in the past for which to look up root ids. Default is None.

    None timestamp_future datetime or None

    Time of a point in the future for which to look up root ids. Not implemented on the server currently. Default is None.

    None

    Returns:

    Type Description dict

    Dict with keys \"future_id_map\" and \"past_id_map\". Each is a dict whose keys are the supplied root_ids and whose values are the list of related root IDs at timestamp_past/timestamp_future.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_delta_roots","title":"get_delta_roots(timestamp_past, timestamp_future=datetime.datetime.now(datetime.timezone.utc))","text":"

    Get the list of roots that have changed between timetamp_past and timestamp_future.

    Parameters:

    Name Type Description Default timestamp_past datetime

    Past timepoint to query

    required timestamp_future datetime

    Future timepoint to query. Defaults to datetime.datetime.now(datetime.timezone.utc).

    now(utc)

    Returns:

    Name Type Description old_roots np.ndarray of np.int64

    Roots that have expired in that interval.

    new_roots np.ndarray of np.int64

    Roots that are new in that interval.

    "},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_oldest_timestamp","title":"get_oldest_timestamp()","text":"

    Get the oldest timestamp in the database.

    Returns:

    Type Description datetime

    Oldest timestamp in the database.

    "},{"location":"api/client/","title":"caveclient.CAVEclient","text":"

    A manager for all clients sharing common datastack and authentication information.

    This client wraps all the other clients and keeps track of the things that need to be consistent across them. To instantiate a client:

    from caveclient import CAVEclient\n\nclient = CAVEclient(datastack_name='my_datastack',\n                    server_address='www.myserver.com',\n                    auth_token_file='~/.mysecrets/secrets.json')\n

    Then

    • client.annotation is an AnnotationClient (see client.annotation)
    • client.auth is an AuthClient (see client.auth)
    • client.chunkedgraph is a ChunkedGraphClient (see client.chunkedgraph)
    • client.info is an InfoServiceClient (see client.info)
    • client.l2cache is an L2CacheClient (see client.l2cache)
    • client.materialize is a MaterializationClient (see client.materialize)
    • client.skeleton is a SkeletonClient (see client.skeleton)
    • client.schema is a SchemaClient (see client.schema)
    • client.state is a neuroglancer JSONService (see client.state)

    All subclients are loaded lazily and share the same datastack name, server address, and auth tokens where used.

    Parameters:

    Name Type Description Default datastack_name str

    Datastack name for the services. Almost all services need this and will not work if it is not passed.

    None server_address str or None

    URL of the framework server. If None, chooses the default server global.daf-apis.com. Optional, defaults to None.

    None auth_token_file str or None

    Path to a json file containing the auth token. If None, uses the default location. See Auth client documentation. Optional, defaults to None.

    default_token_file auth_token_key str

    Dictionary key for the token in the the JSON file. Optional, default is 'token'.

    'token' auth_token str or None

    Direct entry of an auth token. If None, uses the file arguments to find the token. Optional, default is None.

    None max_retries int or None

    Sets the default number of retries on failed requests. Optional, by default 2.

    None pool_maxsize int or None

    Sets the max number of threads in a requests pool, although this value will be exceeded if pool_block is set to False. Optional, uses requests defaults if None.

    None pool_block

    If True, prevents the number of threads in a requests pool from exceeding the max size. Optional, uses requests defaults (False) if None.

    None desired_resolution Iterable[float] or None

    If given, should be a list or array of the desired resolution you want queries returned in useful for materialization queries.

    None info_cache

    Pre-computed info cache, bypassing the lookup of datastack info from the info service. Should only be used in cases where this information is cached and thus repetitive lookups can be avoided.

    None version Optional[int]

    The default materialization version of the datastack to use. If None, the latest version is used. Optional, defaults to None.

    None See Also

    set_session_defaults

    get_session_defaults

    Methods:

    Name Description change_auth

    Change the authentication token and reset services.

    Attributes:

    Name Type Description annotation AnnotationClient

    A client for the annotation service. See client.annotation

    auth AuthClient

    A client for the auth service. See client.auth for more information.

    chunkedgraph ChunkedGraphClient

    A client for the chunkedgraph service. See client.chunkedgraph

    datastack_name str

    The name of the datastack for the client.

    info InfoServiceClient

    A client for the info service. See client.info for more information.

    l2cache L2CacheClient

    A client for the L2 cache service. See client.l2cache

    materialize MaterializationClient

    A client for the materialization service. See client.materialize

    schema SchemaClient

    A client for the EM Annotation Schemas service. See client.schema

    server_address

    The server address for the client.

    skeleton SkeletonClient

    A client for the skeleton service. See client.skeleton

    state JSONService

    A client for the neuroglancer state service. See client.state

    timestamp Optional[datetime]

    The default timestamp to use for queries which rely on a timestamp.

    version Optional[int]

    The default materialization version of the datastack to use for queries which

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.annotation","title":"annotation: AnnotationClient property","text":"

    A client for the annotation service. See client.annotation for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.auth","title":"auth: AuthClient property","text":"

    A client for the auth service. See client.auth for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.chunkedgraph","title":"chunkedgraph: ChunkedGraphClient property","text":"

    A client for the chunkedgraph service. See client.chunkedgraph for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.datastack_name","title":"datastack_name: str property","text":"

    The name of the datastack for the client.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.info","title":"info: InfoServiceClient property","text":"

    A client for the info service. See client.info for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.l2cache","title":"l2cache: L2CacheClient property","text":"

    A client for the L2 cache service. See client.l2cache for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.materialize","title":"materialize: MaterializationClient property","text":"

    A client for the materialization service. See client.materialize for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.schema","title":"schema: SchemaClient property","text":"

    A client for the EM Annotation Schemas service. See client.schema for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.server_address","title":"server_address property","text":"

    The server address for the client.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.skeleton","title":"skeleton: SkeletonClient property","text":"

    A client for the skeleton service. See client.skeleton for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.state","title":"state: JSONService property","text":"

    A client for the neuroglancer state service. See client.state for more information.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.timestamp","title":"timestamp: Optional[datetime] property","text":"

    The default timestamp to use for queries which rely on a timestamp.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.version","title":"version: Optional[int] property writable","text":"

    The default materialization version of the datastack to use for queries which expect a version. Also sets the timestamp to the corresponding timestamp of the version for queries which rely on a timestamp.

    "},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.change_auth","title":"change_auth(auth_token_file=None, auth_token_key=None, auth_token=None)","text":"

    Change the authentication token and reset services.

    Parameters:

    Name Type Description Default auth_token_file str

    New auth token json file path, by default None, which defaults to the existing state.

    None auth_token_key str

    New dictionary key under which the token is stored in the json file, by default None, which defaults to the existing state.

    None auth_token str

    Direct entry of a new token, by default None.

    None"},{"location":"api/config/","title":"Configuration","text":""},{"location":"api/config/#caveclient.set_session_defaults","title":"caveclient.set_session_defaults(max_retries=3, pool_block=False, pool_maxsize=10, backoff_factor=0.1, backoff_max=120, status_forcelist=(502, 503, 504))","text":"

    Set global default values to configure how all clients will communicate with servers. Should be done prior to initializing a client.

    Note that these values are only used when not set at the client level.

    Parameters:

    Name Type Description Default max_retries int

    The maximum number of retries each connection should attempt. Set to 0 to fail on the first retry.

    3 pool_block bool

    Whether the connection pool should block for connections.

    False pool_maxsize int

    The maximum number of connections to save in the pool.

    10 backoff_factor Union[float, int]

    A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). The query will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. For example, if the backoff_factor is 0.1, then will sleep for [0.0s, 0.2s, 0.4s, 0.8s, \u2026] between retries. No backoff will ever be longer than backoff_max.

    0.1 backoff_max Union[float, int]

    The maximum backoff time.

    120 status_forcelist Optional[Collection]

    A set of integer HTTP status codes that we should force a retry on.

    (502, 503, 504) Usage 
    from caveclient import set_session_defaults\n\nset_session_defaults(\n    max_retries=5, # would increase the default number of retries\n    backoff_factor=0.5, # would increase the default backoff factor between retries\n    backoff_max=240, # would increase the default maximum backoff time\n    status_forcelist=(502, 503, 504, 505), # would add 505 to the default list\n)\n\nset_session_defaults() # would revert all defaults to their original values\n
    Notes

    Calling this function will set the default values for all clients created after the call.

    Calling this function with any arguments missing will reset that value to the default value.

    See Also:

    urllib3.util.Retry

    requests.adapters.HTTPAdapter

    "},{"location":"api/config/#caveclient.get_session_defaults","title":"caveclient.get_session_defaults()","text":"

    Get the current default values for session configuration.

    Returns:

    Type Description dict

    Dictionary of current default values for session configuration.

    "},{"location":"api/datastack_lookup/","title":"Datastack lookup","text":""},{"location":"api/datastack_lookup/#caveclient.datastack_lookup.reset_server_address_cache","title":"caveclient.datastack_lookup.reset_server_address_cache(datastack, filename=None)","text":"

    Remove one or more datastacks from the datastack-to-server cache.

    Parameters:

    Name Type Description Default datastack str or list of str

    Datastack names to remove from the cache, by default None

    required filename str

    Name of the cache file, by default None

    None"},{"location":"api/info/","title":"client.info","text":"

    Client for interacting with the info service.

    Methods:

    Name Description annotation_endpoint

    AnnotationEngine endpoint for a dataset.

    get_aligned_volume_info

    Gets the info record for a aligned_volume

    get_datastack_info

    Gets the info record for a datastack

    get_datastacks

    Query which datastacks are available at the info service

    get_datastacks_by_aligned_volume

    Lookup what datastacks are associated with this aligned volume

    image_cloudvolume

    Generate a cloudvolume instance based on the image source, using authentication if needed and

    image_source

    Cloud path to the imagery for the dataset

    refresh_stored_data

    Reload the stored info values from the server.

    segmentation_cloudvolume

    Generate a cloudvolume instance based on the segmentation source, using authentication if needed and

    segmentation_source

    Cloud path to the chunkgraph-backed Graphene segmentation for a dataset

    synapse_segmentation_source

    Cloud path to the synapse segmentation for a dataset

    viewer_resolution

    Get the viewer resolution metadata for this datastack

    viewer_site

    Get the base Neuroglancer URL for the dataset

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.annotation_endpoint","title":"annotation_endpoint(datastack_name=None, use_stored=True)","text":"

    AnnotationEngine endpoint for a dataset.

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True

    Returns:

    Type Description str

    Location of the AnnotationEngine

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_aligned_volume_info","title":"get_aligned_volume_info(datastack_name=None, use_stored=True)","text":"

    Gets the info record for a aligned_volume

    Parameters:

    Name Type Description Default datastack_name str

    datastack_name to look up. If None, uses the one specified by the client. By default None

    None use_stored bool

    If True and the information has already been queried for that dataset, then uses the cached version. If False, re-queries the infromation. By default True

    True

    Returns:

    Type Description dict or None

    The complete info record for the aligned_volume

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastack_info","title":"get_datastack_info(datastack_name=None, use_stored=True)","text":"

    Gets the info record for a datastack

    Parameters:

    Name Type Description Default datastack_name str

    datastack to look up. If None, uses the one specified by the client. By default None

    None use_stored bool

    If True and the information has already been queried for that datastack, then uses the cached version. If False, re-queries the infromation. By default True

    True

    Returns:

    Type Description dict or None

    The complete info record for the datastack

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks","title":"get_datastacks()","text":"

    Query which datastacks are available at the info service

    Returns:

    Type Description list

    List of datastack names

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks_by_aligned_volume","title":"get_datastacks_by_aligned_volume(aligned_volume=None)","text":"

    Lookup what datastacks are associated with this aligned volume

    Args: aligned_volume (str, optional): aligned volume to lookup. Defaults to None.

    Raises: ValueError: if no aligned volume is specified

    Returns: list: a list of datastack string

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_cloudvolume","title":"image_cloudvolume(**kwargs)","text":"

    Generate a cloudvolume instance based on the image source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.

    Requires cloudvolume to be installed, which is not included by default.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_source","title":"image_source(datastack_name=None, use_stored=True, format_for='raw')","text":"

    Cloud path to the imagery for the dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True format_for 'raw', 'cloudvolume', or 'neuroglancer'

    Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.

    'raw'

    Returns:

    Type Description str

    Formatted cloud path to the flat segmentation

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.refresh_stored_data","title":"refresh_stored_data()","text":"

    Reload the stored info values from the server.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_cloudvolume","title":"segmentation_cloudvolume(use_client_secret=True, **kwargs)","text":"

    Generate a cloudvolume instance based on the segmentation source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.

    Requires cloudvolume to be installed, which is not included by default.

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_source","title":"segmentation_source(datastack_name=None, format_for='raw', use_stored=True)","text":"

    Cloud path to the chunkgraph-backed Graphene segmentation for a dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True format_for 'raw', 'cloudvolume', or 'neuroglancer'

    Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"graphene://https://\" type path is used If 'neuroglancer', a \"graphene://https://\" type path is used, as needed by Neuroglancer.

    'raw'

    Returns:

    Type Description str

    Formatted cloud path to the Graphene segmentation

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.synapse_segmentation_source","title":"synapse_segmentation_source(datastack_name=None, use_stored=True, format_for='raw')","text":"

    Cloud path to the synapse segmentation for a dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the dataset to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True format_for 'raw', 'cloudvolume', or 'neuroglancer'

    Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.

    'raw'

    Returns:

    Type Description str

    Formatted cloud path to the synapse segmentation

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_resolution","title":"viewer_resolution(datastack_name=None, use_stored=True)","text":"

    Get the viewer resolution metadata for this datastack

    Parameters:

    Name Type Description Default datastack_name

    If None use the default one configured in the client

    None use_stored

    Use the cached value, if False go get a new value from server

    True

    Returns:

    Type Description array

    Voxel resolution as a len(3) np.array

    "},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_site","title":"viewer_site(datastack_name=None, use_stored=True)","text":"

    Get the base Neuroglancer URL for the dataset

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack to look up. If None, uses the value specified by the client. Default is None.

    None use_stored bool

    If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.

    True

    Returns:

    Type Description str

    Base URL for the Neuroglancer viewer

    "},{"location":"api/l2cache/","title":"client.l2cache","text":"

    Client for interacting with the level2 cache service.

    Methods:

    Name Description cache_metadata

    Retrieves the meta data for the cache

    get_l2data

    Gets the attributed statistics data for L2 ids.

    get_l2data_table

    Gets the attributed statistics data for L2 ids, returned as a dataframe.

    has_cache

    Checks if the l2 cache is available for the dataset

    table_mapping

    Retrieves table mappings for l2 cache.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.cache_metadata","title":"cache_metadata()","text":"

    Retrieves the meta data for the cache

    Returns:

    Type Description dict

    keys are attribute names, values are datatypes

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data","title":"get_l2data(l2_ids, attributes=None)","text":"

    Gets the attributed statistics data for L2 ids.

    Parameters:

    Name Type Description Default l2_ids list or ndarray

    A list of level 2 ids.

    required attributes list

    A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:

    • area_nm2
    • chunk_intersect_count
    • max_dt_nm
    • mean_dt_nm
    • pca
    • pca_val
    • rep_coord_nm
    • size_nm3

    See the tutorial for more description.

    None

    Returns:

    Type Description dict

    Keys are l2 ids, values are data.

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data_table","title":"get_l2data_table(l2_ids, attributes=None, split_columns=True)","text":"

    Gets the attributed statistics data for L2 ids, returned as a dataframe.

    Parameters:

    Name Type Description Default l2_ids list or ndarray

    A list of level 2 ids.

    required attributes list

    A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:

    • area_nm2
    • chunk_intersect_count
    • max_dt_nm
    • mean_dt_nm
    • pca
    • pca_val
    • rep_coord_nm
    • size_nm3

    See the tutorial for more description.

    None split_columns bool

    Whether to split columns with multiple values into separate columns.

    True

    Returns:

    Type Description DataFrame

    A pandas dataframe with the requested attributes as columns and indexed by l2_id.

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.has_cache","title":"has_cache(datastack_name=None)","text":"

    Checks if the l2 cache is available for the dataset

    Parameters:

    Name Type Description Default datastack_name str

    The name of the datastack to check, by default None (if None, uses the client's datastack)

    None

    Returns:

    Type Description bool

    True if the l2 cache is available, False otherwise

    "},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.table_mapping","title":"table_mapping()","text":"

    Retrieves table mappings for l2 cache.

    Returns:

    Type Description dict

    keys are pcg table names, values are dicts with fields l2cache_id and cv_path.

    "},{"location":"api/materialize/","title":"client.materialize","text":"

    Client for interacting with the materialization engine.

    Methods:

    Name Description get_table_metadata

    Get metadata about a table

    get_tables

    Gets a list of table names for a datastack

    get_tables_metadata

    Get metadata about tables

    get_timestamp

    Get datetime.datetime timestamp for a materialization version.

    get_unique_string_values

    Get unique string values for a table

    get_version_metadata

    Get metadata about a version

    get_versions

    Get the versions available

    get_versions_metadata

    Get the metadata for all the versions that are presently available and valid

    get_view_metadata

    Get metadata for a view

    get_view_schema

    Get schema for a view

    get_view_schemas

    Get schema for a view

    get_views

    Get all available views for a version

    ingest_annotation_table

    Trigger supervoxel lookup and root ID lookup of new annotations in a table.

    join_query

    Generic query on materialization tables

    live_live_query

    Beta method for querying cave annotation tables with root IDs and annotations

    live_query

    Generic query on materialization tables

    lookup_supervoxel_ids

    Trigger supervoxel lookups of new annotations in a table.

    map_filters

    Translate a list of filter dictionaries from a point in the

    most_recent_version

    Get the most recent version of materialization for this datastack name

    query_table

    Generic query on materialization tables

    query_view

    Generic query on a view

    synapse_query

    Convenience method for querying synapses.

    Attributes:

    Name Type Description cg_client

    The chunked graph client.

    datastack_name

    The name of the datastack.

    homepage HTML

    The homepage for the materialization engine.

    server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    tables TableManager

    The table manager for the materialization engine.

    version int

    The version of the materialization. Can be used to set up the

    views ViewManager

    The view manager for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.cg_client","title":"cg_client property","text":"

    The chunked graph client.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.datastack_name","title":"datastack_name property","text":"

    The name of the datastack.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.homepage","title":"homepage: HTML property","text":"

    The homepage for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.tables","title":"tables: TableManager property","text":"

    The table manager for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.version","title":"version: int property writable","text":"

    The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.

    Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.views","title":"views: ViewManager property","text":"

    The view manager for the materialization engine.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_table_metadata","title":"get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)","text":"

    Get metadata about a table

    Parameters:

    Name Type Description Default table_name str

    name of table to mark for deletion

    required datastack_name str or None

    Name of the datastack_name. If None, uses the one specified in the client.

    None version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to print out warnings to the logger. Defaults to True.

    True

    Returns:

    Type Description dict

    Metadata dictionary for table

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables","title":"get_tables(datastack_name=None, version=None)","text":"

    Gets a list of table names for a datastack

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client

    None version int or None

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None

    Returns:

    Type Description list

    List of table names

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables_metadata","title":"get_tables_metadata(datastack_name=None, version=None, log_warning=True)","text":"

    Get metadata about tables

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack_name. If None, uses the one specified in the client.

    None version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to print out warnings to the logger. Defaults to True.

    True

    Returns:

    Type Description dict

    Metadata dictionary for table

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_timestamp","title":"get_timestamp(version=None, datastack_name=None)","text":"

    Get datetime.datetime timestamp for a materialization version.

    Parameters:

    Name Type Description Default version int or None

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None datastack_name str or None

    Datastack name, by default None. If None, defaults to the value set in the client.

    None

    Returns:

    Type Description datetime

    Datetime when the materialization version was frozen.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_unique_string_values","title":"get_unique_string_values(table, datastack_name=None)","text":"

    Get unique string values for a table

    Parameters:

    Name Type Description Default table str

    Table to query

    required datastack_name Optional[str]

    Datastack to query. If None, uses the one specified in the client.

    None

    Returns:

    Type Description dict[str]

    A dictionary of column names and their unique values

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_version_metadata","title":"get_version_metadata(version=None, datastack_name=None)","text":"

    Get metadata about a version

    Parameters:

    Name Type Description Default version int or None

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None datastack_name str or None

    Datastack name, by default None. If None, defaults to the value set in the client.

    None

    Returns:

    Type Description dict

    Dictionary of metadata about the version

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions","title":"get_versions(datastack_name=None, expired=False)","text":"

    Get the versions available

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack, by default None. If None, uses the one specified in the client.

    None expired bool

    Whether to include expired versions, by default False.

    False

    Returns:

    Type Description dict

    Dictionary of versions available

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions_metadata","title":"get_versions_metadata(datastack_name=None, expired=False)","text":"

    Get the metadata for all the versions that are presently available and valid

    Parameters:

    Name Type Description Default datastack_name str or None

    Datastack name, by default None. If None, defaults to the value set in the client.

    None expired bool

    Whether to include expired versions, by default False.

    False

    Returns:

    Type Description list[dict]

    List of metadata dictionaries

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_metadata","title":"get_view_metadata(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"

    Get metadata for a view

    Parameters:

    Name Type Description Default view_name str

    Name of view to query.

    required materialization_version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description dict

    Metadata of view

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schema","title":"get_view_schema(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"

    Get schema for a view

    Parameters:

    Name Type Description Default view_name str

    Name of view to query.

    required materialization_version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description dict

    Schema of view.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schemas","title":"get_view_schemas(materialization_version=None, datastack_name=None, log_warning=True)","text":"

    Get schema for a view

    Parameters:

    Name Type Description Default materialization_version Optional[int]

    Version to query. If None, will use version set by client.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description dict

    Schema of view.

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_views","title":"get_views(version=None, datastack_name=None)","text":"

    Get all available views for a version

    Parameters:

    Name Type Description Default version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None datastack_name str

    Datastack to query. If None, uses the one specified in the client.

    None

    Returns:

    Type Description list

    List of views

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.ingest_annotation_table","title":"ingest_annotation_table(table_name, datastack_name=None)","text":"

    Trigger supervoxel lookup and root ID lookup of new annotations in a table.

    Parameters:

    Name Type Description Default table_name str

    Table to trigger

    required datastack_name str

    Datastack to trigger it. Defaults to what is set in client.

    None

    Returns:

    Type Description dict

    Status code of response from server

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.join_query","title":"join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"

    Generic query on materialization tables

    Parameters:

    Name Type Description Default tables list of lists with length 2 or 'str'

    list of two lists: first entries are table names, second entries are the columns used for the join.

    required filter_in_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None

    None filter_out_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None

    None filter_equal_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None

    None filter_greater_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are exclusive upper-bound, by default None

    None filter_less_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are exclusive lower-bound, by default None

    None filter_greater_equal_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are inclusive upper-bound, by default None

    None filter_less_equal_dict dict of dicts

    outer layer: keys are table names inner layer: keys are column names, values are inclusive lower-bound, by default None

    None filter_spatial_dict dict of dicts

    outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None.

    None filter_regex_dict dict of dicts

    outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None

    None select_columns dict of lists of str

    keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated.

    None offset int

    result offset to use. Defaults to None. Will only return top K results.

    None limit int

    maximum results to return (server will set upper limit, see get_server_config)

    None suffixes dict

    suffixes to use for duplicate columns, keys are table names, values are the suffix

    None datastack_name str

    datastack to query. If None defaults to one specified in client.

    None return_df bool

    whether to return as a dataframe default True, if False, data is returned as json (slower)

    True split_positions bool

    whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower)

    False materialization_version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None metadata bool

    toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True desired_resolution Iterable

    What resolution to convert position columns to. Defaults to None will use defaults.

    None random_sample int

    if given, will do a tablesample of the table to return that many annotations

    None log_warning bool

    Whether to log warnings, by default True

    True

    Returns:

    Type Description DataFrame

    a pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_live_query","title":"live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, allow_invalid_root_ids=False, random_sample=None, log_warning=True)","text":"

    Beta method for querying cave annotation tables with root IDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.

    Parameters:

    Name Type Description Default table str

    Principle table to query

    required timestamp datetime

    Timestamp to query

    required joins

    List of joins, where each join is a list of [table1,column1, table2, column2]

    None filter_in_dict

    A dictionary with tables as keys, values are dicts with column keys and list values to accept.

    None filter_out_dict

    A dictionary with tables as keys, values are dicts with column keys and list values to reject.

    None filter_equal_dict

    A dictionary with tables as keys, values are dicts with column keys and values to equate.

    None filter_greater_dict

    A dictionary with tables as keys, values are dicts with column keys and values as exclusive upper-bound.

    None filter_less_dict

    A dictionary with tables as keys, values are dicts with column keys and values as exclusive lower-bound.

    None filter_greater_equal_dict

    A dictionary with tables as keys, values are dicts with column keys and values as inclusive upper-bound.

    None filter_less_equal_dict

    A dictionary with tables as keys, values are dicts with column keys and values as inclusive lower-bound.

    None filter_spatial_dict

    A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds.

    None filter_regex_dict

    A dictionary with tables as keys, values are dicts with column keys and values of regex strings.

    None select_columns

    A dictionary with tables as keys, values are lists of columns to select.

    None offset int

    Value to offset query by.

    None limit int

    Limit of query.

    None datastack_name str

    Datastack to query. Defaults to set by client.

    None split_positions bool

    Whether to split positions into separate columns, True is faster.

    False metadata bool

    Whether to attach metadata to dataframe.

    True suffixes dict

    What suffixes to use on joins, keys are table_names, values are suffixes.

    None desired_resolution Iterable

    What resolution to convert position columns to.

    None allow_missing_lookups bool

    If there are annotations without supervoxels and root IDs yet, allow results.

    False allow_invalid_root_ids bool

    If True, ignore root ids not valid at the given timestamp, otherwise raise an error.

    False random_sample int

    If given, will do a table sample of the table to return that many annotations.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description

    Results of query

    Examples:

    >>> from caveclient import CAVEclient\n>>> client = CAVEclient('minnie65_public_v117')\n>>> live_live_query(\"table_name\", datetime.datetime.now(datetime.timezone.utc),\n>>>    joins=[[table_name, table_column, joined_table, joined_column],\n>>>             [joined_table, joincol2, third_table, joincol_third]]\n>>>    suffixes={\n>>>        \"table_name\":\"suffix1\",\n>>>        \"joined_table\":\"suffix2\",\n>>>        \"third_table\":\"suffix3\"\n>>>    },\n>>>    select_columns= {\n>>>        \"table_name\":[ \"column\",\"names\"],\n>>>        \"joined_table\":[\"joined_colum\"]\n>>>    },\n>>>    filter_in_dict= {\n>>>        \"table_name\":{\n>>>            \"column_name\":[included,values]\n>>>        }\n>>>    },\n>>>    filter_out_dict= {\n>>>        \"table_name\":{\n>>>            \"column_name\":[excluded,values]\n>>>        }\n>>>    },\n>>>    filter_equal_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_greater_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_less_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_greater_equal_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_less_equal_dict\"={\n>>>        \"table_name\":{\n>>>            \"column_name\":value\n>>>        },\n>>>    filter_spatial_dict\"= {\n>>>        \"table_name\": {\n>>>        \"column_name\": [[min_x, min_y, min_z], [max_x, max_y, max_z]]\n>>>    }\n>>>    filter_regex_dict\"= {\n>>>        \"table_name\": {\n>>>        \"column_name\": \"regex_string\"\n>>>     }\n
    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_query","title":"live_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"

    Generic query on materialization tables

    Parameters:

    Name Type Description Default table str

    Table to query

    required timestamp datetime

    Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time.

    required filter_in_dict dict

    Keys are column names, values are allowed entries.

    None filter_out_dict dict

    Keys are column names, values are not allowed entries.

    None filter_equal_dict dict

    Keys are column names, values are specified entry.

    None filter_greater_dict dict

    Keys are column names, values are exclusive upper-bounds.

    None filter_less_dict dict

    Keys are column names, values are exclusive lower-bounds.

    None filter_greater_equal_dict dict

    Keys are column names, values are inclusive upper-bounds.

    None filter_less_equal_dict dict

    Keys are column names, values are inclusive lower-bounds.

    None filter_spatial_dict dict

    Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]].

    None filter_regex_dict dict

    Keys are column names, values are regex strings.

    None select_columns list of str

    Columns to select.

    None offset int

    Offset in query result.

    None limit int

    Maximum results to return (server will set upper limit, see get_server_config).

    None datastack_name str

    Datastack to query. If None, defaults to one specified in client.

    None split_positions bool

    Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower).

    False post_filter bool

    Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True.

    True metadata bool

    Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True merge_reference bool

    Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table.

    True desired_resolution Iterable

    Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata.

    None random_sample int

    If given, will do a tablesample of the table to return that many annotations.

    None log_warning bool

    Whether to log warnings.

    True

    Returns:

    Type Description DataFrame

    A pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.lookup_supervoxel_ids","title":"lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)","text":"

    Trigger supervoxel lookups of new annotations in a table.

    Parameters:

    Name Type Description Default table_name str

    Table to trigger

    required annotation_ids list

    List of annotation ids to lookup. Default is None, which will trigger lookup of entire table.

    None datastack_name str

    Datastack to trigger it. Defaults to what is set in client.

    None

    Returns:

    Type Description dict

    Status code of response from server

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.map_filters","title":"map_filters(filters, timestamp, timestamp_past)","text":"

    Translate a list of filter dictionaries from a point in the future to a point in the past

    Parameters:

    Name Type Description Default filters list[dict]

    filter dictionaries with root_ids

    required timestamp datetime

    timestamp to query

    required timestamp_past datetime

    timestamp to query from

    required

    Returns:

    Type Description list[dict]

    filter dictionaries with past root_ids

    dict

    mapping of future root_ids to past root_ids

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.most_recent_version","title":"most_recent_version(datastack_name=None)","text":"

    Get the most recent version of materialization for this datastack name

    Parameters:

    Name Type Description Default datastack_name str or None

    Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client

    None

    Returns:

    Type Description int

    Most recent version of materialization for this datastack name

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_table","title":"query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)","text":"

    Generic query on materialization tables

    Parameters:

    Name Type Description Default table str

    Table to query

    required filter_in_dict dict

    Keys are column names, values are allowed entries, by default None

    None filter_out_dict dict

    Keys are column names, values are not allowed entries, by default None

    None filter_equal_dict dict

    Keys are column names, values are specified entry, by default None

    None filter_greater_dict dict

    Keys are column names, values are exclusive upper-bound value, by default None

    None filter_less_dict dict

    Keys are column names, values are exclusive lower-bound value, by default None

    None filter_greater_equal_dict dict

    Keys are column names, values are inclusive upper-bound value, by default None

    None filter_less_equal_dict dict

    Keys are column names, values are inclusive lower-bound value, by default None

    None filter_spatial_dict dict

    Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None

    None filter_regex_dict dict

    Keys are column names, values are regex strings, by default None

    None select_columns list of str

    Columns to select, by default None

    None offset int

    Result offset to use, by default None. Will only return top K results.

    None limit int

    Maximum results to return (server will set upper limit, see get_server_config), by default None

    None datastack_name str

    Datastack to query, by default None. If None, defaults to one specified in client.

    None return_df bool

    Whether to return as a dataframe, by default True. If False, data is returned as json (slower).

    True split_positions bool

    Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)

    False materialization_version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None timestamp datetime

    Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version

    None metadata bool

    Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True merge_reference bool

    Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that

    True desired_resolution Iterable[float]

    Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata

    None get_counts bool

    Whether to get counts of the query, by default False

    False random_sample int

    If given, will do a tablesample of the of the table to return that many annotations

    None log_warning bool

    Whether to log warnings, by default True

    True

    Returns:

    Type Description DataFrame

    A pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_view","title":"query_view(view_name, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None)","text":"

    Generic query on a view

    Parameters:

    Name Type Description Default view_name str

    View to query

    required filter_in_dict dict

    Keys are column names, values are allowed entries, by default None

    None filter_out_dict dict

    Keys are column names, values are not allowed entries, by default None

    None filter_equal_dict dict

    Keys are column names, values are specified entry, by default None

    None filter_greater_dict dict

    Keys are column names, values are exclusive upper-bound, by default None

    None filter_less_dict dict

    Keys are column names, values are exclusive lower-bound, by default None

    None filter_greater_equal_dict dict

    Keys are column names, values are inclusive upper-bound, by default None

    None filter_less_equal_dict dict

    Keys are column names, values are inclusive lower-bound, by default None

    None filter_spatial_dict dict

    Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None

    None filter_regex_dict dict

    Keys are column names, values are regex strings, by default None

    None select_columns list of str

    Columns to select, by default None

    None offset int

    Result offset to use, by default None. Will only return top K results.

    None limit int

    Maximum results to return (server will set upper limit, see get_server_config), by default None

    None datastack_name str

    Datastack to query, by default None. If None, defaults to one specified in client.

    None return_df bool

    Whether to return as a dataframe, by default True. If False, data is returned as json (slower).

    True split_positions bool

    Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)

    False materialization_version int

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None metadata bool

    Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.

    True merge_reference bool

    Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that

    True desired_resolution Iterable[float]

    Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata

    None get_counts bool

    Whether to get counts of the query, by default False

    False random_sample int

    If given, will do a tablesample of the of the table to return that many annotations

    None

    Returns:

    Type Description DataFrame

    A pandas dataframe of results of query

    "},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.synapse_query","title":"synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)","text":"

    Convenience method for querying synapses.

    Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.

    Parameters:

    Name Type Description Default pre_ids Union[int, Iterable, ndarray]

    Pre-synaptic cell(s) to query.

    None post_ids Union[int, Iterable, ndarray]

    Post-synaptic cell(s) to query.

    None bounding_box Optional[Union[list, ndarray]]

    [[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset.

    None bounding_box_column str

    Which synapse location column to filter by.

    'post_pt_position' timestamp datetime

    Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version.

    None remove_autapses bool

    Whether to remove autapses from query results.

    True include_zeros bool

    Whether to include synapses to/from id=0 (out of segmentation).

    True limit int

    Number of synapses to limit. Server-side limit still applies.

    None offset int

    Number of synapses to offset query.

    None split_positions bool

    Whether to split positions into separate columns, True is faster.

    False desired_resolution Iterable[float]

    List or array of the desired resolution you want queries returned in useful for materialization queries.

    None materialization_version Optional[int]

    The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.

    None metadata bool

    Whether to attach metadata to dataframe in the df.attr dictionary.

    True

    Returns:

    Type Description DataFrame

    Results of query.

    "},{"location":"api/schema/","title":"client.schema","text":"

    Client for interacting with the schema service.

    Methods:

    Name Description get_schemas

    Get the available schema types

    schema_definition

    Get the definition of a specified schema_type

    schema_definition_all

    Get the definition of all schema_types

    schema_definition_multi

    Get the definition of multiple schema_types

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.get_schemas","title":"get_schemas()","text":"

    Get the available schema types

    Returns:

    Type Description list

    List of schema types available on the Schema service.

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition","title":"schema_definition(schema_type)","text":"

    Get the definition of a specified schema_type

    Parameters:

    Name Type Description Default schema_type str

    Name of a schema_type

    required

    Returns:

    Type Description json

    Schema definition

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_all","title":"schema_definition_all()","text":"

    Get the definition of all schema_types

    Returns:

    Type Description dict

    Dictionary of schema definitions. Keys are schema names, values are definitions.

    "},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_multi","title":"schema_definition_multi(schema_types)","text":"

    Get the definition of multiple schema_types

    Parameters:

    Name Type Description Default schema_types list

    List of schema names

    required

    Returns:

    Type Description dict

    Dictionary of schema definitions. Keys are schema names, values are definitions.

    "},{"location":"api/skeleton/","title":"client.skeleton","text":"

    Client for interacting with the skeleton service.

    Methods:

    Name Description get_skeleton

    Gets basic skeleton information for a datastack

    get_cache_contents

    Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.

    skeletons_exist

    Confirm or deny that a set of root ids have H5 skeletons in the cache.

    get_bulk_skeletons

    Generates skeletons for a list of root ids without retrieving them.

    generate_bulk_skeletons_async

    Generates skeletons for a list of root ids without retrieving them.

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_skeleton","title":"get_skeleton(root_id, datastack_name=None, skeleton_version=3, output_format='dict', log_warning=True, verbose_level=0)","text":"

    Gets basic skeleton information for a datastack

    Parameters:

    Name Type Description Default root_id int

    The root id of the skeleton to retrieve

    required datastack_name str

    The name of the datastack to check

    None skeleton_version int

    The skeleton version to generate and retrieve. Options are documented in SkeletonService. Use 0 for Neuroglancer-compatibility. Use -1 for latest.

    3 output_format string

    The format to retrieve. Options are: - 'dict': A dictionary - 'swc': A pandas DataFrame

    'dict'

    Returns:

    Type Description

    Skeleton of the requested type. See output_format for details.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_cache_contents","title":"get_cache_contents(datastack_name=None, skeleton_version=3, root_id_prefixes=0, limit=0, log_warning=True)","text":"

    Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.skeletons_exist","title":"skeletons_exist(datastack_name=None, skeleton_version=3, root_ids=0, log_warning=True)","text":"

    Confirm or deny that a set of root ids have H5 skeletons in the cache.

    "},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_bulk_skeletons","title":"get_bulk_skeletons(root_ids, datastack_name=None, skeleton_version=3, output_format='dict', generate_missing_skeletons=False, log_warning=True, verbose_level=0)","text":"

    Generates skeletons for a list of root ids without retrieving them.

    Parameters:

    Name Type Description Default root_ids List

    A list of root ids of the skeletons to generate

    required datastack_name str

    The name of the datastack to check

    None skeleton_version int

    The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.

    3"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.generate_bulk_skeletons_async","title":"generate_bulk_skeletons_async(root_ids, datastack_name=None, skeleton_version=None, log_warning=True, verbose_level=0)","text":"

    Generates skeletons for a list of root ids without retrieving them.

    Parameters:

    Name Type Description Default root_ids List

    A list of root ids of the skeletons to generate

    required datastack_name str

    The name of the datastack to check

    None skeleton_version int

    The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.

    None"},{"location":"api/state/","title":"client.state","text":"

    Client to interface with the JSON state service.

    Methods:

    Name Description build_neuroglancer_url

    Build a URL for a Neuroglancer deployment that will automatically retrieve specified state.

    get_neuroglancer_info

    Get the info field from a Neuroglancer deployment

    get_property_json

    Download a Neuroglancer JSON state

    get_state_json

    Download a Neuroglancer JSON state

    save_state_json_local

    Save a Neuroglancer JSON state to a JSON file locally.

    upload_property_json

    Upload a Neuroglancer JSON state

    upload_state_json

    Upload a Neuroglancer JSON state

    Attributes:

    Name Type Description server_version Optional[Version]

    The version of the service running on the remote server. Note that this

    state_service_endpoint str

    Endpoint URL for posting JSON state

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.server_version","title":"server_version: Optional[Version] property","text":"

    The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.state_service_endpoint","title":"state_service_endpoint: str property","text":"

    Endpoint URL for posting JSON state

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.build_neuroglancer_url","title":"build_neuroglancer_url(state_id, ngl_url=None, target_site=None, static_url=False, format_properties=False)","text":"

    Build a URL for a Neuroglancer deployment that will automatically retrieve specified state. If the datastack is specified, this is prepopulated from the info file field \"viewer_site\". If no ngl_url is specified in either the function or the client, a fallback neuroglancer deployment is used.

    Parameters:

    Name Type Description Default state_id int

    State id to retrieve

    required ngl_url str

    Base url of a neuroglancer deployment. If None, defaults to the value for the datastack or the client. As a fallback, a default deployment is used.

    None target_site seunglab or cave - explorer or mainline or None

    Set this to 'seunglab' for a seunglab deployment, or either 'cave-explorer'/'mainline' for a google main branch deployment. If None, checks the info field of the neuroglancer endpoint to determine which to use. Default is None.

    None static_url bool

    If True, treats \"state_id\" as a static URL directly to the JSON and does not use the state service.

    False format_properties bool

    If True, formats the url as a segment_properties info file

    False

    Returns:

    Type Description str

    The full URL requested

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.get_neuroglancer_info","title":"get_neuroglancer_info(ngl_url=None)","text":"

    Get the info field from a Neuroglancer deployment

    Parameters:

    Name Type Description Default ngl_url str(optional)

    URL to a Neuroglancer deployment. If None, defaults to the value for the datastack or the client.

    None

    Returns:

    Type Description dict

    JSON-formatted info field from the Neuroglancer deployment

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.get_property_json","title":"get_property_json(state_id)","text":"

    Download a Neuroglancer JSON state

    Parameters:

    Name Type Description Default state_id int

    ID of a JSON state uploaded to the state service.

    required

    Returns:

    Type Description dict

    JSON specifying a Neuroglancer state.

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.get_state_json","title":"get_state_json(state_id)","text":"

    Download a Neuroglancer JSON state

    Parameters:

    Name Type Description Default state_id int or string

    ID of a JSON state uploaded to the state service. If a string, treats this as the url to the json state.

    required

    Returns:

    Type Description dict

    JSON specifying a Neuroglancer state.

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.save_state_json_local","title":"save_state_json_local(json_state, filename, overwrite=False)","text":"

    Save a Neuroglancer JSON state to a JSON file locally.

    Parameters:

    Name Type Description Default json_state dict

    Dict representation of a neuroglancer state

    required filename str

    Filename to save the state to

    required overwrite bool

    Whether to overwrite the file if it exists. Default False.

    False

    Returns:

    Type Description None"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_property_json","title":"upload_property_json(property_json, state_id=None, max_size=2500000)","text":"

    Upload a Neuroglancer JSON state

    Parameters:

    Name Type Description Default propery_json dict

    Dict representation of a neuroglancer segment properties json

    required state_id int

    ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.

    None max_size Optional[int]

    Maximum size in bytes for the data to upload. Default is 2.5MB. Set to None for no limit.

    2500000

    Returns:

    Type Description int

    state_id of the uploaded JSON state

    "},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_state_json","title":"upload_state_json(json_state, state_id=None, timestamp=None)","text":"

    Upload a Neuroglancer JSON state

    Parameters:

    Name Type Description Default json_state dict

    Dict representation of a neuroglancer state

    required state_id int

    ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.

    None timestamp Optional[time]

    Timestamp for json state date. Requires state_id.

    None

    Returns:

    Type Description int

    state_id of the uploaded JSON state

    "},{"location":"api/testing/","title":"caveclient.tools.testing","text":"

    Functions:

    Name Description CAVEclientMock

    Created a mocked CAVEclient function for testing using the responses library to mock

    get_materialiation_info

    Get the materialization versions and version metadata for the materialization service.

    get_server_versions

    Get the server versions for the services used in testing.

    get_server_information

    Generate the datastack name and server locations used in testing.

    get_api_versions

    Get the API versions for the services used in testing.

    default_info

    Generate a info service info file for testing

    "},{"location":"api/testing/#caveclient.tools.testing.CAVEclientMock","title":"CAVEclientMock(datastack_name=None, global_server=None, local_server=None, info_file=None, chunkedgraph=False, chunkedgraph_server_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, chunkedgraph_api_versions=None, materialization=False, materialization_server_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, materialization_api_versions=None, available_materialization_versions=None, set_version=None, set_version_metadata=None, json_service=False, json_service_server_version=DEFAULT_JSON_SERVICE_SERVER_VERSION, skeleton_service=False, skeleton_service_server_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, schema_api_versions=None, l2cache=False, l2cache_disabled=False, global_only=False)","text":"

    Created a mocked CAVEclient function for testing using the responses library to mock the server responses. This function returns a drop-in replacement for the CAVEclient function that will be able to initialize itself and selected individual service clients with the selected options.

    Note that the test configuration is intended to be purely for pytest purposes and should not actually result in calls to active endpoints.

    Parameters:

    Name Type Description Default datastack_name str

    Name of the test datastack, by default None

    None global_server str

    Test global server address, by default None

    None local_server str

    Test local server address, by default None

    None info_file dictionary

    Info service dictionary for the datastack, by default None

    None chunkedgraph bool

    If True, configures the client to initialize a chunkedgraph subclient, by default False

    False chunkedgraph_server_version str

    Sets the value of the chunkedgraph server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.

    DEFAULT_CHUNKEDGRAPH_SERVER_VERSION chunkedgraph_api_versions list

    List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default None. If None, returns the value in CHUNKEDGRAPH_API_VERSIONS.

    None materialization bool

    If True, configures the client to initalize a materialization subclient, by default False Note that materialization being set to True will also configure the chunkedgraph client.

    False materialization_server_version str

    Sets the value of the materialization server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_MATERIALIZATION_SERVER_VERSON.

    DEFAULT_MATERIALIZATION_SERVER_VERSON available_materialization_versions list

    List of materialization database versions that the materialization client thinks exists, by default None. If None, returns the value in DEFAULT_MATERIALIZATION_VERSONS.

    None materialization_api_versions list

    List of materialization API versions that the materialization client thinks exists, by default None. If None, returns the value in MATERIALIZATION_API_VERSIONS.

    None set_version Optional[int]

    If set, will set the version of the materialization server to the value of set_version, by default None. To work, this version must be in the list of available materialization versions.

    None set_version_metadata Optional[dict]

    If set, will set the version metadata of the materialization server to the value of set_version_metadata. Default value is in DEFAULT_MATERIALIZATION_VERSION_METADATA.

    None json_service bool

    If True, configures the client to initalize a materialization subclient, by default False

    False json_service_server_version _type_

    Sets the value of the json state server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_JSON_SERVICE_SERVER_VERSION.

    DEFAULT_JSON_SERVICE_SERVER_VERSION skeleton_service bool

    If True, configures the client to initalize a skeleton service subclient, by default False

    False skeleton_service_server_version _type_

    Sets the value of the skeleton service version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_SKELETON_SERVICE_SERVER_VERSION.

    DEFAULT_SKELETON_SERVICE_SERVER_VERSION l2cache bool

    If True, configures the client to initialize an l2cache subclient, by default False Note that l2cache being set to True will also configure the chunkedgraph client.

    False l2cache_disabled bool

    If True, allows a subclient to be initialized, but emulates a situation without an L2 cache, by default False Only used if l2cache is True.

    False global_only bool

    If True, only initializes the global services and does not use a datastack, by default False.

    False

    Returns:

    Type Description CAVEclient

    A mocked and initialized CAVEclient object for testing

    Examples:

    To make a basic pytest fixture to test chunkedgraph features with an initialized CAVEclient object in your pytest conftest.py file:

    import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\ntest_datastack = \"test_stack\"\ntest_global_server = \"https://test.cave.com\"\ntest_local_server = \"https://local.cave.com\"\n\n@pytest.fixture()\ndef test_client():\n    return CAVEclientMock(\n        datastack_name=test_datastack,\n        global_server=test_global_server,\n        local_server=test_local_server,\n        chunkedgraph=True,\n    )\n\nYou can also create more complex fixtures with multiple services initialized and specific server versions:\n\n```python\n@pytest.fixture()\ndef fancier_test_client():\n    return CAVEclientMock(\n        datastack_name=test_datastack,\n        global_server=test_global_server,\n        local_server=test_local_server,\n        chunkedgraph=True,\n        chunkedgraph_server_version=\"3.0.2\",\n        materialization=True,\n        materialization_server_version=\"4.21.4\",\n        l2cache=True,\n    )\n
    "},{"location":"api/testing/#caveclient.tools.testing.get_materialiation_info","title":"get_materialiation_info(materialization_versions=DEFAULT_MATERIALIZATION_VERSONS, version_metadata=DEFAULT_MATERIALIZATION_VERSION_METADATA)","text":"

    Get the materialization versions and version metadata for the materialization service.

    Parameters:

    Name Type Description Default materialization_versions list

    List of materialization database versions that the materialization client thinks exists, by default DEFAULT_MATERIALIZATION_VERSONS.

    DEFAULT_MATERIALIZATION_VERSONS version_metadata dict

    Version metadata for the materialization service, by default DEFAULT_MATERIALIZATION_VERSION_METADATA.

    DEFAULT_MATERIALIZATION_VERSION_METADATA

    Returns:

    Type Description dict

    Dictionary with keys: \"materialization_versions\", \"version_metadata\".

    "},{"location":"api/testing/#caveclient.tools.testing.get_server_versions","title":"get_server_versions(chunkedgraph_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, materialization_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, skeleton_service_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, json_service_version=DEFAULT_JSON_SERVICE_SERVER_VERSION)","text":"

    Get the server versions for the services used in testing.

    Parameters:

    Name Type Description Default chunkedgraph_version str

    Version of the chunkedgraph server, by default DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.

    DEFAULT_CHUNKEDGRAPH_SERVER_VERSION materialization_version str

    Version of the materialization server, by default DEFAULT_MATERIALIZATION_SERVER_VERSON.

    DEFAULT_MATERIALIZATION_SERVER_VERSON skeleton_service_version str

    Version of the skeleton service server, by default DEFAULT_SKELETON_SERVICE_SERVER_VERSION.

    DEFAULT_SKELETON_SERVICE_SERVER_VERSION json_service_version str

    Version of the json service server, by default DEFAULT_JSON_SERVICE_SERVER_VERSION.

    DEFAULT_JSON_SERVICE_SERVER_VERSION

    Returns:

    Type Description dict

    Dictionary with keys: \"chunkedgraph_version\", \"materialization_version\", \"skeleton_service_version\", \"json_service_version\". Values are Version objects from packaging.versions.

    "},{"location":"api/testing/#caveclient.tools.testing.get_server_information","title":"get_server_information(datastack_name=TEST_DATASTACK, global_server=TEST_GLOBAL_SERVER, local_server=TEST_LOCAL_SERVER)","text":"

    Generate the datastack name and server locations used in testing.

    Parameters:

    Name Type Description Default datastack_name str

    Datastack value, by default the value in TEST_DATASTACK.

    TEST_DATASTACK global_server str

    Server for global services, by default TEST_GLOBAL_SERVER.

    TEST_GLOBAL_SERVER local_server str

    Server for local services, by default TEST_LOCAL_SERVER.

    TEST_LOCAL_SERVER

    Returns:

    Type Description dict

    Dictionary with keys: \"datastack_name\", \"local_server\", \"global_server\".

    "},{"location":"api/testing/#caveclient.tools.testing.get_api_versions","title":"get_api_versions(chunkedgraph_api_versions=CHUNKEDGRAPH_API_VERSIONS, materialization_api_versions=MATERIALIZATION_API_VERSIONS, schema_api_versions=SCHEMA_API_VERSIONS)","text":"

    Get the API versions for the services used in testing.

    Parameters:

    Name Type Description Default chunkedgraph_api_versions list

    List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default CHUNKEDGRAPH_API_VERSIONS.

    CHUNKEDGRAPH_API_VERSIONS materialization_api_versions list

    List of materialization API versions that the materialization client thinks exists, by default MATERIALIZATION_API_VERSIONS.

    MATERIALIZATION_API_VERSIONS schema_api_versions list

    List of schema API versions that the schema client thinks exists, by default SCHEMA_API_VERSIONS.

    SCHEMA_API_VERSIONS

    Returns:

    Type Description dict

    Dictionary with keys: \"chunkedgraph_api_versions\", \"materialization_api_versions\", \"schema_api_versions\".

    "},{"location":"api/testing/#caveclient.tools.testing.default_info","title":"default_info(local_server=TEST_LOCAL_SERVER)","text":"

    Generate a info service info file for testing

    Parameters:

    Name Type Description Default local_server str

    Name of the local service, by default the value in TEST_LOCAL_SERVER.

    TEST_LOCAL_SERVER

    Returns:

    Type Description dict

    Info file for the datastack.

    "},{"location":"tutorials/","title":"Getting Started","text":"

    AnnotationFramework client is a package for simplifying interactions with HTML services associated with the CAVE (Connectome Annotation Versioning Engine).

    For a larger introduction to CAVE and its services, see the main GitHub organization site: https://github.com/CAVEconnectome

    "},{"location":"tutorials/#installation","title":"Installation","text":"

    The CAVEclient can be installed with pip:

    $ pip install caveclient\n
    "},{"location":"tutorials/#assumptions","title":"Assumptions","text":"

    The code is setup to work flexibly with any deployment of these services, but you need to specify the server_address if that address is not https://globalv1.daf-apis.com/ for each client when initializing it. Similarly, the clients can query the info service for metadata to simplify the interaction with a datastack, but you have to specify a datastack name.

    "},{"location":"tutorials/advanced/","title":"Advanced Usage","text":""},{"location":"tutorials/advanced/#changing-session-configurations","title":"Changing session configurations","text":"

    It is possible to change details of how a client talks to the various servers it needs to interface with. For instance, the CAVEclient will attempt to retry specific failed requests to the server, but will only try a specific number of times, and will only wait specific amounts of time between retries. These values can be changed via the set_session_defaults method. For instance, to change the number of retries to 5, and to increase the delay between subsequent retries, you could do:

    from caveclient import set_session_defaults\n\nset_session_defaults(max_retries=5, backoff_factor=0.5)\n

    Note that this needs to happen before initializing the client for this to work properly. Some of these parameters are also adjustable at the client level.

    To view the current session defaults, you can use the get_session_defaults method:

    from caveclient import get_session_defaults\n\nclient.get_session_defaults()\n

    More information on the available parameters can be found in the API documentation.

    "},{"location":"tutorials/advanced/#writing-unit-tests-using-caveclient","title":"Writing unit tests using caveclient","text":"

    If you have a module that relies on caveclient and you want to write unit tests for it, you can use the caveclient.testing module to help you create an initialized, mocked client with user-specified options. This helps deal with the complex conversation that caveclient has with various services when initializing itself so that you get the appropriate functionality. The function CAVEclientMock offers a number of options for how to build a client, including server versions for various services, available API versions, and more. Importantly, the module provides sensible defaults but also allows you to override them as needed. Note that using testing requires the responses to be installed, which is not a dependency of caveclient.

    The CAVEclientMock function requires you to specify which services you expect to call on in order to ensure that your code is working as expected. For example, to create a pytest fixture that would utilize the chunkedgraph module, but otherwise use default nonsense test values, you could include this in your conftest.py:

    import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\n@pytest.fixture()\ndef test_client():\n    return CAVEclientMock(\n        chunkedgraph=True,\n    )\n

    Then, in your test module, you can use the test_client fixture to get a client. Note that after mocked initialization, the client will attempt normal network requests and thus you should mock the responses. If you only care to get a specific value back from a given function, you can use pytest-mock to mock the response to a given function call. For example, if you have a function do_something_with_roots that takes a caveclient that uses the get_roots function, you could mock the get_roots function so that it return a specific value:

    from pytest_mock import mocker\nfrom conftest import test_client\nfrom my_module import do_something_with_roots\n\ndef test_get_roots(mocker, test_client):\n    mocker.patch.object(test_client.chunkedgraph, 'get_roots', return_value=[1, 2, 3])\n    test_data = # Something appropriate\n    test_output = do_something_with_roots(test_client, test_data)\n    assert test_output == # The expected answer\n

    Note that if you your own datastack info_file that has a different local_server address than the default value (TEST_LOCAL_SERVER variable, which defaults to https://local.cave.com) you will also need to specify a local_server with the same value as in the local_server field of your info file.

    "},{"location":"tutorials/advanced/#specifying-conditions","title":"Specifying conditions","text":"

    While sensible defaults are provided, you can also specify things like server versions to make sure your code works with the versions of the services you expect. For example, let's make a richer mock client that specifies the server versions for the chunkedgraph, materailization, and l2cache services:

    @pytest.fixture()\ndef version_specified_client():\n    return CAVEclientMock(\n        chunkedgraph=True,\n        chunkedgraph_server_version='3.0.1',\n        materialization=True,\n        materialization_server_version='2.0.0',\n        l2cache=True,\n    )\n

    Note that some services like l2cache do not currently use a server version to offer different functionality, and this value is not exposed for them currently. See the API documentation for more information.

    You can also override default values like server_address or datastack_name: 

    @pytest.fixture()\ndef server_specified_client():\n    return CAVEclientMock(\n        datastack_name='my_datastack',\n        server_address='http://my.server.com',\n        chunkedgraph=True,\n        materialization=True,\n    )\n

    If you want to get access to the various default values of server version, datastack name, datastack info file, and api versions, you can use functions get_server_versions, get_server_information, default_info, get_api_versions respectively. Each of these functions will return a dictionary of the values that can be used as a kwargs input into CAVEclientMock. If you specify your own override values, it will take precedence over the default value and you can just use the dictionary in your tests. See the caveclient tests for an example of how to use these functions.

    "},{"location":"tutorials/annotation/","title":"Annotation","text":"

    The AnnotationClient is used to interact with the AnnotationEngine service to create tables from existing schema, upload new data, and download existing annotations. Note that annotations in the AnnotationEngine are not linked to any particular segmentation, and thus do not include any root ids. An annotation client is accessed with client.annotation.

    "},{"location":"tutorials/annotation/#getting-existing-tables","title":"Getting existing tables","text":"

    A list of the existing tables for the datastack can be found with get_tables().

    all_tables = client.annotation.get_tables()\nall_tables[0]\n

    Each table has three main properties that can be useful to know:

    • table_name : The table name, used to refer to it when uploading or downloading annotations. This is also passed through to the table in the Materialized database.
    • schema_name : The name of the table's schema from EMAnnotationSchemas (see below).
    • max_annotation_id : An upper limit on the number of annotations already contained in the table.
    "},{"location":"tutorials/annotation/#downloading-annotations","title":"Downloading annotations","text":"

    You can download the JSON representation of a data point through the get_annotation() method. This can be useful if you need to look up information on unmaterialized data, or to see what a properly templated annotation looks like.

    table_name = all_tables[0]['table_name']      # 'ais_analysis_soma'\nannotation_id = 100\nclient.annotation.get_annotation(annotation_ids=annotation_id, table_name=table_name)\n
    "},{"location":"tutorials/annotation/#create-a-new-table","title":"Create a new table","text":"

    One can create a new table with a specified schema with the create_table() method:

    client.annotation.create_table(table_name='test_table',\n                               schema_name='microns_func_coreg',\n                               voxel_resolution = [1,1,1],\n                               description=\"some text to describe your table\")\n

    The voxel resolution is the units your position columns will be uploaded in [1,1,1] would imply a nm location, where as [4,4,40] would correspond to voxels of that size. If you are uploading points from a neuroglancer session, you want this to match the units of that neuroglancer view.

    Note there are some optional metadata parameters to create_table()

    • notice_text : This is text that will show up to users who access this data as a warning. This could be used to warn users that the data is not complete or checked yet, or to advertise that a particular publication should be cited when using this table.
    • read_permission : one of \"PRIVATE\" which means only you can read data in this table. \"PUBLIC\" (default) which means anyone can read this table that has read permissions to this dataset. So if and only if you can read the segmentation results of this data, you can read this table. \"GROUP\" which means that you must share a common group with this user for them to be able to read. We need to make a way to discover what groups you are in and who you share groups with.
    • write_permission: one of \"PRIVATE\" (default), which means only you can write to this table. \"PUBLIC\" which means anyone can write to this table that has write permissions to this dataset. Note although this means anyone can add data, no annotations are ever truly overwritten. \"GROUP\" which means that you must share a common group with this user for them to be able to write. We need to make a way to discover what groups you are in and who you share groups with.

    If you change your mind about what you want for metadata, some but not all fields can be updated with update_metadata(). This includes the description, the notice_text, and the permissions, but not the name, schema or voxel resolution.

    # to update description\nclient.annotation.update_metadata(table_name='test_table',\n                                  description=\"a new description for my table\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n                                  notice_text=\"This table isn't done yet, don't trust it. Contact me\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n                                  read_permisison=\"PUBLIC\")\n

    New data can be generated as a dict or list of dicts following the schema and uploaded with post_annotation. For example, a microns_func_coreg point needs to have:

    • type set to microns_func_coreg
    • pt set to a dict with position as a key and the xyz location as a value.
    • func_id set to an integer.

    The following could would create a new annotation and then upload it to the service. Note that you get back the annotation id(s) of what you uploaded.

    new_data = {'type': 'microns_func_coreg',\n            'pt': {'position': [1,2,3]},\n            'func_id': 0}\nclient.annotation.post_annotation(table_name='test_table', data=[new_data])\n

    There are methods to simplify annotation uploads if you have a pandas dataframe whose structure mirrors the struction of the annotation schema you want to upload

    import pandas as pd\n\ndf = pd.DataFrame([{'id':0,\n         'type': 'microns_func_coreg',\n         'pt_position': [1,2,3]},\n         'func_id': 0},\n        {'id':1,\n        'type': 'microns_func_coreg',\n        'pt_position': [3,2,1]},\n        'func_id': 2}])\nclient.annotation.post_annotation_df('test_table', df)\n

    Note that here I specified the IDs of my annotations, which you can do, but then its up to you to assure that the IDs don't collide with other IDs. If you leave them blank then the service will assign the IDs for you.

    There is a similar method for updating update_annotation_df()

    "},{"location":"tutorials/annotation/#staged-annotations","title":"Staged Annotations","text":"

    Staged annotations help ensure that the annotations you post follow the appropriate schema, both by providing guides to the field names and locally validating against a schema before uploading. The most common use case for staged annotations is to create a StagedAnnotation object for a given table, then add annotations to it individually or as a group, and finally upload to the annotation table.

    To get a StagedAnnotation object, you can start with either a table name or a schema name. Here, we'll assume that there's already a table called \"my_table\" that is running a \"cell_type_local\" schema. If we want to add new annotations to the table, we simply use the table name with stage_annotations().

    stage = client.annotation.stage_annotations(\"my_table\")\n

    This stage object retrieves the schema for the table and hosts a local collection of annotations. Every time you add an annotation, it is immediately validated against the schema. To add an annotation, use the add method:

    stage.add(\n    cell_type = \"pyramidal_cell\",\n    classification_system=\"excitatory\",\n    pt_position=[100,100,10],\n)\n

    The argument names derive from fields in the schema and you must provide all required fields. Any number of annotations can be added to the stage. A dataframe of annotations can also be added with stage.add_dataframe, and requires an exact match between column names and schema fields. The key difference between this and posting a dataframe directly is that annotations added to a StagedAnnotations are validated locally, allowing any issues to be caught before uploading.

    You can see the annotations as a list of dictionary records with stage.annotation_list or as a Pandas dataframe with stage.annotation_dataframe. Finally, if you initialized the stage with a table name, this information is stored in the stage and you can simply upload it from the client.

    client.annotation.upload_staged_annotations(stage)\n

    Updating annotations requires knowing the annotation id of the annotation you are updating, which is not required in the schema otherwise. In order to stage updated annotations, set the update parameter to True when creating the stage.

    update_stage = client.annotation.stage_annotations(\"my_table\", update=True)\nupdate_stage.add(\n    id=1,\n    cell_type = \"stellate_cell\",\n    classification_system=\"excitatory\",\n    pt_position=[100,100,10],\n)\n

    The update also informs the framework client to treat the annotations as an update and it will use the appropriate methods automatically when uploading client.annotation.upload_staged_annotations.

    If you want to specify ids when posting new annotations, id_field can be set to True when creating the StagedAnnotation object. This will enforce an id column but still post the data as new annotations.

    If you might be adding spatial data in coordinates that might be different than the resolution for the table, you can also set the annotation_resolution when creating the stage. The stage will convert between the resolution you specify for your own annotations and the resolution that the table expects.

    stage = client.annotation.stage_annotations(\"my_table\", annotation_resolution=[8,8,40])\nstage.add(\n    cell_type='pyramidal_cell',\n    classification_system=\"excitatory\",\n    pt_position=[50,50,10],\n)\n
    "},{"location":"tutorials/authentication/","title":"Authentication","text":"

    Authentication tokens are generally needed for programmatic access to our services. The AuthClient handles storing and loading your token or tokens and inserting it into requests in other clients.

    We can access the auth client from client.auth. Once you have saved a token, you probably won't interact with this client very often, however it has some convenient features for saving new tokens the first time. Let's see if you have a token already. Probably not.

    client = CAVEclient()\nauth = client.auth\nprint(f\"My current token is: {auth.token}\")\n
    "},{"location":"tutorials/authentication/#getting-a-new-token","title":"Getting a new token","text":"

    To get a new token, you will need to manually acquire it. For convenience, the function client.auth.get_new_token() provides instructions for how to get and save the token.

    By default, the token is saved to ~/.cloudvolume/secrets/cave-secret.json as a string under the key token. This makes it compatible by default with Cloudvolume projects, which can come in handy. The following steps will save a token to the default location.

    auth.get_new_token()\n
    new_token = 'abcdef1234567890' #This is the text you see after you visit the website.\nauth.save_token(token=new_token)\nprint(f\"My token is now: {auth.token}\")\n

    Note that requesting a new token will invalidate your previous token on the same project. If you want to use the same token across different computers, you will need to share the same token information.

    "},{"location":"tutorials/authentication/#loading-saved-tokens","title":"Loading saved tokens","text":"

    Try opening ~/.cloudvolume/secrets/cave-secret.json to see what we just created.

    If we had wanted to use a different file or a different json key, we could have specified that in auth.save_token.

    Because we used the default values, this token is used automatically when we initialize a new CAVEclient. If we wanted to use a different token file, token key, or even directly specify a token we could do so here.

    client = CAVEclient(datastack_name)\nprint(f\"Now my basic token is: {client.auth.token}\")\n\nclient_direct = CAVEclient(datastack_name, auth_token='another_fake_token_678')\nprint(f\"A directly specified token is: {client_direct.auth.token}\")\n

    If you use a CAVEclient, the AuthClient and its token will be automatically applied to any other services without further use.

    "},{"location":"tutorials/chunkedgraph/","title":"Chunked Graph","text":"

    The chunkedgraph is a dynamic oct-tree connected components supervoxel graph.

    A visual representation of an oct-tree (Wikipedia (WhiteTimberwolf) CC BY-SA 3.0)

    As with any oct-tree, it is organized in hierarchical levels, with the bottom level 1 corresponding to the supervoxels of the segmentations, and the top level being the unique connected components of the supervoxel graph.

    A figure illustrating the spatial chunking and editing of a the chunkedgraph. From Dorkenwald et. al 2021

    The ChunkedGraph client allows one to interact with the ChunkedGraph service, which stores and updates the supervoxel agglomeration graph. This is most often useful for looking up an object root id of a supervoxel or looking up supervoxels belonging to a root id. The ChunkedGraph client is at client.chunkedgraph.

    "},{"location":"tutorials/chunkedgraph/#look-up-a-supervoxel","title":"Look up a supervoxel","text":"

    Usually in Neuroglancer, one never notices supervoxel ids, but they are important for programmatic work. In order to look up the root id for a location in space, one needs to use the supervoxel segmentation to get the associated supervoxel id. The ChunkedGraph client makes this easy using the client.chunkedgraph.get_root_id() method.

    sv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id)\n

    However, as proofreading occurs, the root id that a supervoxel belongs to can change. By default, this function returns the current state, however one can also provide a UTC timestamp to get the root id at a particular moment in history. This can be useful for reproducible analysis. Note below that the root id for the same supervoxel is different than it is now.

    import datetime\n\n# I looked up the UTC POSIX timestamp from a day in early 2019.\ntimestamp = datetime.datetime.utcfromtimestamp(1546595253)\n\nsv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id, timestamp=timestamp)\n

    If you are doing this across lots of supervoxels (or any nodes) then you can do it more efficiently in one request with client.chunkedgraph.get_roots()

    node_ids = [104200755619042523, 104200755619042524,104200755619042525]\nroot_ids = client.chunkedgraph.get_roots(node_ids)\n
    "},{"location":"tutorials/chunkedgraph/#getting-supervoxels-for-a-root-id","title":"Getting supervoxels for a root id","text":"

    A root id is associated with a particular agglomeration of supervoxels, which can be found with the client.chunkedgraph.get_leaves() method. A new root id is generated for every new change in the chunkedgraph, so time stamps do not apply.

    root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id)\n

    You can also query the chunkedgraph not all the way to the bottom, using the stop_layer option

    root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id,stop_layer=2)\n

    This will get all the level 2 IDs for this root, which correspond to the lowest chunk of the hierarchy. An analogous option exists for client.chunkedgraph.get_roots(). This is useful to help find nodes to query within the l2cache, amongst other things.

    "},{"location":"tutorials/chunkedgraph/#other-functions","title":"Other functions","text":"

    There are a variety of other interesting functions to explore in client.chunkedgraph.

    "},{"location":"tutorials/framework/","title":"Introduction","text":""},{"location":"tutorials/framework/#caveclient-one-client-for-all-services","title":"CAVEclient: one client for all services","text":"

    The CAVE Framework consists of a number of different services, each with a specific set of tasks that it can perform through REST endpoints. The CAVEclient is designed to ease programmatic interaction with all of the various endpoints. In addition, most programmatic access requires the use of authentication tokens. In order to collect a given server, datastack name, and user token together into a coherent package that can be used on multiple endpoints, the CAVEclient builds appropriately configured clients for each of the specific services. Each of the individual services has their own specific documentation as well.

    "},{"location":"tutorials/framework/#global-and-local-services","title":"Global and Local Services","text":"

    There are two categories of data in CAVE: Global and local. Local services are associated with a single so-called datastack, which refers to a precise collection of imagery and segmentation data that function together. For example, EM imagery and a specific pychunkedgraph segmentation would be one datastack, while the same EM imagery but an initial static segmentation would be another. Datastacks are referred to by a short name, for instance pinky100_public_flat_v185.

    Global services are those that are potentially shared across multiple different specific datastacks. These include the info service, which can describe the properties of all available datastacks, the authentication service, and the state service that hosts neuroglancer states. Global services are associated with a particular URL (by default http://globalv1.daf-apis.com), but not a single datastack.

    "},{"location":"tutorials/framework/#initializing-a-caveclient","title":"Initializing a CAVEclient","text":"

    Assuming that the services are on http://globalv1.daf-apis.com and authentication tokens are either not being used or set up with default values (see Authentication), a simple CAVEclient that can only access global services can be initialized:

    from caveclient import CAVEclient\n\nclient = CAVEclient()\n

    Just to confirm that this works, let's see if we can get the EM image source from the InfoService. If you get a list of names of datastacks, all is good. If you have not yet set up an authentication token or you get an authentication error, look at Getting a new token for information about how to set up your auth token.

    client.info.get_datastacks()\n

    If you have a specific datastack you want to use, you can initialize your CAVEclient with it. This gives you access to the full range of client functions.

    client = CAVEclient(datastack_name='my_datastack')\n
    "},{"location":"tutorials/framework/#using-other-server-addresses","title":"Using Other Server Addresses","text":"

    If your data is hosted by a different global server, you specify its address when initializing the client.

    client = CAVEclient(datastack_name='my_datastack', server_address='http://global.myserver.com')\n

    By default, if you pass both a server address and a datastack, the client will store the mapping from datastack to server address in the same location as the default for authentication tokens. Once stored, the client will automatically use the correct server address for the datastack if none is provided. You can override storing the server address by passing write_server_address=False. Datastacks can be removed from the cache using

    caveclient.datastack_lookup.reset_server_address_cache(datastack_name).

    "},{"location":"tutorials/framework/#accessing-specific-clients","title":"Accessing specific clients","text":"

    Each client can be accessed as a property of the main client. See the documentation at left for the capabilities of each. Assuming your client is named client, the subclients for each service are:

    • Authentication Service : client.auth
    • AnnotationEngine : client.annotation
    • PyChunkedGraph : client.chunkedgraph
    • InfoService : client.info
    • EM Annotation Schemas : client.schemas
    • JSON Neuroglancer State Service : client.state
    • Skeleton Service : client.skeletonservice
    "},{"location":"tutorials/info/","title":"Info Service","text":"

    A datastack has a number of complex paths to various data sources that together comprise a datastack. Rather than hardcode these paths, the InfoService allows one to query the location of each data source. This is also convenient in case data sources change.

    An InfoClient is accessed at client.info.

    client = CAVEclient(datastack_name)\nprint(f\"This is an info client for {client.info.datastack_name} on {client.info.server_address}\")\n
    "},{"location":"tutorials/info/#accessing-datastack-information","title":"Accessing datastack information","text":"

    All of the information accessible for the datastack can be seen as a dict using get_datastack_info().

    info.get_datastack_info()\n

    Individual entries can be found as well. Use tab autocomplete to see the various possibilities.

    info.graphene_source()\n
    "},{"location":"tutorials/info/#adjusting-formatting","title":"Adjusting formatting","text":"

    Because of the way neuroglancer looks up data versus cloudvolume, sometimes one needs to convert between gs:// style paths to https://storage.googleapis.com/ stype paths. All of the path sources in the info client accept a format_for argument that can handle this, and correctly adapts to graphene vs precomputed data sources.

    neuroglancer_style_source = info.image_source(format_for='neuroglancer')\nprint(f\"With gs-style: { neuroglancer_style_source }\")\n\ncloudvolume_style_source = info.image_source(format_for='cloudvolume')\nprint(f\"With https-style: { cloudvolume_style_source }\")\n
    "},{"location":"tutorials/l2cache/","title":"Level 2 Cache","text":"

    To understand the level 2 cache, you must understand the structure of the chunkedgraph so see the chunkedgraph tutorial.

    Nodes on the second level or layer of the graph, corresponds to all the supervoxels that are locally connected to one another within a single level 2 spatial \"chunk\" of the data. The Level 2 Cache, is a service whose job it is to track and update relevant statistics about every level 2 node within the a chunkedgraph. The source code of this service can be found here.

    "},{"location":"tutorials/l2cache/#finding-level-2-nodes","title":"Finding Level 2 Nodes","text":"

    The chunkedgraph can be used to find the level2 nodes of a rootID using a stop_layer=2 keyword argument on the client.chunkedgraph.get_leaves(). Conversely the level 2 node of a supervoxel can be found using the same keyword argument of client.chunkedgraph.get_roots(). Note if you don't specify a timestamp it will give you the level2 node that is presently associated with the object.

    "},{"location":"tutorials/l2cache/#statistics","title":"Statistics","text":"

    The statistics that are available are:

    • area_nm2: The surface area of the object in square nanometers. Does not include border touching voxels
    • size_nm3: The volume of the object in cubic nanometers, based on counting voxels in the object.
    • max_dt_nm: The maximum edge distance transform of that object in nanometers. Meant to capture the maximum \"thickness\" of the voxels in the node.
    • mean_dt_nm: The average edge distance transform of that object in nanometers. Meant to capture the average \"thickness\" of voxels in that node.
    • rep_coord_nm: A list of x,y,z coordinates in nanometers that represent a point within the object that is designed to be close to the \"center\" of the object. This is the location of the max_dt_nm value.
    • chunk_intersect_count: A 2 x 3 matrix representing the 6 sides of the chunk, and whose values represent how many voxels border that side of the chunk. Meant to help understand significant the borders with other chunks are. Ordering is the [[x_bottom, y_bottom, z_bottom],[x_top, y_top, z_top]] where {xyz}_bottom refers to the face which has the smallest values for that dimension, and {xyz}_top refers to the face which has the largest.
    • pca A 3x3 matrix representing the principal components of the xyz point cloud of voxels for this object. Ordering is NxD where N is the components and D are the xyz dimensions. Meant to help desribe the orientation of the level 2 chunk. Note that this is not calculated for very small objects and so might not be present for all level 2 nodes. You will see that its availability correlates strongly with size_nm3.
    • pca_val The 3 principal component values for the PCA components.
    "},{"location":"tutorials/l2cache/#retrieving-level-2-statistics","title":"Retrieving Level 2 Statistics","text":"

    Level 2 stats about nodes can be retreived using the client.l2cache.get_l2data() method. It simply takes a list of level 2 nodes you want to retrieve. Optionally you can specify only the attributes that you are interested in retrieving which will speed up the request.

    "},{"location":"tutorials/l2cache/#missing-data","title":"Missing Data","text":"

    The service is constantly watching for changes made to objects and recalculating stats on new level2 nodes that are created, in order to keep its database of statistics current. This however takes some time, and is subject to sporadic rare failures. If you request stats on a level 2 node which are not in the database, you will receive an empty dictionary for that node. This will immediately trigger the system to recalculate the statistics of that missing data, and so it should be available shortly (on the order of seconds) if systems are operational. Please note that PCA is not calculated for very small objects because it is not meaningful. So if you are interested in differentiating whether PCA is not available because it hasn't been calculated, vs when its not available because it is not possible to calculate, you should ask for at least one other non PCA statistic as well. You will see that its availability correlates strongly with size_nm3.

    "},{"location":"tutorials/l2cache/#use-cases","title":"Use Cases","text":""},{"location":"tutorials/l2cache/#calculate-total-area-and-volume-of-cells","title":"Calculate Total Area and Volume of Cells","text":"

    Say you want to calculate the total surface area and volume of a object in the dataset. The areas and volume of each component can simply be added together to do this.

    import pandas as pd\nroot_id = 648518346349541252\nlvl2nodes = client.chunkedgraph.get_leaves(root_id,stop_layer=2)\nl2stats = client.l2cache.get_l2data(lvl2nodes, attributes=['size_nm3','area_nm2'])\nl2df = pd.DataFrame(l2stats).T\ntotal_area_um2=l2df.area_nm2.sum()/(1000*1000)\ntotal_volume_um3 = l2df.size_nm3.sum()/(1000*1000*1000)\n

    By utilizing the bounds argument of get_leaves, you can also do simple spatially restricted analysis of objects. In fact, because you have data on each level2 node individually, you can segregate the neuron using any labelling of its topology.

    "},{"location":"tutorials/l2cache/#skeletonization","title":"Skeletonization","text":"

    Level 2 nodes have \"cross chunk\" edges within the chunkedgraph which represent what level 2 nodes that object is locally connected to. This forms a graph between the level 2 nodes of the object that can be retrieved using the chunkedgraph function client.chunkedgraph. This graph represents a topological representation of the neuron at the resolution of individual chunks, and is guaranteed to be fully connected, unlike a voxel or mesh representation of the neuron which can have gaps where there are defects in the segmentation volume or incorrectly inferred edges at self contact locations.

    The level 2 graph can be turned into a skeleton representation of a neuron using a graph based TEASAR like algorithm as described for skeletonizing meshes in this MeshParty Documentation. There is an implementation of this approach that utilizes the chunkedgraph and the L2cache if available here and on pypi as pcg-skel. In this implementation the l2cache is used to more accurately place the level 2 nodes in space using the rep_coord_nm value.

    Note that there is detailed documentation on the Skeleton Client interface at the skeletonization tutorial.

    "},{"location":"tutorials/l2cache/#trajectory-distributions","title":"Trajectory Distributions","text":"

    If one is interested in the bulk direction of processes in a region of the brain, one can start with supervoxels in a region, find level 2 nodes that correspond to them, filter out components based on size, (or other criteria such as whether they are part of objects that have components in some other brain area) and look at the distribution of PCA components to understand the directions that those processes are moving within that region of space.

    "},{"location":"tutorials/materialization/","title":"Materialization","text":"

    The Materialization client allows one to interact with the materialized annotation tables, that were posted to the annotation service (the annotations tutorial).

    To see the entire class visit the API doc.

    The service regularly looks up all annotations and the segids underneath all the boundspatialpoints. You can then query these tables to find out the IDs that underlie the annotations, or the annotations that now intersect with certain IDs.

    For example, one common pattern is that you have identified a cell based on the location of its cell body, and you have an annotation there.

    You want to know what are the inputs onto the cell, so you first query the annotation table with your soma annotation, asking for the current ID underneath that soma. Then you query a synapse table for all synapse annotations that have a post-synaptic ID equal to the ID from your soma annotation.

    In this way your code stays the same, as the proofreading changes and you can track the connectivity of your cell over time.

    "},{"location":"tutorials/materialization/#initializing-the-client","title":"Initializing the client","text":"

    By default when you initialize the overall client, it will choose the most recent materialization version available. This may or may not be desirable depending on your use case. If your code involves using specific IDs then you should be using a specific version that is tied to a timepoint where those IDs are valid.

    To see what versions are available, use the client.materialize.get_versions() function.

    client.materialize.get_versions()\n

    Each version has a timestamp it was run on as well as a date when it will expire. You can query all this metadata for a specific version using client.materialize.get_version_metadata() or all versions using client.materialize.get_versions_metadata().

    To change the default version, alter the .version property of the client. This will change the version for all subsequent calls which expect one, unless you specify a different version in the method call. Note that this also sets the timestamp property of the client to the timestamp of the version for methods which expect a timestamp.

    client.version = 9\n

    You can also specify the version when you initialize the client, e.g.:

    client = CAVEclient('minnie65_public', version=661)\n

    Or, you can specify the version when making a particular method call.

    "},{"location":"tutorials/materialization/#browsing-versions","title":"Browsing versions","text":"

    To see what tables are available in a version you can use client.materialize.get_tables().

    If you want to read about the description of what that table is, use the annotationengine client client.materialize.get_table_metadata().

    If you want to read more about the schema for the annotation table use the schema service client.schema.schema_definition().

    Note, the materialization service has a human readable webpage that links to the other services that might be more convenient for you to browse, to get a link there in ipython display client.materialize.homepage

    for some important tables, the info service has a pointer to which table you should use in the metadata for the datastack. client.info.get_datastack_info()['synapse_table'] and client.info.get_datastack_info()['soma_table'].

    To see how many annotations are in a particular table use

    nannotations=client.materialize.get_annotation_count('my_table')\n
    "},{"location":"tutorials/materialization/#querying-tables","title":"Querying tables","text":"

    To query a small table, you can just download the whole thing using client.materialize.query_table() which will return a dataframe of the table.

    Note however, some tables, such as the synapse table might be very large 200-300 million rows and the service will only return the first 200,000 results, and not in a deterministic manner. NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.

    To just get a preview, use the limit argument (but note again that this won't be a reproducible set)

    df=client.materialize.query_table('my_table', limit=10)\n

    For most applications, you will want to filter the query in some way.

    We offer seven kinds of filters you can apply: filter_in_dict filter_out_dict, filter_equal_dict, filter_greater_dict, filter_less_dict, filter_greater_equal_dict, and filter_less_equal_dict. For query_table each is specified as a dictionary where the keys are column names, and the values are a list of values (or single value in the case of filter_equal).

    So for example to query a synapse table for all synapses onto a neuron in flywire you would use

    synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n                                  filter_equal_dict = {'post_pt_root_id': MYID})\n

    The speed of querying is affected by a number of factors, including the size of the data. To improve the performance of results, you can reduce the number of columns returned using select_columns.

    So for example, if you are only interested in the root_ids and locations of pre_synaptic terminals you might limit the query with select_columns. Also, it is convenient to return the with positions as a column of np.array([x,y,z]) coordinates for many purposes. However, sometimes you might prefer to have them split out as separate x, y, z columns. To enable this option use split_columns=True. split_columns=True is faster, as combining them is an extra step.

    synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n                                  filter_equal_dict = {'post_pt_root_id': MYID},\n                                  select_columns=['id','pre_pt_root_id', 'pre_pt_position'],\n                                  split_columns=True)\n
    "},{"location":"tutorials/materialization/#desired-resolution","title":"Desired Resolution","text":"

    Often you want to have position information in different units. For example, to consider synapse locations or soma locations, you might want to have positions in nanometers or microns.

    To create neuroglancer views, you might want positions in integer voxels of a size that aligns with the resolution you are used to using Neuroglancer at.

    Annotation tables can be created and uploaded in varying resolutions according to whatever the user of the table felt was natural. This information is available in the metadata for that table. In addition, you may pass desired_resolution as a keyword argument which will automatically convert all spatial positions into voxels of that size in nanometers.

    So if you want positions in nanometers, you would pass desired_resolution=[1,1,1]. If you want positions in microns you would pass desired_resolution=[1000,1000,1000]. If you want positions in 4,4,40nm voxel coordinates to use with cloud-volume or neuroglancer you would pass desired_resolution=[4,4,40].

    "},{"location":"tutorials/materialization/#spatial-filters","title":"Spatial Filters","text":"

    You can also filter columns that are associated with spatial locations based upon being within a 3d bounding box.

    This is done by adding a filter_spatial_dict argument to query_table. The units of the bounding box should be in the units of the voxel_resolution of the table (which can be obtained from client.materialize.get_table_metadata()).

    bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nsynapse_table = client.info.get_datastack_info('synapse_table')\ndf=client.materialize.query_table(synapse_table,\n                                  filter_equal_dict = {'post_pt_root_id': MYID},\n                                  filter_spatial_dict = {'post_pt_position': bounding_box})\n
    "},{"location":"tutorials/materialization/#synapse-query","title":"Synapse Query","text":"

    For synapses in particular, we have a simplified method for querying them with a reduced syntax. client.materialize.synapse_query() lets you specify pre and post synaptic partners as keyword arguments and bounding boxes. The defaults make reasonable assumptions about what you want to query, namely that the synapse_table is the table that the info service advertises, and that if you specify a bounding box, that you want the post_pt_position. These can be overridden of course, but the above bounding box query is simplified to.

    NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.

    bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\ndf=client.materialize.query_table(post_ids = MYID,\n                                  bounding_box=bounding_box)\n
    "},{"location":"tutorials/materialization/#live-query","title":"Live Query","text":"

    In order to query the materialized tables above you can only use IDs that were present at the timestamp of the materialization. If you query the tables with an ID that is not valid during the time of the materialization you will get empty results.

    To check if root_ids are valid at your materialization's timestamp, you can use client.chunkedgraph.is_latest_roots()

    import numpy as np\nmat_time = client.materialize.get_timestamp()\nis_latest = client.chunkedgraph.is_latest_roots([MYID], timestamp=mat_time)\nassert(np.all(is_latest))\n

    If you need to lookup what happened to that ID, you can use the chunkedgraph lineage tree, to look into the future or the past, depending on your application you can use client.chunkedgraph.get_lineage_graph().

    Again, the ideal situation is that you have an annotation in the database which refers to your objects of interest, and querying that table by the id column will return the object in the most recent materialization.

    However, sometimes you might be browsing and proofreadding the data and get an ID that is more recent that the most recent version available. For convenience, you can use client.materialize.live_query().

    to automatically update the results of your query to a time in the future, such as now. For example, to pass now, use datetime.datetime.now(datetime.timezone.utc). Note all timestamps are in UTC throughout the codebase.

    import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.live_query(synapse_table,\n                                  datetime.datetime.now(datetime.timezone.utc),\n                                  filter_equal_dict = {'post_pt_root_id': MYID})\n

    This will raise an ValueError exception if the IDs passed in your filters are not valid at the timestamp given

    You can also pass a timestamp directly to query_table and it will call live_query automatically.

    import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n                                  timestamp=datetime.datetime.now(datetime.timezone.utc),\n                                  filter_equal_dict = {'post_pt_root_id': MYID})\n

    Also, keep in mind if you run multiple queries and at each time pass datetime.datetime.now(datetime.timezone.utc), there is no guarantee that the IDs will be consistent from query to query, as proofreading might be happening at any time. For larger scale analysis constraining oneself to a materialized version will ensure consistent results.

    Versions have varying expiration times in order to support the tradeoff between recency and consistency, so before undertaking an analysis project consider what version you want to query and what your plan will be to update your analysis to future versions.

    "},{"location":"tutorials/materialization/#content-aware-interface-experimental","title":"Content-aware Interface (Experimental)","text":"

    As of version 5.8.0, we have introduced a new interface to query tables and views. This interface might have small but breaking changes in the near future. :::

    In order to make the querying interface more consistent across tables, we have introduced an additional alternative interface to filtering and querying data via the client.materialize.tables object. When you instantiate this object, this object finds all of the existing tables and the list of their columns and lets you filter the tables as arguments in the function with suggestions. Moreover, the filtering arguments and the querying arguments are separated into two.

    Let's see how this works with a simplest example --- downloading a table called nucleus_detection_v0. First, we reference the table as a function and then we run the query --- this is exactly the same as client.materialize.query_table('nucleus_detection_v0').

    client = CAVEclient('minnie65_public')\nnuc_df = client.materialize.tables.nucleus_detection_v0().query()\n

    Where things differ is when we add filters. If we want to query based on a set of values for the field \"id\", for example, we add that as an argument:

    my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(id=my_ids).query()\n

    Where in this example the id= queries the column id based on the schema. These values can be either individual elements (i.e. an integer or a string) or a list/array of elements, and any field can be used. The tooling will automatically sort out how to format the filtering appropriately when running the query. Importantly, the filtering is identical between querying all types of tables and queries. To see the complete list of fields that can be queried, you can tab-autocomplete or in Jupyter or IPython glance at the docstring with client.materialize.tables.nucleus_detection_v0?.

    In addition to filtering by one or many values, you can also do spatial queries. To find only annotations within a particular bounding box, you need to specify the spatial point (e.g. pt_position). For each such spatial point, there will be an argument {spatial_point_position}_bbox (e.g. pt_position_bbox) that accepts a 2x3 list of coordinates specifying the upper and lower bounds for the query. For example, to find all nucleus centroids within a particular bounding box, it would be:

    bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n    pt_position_bbox=bounding_box\n).query()\n

    If you are not using any filters, you can omit the parenthesis and use the get_all or get_all_live functions directly, which act similarly to the query and live_query functions respectively. The first example could be rewritten as:

    nuc_df = client.materialize.tables.nucleus_detection_v0.get_all()\n

    If you want to list all available fields, you can use the .fields attribute. Similarly, you can get all numeric fields with the .numeric_fields attribute and all spatial fields (allowing bounding box queries) with .spatial_fields.

    nuc_df = client.materialize.tables.nucleus_detection_v0.spatial_fields\n

    If you need to specify the table programmatically, you can also use a dictionary-style approach to getting the table filtering function. For example, an equivalent version of the above line would be:

    my_ids = [373879, 111162]\nmy_table = 'nucleus_detection_v0'\nnuc_df = client.materialize.tables[my_table](id=my_ids).query()\n

    The query function can also take arguments relating to timestamps or formatting where they act just like in the other query method. In particular, the arguments that apply to query are: select_columns, offset, limit, split_positions, materialization_version, timestamp, metadata, desired_resolution, and get_counts. For example, to add a desired resolution and split positions in the above query, it would look like:

    my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n    id=my_ids\n).query(\n    split_positions=True,\n    desired_resolution=[1,1,1],\n)\n

    Inequalities can also be used in filtering numeric columns. Here, you can pass a dictionary instead of a list of values, with the keys being inequality operators (\">\", \">=\", \"<\", and \"<=\") and the values being the comparison. For example, to query for all nuclei with a volume greater than 1000:

    client.materialize.tables.nucleus_detection_v0(\n    volume={\">\": 1000}\n).query()\n

    You can also use multiple inequalities in the same dictionary to filter within a range. For example, to query for all nuclei with a volume between 500 and 750:

    client.materialize.tables.nucleus_detection_v0(\n    volume={\">\": 500, \"<\": 750}\n).query()\n

    If you want to do a live query instead of a materialized query, the filtering remains identical but we use the live_query function instead. The one required argument for live_query is the timestamp.

    my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n    id=my_ids\n).live_query(\n    timestamp=datetime.datetime.now(datetime.timezone.utc),\n)\n

    The live query functions have similar but slightly different arguments: timestamp (required), offset, limit, split_positions, metadata, desired_resolution, and allow_missing_lookups.

    Note that way that IPython handles docstrings means that while you can use ? to get the docstring of the filtering part of the function, you can't simply do something like client.materialize.tables.nucleus_detection_v0().query?. It will tell you the function can't be found, because technically the query function does not yet exist until the table filtering function is called.

    Instead, if you want to glimpse the docstring of the query or live_query functions, you need to split it into two lines:

    qry_func = client.materialize.tables.nucleus_detection_v0().query\nqry_func?\n

    Finally, if the project you are working with has views, a similar interface is available to them via client.materialize.views. Currently views are not compatible with live query, and so only the .query function is available.

    "},{"location":"tutorials/schemas/","title":"EM Annotation Schemas","text":"

    The EMAnnotationSchemas client lets one look up the available schemas and how they are defined. This is mostly used for programmatic interactions between services, but can be useful when looking up schema definitions for new tables.

    "},{"location":"tutorials/schemas/#get-the-list-of-schema","title":"Get the list of schema","text":"

    One can get the list of all available schema with the schema method. Currently, new schema have to be generated on the server side, although we aim to have a generic set available to use.

    client.schema.get_schemas()\n
    "},{"location":"tutorials/schemas/#view-a-specific-schema","title":"View a specific schema","text":"

    The details of each schema can be viewed with the schema_definition method, formatted as per JSONSchema.

    example_schema = client.schema.schema_definition('microns_func_coreg')\nexample_schema\n

    This is mostly useful for programmatic interaction between services at the moment, but can also be used to inspect the expected form of an annotation by digging into the format.

    example_schema['definitions']['FunctionalCoregistration']\n
    "},{"location":"tutorials/skeletonization/","title":"Skeletonization","text":"

    The skeleton service enables you to generate and retrieve skeletons from the server. This saves you the trouble of performing skeletonization routines on your local machine, while also saving you the trouble of understanding many of the details inside of skeletonization. Furthermore, the server offers the ability to generate multiple skeletons simultaneously, in parallel. And lastly, the server caches each skeleton that is generated so that it only ever needs to be generated once. All subsequent requests for the same skeleton will simply return the cached skeleton with notable turn-around-time improvements.

    "},{"location":"tutorials/skeletonization/#initializing-the-client","title":"Initializing the client","text":"

    The simplest way to initialize the CAVEclient is by merely providing the datastack of interest:

    import caveclient as cc\n\nclient = cc.CAVEclient(<datastack_name>)\n

    With a CAVEclient built, you can now investigate the current build version of the SkeletonClient:

    client.skeleton.get_version()\n

    And you can see a list of available skeleton versions. In most cases you will want to use the highest (most recent) version provided.

    client.skeleton.get_versions()\n
    "},{"location":"tutorials/skeletonization/#retrieving-a-skeleton","title":"Retrieving a skeleton","text":"

    Retrieve a skeleton using get_skeleton(). The simplest usage is:

    sk = client.skeleton.get_skeleton(\n    <root_id>,\n    output_format=<output_format>,\n)\n

    where the availale output_formats (described below) are:

    • 'dict' (default if unspecified)
    • 'swc' (a Pandas Dataframe)

    If the skeleton doesn't exist in the server cache, it may take 20-60 seconds to generate the skeleton before it is returned. This function will block during that time. Any subsequent retrieval of the same skeleton should go very quickly however.

    To specify a nondefault skeleton version:

    sk = client.skeleton.get_skeleton(\n    <root_id>,\n    skeleton_version=<sk_version>,\n    output_format=<output_format>,\n)\n

    To specify a specific datastack:

    sk = client.skeleton.get_skeleton(\n    <root_id>,\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n    output_format=<output_format>,\n)\n
    "},{"location":"tutorials/skeletonization/#peering-into-the-contents-of-the-cache","title":"Peering into the contents of the cache","text":"

    Most end-users shouldn't need to use the following function very much, but to see the contents of the cache for a given root id, set of root ids, root id prefix, or set of prefixes:

    get_cache_contents(\n    root_id_prefixes=<root_id>,\n)\n

    You can also add additional parameters as needed:

    get_cache_contents(\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n    root_id_prefixes=<root_id>,\n)\n

    The primary parameter, root_id_prefixes, can be a list of root ids:

    get_cache_contents(\n    root_id_prefixes=[<root_id>, <root_id>, ...],\n)\n

    The primary parameter can also be a root id prefix, which will match any associated root ids. Since this could potentially return a large list of results, there is an optional limit parameter so you don't overwhelm the memory of your processing environment, e.g., a Jupyter notebook or some other Python script running on your local machine:

    get_cache_contents(\n    root_id_prefixes=<root_id_prefix>,\n    limit=<limit>,\n)\n

    Note that the limit only constraints the size of the return value. The internal operation of the function will still receive the full list when it passes the prefix on to CloudFiles. Consequently, calling this function for a short prefix may block for a long time.

    And of course you can also pass in a list of prefixes (or a mixture of full ids and partial prefixes):

    get_cache_contents(\n    root_id_prefixes=[<root_id_prefix>, <root_id_prefix>, ...],\n    limit=<limit>,\n)\n
    "},{"location":"tutorials/skeletonization/#querying-the-presence-of-a-skeleton-in-the-cache","title":"Querying the presence of a skeleton in the cache","text":"

    The function shown above isn't necessarily the most direct way to simply inquire whether a skeleton exists in the cache for a given root id. For that purpose, the following function is better suited:

    skeletons_exist(\n    root_ids=<root_id>,\n)\n

    Or:

    skeletons_exist(\n    root_ids=[<root_id>, <root_id>, ...],\n)\n

    Note that this function doesn't accept prefixes, as supported by cache_query_contents(). Only full root ides are supported. When querying with as single root id, the return value will be a boolean. When querying with a list of ids, the return value will be a Python dictionary mapping from each id to a boolean.

    This function also takes the same optional parameters described above:

    skeletons_exist(\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n    root_ids=<root_id>,  # Or [<root_id>, <root_id>, ...],\n)\n
    "},{"location":"tutorials/skeletonization/#retrieving-multiple-skeletons","title":"Retrieving multiple skeletons","text":"

    You can retrieve a large set of skeletons in a single function call:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n)\n

    If any skeletons are not generated yet, the default behavior is to skip those root ids and only return skeletons that are already available. But you can override this default behavior:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    generate_missing_skeletons=[False|True],\n)\n

    Any root ids for which skeletonization is required will be generated one at a time, at a cost of 20-60 seconds each. Consequently, there is a hard-coded limit of 10, after which all subsequent missing skeletons will not be returned.

    By default, skeletons are returned in JSON format. However SWC is also supported, thusly:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    output_format=<\"json\"|\"swc\">\n)\n

    And the usual defaults can be overridden again:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n)\n
    "},{"location":"tutorials/skeletonization/#generating-multiple-skeletons-in-parallel","title":"Generating multiple skeletons in parallel","text":"

    get_bulk_skeletons() is not an effective way to produce a large number of skeletons since it operates synchronously, generating one skeleton at a time. In order to generate a large number of skeletons it is better to do so in parallel. The following function dispatches numerous root ids for skeletonization without returning anything immediately. The root ids are then distributed on the server for parallel skeletonization and eventual caching. Once they are in the cache, you can retrieve them. Of course, it can be tricky to know when they are available. That is addressed further below. Here's how to dispatch asynchronous bulk skeletonization:

    generate_bulk_skeletons_async(\n    root_ids=[<root_id>, <root_id>, ...],\n)\n

    And with the usual overrides:

    get_bulk_skeletons(\n    root_ids=[<root_id>, <root_id>, ...],\n    datastack_name=<datastack_name>,\n    skeleton_version=<sk_version>,\n)\n
    "},{"location":"tutorials/skeletonization/#retrieving-asynchronously-generated-skeletons","title":"Retrieving asynchronously generated skeletons","text":"

    In order to retrieve asynchronously generated skeletons, it is necessary to poll the cache for the availability of the skeletons and then eventually retrieve them. Here's an example of such a workflow:

    # Dispatch multiple asynchronous, parallel skeletonization and caching processes\ngenerate_bulk_skeletons_async(root_ids)\n\n# Repeatedly query the cache for the existence of the skeletons until they are all available\nwhile True:\n    skeletons_that_exist = client.skeleton.skeletons_exist(root_ids=rids)\n    num_skeletons_found = sum(skeletons_that_exist.values())\n    if num_skeletons_found == len(rids):\n        break\n    sleep(10)  # Pause for ten seconds and check again\n\n# Retrieve the skeletons (remember, SWC is also offered)\nskeletons_json = get_bulk_skeletons(root_ids)\n
    "},{"location":"tutorials/state/","title":"JSON Neuroglancer State Service","text":"

    We store the JSON description of a Neuroglancer state in a simple database at the JSON Service. This is a convenient way to build states to distribute to people, or pull states to parse work by individuals. The JSON Client is at client.state

    client.state\n
    "},{"location":"tutorials/state/#retrieving-a-state","title":"Retrieving a state","text":"

    JSON states are found simply by their ID, which you get when uploading a state. You can download a state with get_state_json.

    example_id = 4845531975188480\nexample_state = client.state.get_state_json(test_id)\nexample_state['layers'][0]\n
    "},{"location":"tutorials/state/#uploading-a-state","title":"Uploading a state","text":"

    You can also upload states with upload_state_json. If you do this, the state id is returned by the function. Note that there is no easy way to query what you uploaded later, so be VERY CAREFUL with this state id if you wish to see it again.

    Note: If you are working with a Neuroglancer Viewer object or similar, in order to upload, use viewer.state.to_json() to generate this representation.

    example_state['layers'][0]['name'] = 'example_name'\nnew_id = client.state.upload_state_json(example_state)\n
    test_state = client.state.get_state_json(new_id)\ntest_state['layers'][0]['name']\n
    "},{"location":"tutorials/state/#generating-a-neuroglancer-url","title":"Generating a Neuroglancer URL","text":"

    Once you have a state ID, you want to turn it into a well-formatted link. So you don\\'t have to remember all the endpoints, we can do this from the state client.

    ngl_base = 'neuromancer-seung-import.appspot.com'\nclient.state.build_neuroglancer_url(new_id, ngl_base)\n

    Note that the neuroglancer base can be found in the info service under client.info.viewer_site().

    "}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 9f6c6ec9..6f20efcf 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,126 +2,126 @@ https://caveconnectome.github.io/CAVEclient/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/changelog/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/contributing/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/glossary/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/installation/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/annotation/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/auth/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/chunkedgraph/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/client/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/config/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/datastack_lookup/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/info/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/l2cache/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/materialize/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/schema/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/skeleton/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/state/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/api/testing/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/advanced/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/annotation/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/authentication/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/chunkedgraph/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/framework/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/info/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/l2cache/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/materialization/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/schemas/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/skeletonization/ - 2024-12-12 + 2024-12-13 https://caveconnectome.github.io/CAVEclient/tutorials/state/ - 2024-12-12 + 2024-12-13 \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 8bbe121b..1b34d5fb 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ