Working Document for Websockets protocol and ClientDataStore

ClientDataStore operations

Stores actual columns of data on the client. Used by lightweight "data source" models. Supports performing the following CRUD operations. ClientDataStore objects are not part of any Bokeh Document, their loading/updating is handled as a separate concern. The motivation for this is to allow Bokeh Documents to only contain lightweight models, so that loading/saving the document is a very fast operation, and so that reloading or updating the data only happens when it actually needs to happen.

• create new columns

• replace all columns

• replace selected columns

• append to all columns

• append to all columns, roll buffer

• patch arbitrary indices

• delete columns

The ClientDataStore object will present a JS API for these operations. Separately, a Websocket protocol will support performing these operations from a Bokeh server.

Bokeh._data_store

Index of to ClientDataStore, does not live in any Bokeh Document.

ClientDataSource

A lightweight data source object that lives in a Bokeh Document. Is configured with, or can reference a ClientDataStore in Bokeh._data_store

Array Header

Currently to include a NumPy array in a data source, it is: converted to python list, which is converted to JSON, this is sent to the server over HTTP and then stored in redis as JSON. The data is later extracted from redis, and sent to the client, which turns it back in to a (non-typed) array. By using Websocket binary messages and sending light array metadata, many above the above steps can be simplified or eliminated. Redis, e.g. can store the binary data directly, and on the client it can be immediately viewed as a typed array.

{
    type: <type>   # float32, float64, int32, int64, int8, uint8
    shape: [<int32>, <int32>, <int32>]  # nulls for missing dims
    size: <int>    # array size in bytes
}

Need to define endiannes somewhere. Propose fixing this on the server.

WebSocket Protocol

<int32>     # header size, N bytes

<N bytes>   # utf-8 header, interpreted as JSON

[<payload>] # optional binary payload described by header