How is input cue factored in by MARBLE?

[stedeny]

Hello,

We are trying to use MARBLE for a monkey reach task, but we don't understand how the reach direction cue should be encoded/provided to the model. Especially, is a single dynamical system fit to all conditions simultaneously? If that is the case, what is to be expected if the neural recordings start before the reach direction cue is provided to the monkey (resting state activity). How does the dynamical system differentiate between the reach directions in that situation? How are reach directions factored in the model?

Thank you for your help!

[agosztolai]

Hi,

I understand you want to use the 'reach direction cue' to supervise the model. Is that correct?

In general, you may proceed in two ways

1. Unsupervised way: Mimic what we do in the monkey reaching case study. Split your training data (neural firing rates) into conditions and feed them as two data lists into MARBLE.construct_dataset(anchor=pos, vector=x). Here, you expect the model to take your discrete training conditions and infer a continuous manifold in latent space from them. You may want to set emb_norm=True to force a spherical layout, but, in general, it is not needed.

2. Supervised way: You supervise the model with the 'reach direction cue'. If the cue is a spatial direction (x,y) or (x,y,z), you could append this to your neural data. For example, if x is an nxd matrix, you would extend it to be an nx(d+3) matrix. Make sure you normalise appropriately or the model will likely ignore the neural or behavioural data.

In our case, we could model a brief resting state activity, but only at higher output dimensions (out_channels). It might be a good idea to play with out_channels: a lower value will give you better visual interpretability, but lower decodability (especially for resting state activity). If the model is trained well, in the resting state the model will not differentiate between reach directions (unless you have some kind of preparatory activity).

Let me know if you have questions.

[stedeny]

Thank you for the quick reply! What I wonder with is, what is the splitting into conditions in solution 1 implying. How will the model use that exactly? Thanks.

[agosztolai]

A condition refers to neural dynamics (i.e., trials) which share input patterns. So I would split your dataset into a set of conditions, e.g., a set of reach directions or a set of other task outcomes would work, where you know that inputs are consistent. You do not have to know how these conditions are related. You also do not need to inject knowledge about where the data in each condition ends up in latent space, but you can (see supervised way point 2 in my previous message). But you need to define several trials for each condition, which are consistent to allow feature extraction. Then at training time, they are jointly embedded and the algorithm figures out how the conditions are related, whether there is some underlying global structure or not, etc.

Happy to clarify further if needed.

[stedeny]

Thank you for your help! But there is still something I misunderstand with the model. I think of your model as a dynamical system, which tries to learn its parameters to emulate another dynamical system, which has generated the data. Why would your dynamical system be fed as an input the data itself at test time, as opposed to simply some initial conditions and inputs. And how can your model provide different trajectories for trials of the same condition? Shouldn't the condition (ie. input to the dynamical system + initial condition of the system) fully determine the trajectory?

[stedeny]

(I have the same conceptual issue with LFADS btw, which takes as an input the data itself at test time)

[agosztolai]

MARBLE is not a dynamical system. It's a representation learning method for data sampled from dynamical systems. It is like PCA, but for dynamics over non-linear spaces: you both learn the non-linear space (manifold) and the dynamics. You put in samples from a dynamical system and it spits out compressed, low-d latent states that explain a lot of variance about your high-d dynamics. It has no temporal prediction/generative ability like a dynamical model.

[stedeny]

Thank you, that explains a lot. So i guess we can't use MARBLE for a fixed point analysis of the dynamical system that has generated the data?

[agosztolai]

It depends on what you mean by f.p. analysis. If you want to do classical f.p. detection/stability analysis, then this is not the intended use for MARBLE. If you want to detect bifurcations in a data-driven way, you can represent each parameter condition separately and then look for discontinuity in low-d structures, like we do in the paper.

[stedeny]

In our case we are interested in characterizing the attractors around which the neural trajectories gravitate, for each condition. We want to know if some conditions share the same attractor. We want to know that about the underlying dynamical system that has generated the data.

[agosztolai]

You can try embedding the conditions jointly as I suggested to see if a structure you expect emerges in latent space. You can also try our RVGP method to extrapolate the vector field from your data. Otherwise, you might want to do classical f.p. analysis.

[stedeny]

OK thank you very much for your help! Much appreciated. Will look into this.