Shift in plotted mesh in flywire

[YijieYin]

The meshes plotted by gal4line2flywire script sometimes have shifts anteriorly.


All the code comes from the gal4line2flywire example, except: ALad1 (VFB_00020032) may be a good example as you can see that some of the plotted mesh is outside the brain mesh. Here's a flywire link with the brain mesh. You could use this as the shorturl.
However this shift doesn't always happen, and I'm not sure what triggers it. Please let me know if you'd like me to paste all the code I ran here.

[YijieYin]

This DM5 mesh has been shifting for me, having tried many times. Here's all the code I ran:

import navis
import fafbseg
import flybrains

import os
import numpy as np
import pandas as pd
import copy

from pyroglancer.localserver import startdataserver, closedataserver
from pyroglancer.flywire import flywireurl2dict, add_flywirelayer
from pyroglancer.utils import pointcloud2meshes

import trimesh as tm
import open3d as o3d 


from vfb_connect.cross_server_tools import VfbConnect
vc = VfbConnect()
vfb_id = 'VFB_00020020' 
# 'VFB_00020048': DM6
# 'VFB_00020085': DM4
# VFB_00020010: SMPpv1
# VFB_00020051: SLPal3
# VFB_00020052: SLPal4
# VFB_00020059: SLPal5
# VFB_00020031: PSp3
# VFB_00020071: VLPp2
# VFB_00020203: DM5 (cleaner) 
# VFB_00020020: DM5 (messy)
# VFB_00020023: SLPpl2
# VFB_00020008 ALl1
vcinfo = vc.neo_query_wrapper.get_anatomical_individual_TermInfo([vfb_id])
image_folder = vcinfo[0]['channel_image'][0]['image']['image_folder'] + 'volume.obj'
image_folder

vfb_mesh = tm.load_remote(image_folder)
vfb_mesh 
mesh_files = '/Users/yijieyin/Documents/Python/Pyroglancer/gal4_lines/' + vfb_id + '.ply'
mesh_files
tempval = vfb_mesh.export(mesh_files)

pcd = o3d.io.read_point_cloud(mesh_files)
pcd.estimate_normals()

# estimate radius for rolling ball
distances = pcd.compute_nearest_neighbor_distance()
avg_dist = np.mean(distances)
radius = 3 * avg_dist

# This will take some time. This is generating a point cloud from the nrrd file 
mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_ball_pivoting(pcd,
           o3d.utility.DoubleVector([radius, radius * 2]))

# create the triangular mesh with the vertices and faces from open3d
tri_mesh = tm.Trimesh(np.asarray(mesh.vertices), np.asarray(mesh.triangles),
                          vertex_normals=np.asarray(mesh.vertex_normals))

# Checks if the mesh is broken 
tm.convex.is_convex(tri_mesh)

vfb_mesh = navis.Volume(tri_mesh) 


jrc2018u_mesh = navis.Volume(flybrains.JRC2018U.mesh)
vfb_mesh.color = 'blue'
vfb_mesh
fig, ax = navis.plot2d([vfb_mesh,jrc2018u_mesh], method='2d')
fig.gca().invert_yaxis()
vfbmesh_xyz_02 = pd.DataFrame(list(map(np.ravel, vfb_mesh.vertices)), columns=['x', 'y', 'z'])
vfb2flywiremesh = copy.deepcopy(vfb_mesh)
# Transform from Janelia Research Campus 2018 Unisex into Flywire space
vfb2flywiremesh_xyz = navis.xform_brain(vfbmesh_xyz_02[['x','y','z']],source='JRC2018U', target='FLYWIRE')
vfb2flywiremesh.vertices = vfb2flywiremesh_xyz
vfb2flywiremesh 


# You should always close this even if you shut down the terminal 
startdataserver()
# An empty url containing the brain mesh 
shorturl = 'https://ngl.flywire.ai/?json_url=https://globalv1.flywire-daf.com/nglstate/4757698205712384' 


# just a random number to show in FlyWire
vfb2flywiremesh.id = 1
# Note the type here is volumes 
# The link this cell gives is the one that has the pre-computed mesh. 
shorturl = add_flywirelayer(flywireurl2dict(shorturl), layer_kws = {'type': 'volumes','name': 'DM5',
                                                        'source': vfb2flywiremesh,'color': 'white'})


# Make sure you run this at the end: 
closedataserver()

[SridharJagannathan]

Hi @YijieYin :
I had a detailed look at this issue:
While adding this mesh using pyroglancer produced the following figure in 3d:

If you carefully look the mesh is exceeding the neuropil boundaries at regions near medulla and anterior ventrolateral protocerebrum
Now if you go back to the vfb and look at the actual point cloud mesh, you can see below:
vfb link

you can see that the point cloud mesh (green) was not registered properly in regions near medulla (red) and anterior ventrolateral protocerebrum (red).
This is the primary issue for the mesh being not transformed properly.
Also if the points are closer to the neuropil boundaries, they would be a bit squished, you can see I have noted this problem before and have raised an issue a while ago here:
navis-org/navis-flybrains#2

[SridharJagannathan]

So in conclusion, I would do the following to fix this issue:

1. Perform re-registration of the point cloud mesh DM5 clone again, and check if it fits well within the JRC2018U template. This could be done by the VFB people.
2. Then try the pyroglancer again.
3. If steps 1,2 doesn't fix this, then ask for a solution to the registration at neuropil boundary issue with The registration from hemibrain to FAFB is squished navis-org/navis-flybrains#2

[SridharJagannathan]

Re-referenced the issue to here: VirtualFlyBrain/VFB2#310
The main problem seems to be the registration issue.

[SridharJagannathan]

Further investigation reveals that some of these clones are actually registered to JFRC2 and not JFRC2018U template.
@YijieYin: It seems the prediction of @jefferis is right, in that these clones are indeed belong to the JFRC2 space.
For example, I downloaded VFB_00020020 from the virtualflybrain website and simply plotted them in the the JFRC2 neuropil. See below:

Further, when I transform them to the flywire space and inject them using pyroglancer, they are well within the mesh:

i was quickly able to find a match like below after some probing:

So the part of the code that you need to change to achieve this transformation would be:

vfb2flywiremeshes['VFB_00020020'] = navis.xform_brain(vfbmeshes_dict['VFB_00020020'],source='JFRC2', target='JFRC2010')
vfb2flywiremeshes['VFB_00020020'] = navis.xform_brain(vfb2flywiremeshes['VFB_00020020'],source='JFRC2010', target='FLYWIRE')

The main issue seems to be that virtualflybrain mentions that the clones are aligned to JRC2018Unisex space in reality it seems they are aligned to JFRC2. So I think virtualflybrain needs to cross check and correct these references.

@Robbie1977: FYI