bitECS: image + controls movability and object menu visibility

[takahirox]

Reported by Imaginer @Exairnous on Discord

Description

The behaviors of Hubs image component + controls are different between with and without the new loader.

When dropping-in the asset,

• Without the new loader
  • Not movable
  • No object menu on hover + spacebar
• With the new loader
  • Movable
  • Object menu on hover + spacebar

As scene object

• Without the new loader
  • Object menu on hover + spacebar
• With the new loader
  • No object menu on hover + spacebar

Testing asset from Imaginer image_with_controls.zip
And a room for testing scene object https://hubs.mozilla.com/qm8bAh6/bleak-loyal-meetup (Left with controls, Right without controls)

To Reproduce
Steps to reproduce the behavior:

For drop-in object

1. Enter a room with the new loader enabled
2. Upload the example asset
3. Try to move the object and confirm it moves
4. Hover the cursor on the object, hold the space bar, and confirm that object menu appears
5. Try the steps above without the new loader enabled, and confirm that the behaviors are different

For scene object

1. Enter https://hubs.mozilla.com/qm8bAh6/bleak-loyal-meetup with the new loader enabled
2. Hover the cursor on the left object and press the space bar
3. Confirm that no object menu appears
4. Try the steps above without the new loader enabled, and confirm that object menu appears

Expected behavior

Ideally the behaviors are the same between with and without the new loader

Or I first want to think of the correct behavior... I guess object menu should be shown on image + controls object? If so, with the new loader the behavior for drop-in seems correct while it doesn't seem correct for scene object.

Hardware

• Device: Desktop
• OS: Windows
• Browser: Chrome

[Exairnous]

with the new loader the behavior for drop-in seems correct while it doesn't seem correct for scene object.

Agreed.

[Exairnous]

Also note that when hovering over the scene object the cursor doesn't turn blue in the new loader like it does in the old loader.

[keianhzo]

Thanks for the feedback @Exairnous

New Loader

https://hubs.mozilla.com/idbvNmZ/loud-grounded-nation/

Scene element

Object	Grabbable	Can Be Pinned	Object Menu	Link Menu
Image + Control	N/A	N/A	N/A	NO (BUG, we should show it)
Image + No Control	N/A	N/A	N/A	NO
Image + Control + Link	N/A	N/A	N/A	YES
Image + No Control + Link	N/A	N/A	N/A	YES (link overrides no controls)

Drop-in glTF

Object	Grabbable	Can Be Pinned	Object Menu	Link Menu
Image + Control	YES	YES	NO (BUG, we should show it)
Image + No Control	YES	YES	NO
Image + Control + Link	NO	YES	YES
Image + No Control + Link	NO	YES	YES (link overrides no controls)

Old Loader

https://hubs.mozilla.com/oLeTtP4/wellmade-quirky-huddle

Scene element

Object	Grabbable	Can Be Pinned	Object Menu	Link Menu
Image + Control	N/A	N/A	Reduced	NO
Image + No Control	N/A	N/A	N/A	NO
Image + Control + Link	N/A	N/A	N/A	YES
Image + No Control + Link	N/A	N/A	N/A	NO

Drop-in glTF

Object	Grabbable	Can Be Pinned	Object Menu	Link Menu
Image + Control	YES	NO	Only through Object side panel	Reduced
Image + No Control	YES	YES	YES	NO
Image + Control + Link	NO	Only through Object side panel	Reduced	YES
Image + No Control + Link	NO	NO	YES	NO

Based on the table above I feel that the new loader behavior is more consistent.

• If either controls is enabled or the object has an additional linkcomponent attached. We show the link menu. From the component docs in Blender:

So not showing the link menu for Image + Control int he old loader seems not correct to me.

• The link component overrides the image control link in the link menu.
• Dropped-in objects can always be pinned from the object menu.
• We always show the full object menu for all image and image+link objects. In the old loader we show a reduced version in some cases which is not very consistent.
• If you add a link object to an image object we show the link menu for the link component link. In the old loader you can't access that link.
• Dropped-in objects int he old loader are randomly grabbable. New loader has a bug that doesn't let the object to be grabbed when the link menu is showing and it show oh hover so they can never be hovered. This has already been mentioned and we should file a bug to fix for both the object menu and the link menu.

The only actionables that I see here are:

• We need to fix the bug where Image + Control doesn't show the link menu in the new loader. Bitecs-image-controls-link-menu #6392
• We need to fix the menus so you can always grab objects unless you are hovering the buttons, This should fix it for link menus: Look for the holdable media root when grabbing an object #6393 We still need to find a solution for object-menus.

Does that make sense or am I missing things?

[Exairnous]

You're welcome.

My current testing in Chromium doesn't seem to completely agree with your table. Although it's a bit hard right now because the objects in your test scene (which are very good) seem to keep locking the interaction capabilities of my mouse, e.g. I will suddenly no longer be able to grab things, hit buttons turn my view, but Hubs isn't frozen and I can turn my view with the right mouse button (or sometimes it focuses on the last touched object whenever I right click, no matter where my cursor is). This is happening in both loaders, and focusing on an object also triggers it sometimes.

In the old loader I get this error message (but I don't think it happens in the new loader):

TypeError: Cannot read properties of null (reading 'charAt')
    at media-utils.js:608:17
    at Generator.next (<anonymous>)
    at Nc (hub-a72d9512fa6e153c06a4.js:1:243323)
    at s (hub-a72d9512fa6e153c06a4.js:1:243527)
    at hub-a72d9512fa6e153c06a4.js:1:243588
    at new Promise (<anonymous>)
    at hub-a72d9512fa6e153c06a4.js:1:243467
    at ud (media-utils.js:668:2)
    at media-utils.js:606:39
    at media-loading.ts:267:28

But back to the disagreements between my latest tests and your table

First off, what do you mean by "Link Menu"? Do you mean the hover button or the vertical menu containing: refresh, open link (opens the image link as opposed to downloading the glb), magnifying glass (creates a copy that follows you around), reticule (focuses on the object, iirc)?
For scene elements:
New loader: the vertical menu doesn't show at all (bug), the hover button shows on both of the link ones (correct)
Old loader: the vertical menu shows on ones with controls (correct), the hover button shows on link with controls (correct), but not on the link without controls (bug)

Secondly, what do you mean by "Object Menu"? Do you mean the spacebar menu or the menu box at the bottom center of your screen when you focus on something?
For in-room objects:
New loader: the spacebar menu seems to come up on all your pinned objects. The central menu box also seems to appear for all the objects, but checking through the Objects panel I see one image and one object for each item and the images don't seem to inherit the correct pin/delete state.
Old loader: the spacebar menu seems to come up on all your pinned objects, but the link menu also shows up on the images with controls when you hover over the image directly (as opposed to the text). The central box appears for all objects and there is only one object in the object list for each pinned image glTF. The old loader seems more correct in general here but it gets more complicated if you only have an image in a dropped in glTF as opposed to an image plus something else, such as your text here.

Aside from that, I think the dropped in glTFs should always be grabbable (which I think they are in the new loader, but I can't test right now).

This is one of the more complex systems in Hubs, so it's hard to figure out the intricacies and I'm not sure I'm doing a good job of it here. Would it be possible to schedule a meeting to go over this in Hubs directly (possibly invite Jim too, not sure)?

As for the PRs, it seems from your description like they make sense and are in line with what I've written here. If they're only going to affect the new loader that's fine, but if they would affect the old loader as well then it might be good to wait a little bit.

[keianhzo]

My current testing in Chromium doesn't seem to completely agree with your table. Although it's a bit hard right now because the objects in your test scene (which are very good) seem to keep locking the interaction capabilities of my mouse, e.g. I will suddenly no longer be able to grab things, hit buttons turn my view, but Hubs isn't frozen and I can turn my view with the right mouse button (or sometimes it focuses on the last touched object whenever I right click, no matter where my cursor is). This is happening in both loaders, and focusing on an object also triggers it sometimes.

Yes, this was due to a bug that I've fixed here: #6393 Should be fixed in production now.

First off, what do you mean by "Link Menu"? Do you mean the hover button or the vertical menu containing: refresh, open link (opens the image link as opposed to downloading the glb), magnifying glass (creates a copy that follows you around), reticule (focuses on the object, iirc)?

I mean the hover button that lets you open the link, either provided by the image src iteself or by the link component in case the object has also a link component.

Secondly, what do you mean by "Object Menu"? Do you mean the spacebar menu or the menu box at the bottom center of your screen when you focus on something?

I mean the object menu when you click either tab or spacebar.

New loader: the spacebar menu seems to come up on all your pinned objects. The central menu box also seems to appear for all the objects, but checking through the Objects panel I see one image and one object for each item and the images don't seem to inherit the correct pin/delete state.

Yeah this is a different bug, I've opened an issue for it: #6407

Aside from that, I think the dropped in glTFs should always be grabbable (which I think they are in the new loader, but I can't test right now).

Yes, this should be now the case.

As for the PRs, it seems from your description like they make sense and are in line with what I've written here. If they're only going to affect the new loader that's fine, but if they would affect the old loader as well then it might be good to wait a little bit.

This PRs should only affect the new loader code path.

This is one of the more complex systems in Hubs, so it's hard to figure out the intricacies and I'm not sure I'm doing a good job of it here. Would it be possible to schedule a meeting to go over this in Hubs directly (possibly invite Jim too, not sure)?

I agree and one of the goals is to simplify and add some consistency to the menu system even if we diverge from the original code as I don't this (looking at this table) that is very consistent. And yes, I think it would be a good idea to gather more feedback to simplify all this menus.