support HTML object/anchor/portal

[ShaMan123]

Overview

This is a tracking discussion for implementing a way to handle HTML with fabric.
Many devs want this feature, myself included.
We will try to define a spec of what this object does. It's purpose is to make the HTML element look and feel as if it is part of the canvas.
There are simple use cases that the best practice towards solving them would be using a Control as an anchor to the HTML object, e.g. dropdown toolbar.
However there are cases where this is insufficient:

• a need for the HTML element to have controls like all fabric objects
• a need for the object to be visible on canvas at all times

Spec

• respect object stacking
  • using clip-path css to clip the HTML element with all the objects on top of in the object stack produced a promising POC
• interactivity:
  • selection
  • controls
  • events
• deep customization
  • insert elements to the document between lower and upper canvases it will give the dev some freedom of customizing visuals if they use the top context to render what they want over the HTMLObject

Workarounds

• render html to an image and render that on canvas. This, no doubt, comes with disadvantages.
• use the Control API as mentioned above

[asturur]

The workarounds you describe would be the solution as today.
Which are the disadvantages?

I understand that making an html element pixles, freezes it, so no hover effects, no click feedback, nothing.
Outside the controls that i fully understand the need for, and are definitely an easy use case to demostrate, for the cases of an object being actually an html element, can you elaborate on the use case? ( you spoke about one briefly before flesh it out )

[melchiar]

Just to preempt what will likely be some common questions with this functionality, HTMLObjects will not have all of the same capabilities as current fabric objects. Since this new class would only be anchored to the DOM and not drawn to the canvas itself, a few notable limitations come to mind.

No Rasterization
While it is technically possible to convert basic html to an image using a library such as html-to-image, it's not possible to capture content from an iframe such as an embedded youtube video. I think we can add some customizability that would allow the dev to specify what get's rendered when calling toDataURL on a canvas containing an HTML element (in case you want to specify a placeholder image like a youtube thumbnail or icon), just know that HTMLObjects will not be rasterizable like other fabric objects. Note as well this will also mean no reading color data from specific pixel coordinates, so therefore things like eyedroppers won't work.

Stacking Limitations
HTMLObjects would be drawn above the canvas, and therefore would present some issues if we try to integrate full object stacking with other fabric objects (ie. drawing fabric objects on top of an HTMLObject). While we could possibly look at tricks like layering canvases at each z-depth or clipping HTMLObjects to simulation occlusion, I think the resulting performance issues would outweigh the potential benefits. I feel the most logical solution would be to restrict HTMLObjects to the top, although still allow them to stack with each other within their own stacking order. If serialized, they could be listed in a separate array from the current Fabric JSON structure (eg. {"version":"4.6.0","htmlObjects":[],"objects":[],"background":"white"}).

Which brings me to the final issue...

Serialization XSS Concerns
Allowing HTMLObjects to be serialized adds the risk xss vulnerabilities if malicious code is injected into the htmlObject. I don't really see a good way around this other than disabling serialization for HTMLObjects entirely.

[ShaMan123]

Use case for needing actual HTML

• custom objects - e.g. math editor
• inputs
• iframes as @melchiar pointed out

css clipping POC

You can see the math editor which is HTML and the selected fabric object clipping it while moving

No Rasterization

Note as well this will also mean no reading color data from specific pixel coordinates, so therefore things like eyedroppers won't work.

@melchiar valid point. I don't know how the native eyedroppers work or if we can access them (quite sure we can't). Cause if there is some option we can add logic to handle this from within fabric.

Stacking Limitations

I think that stacking in the essence of the spec. That's what I really need when I try to integrate it into a fabric app. Without it I remain with toolbars and I don't see what the impl does.

Serialization XSS Concerns

I am not knowledgeable in security risks. However, passing HTML elements poses no issues, same as Image receives a HTMLImageElement or video. fabric should not handle any HTML tasks. The maximum it should do is toObject print out element.outerHTML (can be argued against and for passing the element itself) but fromObject will receive an element and not html text. The dev should handle that by themselves.

[melchiar]

@melchiar valid point. I don't know how the native eyedroppers work or if we can access them (quite sure we can't). Cause if there is some option we can add logic to handle this from within fabric.

Eyedroppers call ctx.getImageData to get the color value at specific pixel coordinates. To my knowledge, there's no way to do this in javascript for regular html elements.

I think that stacking in the essence of the spec. That's what I really need when I try to integrate it into a fabric app. Without it I remain with toolbars and I don't see what the impl does.

The clippath method in your POC looks like it works well in a very simple use case, but I think that method will fall apart if the foreground objects use dropshadows, transparency, or a globalCompositeOperation value other than source-over. I don't think there's a perfect way to simulate partial occlusion, though using css mask-image could give us a rough approximation of it since it allows for partial transparency. Even so though, I don't think even masking is a viable solution.

Consider this example.

Here's an mockup showing a YouTube video with a 75% opacity rectangle and text with a 50% opacity shadow above it:

To simulate this in Fabric with an HTMLObject, you'd need to generate a clipmask that looks something like this:

Which would make the HTMLObject look like this:

Bring everything together and you'd get this:

The end result isn't terrible, but it's not great either since the effect causes the partially transparent objects to wash out. Now imagine having multiple foreground objects over a background HTMLObject, and the performance impact of having to generate masks based on every object in the canvas, and refresh those masks in real-time whenever foreground objects are transformed.

I am not knowledgeable in security risks. However, passing HTML elements poses no issues, same as Image receives a HTMLImageElement or video. fabric should not handle any HTML tasks. The maximum it should do is toObject print out element.outerHTML (can be argued against and for passing the element itself) but fromObject will receive an element and not html text. The dev should handle that by themselves.

I'm definitely not strong on security either, though I've always understood that allowing a web application to accept html from the user along with any scripts it may contain is fraught with peril. Perhaps someone more knowledgeable on web security could provide some insight on whether we can do this in a safe way.

[ShaMan123]

I thought of this a day ago and I came up with something interesting.
We can create a renderer for this case that renders the object stack as separate HTML canvas elements with the cache canvas of each object while applying transformations via css. Then it will use the DOM, becoming more HTML5 than a strict canvas app.
In terms of performance I don't see a hit.
In terms of compatibility with current functions... If we keep the top context concept we can keep drawing as is.
Text logic can be dumped in favor of HTMLTextArea, Image, Group, etc. but kept for StaticCanvas.
We could extract the Renderer logic, as suggested by @asturur, to a dedicated class and have this mechanism opt in.
Filtering and clipping can be done via css. Composite operations can be done with Groups and rendered as a single object/canvas.
Controls can be converted to use HTML controls and since they change transform props or things that can be applied to css I don't see a problem. Even sounds better perf.
Object stacking will be easy to customize using z-index...
Am I missing something?
Sounds very flexible to me.
Input was a thing I wanted to add to canvas. It can be done from scratch in fabric but using the DOM is a lot easier and richer and out of the box.
It is a BIG change in concept but not in code IMO.
Thoughts?

[ShaMan123]

This can also solve #7874

[ebigtm]

@ShaMan123 sorry for bumping late to the party, sounds amazing, I need to represent alot of html elements inside the canvas, for now im rendering it to an image but any other solution would be better lol

[ShaMan123]

Yeah, the cache canvas is an image.
So we do it the other way around.
Instead of rendering html to canvas we render canvas to html

[ebigtm]

I might be wrong but from my testing with a similar solution you lose the benefits of snappy zoom and pan and instead you get the laggy html solution for that, im talking of course about incorporating html elements inside

[ShaMan123]

Please elaborate.
I think that going in this direction means using CSS to impl zoom/pan and control actions also