[PLAYBACK] QUESTION: Frame processing

[antithing]

I have a bit of a edge use case, and am looking to add opencv and directx processing to frames as they are played in olive.

I would like to 'intercept' every frame as it plays, and add my own processing to it (convert to cv::Mat and process / convert to directx and send to a third party application)

I am going to assume that the best way to do this is by creating new nodes, so I have two questions:

1. What are the steps to create a new node? I have created the class cpp and h, and added an entry in factory.cpp, but I have hit a compile error.

2. How is the frame data stored in the nodes? eg, if I want to convert to cv::Mat, what am I converting from?

Thank you very much, and happy holidays!

[itsmattkc]

What are the steps to create a new node? I have created the class cpp and h, and added an entry in factory.cpp, but I have hit a compile error.

This is correct, but make sure you're adding to both the enum in factory.h and the switch statement in factory.cpp.

How is the frame data stored in the nodes? eg, if I want to convert to cv::Mat, what am I converting from?

The frames are all processed as OpenGL textures, however there is a lot of abstraction around them because it's expected that Olive will support other backends (Vulkan, etc.) and/or drop OpenGL in the near future. The object will be a Texture object, and you can pull/push data to and from it with Texture::Download and Texture::Upload. This isn't how the nodes usually operate on data (they usually don't operate directly and instead push "shader jobs"), but it should work. I would expect this to be fairly slow, especially if you're pushing it to a DirectX texture and back in the process (any way you can do what you want to do in OpenGL?).

[antithing]

Thank you! I have to remove the Q_OBJECT line from my header for some reason, but now my node compiles.

How can I access the Texture data from inside a node? Do I use :

TexturePtr texture = job.GetValue(kTextureInput).data().value<TexturePtr>();
      if (texture)      
      {
          Texture tex = *texture.get();  
          //download?
      }

How can i get the OpenGL texture from that?

I see a tex.Upload(), but not a Download.

Thanks again!

[itsmattkc]

Thank you! I have to remove the Q_OBJECT line from my header for some reason, but now my node compiles.

The Q_OBJECT macro is supposed to trigger other processing from Qt/CMake, but sometimes it doesn't happen automatically. Try adding it back and then re-running CMake, cleaning the build tree, and rebuilding.

I see a tex.Upload(), but not a Download.

Pull from the repo again, I added Texture::Download recently.

[antithing]

In the master branch, I only see upload..

olive/app/render/texture.cpp

Line 36 in eab3148

void Texture::Upload(void *data, int linesize)

Is this the wrong branch to use?

Thanks again

[itsmattkc]

That's bizarre, I could have sworn I'd written and committed Download two days ago. Oh well, it's there now.

[antithing]

Great thank you! That gives me the frame buffer in *data, right? What format is that in?

Would you be able to give me an idea of how to call this?

Thanks again for your help

[itsmattkc]

The easiest thing would be to use Olive's "Frame" object. That will ensure a CPU buffer is allocated correctly to download data into into (in Olive terminology, a "Frame" is a CPU buffer and a "Texture" is a GPU buffer). This is the code you would use:

FramePtr frame = Frame::Create();
frame->set_video_params(texture->video_params());
frame->allocate();
texture->Download(frame->data(), frame->linesize_pixels());

Then your raw image data will be accessible in frame->data(). The data will always be packed and will be aligned to linesize_pixels()/linesize_bytes(), but to be more specific you'll need to poll frame->format() or texture->format() to determine the pixel format (usually half-float, but will depend on the circumstance and what the user has selected) and frame->channel_count() or texture->channel_count() to determine if it's grayscale, RGB or RGBA.

How you get that into OpenCV I'm not sure, I'm not familiar with that library. If you need to, Frames can be converted to other pixel formats using frame = frame->convert(format); However, this will be slow because it's happening on the CPU, and as I mentioned you should expect all of this to be fairly slow anyway because it involves a CPU/GPU roundtrip (multiple if you're sending the frame to Direct3D too).

[sobotka]

@itsmattkc Any chance to open up Discussions and move issues that are of this sort there for future contributors?

[itsmattkc]

Ah I wasn't aware I had to enable that manually (on my end it shows the tab all the time, but then links to a "getting started" page when I click it). Yeah that would probably be better for things like this

[sobotka]

Labels carry over too! This is excellent. Congratulations on Olive’s first Discussion thread!

[antithing]

Wooo Discussions! Thanks guys :)

I am using your code snippet, and had to make one change to get it to compile, just checking that this is correct:

 FramePtr frame = Frame::Create();
frame->set_video_params(texture->params()); // instead of video_params
frame->allocate();
texture->Download(frame->data(), frame->linesize_pixels());    

Additionally, when I am using a 1920x1080 video, and i print the width and height in my node:

std::cout << frame->height() << " " << frame->width() << std::endl;

I get 360 640.

Why would this be? Is there a resize happening automatically somewhere?

And one last thing, if I stick with GL, how can I retrieve a GLuint / frame buffer?

I am looking at integrating a function that requires these inputs :

  GLuint TextureID,
  GLuint TextureTarget,
unsigned int width,
unsigned int height
GLuint HostFBO

Thanks for your patience, and happy new year!