Skip to content

Commit

Permalink
Python docs done
Browse files Browse the repository at this point in the history
  • Loading branch information
jonahmvp authored and sfeiwong committed Sep 30, 2024
1 parent ae703ee commit 14a4999
Showing 1 changed file with 62 additions and 48 deletions.
110 changes: 62 additions & 48 deletions content/en/docs/bmf/multiple_features/data_backend/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ See example code snippet:
##### AVFrame to VideoFrame
1. use `private_attach` set AVFrame as private_data of VideoFrame
2. src_dp set media_type with value MeidaType::kAVFrame
2. src_dp set media_type with value MediaType::kAVFrame
3. use `bmf_convert` to convert
4. get the return VideoFrame
Expand All @@ -110,33 +110,38 @@ See example code snippet:
```

### Python Interface
The `bmf_convert` functionality is provided in Python via pybind11 bindings to the C++ implementation. Usage is much the same as the C++ snippets above. Some legacy data conversion examples are included below where the conversion is not currently supported by `bmf_convert`.
Data conversion is available in Python via pybind11 binding to the `bmf_convert` C++ implementation.

```python
from bmf import *
```

Alternatively, import the specific classes/methods:
```python
from bmf import MediaDesc, MediaType, bmf_convert
```

#### Scale and colorspace conversion
See the following example of using `bmf_convert` to convert a video frame from RGB to NV12 and scale the image to half-width and half-height:
The following snippet shows a colorspace conversion from RGB to NV12 and scale conversion to half-width and half-height:

```python
from bmf import *
import bmf.hml.hmp as mp
from bmf.lib._bmf.sdk import MediaDesc, bmf_convert

# construct a video frame
width = 640
height = 360
RGB = mp.PixelInfo(mp.PixelFormat.kPF_RGB24, mp.ColorSpace.kCS_BT709)
vf = VideoFrame(width, height, pix_info=RGB)

# generate a media description of the converted media
# ... half-width, half-height
# ... NV12 pixel format and BT470BG color space
# generate a media description of the desired converted media
dst_md = MediaDesc().width(width//2).height(height//2)
dst_md.pixel_format(mp.PixelFormat.kPF_NV12).color_space(mp.ColorSpace.kCS_BT470BG)

# do the conversion with `bmf_convert(...)`
# do the conversion
out_vf = bmf_convert(src_vf, MediaDesc(), dst_md)
```

Currently `bmf_convert` does not support rotate, or color depth conversions. However, this functionality can be achieved using the following legacy functions:
Legacy conversion functions are also available in the BMF Python module (rotate currently not supported by `bmf_convert` interface).

`bmf.hml.hmp.img.rgb_to_yuv`

Expand All @@ -148,25 +153,31 @@ Currently `bmf_convert` does not support rotate, or color depth conversions. How

`bmf.hml.hmp.img.rotate`

The following legacy snippet does a color space conversion from RGB to NV12, and allows for specifying the color range:

```python
The following snippet uses the legacy conversion function to achieve the same conversion shown previously:
```python
from bmf import *
import bmf.hml.hmp as mp

NV12 = mp.PixelInfo(mp.PixelFormat.kPF_NV12, mp.ColorSpace.kCS_BT470BG, mp.ColorRange.kCR_MPEG)
# construct a video frame
width = 640
height = 360
RGB = mp.PixelInfo(mp.PixelFormat.kPF_RGB24, mp.ColorSpace.kCS_BT709, mp.ColorRange.kCR_MPEG)
src_vf = pkt.get(VideoFrame)
out_frame = mp.Frame(src_vf.frame().width(), src_vf.frame().height(), NV12, device='cuda')
mp.img.rgb_to_yuv(out_vf.frame().data(), src_vf.frame().plane(0), NV12, mp.kNHWC)
vf = VideoFrame(width, height, pix_info=RGB)

# construct output pixel info and output frame
NV12 = mp.PixelInfo(mp.PixelFormat.kPF_NV12, mp.ColorSpace.kCS_BT470BG, mp.ColorRange.kCR_MPEG)
out_vf = VideoFrame(vf.frame().width(), vf.frame().height(), pix_info=NV12)

# do the conversion
mp.img.rgb_to_yuv(out_vf.frame().data(), vf.frame().plane(0), NV12, mp.kNHWC)
```

#### Device memory transfer
Using `bmf_convert`:
```python
cpu_vf = pk.get(VideoFrame)
dst_md = MediaDesc().device(mp.kCUDA)
dst_md = MediaDesc().device(mp.Device('cuda:0'))
gpu_vf = bmf_convert(cpu_vf, MediaDesc(), dst_md)
#...
```

Or using the legacy approach:
Expand All @@ -188,35 +199,11 @@ Sample code:
```

#### Conversion between VideoFrame and third-party data structure
In python API, those types of third-party data type are supported:
- VideoFrame, which is the general class of video frame in BMF. And `VideoFrame` includes `Frame` as member
The general class of video frame in BMF is `VideoFrame`, and contains `Frame` as a member. The python API supports conversion between `VideoFrame` and the following third-party data types:
- numpy
- torch

The media_type property on the MediaDesc instance is used to identify the data type when doing conversions with `bmf_convert`. The following options are available:

`bmf.lib._bmf.sdk.kBMFVideoFrame` - BMF video frame

`bmf.lib._bmf.sdk.kAVFrame` - ffmpeg frame

`bmf.lib._bmf.sdk.kATTensor` - torch

`bmf.lib._bmf.sdk.kTensor`

`bmf.lib._bmf.sdk.kCVMat` - opencv mat

**NOTE**: conversions must be to-or-from a BMF video frame or a runtime error will be raised.

Convert ffmpeg AVFrame to BMF Video Frame:
```python
from bmf import *
import bmf.hml.hmp as mp
from bmf.lib._bmf.sdk import MediaDesc, bmf_convert, kAVFrame

vf = pkt.get(VideoFrame) # grab input AVFrame
src_md = MediaDesc().media_type(kAVFrame)
out_vf = bmf_convert(vf, src_md, MediaDesc())
```
**NOTE**: conversions must be to-or-from a BMF Video Frame or a runtime error will be raised.

`bmf.hml.hmp.Frame.numpy` converts a BMF Frame to numpy, and Frame can be included in VideoFrame

Expand All @@ -226,9 +213,10 @@ Sample code:
```python
from bmf import *
import numpy as np
np1 = np.array(obj)

npa = np.array(obj)
rgb = mp.PixelInfo(mp.kPF_RGB24)
frame = mp.Frame(mp.from_numpy(npa, rgb)
frame = mp.Frame(mp.from_numpy(npa, rgb))
vf = VideoFrame(frame)
#...video frame process
np0 = vf.frame().plane(0).numpy()
Expand All @@ -246,6 +234,7 @@ Sample code:
```python
from bmf import *
import torch

vf = pkt.get(VideoFrame)
torch_data = vf.frame().data().torch()
#...torch process
Expand All @@ -256,6 +245,7 @@ Sample code:
from bmf import *
import bmf.hml.hmp as mp
import torch

vf = pkt.get(VideoFrame)
rgb = mp.PixelInfo(mp.kPF_RGB24)
torch_vf = torch.from_dlpack(vf.reformat(rgb).frame().plane(0))
Expand All @@ -264,4 +254,28 @@ Sample code:
frame = mp.Frame(mp.from_torch(torch_vf.contiguous()), RGB)
bmf_vframe = VideoFrame(frame)
#...
```
```

#### Underlying data type conversion
Conversion of the underlying data type of the BMF Video Frame is also possible using `bmf_convert` similar to the C++ examples above. The `media_type` property on the MediaDesc instance is used to identify the underlying data type when doing conversions with `bmf_convert`. The following options are available:

`bmf.lib._bmf.sdk.kBMFVideoFrame` - BMF video frame

`bmf.lib._bmf.sdk.kAVFrame` - ffmpeg AVFrame

`bmf.lib._bmf.sdk.kATTensor` - torch

`bmf.lib._bmf.sdk.kTensor`

`bmf.lib._bmf.sdk.kCVMat` - opencv mat

Convert ffmpeg AVFrame to BMF Video Frame:
```python
from bmf import *
from bmf.lib._bmf.sdk import kAVFrame

vf = pkt.get(VideoFrame) # grab input AVFrame
src_md = MediaDesc().media_type(kAVFrame)
out_vf = bmf_convert(vf, src_md, MediaDesc())
#...VideoFrame process
```

0 comments on commit 14a4999

Please sign in to comment.