CinemaDNG/video mode

[matt-from-wii-sports-2]

Would it be possible to make it work with CinemaDNG sequences, moving frame by frame stacking n frames before and after the target frame? Like if you choose to stack 5 frames total in some settings menu it would stack 2 frames before the current frame, the current frame and 2 frames after the current frame, then do the same with the next frame in a sequence. I understand it'll take a long time to process everything and that some new GUI may be needed

[Alex-Vasile]

It would definitely be possible do something like that, but I think that would be beyond the scope of this software.

Some other changes would have to be made, specifically:

• the currently alignment method is not temporally stable. The alignment of each tile is independent of every other tile and of every other image. This would likely need to be redone
• the current alignment also struggles with certain textures, issues which would likely become more apparent on video, especially when coupled with the issue above.
• handling of changing exposure between frames. Currently we merge into the darkest frame if a bracket shot is detected, but in this case it would always have to be the "current" frame.

The time would actually likely not be that bad. I expect it could manage real time conversion since the video files would be what 4K? 6K? 8K would be a bit of a struggle.

[Alex-Vasile]

Do you have any programming experience? The application has a command line interface which you could script this functionality with.

Alternatively, are there any publicly available clips to test this on?

[matt-from-wii-sports-2]

Thanks for the clarification.

Do you have any programming experience? The application has a command line interface which you could script this functionality with.

I did not know there was a CLI, do you mean https://github.com/martin-marek/hdr-plus-pytorch? If not, is there documentation available somewhere? I would definitely like to try it with this, robust implementation

Alternatively, are there any publicly available clips to test this on?

"Mountain Bike" from BMPCC Gallery is publicly available and CinemaDNG. Or, alternatively, there is MotionCam app for Android phones that can film CinemaDNG. And it can also do frame stacking but only when converting from their own container format, not with a normal DNG sequence, and I don't know what algorithm they use.

[Alex-Vasile]

Thanks for the clarification.

Do you have any programming experience? The application has a command line interface which you could script this functionality with.

I did not know there was a CLI, do you mean https://github.com/martin-marek/hdr-plus-pytorch? If not, is there documentation available somewhere? I would definitely like to try it with this, robust implementation

The "CLI" is in https://github.com/martin-marek/hdr-plus-swift/blob/main/burstphoto/cli.swift . It's not really a CLI since it uses hardcoded values. You will also have to build the package from source. See #5 (comment) for how to set it up to not require a developer account to run.

I will look take spend a bit of time changing the CLI to be a real CLI, and see if It could be packaged into the main application itself (even if as a second binary).

Alternatively, are there any publicly available clips to test this on?

"Mountain Bike" from BMPCC Gallery is publicly available and CinemaDNG. Or, alternatively, there is MotionCam app for Android phones that can film CinemaDNG. And it can also do frame stacking but only when converting from their own container format, not with a normal DNG sequence, and I don't know what algorithm they use.

I'll take a look at that mountain bike example and play around with it.

On an initial test, it looks like something is wrong with the exposure control when using these files. So I reccomend you leave the exposure_control setting to Off.

Keep in mind that the interface is not meant for this, and will be slower than it would have to be since for each image it will re-load all N+1 files (centre and N files before and after) when processing each image.

[Alex-Vasile]

If you replace

hdr-plus-swift/burstphoto/cli.swift

Lines 60 to 63 in 67e1e06

	// align+merge
	let out_url = try perform_denoising(image_urls: image_urls, progress: progress, merging_algorithm: merging_algorithm, tile_size: tile_size, search_distance: search_distance, noise_reduction: noise_reduction, exposure_control: exposure_control, output_bit_depth: output_bit_depth, out_dir: out_dir, tmp_dir: tmp_dir)
	print("Image saved in:", out_url.relativePath)

with:

            let N = 2
            for i in N..<image_urls.count-N {
                // ProcessingProgress is only useful for a GUI, but we have to instantiate one anyway
                let progress = ProcessingProgress()
                
                let urls = Array(image_urls[i-N...i+N])
                
                // align+merge
                let out_url = try perform_denoising(image_urls: urls, progress: progress, merging_algorithm: merging_algorithm, tile_size: tile_size, search_distance: search_distance, noise_reduction: noise_reduction, exposure_control: exposure_control, output_bit_depth: output_bit_depth, out_dir: out_dir, tmp_dir: tmp_dir)
               
                print("Image saved in:", out_url.relativePath)
            }

This will do what you're asking for. Since the loop is happening inside Burst Photo, it can take advance of the cache we've build into it. With that, I'm getting processing times of ~0.45s/frame with N=2 (i.e. frame + 2 frames before and 2 frames after) and mode "Fast". Of that 0.45s, 0.075s is reading the next frame from disk and 0.35 is writing it to disk. You could change it to mode "High Quality" and that time bumps up to ~0.65s/frame.

Overall, the process is really limited by how we currently write DNGs and their metadata back to disk.

[matt-from-wii-sports-2]

Thank you for the code, I could successfully compile the CLI.app and run it

In the limited testing in High Quality mode, small tile size and high search radius there's quite a bit of wobbling in the output. So you were right it will need some changes to be usable, but at least the speed is indeed better than I expected and the motion clarity is generally pretty good

If you make any changes regarding temporal stability in the future, I would be glad to test them too

[Alex-Vasile]

Thank you for the code, I could successfully compile the CLI.app and run it

In the limited testing in High Quality mode, small tile size and high search radius there's quite a bit of wobbling in the output. So you were right it will need some changes to be usable, but at least the speed is indeed better than I expected and the motion clarity is generally pretty good

If you make any changes regarding temporal stability in the future, I would be glad to test them too

I also recommend you turn search distance lower unless there's a lot of movement in your scene between frames. The search distance has a tendency to do that.

How many images were you using for N?

Also keep in mind that the script above won't output the first N frames, so you won't be able to compare frame by frame without padding.

[matt-from-wii-sports-2]

I also recommend you turn search distance lower unless there's a lot of movement in your scene between frames. The search distance has a tendency to do that.

Unfortunately doesn't seem to help much

How many images were you using for N?

2 for all tests, which should be a good compromise between noise level and motion clarity IMO

Also keep in mind that the script above won't output the first N frames, so you won't be able to compare frame by frame without padding.

Yes, sure, I was comparing the results in the middle of the sequence