bundle_adjust pinhole cameras with GCPs seen in multiple images

[ShashankBice]

Describe the bug
When using bundle_adjust for pinhole cameras with GCPs which are seen in 2+ images, the input camera models are not honoured to triangulate points for the control network, and the gcps are used to initiate the pinhole cameras. This results in initial triangulated points with very high reprojection errors and awkward height values. When the same set of cameras are used without GCPs, the initially triangulated points are way better. The resultant camera obtained bundle_adjustment is not very good. The behaviour was suppressed when using --disable-pinhole-gcp-init, and the triangulated points returned back to normalcy. However, should not this be the default ? Why should we be initialising cameras from gcps by default, until we are doing some camera resection problem?
I am not sure if this happens with other pinhole sensors though, so maybe we can check with already available gcp file which are seen in multiple images and pinhole cameras to confirm them.
To Reproduce

• Case 1 (GCP): Run `bundle_adjust img1.tif img2.tif img1.tsai img2.tsai gcp.gcp --num-passes 1 --max-iterations 0 --datum WGS84 -o ba_gcp/run'. Look at the initial_residual_pointmap_log.csv.
• Case 2 (No GCP): Run `bundle_adjust img1.tif img2.tif img1.tsai img2.tsai --num-passes 1 --max-iterations 0 --datum WGS84 -o ba_gcp/run'. Look at the initial_residual_pointmap_log.csv
• Case 3 (GCP with --disable-pinhole-gcp-init): Run `bundle_adjust img1.tif img2.tif img1.tsai img2.tsai gcp.gcp --num-passes 1 --max-iterations 0 --disable-pinhole-gcp-init --datum WGS84 -o ba_gcp/run'. Look at the initial_residual_pointmap_log.csv.

Expected behavior
Case 2 pointmap screenshot:

Case 1 pointmap screenshot:

Case 3 pointmap screenshot:

Error Logs, Terminal Captures, Screenshots
If applicable, please give us as much information as you can to help explain your problem.

Your Environment (please complete the following information):

• OS: Linux Ubuntu 12 or 14 (@seth-planet can confirm)
• ASP Version: 2.7.0
• Any other environment information that might be helpful?

Additional context
One more point of concern is the that the GCPs are 100s of km away from the other triangulated points, are your Lat/lon value swapped warning message. We checked that this is not the case and the GCPs and triangulated points are reasonably close, but still this warning appears. Log screenshot for context:

When should we be paying attention to this warning, and should we make it more meaningful ?

The experiments were primarly conducted by Seth Price, please chime in if I missed something. Hopefully the cameras produced with --disable-pinhole-gcp-init flag are good for stereo for now.

[ScottMcMichael]

The idea behind GCP initialization of pinhole cameras is that pinhole cameras would be less likely to have good nav data available than the other types of cameras we process and that GCPs, if available, would probably be the most accurate source of position information. In the case where your nav data is good hopefully the GCP are consistent with your camera information. I am not sure what is going on in this case. Probably the best place to start investigating is why that warning appears. Can you post your GCP files and also the output files when you run with --disable-pinhole-gcp-init and --num-passes 0 ?

[seth-planet]

The output files are 30-80 MB compressed. The log file is huge due to ~181k triangulation warnings like above. I'll look into getting the files posted to a URL you can access.

[seth-planet]

In this case we are working with SkySat cameras flying at 400-450km, which should have pretty good positional data. The GCP data tends to be on the order of 4 meters RMSE, while the pixel size is 0.8 meters.

[seth-planet]

Ok, I've assembled the input gcp file and all output here:
https://hello.planet.com/data/s/NQYb5HzzHGTFFAK

Let me know if you have trouble with the link or you'd like additional data.

[oleg-alexandrov]

There is probably some inconsistency between GCP and the cameras. While the GCP data may have 4 meters of RMSE, the very large focal length may amply some issues.

It would be useful to run indeed bundle adjustment with --disable-pinhole-gcp-init, to not use the GCP for initialization of cameras, and then, when looking at the initial_residual_pointmap_log.csv to also look at the very last lines in this file, which have the reprojection errors for your GCP. Those lines should have the comment "# GCP" at their ends, in any recent software.

It will be interesting to see what reprojection errors (column 4) and triangulated heights are (column 3), and how those compare to the heights from the gCP.

[seth-planet]

I've added the initial* files to the above shared folder. You should be able to download them and inspect them yourself. As a preview however, this looks reasonable for the 4 meter RMSE:

$ tail run-initial_residuals_no_loss_function_pointmap_point_log.csv
131.02542853782623, -25.2861325394573697, 513.000000000898581, 3.87142390157188743, 2 # GCP
131.024258029893701, -25.423756495658008, 539.999999999903935, 5.35001516754978468, 2 # GCP
131.040258746808291, -25.3192303322315091, 520.000000001177682, 4.09302981909141295, 2 # GCP
131.008059828507385, -25.389923336044621, 530.999999999273314, 4.18466419261611122, 2 # GCP
131.01598480488326, -25.4216162420684242, 540.999999998733529, 2.90952171032108708, 2 # GCP
131.018339974764302, -25.4005513766191804, 536.000000000944965, 2.87801493190085012, 2 # GCP
131.062062801160806, -25.3189921060249183, 516.000000000161322, 5.222560524683729, 2 # GCP
131.053391389526752, -25.3802718396675893, 541.000000000790237, 11.1916935286371881, 2 # GCP
131.047091713190639, -25.2726592004568822, 510.000000000697412, 3.81290734471390635, 2 # GCP
131.03186322905924, -25.4033771839609166, 534.000000000797172, 3.33162061142687094, 2 # GCP

$ tail run-final_residuals_no_loss_function_pointmap_point_log.csv
131.025428492008416, -25.2861328907924481, 513.013840591504731, 8.90114691130327174, 2 # GCP
131.024257663766832, -25.4237532073773949, 539.958598259552446, 0.462549387755650798, 2 # GCP
131.040258578774285, -25.3192307581062224, 519.999553187219476, 7.28239918435815525, 2 # GCP
131.008060938511903, -25.3899238512416616, 531.027361948209773, 3.64019641191205778, 2 # GCP
131.01598707350044, -25.4216152006231049, 540.996381043433416, 1.72797940639071612, 2 # GCP
131.018341775488096, -25.4005531812823513, 535.996696951285344, 1.83420441017139524, 2 # GCP
131.06206247852495, -25.3189924973232721, 515.995998071597342, 7.58473776622466289, 2 # GCP
131.053391718795211, -25.3802724951649523, 541.014818396716009, 7.13777100383367724, 2 # GCP
131.04709155596845, -25.272659470722143, 510.008533096952476, 12.9310453479833249, 2 # GCP
131.031857594137364, -25.4033795744926572, 533.975778365604469, 0.220020713315705052, 2 # GCP```

[oleg-alexandrov]

Our logic for initiating the cameras from GCP likely needs to be made more robust by using some kind of RANSAC-based method with outlier filtering. As it is, it is a least-squares fit.

You have a very bad GCP there:

131.02209948814766, -25.2627184316647586, 509.999999999894442, 615.098370341909117, 2 # GCP

Look at the last number.

Here's another one:

131.018006248468964, -25.2661551028768017, 509.999999999686622, 298.306044629161136, 2 # GCP

And there's more.

Here are more numbers from that file:

130.829177790193313, -25.6693053174450334, 71039.2515870268981, 14861.296234854779, 4
130.750221677652831, -25.7669259473323464, 99859.320379507466, 18594.5813775474853, 3

In short, this seems to be a combination for some unreliability in GCP, initial cameras, and our insufficiently robust logic.

[oleg-alexandrov]

Likely this line:

https://github.com/NeoGeographyToolkit/StereoPipeline/blob/master/src/asp/Tools/bundle_adjust_misc_functions.h#L1094

will need changing, while using a new code to fit a 3D transform to two set of points while robust to outliers. I hope to get to it at some point, but I don't know when.

But one would need to verify first that removing the outliers in the GCP or at least using a very small set of most reliable GCP bypasses the problem.

[seth-planet]

Oh wow. I don't know where those GCPs came from. I had only put a check in for horizontal distance. I'll investigate further. In the future, would it be possible to get more information in the warning messages that would help me debug? As it is, they are frustratingly vague when it comes to bad GCPs.