FragmentOptimizer: matrix not positive definite

[algoterranean]

When running this code (both my compiled version and the pre-compiled binaries available at the Reconstruction site) with the provided demo.sh, I run into some issues in FragmentOptimizer, resulting in the output giving a bunch of NANs.

For example, every entry in pose.txt is of form

0	0	1
1.#QNAN000 1.#QNAN000 1.#QNAN000 1.#QNAN000
1.#QNAN000 1.#QNAN000 1.#QNAN000 1.#QNAN000
1.#QNAN000 1.#QNAN000 1.#QNAN000 1.#QNAN000
1.#QNAN000 1.#QNAN000 1.#QNAN000 1.#QNAN000

and output.ctr is of form

1.#QNAN00000 1.#QNAN00000 1.#QNAN00000

Any ideas on what is going on here?

The output from FragmentOptimizer looks like:

FragmentOptimizer.exe --slac --rgbdslam ../sandbox/init.log --registration ../sandbox/reg_output.log --dir ../sandbox/pcds/ --num $numpcds --resolution 12 --iteration 10 --length 4.0 --write_xyzn_sample 10
Parameters: weight 1.00000, resolution 12, piece number 57, max iteration 10
IPose initialized.
Read ../sandbox/pcds/cloud_bin_1.pcd ... get 137524 points.
Read ../sandbox/pcds/cloud_bin_3.pcd ... get 188961 points.

snip

Read ../sandbox/pcds/corres_0_1.txt, get 120912 correspondences.
Read ../sandbox/pcds/corres_1_2.txt, get 94185 correspondences.

snip

Processing 224 :   224 ... Done.
Data error score is : 17028297.21
CHOLMOD warning: matrix not positive definite
SLAC optimization.
Regularization error score is : 0.00
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 224 :   224 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Neat Optimization took 601446ms.
Save ctr to file output.ctr ... Done.
Save sample pcd into sample.pcd ... Optimize All took 618325ms.

[zhangxaochen]

I have the same problem, @algoterranean do you have any progress with this?
I was testing this project with a simple mock oni file with only 361 frames of synthetic depth images and fake color images, the scanning trajectory is a irregular circle, and when running FragmentOptimizer.exe my console/shell output is:

$ $numpcds
bash: 7: command not found

zhangxaochen@zhangxaochen-PC /cygdrive/d/Users/zhangxaochen/Documents/indoor-executables-1.1/bin
$ ./FragmentOptimizer.exe --slac --rgbdslam ../sandbox/init.log --registration
../sandbox/reg_output.log --dir ../sandbox/pcds/ --num $numpcds --resolution 12
 --iteration 10 --length 4.0 --write_xyzn_sample 10
SLAC optimization.
Parameters: weight 1.00000, resolution 12, piece number 7, max iteration 10
IPose initialized.
Failed to find match for field 'rgb'.
Failed to find match for field 'rgb'.
Failed to find match for field 'rgb'.
Failed to find match for field 'rgb'.
Read ../sandbox/pcds/cloud_bin_0.pcd ... get 10345 points.
Failed to find match for field 'rgb'.
Failed to find match for field 'rgb'.
Failed to find match for field 'rgb'.
Read ../sandbox/pcds/cloud_bin_3.pcd ... get 16454 points.
Read ../sandbox/pcds/cloud_bin_5.pcd ... get 16812 points.
Read ../sandbox/pcds/cloud_bin_6.pcd ... get 16843 points.
Read ../sandbox/pcds/cloud_bin_2.pcd ... get 16350 points.
Read ../sandbox/pcds/cloud_bin_1.pcd ... get 13297 points.
Read ../sandbox/pcds/cloud_bin_4.pcd ... get 16444 points.
Read ../sandbox/pcds/corres_2_3.txt, get 14751 correspondences.
Read ../sandbox/pcds/corres_5_6.txt, get 16814 correspondences.
Read ../sandbox/pcds/corres_0_6.txt, get 10987 correspondences.
Read ../sandbox/pcds/corres_4_6.txt, get 16519 correspondences.
Processing 4 :     4 ... Done.
Data error score is : 5838.06
CHOLMOD warning: matrix not positive definite
Regularization error score is : 0.00
Processing 4 :     4 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 4 :     4 ... Done.
Data error score is : -1.#J
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 4 :     4 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 4 :     4 ... Done.
Data error score is : -1.#J
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 4 :     4 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 4 :     4 ... Done.
Data error score is : -1.#J
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 4 :     4 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Processing 4 :     4 ... Done.
Data error score is : -1.#J
CHOLMOD warning: matrix not positive definite
Regularization error score is : -1.#J
Processing 4 :     4 ... Done.
Data error score is : 1.#R
CHOLMOD warning: matrix not positive definite
Regularization error score is : 1.#R
Neat Optimization took 9604ms.
Save ctr to file output.ctr ... Done.
Save sample pcd into sample.pcd ... Optimize All took 9691ms.

the output file output.ctr and sample.pcd are both invalid (I checked them using notepad++ text editor). Contents of output.ctr & pose.log are as @algoterranean described above.

I've uploaded a package of my input and output here: https://drive.google.com/file/d/0B4vahSr3aGadM2dWaWRZS0NyNDg/view?usp=sharing
Much appreciated if authors of Robust Reconstruction of Indoor Scenes, CVPR 2015 could offer some help ;-) //@qianyizh @qianyi-zhou

[zhangxaochen]

UPDATE 1:
On my simple test scene oni, I noted that 8 corres_xx.txt have been generated, and the console outputs only 4 of them:

Read ../sandbox/pcds/corres_2_3.txt, get 14751 correspondences.
Read ../sandbox/pcds/corres_5_6.txt, get 16814 correspondences.
Read ../sandbox/pcds/corres_0_6.txt, get 10987 correspondences.
Read ../sandbox/pcds/corres_4_6.txt, get 16519 correspondences.

Then I realize that's because of blacklist_pair_num_ defaults to 10000, which is unsuitable for my small mock scene. I then added --blacklistpair 3000 to cmd params, and tried --rigid instead of --slac or the default --nonrigid, and finally get rid of the 1.#QNAN000 & -1.#J errors.
Now the file output.ctr looks like:

-5.1016936793 -9.9859573933 7.8406904559
-5.1098440328 -10.1346105529 7.5424507519
... //15379 total lines
-312.2567137439 102.6994950720 -122.2342856582
-312.3756642018 102.9904097349 -122.3453400520
-312.4946146597 103.2813243978 -122.4563944458

Since PCLException error then occurred when call Integrate as shown below, I'm still not sure if the contents of output.ctr are correct or not :

$ ./Integrate.exe --pose_traj ../sandbox/pose_slac.log --seg_traj ../sandbox/100-0.log --ctr ../sandbox/output.ctr --num ${numpcds} --resolution 12 --camera synDep.param -oni ../sandbox/input.oni --length 4.0 --interval 50
Registration mode: On
Camera model set to (fx, fy, cx, cy, icp_trunc, int_trunc):
        609.28, 609.28, 320.00, 240.00, 5.00, 5.00
Trajectory created from pose and segment trajectories.
Frames processed : 100 / 350
Frames processed : 200 / 350
Frames processed : 300 / 350
PCLException
Integrate All took 12120ms.

[algoterranean]

@zhangxaochen I was able to get the provided sample scene working by using the github code and changing the FragmentOptimizer to use a Simplicial solver instead of the Supernodal one. This is just one line of code: In OptApp.cpp, change instances of

Eigen::CholmodSupernodalLLT< SparseMatrix, Eigen::Upper > solver;

to

Eigen::CholmodSimplicialLLT< SparseMatrix, Eigen::Upper > solver;

However, I too am working on my own small synthetic scene and I still have problems with SLAC with this data. I have not figured out a solution yet but, like you, had success (that is, at least a resulting mesh, however terrible it was) with the rigid and nonrigid options. I suspect my correspondences earlier in the pipeline are causing an issue but I need to investigate more.

I will post an update here when I find out more.

[zhangxaochen]

@algoterranean great to have someone to discuss, though few people tackles with running this repo...
Thx for ur advice, I'll check it as soon as I can.
BTW, would you like to leave your contact information(e.g., email, QQ, wechat, IRC channel@freenode , etc. ) so I could keep touch directly? If you don't mind, mine email address: [email protected].

[zhangxaochen]

@algoterranean my test results:

1. using cmdline param --rigid: after changing CholmodSupernodalLLT to CholmodSimplicialLLT, the output.ctr looks better, and the following integrate.exe ... succeeds, but the reconstruction result looks terrible:
   

2. though using CholmodSimplicialLLT instead, --nonrigid still not work, neither --slac does.

[algoterranean]

@zhangxaochen I am getting close. One thing to look at is the configuration settings used for GlobalRegistration. These settings are not available via the command line but are in the file alignment.config. Changing some of these values might improve your registration results and thus your final model.

Will post more in a day or two when I have more.

[zhangxaochen]

@algoterranean Hi~
I'm following your hint and reading the file alignment.config, yet not quite understand all it's parameters.
I tried changing inlier_number from default 30000 to 3000 on my small synthesized scene, and then run the whole pipeline. The final output mesh_1.ply changed a bit, but not improved, it looked even worse to me.
How did you edit the alignment.config file? Is my editting inlier_number=3000 proper or not? (how can I judge?)
And, have you any progress last week?

[hitsjt]

@algoterranean @zhangxaochen ，do you have made any progress?I also encountered this problem.
PS:I have wrote an email to you@zhangxaochen.

[SingularTnT]

I wonder do you have the problem like this when I running the FragmentOptimizer.exe:
...FragementOptimizer.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory
I don't know what the symbol "?" refers to ?
Thank you very much!

[dcazzato]

Hi, I am having the same problem, therefore the program "Integrate" cannot see a valid output and I cannot complete the whole pipeline. I also tried with other oni files from "Augmented ICL-NUIM Dataset", even that ones that need the other script (demo_val3.sh). Exactly the same issue. Any progress or solution? Are the authors planning to release updates?

@SingularTnT : if you run the exe from a classical MS-DOS window, of course calling directly the program with the proper arguments instead of the whole script, you should see the error instead of having a "?".