Skip to content

GSoC 2018

Jump to bottom
Utkarsh Tripathi edited this page Aug 14, 2018 · 2 revisions

Google Summer of Code 2018

This page covers the overview of work done during the last few months on this project along with respective pull requests, targets met and future work. Other pages of this wiki (Introduction, Generating DLL and Unity Demo) contain detailed explanation on how to use ViSP functions in Unity by demonstrating Augmented Reality and Model-based Tracking.

Mentors for this project:

  • Marc Pouliquen
  • Florian Berton

Relevant links for this project:

Work timeline:

I started working on this project in mid-February and the first part was to get familiar with the work done on this project last year, which involved: learning to use unmanaged C++ native plugins within managed environment of Unity and then testing the code for 3D to 2D projection (using ViSP functions in a native C++ wrapper plugin) within Unity. Also, during this time, I worked on creating a tutorial using ViSP and MATLAB issue #143 and proposed a solution in the PR #321 which is available here.

By 21st of Feb, I had implemented the C++ wrapper that interfaces with vpDetectorAprilTag class and return 0/1 depending on whether an AprilTag was within field of view of physical camera. An important point to note here was that the image was flipped horizontally to compensate the difference in coordinate systems in Unity and ViSP.

In the next step, I calibrated the virtual camera in Unity to adapt Unity camera to the intrinsics of the real camera (Camera parameters can be set in Unity and they will be sent to the ViSP wrapper). By the first week of March, I had created a video, where a GameObject (sphere) was inserted at the centre of AprilTag and by the last week of March, rotation of the object (a cube) was also accomplished. There was an issue with occlusion of AprilTag with the inserted cube and it was resolved later using the concept of pivot point in Unity.

First demonstration

Notice in the video that frame-rate observed in Unity is very slow and this issue was resolved later by limiting the number of function calls to wrapper from Unity to just one. From mid-May until mid-June, I enhanced cube augmentation based on AprilTag in Unity but this time using an Orthographic camera instead of Perspective camera as suggested by Marc. Orthographic camera was geometrically simpler to use for augmentating game object because we do not worry about projective geometry. Even though the image of AprilTag is perspective, we found experimentally that this assumption is valid in the general video context because AprilTag was not perspectively distorted even at reasonably close distance from the physical camera. I also incorporated scaling of the object based on the size of AprilTags' diagonals. Using these methods, we found that the results were very realistic, and the cube completely covered the AprilTag as expected.

After this part, we started working on the Model-based tracking. We made a cube of specific dimension (side length 12 cm), and an AprilTag was pasted on one of its face. We also have a 3D wireframe CAD diagram of the cube which we want to augment in the Unity scene. Model-based tracking relies on AprilTag for pose estimation of physical object and once object’s pose is estimated, the tracking algorithm starts tracking the physical object in real-time using features such as edges, texture etc. until the estimation error is large enough. If the error in pose estimation exceeds a pre-defined threshold, the pose estimation algorithm based on AprilTag is invoked again.

To make the system user friendly, I have created an interface in Unity where the user can set the camera parameters, tracker type and edge visibility. This work was completed by mid-July and a clear demonstration video was uploaded on Youtube. We found that there was an issue that if the edges that were not detected (such as outside camera field of view), the Lines in Unity did not disappear and stayed on the scene. We resolved this issue by the end of July by using a flag that tracks visibility of edges in ViSP.

Second demonstration

This part was really interesting as it involved understanding the functions and then implementing them in Unity as well as in the wrapper. Towards the end of July and in the first week of August, I worked on the last task to run the application made in Unity (with ViSP C++ plugin) on Android. In this task I had to use Unity to cross-compile the ViSP plugin for Android, however, due to incompatibility of Unity's Android Native Development Kit (NDK) version r13 and ViSP NDK version requirement r16+, the Android application could not be compiled for Android. But, I did cross-compile a test application with Unity without ViSP and run it on Android. Unity cross-compiles C# code and links C++ plugins using IL2CPP (Intermediate Language To C++). IL2CPP requires Android NDK specifically version r13 otherwise it gives error. Also note that, even with the latest Unity (2018), IL2CPP supports only Android NDK r13. If ViSP can extend its support for Android NDK version r13 which latest Unity requires, then it would be straightforward to cross-compile the same ViSP plugin that we have developed for Windows Desktop. I would be more than happy to support ViSP with this work in future.

Future work

Using this project as a template, we can now literally port any functionality that ViSP supports that can benefit the community that uses Unity for AR application development. Some of them are listed below:

  • Porting AR demonstration to mobile devices particularly Android
  • Markerless model-based tracking using ViSP in Unity
  • Multiple AprilTag tracking
  • Animated virtual objects and Physics

1. Introduction

2. ViSP wrapper plugin

3. Unity AR Demo

4. GSoC 2018

Clone this wiki locally