Geo Voyage is a virtual and mixed reality app created to show developers how to build apps that make Quest users excited about learning and using the new Meta Spatial SDK. This app transforms your space into a global exploration platform. You may spin the stylized 3D globe, drop a pin anywhere, and dive into immersive experiences with 360 degree images and daily challenges. Geo Voyage’s AI companion will guide you through your journey, answering your questions and helping you discover fascinating facts about our planet.
Some of the goals of this project are to:
- Develop an experience that inspires developers to build with this application framework.
- Create an open source project that demonstrates best practices and clean development.
- Provide documentation that helps developers navigate the project and understand the intricacies of the development – including detailed breakdowns of how different aspects of the project have been implemented, how they were optimized for Quest, and best practices for content creation.
This document also serves as a guide for setting up the app to build and run locally on your machine, and provide a high-level overview of the application architecture and features. More in-depth documentation on the integration of cloud services and implementation of core systems within the app can be found in the official Horizon documentation here.
- Geo Voyage
- Project Overview
- Table of Contents
- Health and Safety Guidelines
- Design Flow
- Device Compatibility
- Key Components
- Getting Started
- Running the Project
- Main Scene
- Other Scenes
- Dependencies
- License
- Attributions
When building mixed reality experiences, we highly recommend evaluating your content from a health and safety perspective to offer your users a comfortable and safe experience. Please read the Mixed Reality H&S Guidelines before designing and developing your app using this sample project, or any of our Presence Platform features.
Developers should avoid improper occlusion, which occurs when virtual content does not respect the physicality of the user’s environment. Improper Occlusion can result in a misperception of actionable space.
- See Occlusions with Virtual Content
- To avoid improper occlusion, developers should ensure that users have (1) completed Space Setup and (2) granted Spatial Data permission (setup design) to allow proper occlusion in content placement, mesh collisions, and air navigation.
Using semi-transparent content lets the user have a better view of their physical space and reduces the occlusion of objects or people that are not part of the scanned mesh.
- Spatial data won’t incorporate dynamic elements of a user’s living space (for example, a chair that was moved after capture or a moving person/pet in the space).
- Uncaptured dynamic elements may be occluded by virtual content, making it more difficult for a user to safely avoid such hazards while engaged in the mixed reality experience.
Respect the user’s personal space. Avoid having virtual content pass through their body or loom close to their face. When content crosses into a user’s personal space they may experience a psychological or visual discomfort, or take actions to avoid the virtual content that may increase the risk of injury or damage (for example, backing up into a wall or chair). Dynamic virtual content may also distract the user from their surroundings.
This application experience consists of 4 different "play modes":
- Ask Earth – speak a question related to Earth ecology, culture, or history, and get a text response from your AI assistant – powered by Wit.ai, Meta's Llama 3 8B LLM, and AWS Bedrock.
- Explore – spin the globe around, find and drop a pin on a place of interest, and view information about a notable city or landmark near that location – powered by the Google Maps Geocoding API, Llama 3 8B, and AWS Bedrock. If available at that location, you may also transport yourself there by entering a virtual 360 degree view of the corresponding place on Earth – powered by the Google Map Tiles API. Additionally, select 3D models of notable landmarks placed around the globe to view their brief descriptions – pre-generated by Llama 3 8B.
- Daily Quiz – challenge yourself with 5 daily quiz questions related to earth geography and cultures – pre-generated by Llama 3 8B. Compare your results with your friends, and return the following day for new questions.
- Today in History – learn about one notable event in history that occurred on today's date – powered by Llama 3 8B and AWS Bedrock. Return the following day for a new fact.
Meta's Spatial Framework is compatible with the following devices:
- Quest 3
- Quest Pro
- Quest 2 (supported, but not recommended)
The Quest 3 was primarily used for testing throughout the development of this project. Attempting to run this application on unsupported devices is not recommended, and may result in errors and/or undesirable performance or application behavior.
In order to build and run this application, ensure that you have one of the Quest headset models listed above, and that the steps to setup your headset for development have been completed.
The main features of this application can be divided into the following 4 areas. Each of these areas has detailed documentation on their implementation and usage in the pages respectively linked.
This application was built with Android Studio Koala (2024.1.1) and Kotlin (2.0.0). To begin, make sure you've downloaded and installed the required software.
As per the Meta Quest documentation regarding Android SDK version requirements for submitting your app to the Meta Horizon Store, this application uses the following SDK versions:
android:minSdkVersion29android:targetSdkVersion32
Note: even though we are targeting API level 32, the
compileSdkversion has been set to 34 to enable support for some Jetpack Compose and Material 3 dependencies.
These SDK versions can be downloaded and installed via the Android Studio SDK Manager window, accessible through the Tools/SDK Manager menu item. Note that these SDK version numbers are specified in the app/build.gradle.kts gradle script, and the app AndroidManifest.xml.
Wit.ai is used by this application to perform speech to text transcription, as well as understand what the user is intending. Follow these steps to ensure that Wit.ai integration will properly function.
- Follow the Quickstart guide here to set up your Wit.ai app and train it. Note that the Wit.ai intents, entities, and traits used by this application are defined as enumerations in the Android project files located in the
:app/com.meta.pixelandtexel.geovoyage/services/witai/enumsdirectory. You may choose to name the intents, entities, and traits defined in your Wit.ai app to match the names of the enumeration values in those code files, or rename the values in the code. To learn more about this application's integration of Wit.ai and its usage, see this documentation. - Get your Wit.ai app Client Access Token from the Settings page of your app in the Wit.ai web dashboard. This can be accessed by selecting the
Management/Settingsmenu item on the left side. Once you've opened the Settings page, you should see the Client Access Token value there. You'll need it in the below step on adding your app secrets to the Android project. The Wit.ai HTTP API uses OAuth2 to authenticate requests via a bearer token in theAuthorizationheader value. Note that access tokens are app and user specific, and should not be shared or posted publicly. More info on Wit.ai HTTP API can be found here.
A core part of this application is its ability to prompt Meta's Llama 3 8B LLM. This is used for several purposes throughout the application. To read more detail about that, see this page. In order to build and run this application, you'll need to take these steps to ensure Llama querying functions.
This application has built-in support for invoking the Llama via two different methods:
- AWS Bedrock – AWS's solution for invoking various LLM models, as well as the capability to train and upload your own model for querying. This option has a cost associated with it which should be considered.
- Ollama is a free option, but requires you to set up a server to run the tool. You could also run ollama locally on a device on your network and connect to it using the device's IP address and port number – an option we found useful during development. See the instruction below on running ollama via docker. Alternatively, you can use the executable available on the ollama website.
- You must first create an AWS account if you don't already have one.
- Next, login to the AWS console, and navigate to the Bedrock service.
- At the bottom of the left side navigation bar, select the
Bedrock configurations/Model accessmenu item. Scroll down to make sure that access has been granted to the Llama 3 8B Instruct model. If not, you may have to select the "Modify model access" button at the top of the page to gain access. - Next, you'll create a custom permissions policy which only grants
InvokeModelWithResponseStreamaccess on the"arn:aws:bedrock:<your region>::foundation-model/meta.llama3-8b-instruct-v1:0"resource (replace<your region>with the region within which you setup your AWS account and Bedrock service). Follow these instructions to create the policy, and remember the name you've given the policy – e.g.InvokeBedrockLlama3Model. Your policy JSON should look similar to this (except, perhaps, the region name):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": ["bedrock:InvokeModelWithResponseStream"],
"Resource": [
"arn:aws:bedrock:us-east-1::foundation-model/meta.llama3-8b-instruct-v1:0"
]
}
]
}- Next, follow the instructions here to create and configure an IAM user.
- You may assign a descriptive User name of your choice – e.g.
LlamaServiceAccount - It is not necessary to grant the user access to the cloud console.
- For this new user, assign the policy you just created to the new user.
- You may assign a descriptive User name of your choice – e.g.
- Finally, you must generate an access key for this new AWS IAM user, and save those credentials.
- Navigate to the IAM service dashboard, and then select the
Access management/Usersmenu item on the left side navigation bar. - Find and select the user you just created to view their account page.
- Select the
Security credentialstab on the mid-page navigation, and scroll down to theAccess keyscard. - Select the "Create access key" button on the right side.
- Select the "Other" option as your use-case.
- Add a description – e.g.
Service account access key for invoking Llama3 in Geo Voyage project. - Select the "Create access key" button, and save the "Secret access key" value someplace secure, as you will not be able to access that value after you leave the webpage. You'll need the access key and secret access key in the below step on adding your app secrets to the Android project.
- Navigate to the IAM service dashboard, and then select the
- Note that you should always follow best practices for keeping your access key secure.
To quickly get ollama running locally, you may utilize the provided docker-compose script. First, install docker locally and then open terminal in the project folder. Run the following command to create the Ollama server:
docker compose up -dNext, execute the below command to run the Llama 3 model model:
docker exec -it ollama ollama run llama3This will download the latest llama3 model locally in the data folder and expose it to your network at port 11434.
- Sign up or in to the Google Cloud Platform.
- Navigate to your Google Maps Platform APIs & Services page, and enable the "Geocoding API" and "Map Tiles API".
- Navigate to the Google Maps Platform Credentials page and create a new API key by selecting the "Create Credentials" dropdown at the top menu bar, and selecting the "API key" option.
- After creating it, select the "Edit API key" option from the Actions list which becomes visible after clicking on the kebab button on the right side of your new API key in the "API Keys" list in the middle of the webpage.
- Name the key something meaningful – e.g. "Geo Voyage Maps API Key"
- Restrict the key to the APIs you enabled in the previous steps.
- Save the "API key" value someplace secure, as you will need it in the below step on adding your app secrets to the Android project.
The cloud services outlined below in Dependencies require access tokens. These tokens must be provided in a secrets.properties file, which is ignored by git. Those key/value pairs are parsed in the Android project app/build.gradle.kts gradle script, and are injected into the application's generated BuildConfig.java file to be accessed by the application code.
Start by duplicating the secrets.properties.example file, and renaming it to secrets.properties. In this new file (pictured below), replace the values with your access tokens. Note that depending on which service you have chosen to provide Llama 3 invocation in the instructions above, you must only provide secrets for one of the given values below – AWS Bedrock keys or your Ollama server URL (including port number).
# Token for Wit.ai client REST API
WIT_AI_CLIENT_ACCESS_TOKEN=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
# (optional) URL to ollama server
OLLAMA_SERVER_URL=https://your-domain:port
# AWS Bedrock keys
AWS_BEDROCK_ACCESS_KEY=BBBBBBBBBBBBBBBBBBBB
AWS_BEDROCK_SECRET_KEY=CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
# Google Maps API key
GOOGLE_MAPS_API_KEY=DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDAfter you have completed the steps in Getting Started, you are ready to build and run the app locally. Open the project in Android Studio by selecting the File/Open... menu item, and selecting the top-level project directory in the file browser.
After the project has successfully opened, make sure to Sync the project with the gradle files. You can do this by selecting the Sync icon near the end of the top menu bar. This may take some time as dependencies are downloaded.
Next, ensure that your headset is plugged in to your computer, and that the device appears and is selected in the target devices dropdown – also in the top menu bar.
Next, select the Make Module icon in the top menu bar to compile the project code. Note that this step will generate the BuildConfig.java file, which should include the access token key values you entered into your secrets.properties file.
Finally, select Run from the top menu bar or from the Run menu item. The application should install onto your device and launch.
The primary scene is composed of the 3D objects and UI panel which load and are displayed when the application starts. The entities in this scene and their attached components are primarily defined in the res/xml/scene.xml file, and are loaded with the command inflateIntoDataModel(R.xml.scene) in the MainActivity.onSceneReady function.
This scene consists of 5 entities (1 panel and 4 3D objects), detailed below.
This application uses one main panel object for all UI. The layout and UI elements were created with Jetpack Compose, and are rendered into the scene via a Spatial SDK PanelSceneObject. The design follows Material Design standards, and was engineered to be similar to a landscape tablet screen view.
This panel floats to the right of the globe, and is the main navigation means for the user. Tapping on the list items on the right side of this panel initiates the 4 corresponding play modes – each with their own content which appears in the middle of the panel on elevated green surfaces. Additionally, the cog button on the upper right exposes a Settings menu with various controls over application behavior, while the title text at the top displays persistent relevant information about or of the current play mode.
Globe Models – the globe object is the central focus of the 3D scene. It is composed of 3 separate entities: the globe, the trees atop the globe surface, and the clouds spinning over the globe surface. The clouds are separate from the globe model in order to rotate independently, and the trees model was separated so it could be hidden programmatically during different parts of the Explore play mode.
Pin model – the pin object is initially hidden, and only appears when the user enters the Explore play mode, and selects a spot on the globe.
Landmark Entities – In addition to the 3D mesh and panel entities listed above, a number of mesh entities are created when the app first begins in the LandmarkSpawnSystem.kt file (more info on that here). Each of these entities represent different iconic landmarks around the world, and appear across all of the continents.
In addition to the main scene detailed above, additional test scenes may be found in the same directory with the naming convention res/xml/scene_<name>.xml:
res/xml/scene_globe_only.xml– this scene contains only the globe entity and its associated clouds and trees entities. It can be used to test the customSpinnableandGrabbableNoRotationcomponents and systems without the UI panels getting in the way.
The following cloud services are used by this application:
- Wit.ai – for speech-to-text transcription, and Entity and Intent understanding extraction
- AWS Bedrock – Llama 3 8B querying
- Google Maps API – Geocoding API and Street View Tiles API
Optional:
- Ollama – an alternative to AWS Bedrock, for serving your Llama 3 model over local network, or for connecting to a remote server running ollama.
These dependencies and their corresponding versions can be found in the app/build.gradle.kts and libs.versions.toml files in the Android project.
- Kotlin 2.0.0
- Meta's Spatial Framework
- gson – for deserializing JSON response objects from the aforementioned cloud services
- compose-markdown – for displaying markdown-formatted responses from Llama3 in Jetpack Compose
- AWS Kotlin SDK – for integrating AWS Bedrock for Llama 3 querying
- Jetpack Compose – for building modern Android UI with Material Design 3 support
MIT License
Copyright (c) 2024 Meta
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
| Dependency | Source | License |
|---|---|---|
| okhttp | GitHub repo | Apache License 2.0 |
| gson | GitHub repo | Apache License 2.0 |
| compose-markdown | GitHub repo | MIT License |