Background
This article is intended as a brief introduction to the system. It is not meant to be a comprehensive usage or development guide. Understanding the concepts here will allow you to understand more focused technical discussions or documentation.
The Campaign Information Management System (CIMS, pronounced "sims") is a collection of tools originally developed to serve Medical Care Development's Bioko Island Malaria Control and Equatorial Guinea Malaria Vaccine projects in Malabo, Equatorial Guinea. It continues to be used there for the combined successor project, the Bioko Island Malaria Elimination Program. The system is employed as a means to collect information about all households and individuals on the island. It is used for multiple simultaneous campaigns across functional units such as entomology, indoor residual spraying, spatial mapping, etc. It provides fieldworkers a functional tool to collect this information accurately and reliably using mobile computing devices with intermittent network connectivity.
The following diagram shows a simplified work flow using the system. We will revisit this with a more detailed diagram once relevant concepts have been introduced.
Describing the flow from left-to-right:
- A campaign manager, or deployer, creates or updates a file containing configuration specifying how the system should implement the campaign. They upload it to the CIMS server.
- Configuration and data is downloaded to mobile devices by one or more fieldworkers. They use it to collect information in the field where they may not have Internet access.
- Once they have Internet connectivity, fieldworkers upload the collected data to the server where it is processed.
- Collected data is downloaded from the server by analysts or decisions makers. They then use their preferred tools to build reports for decision support.
This is generally how the CIMS works. However, as mentioned previously, this description is simplified and is provided here to give an initial feel for what the system does. Before we can talk about what the system actually does we need to introduce some concepts.
The CIMS leverages software from the open source data collection system Open Data Kit (ODK). This is both a strength and a weakness. These tools are well-known and the interfaces are simple for many users to comprehend, but the system has some significant limitations. Understanding these limitations is important. Only by doing so can you work with the system and not against it. Before we get into that, let's introduce the basics of ODK.
In ODK, forms are files that define the structure, behavior, and language of a survey or questionnaire. It is used by the data collection application, ODK Collect, as a template to generate form instances.
A form instance is the basic unit of data collection in ODK. Each instance corresponds to a survey session with a data collector. It is a file containing the answers to questions defined in the form used to generate it. Form instances contain no information about the questions asked. They only contain the results. This means form instances can only be fully understood when interpreted in the context of the form used to generate them. A trivial example is that selection questions may not have human-readable labels like "Captured" or "Not captured" and rather could simply contain values like '1' or '2'. If someone wanted to swap the values of these selections in the form, they could. You would not know the meaning of the value '1' without the original form definition for the instance. Mostly, these types of problems can be avoided by following some rules or conventions when working with ODK forms. However, these come with some severe conseqences for certain types of data collection efforts.
The close relationship between forms and form instances means that understanding the data in an instance is critically dependent on the form used to generate it. Changing a form after instances have been created from it can change the meaning of data already collected. For this reason, the ODK community recommends conventions or rules to follow to avoid such problems:
- Forms, once deployed, are unchangeable with minor exceptions.
- Form instances, once submitted, are unchangeable.
- As long as changes to a form will not change the interpretation of existing form instances, the updated form can be deployed as another version of the same form. For example, fixing spelling mistakes in question prompt.
- If changes to a form will change interpretation of existing form instances, the form must be changed and deployed as a new form, not as a new version of the original.
Form instances being unchangeable presents presents an obvious challenge to the goal of the CIMS, which is to represent real-world objects that change over time.
ODK is not enough to be able to easily track real-world entities like individuals and locations over time. Its ease of use and form development features are great for surveys, but not being able to update data collected directly (remember, form instances are unchangeable once submitted) does not work well for maintaining the same objects over time. Overcoming that limitation is a primary function of the CIMS. It adds facilities around ODK to enable tracking things such as individuals and locations over time. The following concepts are important for understanding and discussing the system.
In contrast to ODK's forms, the CIMS maintains a database of real-world entities. The most important are:
- Hierarchy: an organizational hierarchy, under which households are located
- Households: geo-locatable locations where individuals reside
- Individuals: people traceable back to a household
These are typically created and updated using ODK. They constitute an organization's collected data. The following entities are also present in the system, but constitute details for the operation of the data collection system and are not collected data.
- Users: people that will interact with the CIMS server without using the mobile client
- Devices: mobile data colection devices possibly shared by one or more fieldworkers to collect data
- Fieldworkers: people with the ability to collect data on a mobile device
- Campaigns: Uniquely-named interventions. In the system, a campaign encapsulates all the users, devices, fieldworkers, forms, and behavior required to manage a campaign.
These are typically created and managed using the CIMS web application. Last, but not least, the system maintains data concerning ODK forms and form instances, called "form submissions" in the CIMS.
Before we can do a detailed walk-through of the system flow, it will help to understand, at a basic level, the components of the CIMS system. The CIMS is comprised of multiple software applications.
CIMS Server is a software application that is run in a data center, like Amazon Web Services. It responsible for delivering content to and accepting form submissions from the CIMS Mobile application. It manages its important data using an external relational database, also hosted in a data center.
On the tablet, the CIMS is mainly comprised of two applications: CIMS Mobile and CIMS Forms.
The CIMS Mobile application is the primary interface used by fieldworkers for data entry. The application augments the form-based functionality of the ODK-based Forms app with support for creating and maintaining entities like location and individual, along with other features. It pre-fills forms with offline data based on where you launch them in an entity hierarchy. It also communicates with CIMS Server for authenticating the device, for synchronizing data captured by other devices and for offline fieldworker login.
The CIMS Forms application is based on ODK Collect and it handles all form data entry and form download/upload tasks. It is invoked by CIMS Mobile for certain activities, making it appear to be part of a single application. It communicates separately to CIMS Server using ODK protocols. It has limited ability to interact with the CIMS Mobile application.
There are a few other services and applications that can be used as part of the CIMS. These are not essential for understanding the system conceptually, but are mentioned here since they are likely to be encountered in a real deployment.
A conversion service that can convert form definitions written using XLSForm into ODK's native XML format. This service is primarily provided as an online conversion service for CIMS Server so it can support direct upload of XLSForm. However, it can often be helpful and/or necessary for users to perform a conversion manually. Examples of these can be when using functionality only available in the native format, or when trying to diagnose problems with automatic conversion.
Similar to Convert, it is an online validation service for ODK's native XML form format. This is provided mostly as a service for CIMS Convert, but it can be used directly when form authors want to validate a hand-edited XML form definition.
CIMS Server supports the full ODK Briefcase protocol so you can use the official ODK Briefcase application to upload and download forms and form instances (submissions) from CIMS Server. Typically, Briefcase is used to pull raw form submissions from the server. Once fetched, the user can export the submissions into tabular CSV files for importing and analyzing in other tools. This facility can be particularly useful when you are not interested in real-time updates to the data collected and you do not need information about entities like location and individual.
Sidecar is software that can be deployed on an inexpensive computer, such as a Raspberry Pi, to a local network where mobile devices will be synchronizing. It can be critical in certain deployments, especially when bandwidth is limited and the number of mobile devices on that connection is high. It provides efficient data synchronization services on a local network by allowing devices on the local network to sync to it rather than the remote server. This can drastically reduce Internet bandwidth usage when there are a large number of devices.
For example, without sidecar, if 25 devices in the same office attempt to synchronize a database of 100MB, collectively they will consume 25 x 100MB of data, or roughly 2.5GB. With sidecar, only the sidecar device syncs the database down, not the mobile devices. The mobile devices sync only to the sidecar, so only a single download of 100MB is performed. In addition to efficient use of data, Sidecar also can make syncing feel much faster for users, since local area network speeds are considerably faster than remote speeds.
To track entities using ODK forms, the CIMS takes advantage of some key capabilities of the ODK-based mobile app, CIMS Forms:
- Registering and deleting forms from another app
- Editing a form from another app with the ability to get the result on success
- Sending forms from another app
- Designing forms with questions that can launch another app
Using the CIMS Mobile app, users navigate or search a hierarchy of known entities. Once the entity of interest is located, they launch a new form instance for that entity. CIMS Mobile generates new form instances for CIMS Forms and, depending on the campaign, can embed identifiers to existing entities. This allows the system to ensure that forms reference the related entities without requiring any action from the user. CIMS Mobile registers the generated instance with CIMS Forms and launches it for editing by the user. Upon completion of a form instance, CIMS Mobile gets a result notification in return and using information contained in the completed instance, again depending on the campaign, it can create entity records locally. This is how a network of entities can be constructed offline on the mobile device using ODK. However, ODK forms and created entities on the mobile device are temporary. Only when completed forms are submitted to the server is a similar process performed to integrate the form data into a shared server database. That server database contains information from all mobile devices and is used to periodically provide data synchronization to ensure devices stay up-to-date with entities that other devices collected.
Let's revisit the simple flow diagram we started with and add some more detail on how the system is typically used. Here is that diagram from before, updated to include a few more details: a numbered ordering to show the relative order of steps in the process, and some miniature diagrams at each step to make the important item or process occurring at that step more clear.
Before we start, it is worth noting that the diagram here does not represent all possible flows supported by the system. It represents the most common flow for campaigns that will model entities like individuals or households. Simpler flows are common for forms that don't need to create or update entities and instead only wish to attach collected data to existing ones. With that said, let's break down the diagram.
To setup a campaign, an operator with campaign privileges creates a new campaign. This starts with creating a new, uniquely named, campaign record using the CIMS web application. Once the campaign exists, the operator uploads the campaign configuration consisting of ODK forms and a campaign file.
The campaign file is a zip file containing configuration necessary for the CIMS systems to implement the campaign on both mobile and server-side. Currently, the system supports specifying the following in the campaign file:
- Navigation Hierarchy (Mobile): The Mobile app expects a hierarchy of uniform depth with named levels. Using the campaign file, a campaign defines hierarchy levels to match the campaign area or activity.
- Navigation Modules (mobile): Navigation modules define how forms are bound to the hierarchy in the Mobile app, how forms should be processed (using form builders and consumers, discussed shortly), and other details like label translations.
- Form Builders (mobile): Form builders define procedures for how the CIMS Mobile app should pre-populate an ODK form given its launch context. This is how the system can embed entity information in ODK forms.
- Form Consumers (mobile): Form consumers define procedures for how to consume completed ODK forms. When a campaign is maintaining entities with the forms, consumers typically are what performs creation and updating.
- Translations (mobile and server): Allows specifying translatable messages to support multi-language campaigns.
- Database Export (server): Defines a mobile database export procedure for mobile data synchronization. Campaigns can customize the schedule and content of the mobile database delivered to mobile clients during sync.
- Form Processors (server): Form processors define the procedure for consuming a form on the server. These are much like Form Consumers on the mobile client, but the procedures often differ due to complexities of dealing with submissions from multiple mobile clients.
In the flow diagram, the document and gear icons represent the ODK forms and various behavior to handle them specified in the campaign file.
Once the campaign file is uploaded, the CIMS Server has everything it needs to build its first campaign database for the mobile client. It uses this configuration to build the initial database so that mobile clients will properly download their initial content during setup.
In the flow diagram, the large cylinder represents the shared campaign database, the gear represents the Database Export defined for the campaign, and the small cylinder represents the mobile database generated as the result of running the Database Export on the shared database.
When a device is first setup for a campaign or when campaign data or configuration changes, the mobile device will automatically download the relevant campaign and data updates. This is what ensures that campaign devices see data that other devices, and that they are running the correct procedures for the campaign.
Once campaign configuration and data is up to date on the device, it can be used to perform data entry. The general procedure for this is described briefly in the section "How It Works, For The Impatient". After the fieldworker signs into the mobile app using their unique fieldworker login, the user selects a relevant navigation module, navigates the hierachy or searches to find an entity of interest and launches a form by selecting a form launcher button. This causes the application to generate a new form, pre-populate it using the Form Builder specified in the campaign file, and launch the form for editing in the CIMS Form app.
After successfully editing a new form, the fieldworker completes it in the CIMS Form app. This notifies the CIMS Mobile app which then runs the Form Consumer specified in the campaign file. Depending on the consumer, this could create new entities in the hierarchy. The fieldworker then may proceed to navigate to the new entities and continue the data entry process by launching form from those entities.
The mini diagram next to the users with a tablet shows a cylinder representing the mobile database, a gear labeled 4* representing the form builder, a document representing the generated form instance, a gear labeled 5 representing the form consumer, which then can modify the mobile database with new or updated entities.
Step 4 and 5 can be performed in a loop for as long as necessary to capture the data offline.
As mentioned briefly in the section "How It Works, For The Impatient", ODK forms and entities created or updated on the mobile device are temporary. The changes are made permanent and are shared by submitting the forms to the server.
To integrate form submissions into the shared database, CIMS Server periodically processes uploaded form submissions using the Form Processors specified in active campaigns. A form processor is similar to the form consumer on the mobile side, but instead of dealing with just the local mobile database, form processors have to properly handle form submissions from multiple devices and successfully merging (or rejecting) the changes into the shared database. After this step, the flow can follow two possible paths: the mobile database is updated and mobile devices updated via 2*, or the data collection process may be complete and we can continue to Step 8.
Depending on the campaign requirements, step 8 may happen concurrently with active campaign activities or after. The method of accessing the campaign data may be different depending on the scenario as well. For operational users, real-time updates to the data are the most helpful, for example live dashboards that can be consulted regularly for feedback on campaign progress or quality. For modelers or other traditional analysts, they may prefer to have more complete finalized information at the end of the campaign. In both cases, the form submissions and/or entities created during the campaign are accessed from the system (either using ODK tools, or by traditional relational database access) and used by the analysts preferred tools. The CIMS intentionally does not attempt to handle analysis directly, since there are far better tools out there for that purpose.