Skip to content

Latest commit

 

History

History
655 lines (482 loc) · 50.9 KB

12-ahrq-step-up-app-challenge.md

File metadata and controls

655 lines (482 loc) · 50.9 KB
challenge-title layout permalink challenge-id status sidenav card-image agency-logo tagline agency partner-agencies-federal external-url total-prize-offered-cash type-of-challenge submission-start submission-end fiscal-year legal-authority challenge-manager challenge-manager-email description prizes rules judging how-to-enter prize partners-non-federal submission-link
AHRQ Step Up App Challenge
front-matter-data
/challenge/ahrq-step-up-app-challenge/
12
closed
true
/assets/images/cards/ahrq-stepup-logo.jpg
ahrq-logo.jpg
Advancing Care Through Patient Self-Assessments
Agency for Healthcare Research and Quality
$250,000
Software and apps
08/13/2018 09:00 AM
09/24/2018 11:59 PM
FY18
America COMPETES Act
Bryan Kim
### Video Message From AHRQ Director Gopal Khanna About Entering the Step Up App Challenge **\*\****Miss the Step Up App Challenge: Phase 1 Informational Webinar?**[Check out the slides presented during the August 23 webinar](https://www.ahrq.gov/sites/default/files/wysiwyg/stepup_Challenge/step-up-challenge-phase-1-webinar-slides.pdf)\*\**** #### **The Background** Patient-reported outcomes (PROs) are vital to understanding the complex needs of a patient. For this challenge, PRO data are defined as “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or other medical expert.” In other words, PRO data come from patient self-assessments and look beyond the clinical data to assess how a patient feels about his or her health. #### **The Problem** While PRO data have proven useful to healthcare providers, they are not widely used in clinical settings. Recently, some digital tools have been developed to streamline the collection of PRO data. However, those digital tools are not widely adopted due to challenges related to clinician workflow and patient ease with the tools. Also, researchers who want to analyze PRO data across different practices or health systems often encounter the issue that data are not collected in a standardized manner. Creating a digital tool to efficiently collect, aggregate, and share PRO data is critical to advancing the quality of care. #### **Phase 1 Winners** The Agency for Healthcare Research and Quality (AHRQ) Step Up App Challenge is seeking innovative technologists to develop user-friendly apps that collect standardized PRO data for clinical and research purposes. **Phase 1 winners announced:** **AgilisIT:** AgilisITTM's solution focuses on patient engagement, ease of use for physicians, and EHR integration in order to encourage the digital collection of PRO data. The patient centered app offers both mobile and web applications, enabling patients to receive reminders and track their own progress while providers can receive patient health notifications and easily review data to prepare treatment plans based on patient priorities. **Asymmetrik:** Asymmetrik’s Phoenix Integration PlatformTMallows healthcare providers to use FHIR®to leverage existing investments in data infrastructure, simplify data analytics to make informed decisions across health systems, and deploy secure, data-driven applications for researchers, doctors, and patients. Built on the award-winning Phoenix ServerTM, Asymmetrik’s patient engagement solution can help healthcare providers improve patient engagement and care coordination by exchanging PRO data with patients, targeting patients with low response rates or certain medical conditions, and analyzing responses in real time. **Booz Allen Hamilton:** DEXi Care MapTMis an application designed to allow patients to interact directly with their providers including inputs from wearables, and has an easy-to-understand interface viewable to both the patient and provider that tracks progress. Using a combination of artificial intelligence (AI), machine intelligence (MI), and natural language processing (NLP), DEXi Care Map can provide tailored information to patients and providers and is able to parse user input and feedback into structured data. **Health Wizz:** The Health WizzTMsolution incentivizes patients to provide sought after data through "Campaigns" (e.g., Chronic Disease Management Campaign, Patient Reported Outcome Campaign, etc.). Participants can choose to accept Campaign invitations and can receive rewards for participation. The data can be entered and tracked on the patient's mobile device or wearable and can be transferred into an EHR system accessible to clinicians. **1upHealth:** ProUpTMis a secure, patient facing application developed by 1upHealth that facilitates seamless integration between EHR systems and PRO data collected via easy to complete PRO questionnaires. **Qidza Inc:** Qidza's ProNogginTMmobile app is designed to improve the collection of PRO data by addressing health literacy gaps via simple designs, increasing patient engagement via personalization, and improving data aggregation and sharing via FHIR integration. **cliexa:** Tailored for tracking chronic disease activity, cliexa®’s Platform for Chronic Care Management enables clinicians to fully leverage patient-reported data through allowing assessment customization, EMR integration, facial and voice recognition using AI, FDA linked databases for medication tracking functions, body maps and pain diagrams, and a reimbursement framework. **PEER Technologies, PLLC:** PEER Technologies utilizes the Shared Decision Making (SDM) process between patients and providers in a user-friendly application that strives to improve patient knowledge and satisfaction through producing benchmarked PRO scores for patients and risk/benefit evaluation when discussing surgery. Patients easily enter PRO data regarding non-specific low back pain and are presented with comprehendible recommendations including personalized estimates of normative outcomes based on patients’ age, sex and other features. **PRISM:** PRISMTMis an application developed by a multi-disciplinary team that enhances the quality of clinical discussion between healthcare providers and patients by providing the option for patients to fill out PRO questions before or during office visits via a mobile application or web site. Patients can compare their health trends to the overall population while providers and healthcare organizations have access to the PRO data from within the EHR. Providers have the ability to send individual reminders to patients and they can use dashboards to track population trends and patient response rates. **Social & Scientific Systems PatientPort:** PatientPortTMis a secure, patient-centric application leveraging a user-friendly interface to maximize engagement. The application is built to provide interoperability with user devices, regardless of operating system and can be integrated with commercially available EHR products to promote adoption by both health systems and patients. The ten Phase 1 winners will now move to Phase 2 Development. The teams will compete for $35,000 for the first place winner, $30,000 for the second place winner, and $25,000 for the third place winner. The applications will be evaluated based on the technical merit, usability and functionality, and deployability of their fully functioning application. - - - #### **Submission Requirements** **Overview of Phase 1 – Proposals** The proposal phase allows participants to present plans and frameworks that demonstrate how they plan to carry out the application development. In this phase, participants should describe the technical, operational, and financial aspects of their proposed approaches. The main goal of Phase 1 is for participants to show the originality, feasibility, and executability of their application development approaches. **Phase 1 Submission Requirements:** * Applications should be in English and all documents uploaded in PDF format * Submit by the deadline of **September 24, 2018** using the online platform:**<https://www.challenge.gov/ahrq-step-up-app-challenge>** * The submission should be made up of, minimally, a title and two uploaded .pdf files. The two files are described in more detail below. You may also optionally include a picture/logo (for instance a logo), and a company’s URL * **Business case (5-page maximum)** * Include an executive summary stating the value proposition: * Describe the specific problem being solved * Analysis of the existing market and market opportunity * Describe how the proposed application will solve the problem * Describe the customers/users * Describe the methods and technologies proposed to develop the application * Describe the business model: * Describe the specific problem being solved * Analysis of the existing market and market opportunity * Describe how the proposed application will solve the problem * Describe the customers/users * Development plan and timeline * Describe key activities and resources required to build the application * Plan to make the application readily available to patients. For example, to be used on existing mobile platforms or deployed on a public-facing website * Define metrics for success (e.g., number of users of the application, money saved by using the application, time saved, increases in patient-reported outcome data exchanges between patients and providers) * Show potential risks and mitigation strategies, including security constraints * Describe participant roles, responsibilities, and capabilities. A multi-disciplinary team is encouraged. * **Include a brief deck presentation PDF of the application and use case(s) to visually present the project (10 slides maximum)** * Content: * Briefly describe the proposed approach and how the participants will develop an application using the FHIR technical specifications and PROMIS® physical functioning measures provided by AHRQ * Explain the competitive advantage of the approach: * Innovative and does not infringe on in-market solutions * Includes an advanced application technology that has the potential to improve the collection and use of PRO data * Designed for patient engagement * Give an example use case * Show proposed development workflow and deliverables * An architecture diagram or framework * The above documents should discuss how you will comply with the Health Insurance Portability and Accountability Act (HIPAA) if applicable. **Overview of Phase 2 – Development** Phase 1 winners will advance to Phase 2 focused on developing the application using the FHIR technical specifications and PROMIS® resources provided by AHRQ. The application should have the capability to do CATs which requires the device to be connected with a server for real-time question calling and scoring (online administration). Participants can use the **[Assessment Center Application Programming Interface](http://www.healthmeasures.net/index.php?option=com_content&view=category&layout=blog&id=170&Itemid=1138)** (API) to build an application that enables CAT. In addition to CAT, participants are highly encouraged to also include the capability to administer the selected PROMIS® short form v1.0 – Physical Function 10a (offline administration). Below are the references to PROMIS® API, CAT, and selected short form. A webinar on how to use the PROMIS® API will be held at the beginning of Phase 2. Register for credentials to use the API: **[https://www.assessmentcenter.net/ac_api](https://protect2.fireeye.com/url?k=908ee238d02fcffe.908ed307-f175b956c34155c8&u=https://www.assessmentcenter.net/ac_api)** Documentation on how to administer PROMIS® items using the API: **<https://www.assessmentcenter.net/ac_api/2012-01/Docs/API_tech_walkthrough.docx>** The goal of Phase 2 is to present a viable application to collect standardized PRO data in ambulatory care settings including both primary care and specialty care practices. Participants need to demonstrate that the app could export standardized data which would facilitate future data integration into an EHR or IT system. Participants are required to undertake user testing of the application during Phase 2. Participants must provide evidence of testing with at least five people. A larger amount of time spent with each tester, greater formal rigor, and the number and diversity of people used for testing will result in a more positive assessment under the judging criteria. Evidence demonstrating user testing could include sample feedback, quotes, or pictures, and should include how it affected the development of the application. **Phase 2 Submission Requirements** The application must be submitted by the deadline, February 15, 2019, in PDF format. The submission should include the following: * Implementation plan (up to 10 pages): * Describe key activities and resources required to deploy and pilot the application. * Refine and incorporate relevant information from business case submission in Phase 1. * Provide evidence of user testing including how it affected the application development (up to 5 pages). * Provide completed Attestation Questionnaire along with a trace log. * Apps must incorporate the following usability, functionality, and technical requirements: * Computerized adaptive testing (CAT). * One or two samples of using the Assessment Center API to do CAT. * Adaptable to PRO domains other than physical functioning (e.g., pain, mental health). * Simple design, easy navigation, and intuitive to use. * Personalized content. * **Not** connected with any social media applications. * **HIPAA compliance if applicable.** * Apps must employ the following device-based security ONLY if the developer chooses to store ANY patient data on the device (username, password, or Face ID are not considered as patient data): * The PRO data must be encrypted at rest, even when the device is unlocked. * Follow specifications and current recommendations for AES-256 bit encryption. * The key must be saved in the Keychain or SharedPreferences for iOS and Android respectively. * Authorization Protocol: * Demonstrate the authorization protocol by using the "OAuth 2.0 Username and Password Grant Flow" and HL7 patient-facing app standalone launch sequence. * Data communication: * Demonstrate the ability to represent the collected PRO data as either JavaScript Object Notation (JSON) or Extensible Markup Language (XML). * Technical ability to do video demonstration of the application via video demonstration to display the application’s device and data security, usability, accessibility, ease of data collection and HIPAA compliance if applicable. * Export and deposit standardized PRO data to a FTP site selected by AHRQ. **Overview of Phase 3 – Pilot Test** The final phase will involve pilot testing the application in “real-life” clinical settings. The grand prize winner in Phase 2 is required to collaborate with MedStar Health to pilot test the application in nine practice settings in the District/Maryland/Virginia area. If the grand prize winner cannot be at the pilot sites physically, MedStar could facilitate remote access for the collaboration. There will be a mix of primary care and specialty care practices. The goal of Phase 3 is to assess the usability of the application. This phase will also test the feasibility of implementation, the scalability of the application, and acceptance by stakeholders. **Phase 3 Requirements** * Work with MedStar Health on the pilot test including setting up the IT infrastructure, modifying the application if needed, and troubleshooting issues during data collection * Pilot Test Period: **Monday, February 25, 2019 – Monday, September 9, 2019** * Demonstrate achievement of milestones to receive the final prize payment * Post an open source version of the application’s code on the GitHub source repository to make the code publicly available * Successful data collection from all pilot sites * Successful export of standardized data at all pilot sites * Submit monthly progress report of the pilot test (25th of every month from Feb – Sept, 2019) * Submit a final report by **September 13, 2019** on lessons learned, recommendations, and concrete next steps for sustainability including commercialization and/or broadened use and how to attract patients and/or providers to adopt and use the application
**Phase 1** Up to 10 winners receive up to $12,000 each. **Phase 2** * 1st place winner receives up to $35,000 * 2nd place winner receives up to $30,000 * 3rd place winner receives up to $25,000 **Phase 3** Grand Prize Winner receives $40,000 after successful pilot test.
#### Eligibility Rules for Participating in the Challenge To be eligible to win a prize under this Challenge, an individual or entity: 1. Shall have registered to participate in the Challenge under the rules promulgated by the Agency for Healthcare Research and Quality. 2. Shall have complied with all the stated requirements of the Step Up App Challenge 3. In the case of a private entity, shall be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States. 4. May not be a Federal entity or Federal employee acting within the scope of their employment. 5. Shall not be an HHS employee working on their applications or Submissions during assigned duty hours. 6. Shall not be an employee of the Agency for Healthcare Research and Quality. 7. Federal grantees may not use Federal funds to develop COMPETES Act Challenge applications unless consistent with the purpose of their grant award. 8. Federal contractors may not use Federal funds from a contract to develop COMPETES Act Challenge applications or to fund efforts in support of a COMPETES Act Challenge Submission. 9. A product may be disqualified if it fails to function as expressed in the description provided by the Participant, or if it provides inaccurate or incomplete information. 10. If applicable, the proposed application must be HIPAA compliant to be eligible for entry into the Challenge. 11. All individual members of a team must meet the eligibility requirements. 12. Shall not be currently under contract with the Office of the National Coordinator for Health Information Technology (ONC) to perform work related to electronic capture of patient-reported outcomes or be a subcontractor under such contract. An individual or entity shall not be deemed ineligible because the individual or entity used Federal facilities or consulted with Federal employees during a Challenge if the facilities and employees are made available to all individuals and entities participating in the Challenge on an equitable basis. By participating in this Challenge, Participants: 1. Agree to assume any and all risks and waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from their participation in this prize contest, whether the injury, death, damage, or loss arises through negligence or otherwise. 2. Agree to obtain liability insurance, or demonstrate financial responsibility, in the amount of $500,000, for claims by (a) a third party for death, bodily injury, or property damage, or loss resulting from an activity carried out in connection with participation in a Challenge, with the Federal Government named as an additional insured under the registered participant’s insurance policy, and (b) the Federal Government for damage or loss to Government property resulting from such activity. 3. Agree to indemnify the Federal Government against third party claims for damages arising from or related to Challenge activities. - - - #### General Submission Requirements In order for a Submission to be eligible to win this Challenge, it must meet the following requirements: 1. No HHS or AHRQ logo—They must not use HHS' or AHRQ's logos or official seals and must not claim endorsement. 2. Functionality/Accuracy—An application may be disqualified if it fails to function as expressed in the description provided by the participant, or if it provides inaccurate or incomplete information. 3. Security—Submissions must be free of malware. Participant agrees that AHRQ may conduct testing on the application(s) to determine whether malware or other security threats may be present. AHRQ may disqualify the application(s) if, in AHRQ's judgment, the application may damage government or others' equipment or operating environment.
#### **Basis Upon Which Winner Will be Selected** **Phase 1 Evaluation Criteria:** * **Team/Participant Capabilities (20%)** * There is appropriate expertise and capability to bring the idea to the development stage. A multi-disciplinary team is encouraged. * The team and/or participant has the resources available to carry out the proposed work * **Impact (30%)** * The proposed Solution has the potential to improve the state of PRO collection, aggregation, and reporting * The proposed Solution describes the value proposition to users * There is a clear plan to make the proposed Solution readily available to patients and providers on existing mobile platforms, or customer-facing websites * **Feasibility (30%)** * The proposed Solution demonstrate a reasonable and credible approach to accomplish the proposed objectives, tasks, outcomes, and deliverables * The proposed Solution addresses a reasonable pathway or timeline to application development * The proposed Solution clearly defines potential risks and mitigation strategies including security constraints * The proposed Solution includes an architecture diagram or framework that is feasible and executable for deployment * **Originality (20%)** * The proposed Solution is innovative and does not infringe on in-market solutions * The proposed Solution includes an advanced application technology that has the potential to improve the collection and use of PRO data * The proposed solution is designed for patient engagement **Phase 2 Evaluation Criteria:** * **Technical Merit (40%)** * The application demonstrates technical compliance regarding * Device-based security * Authentication security * Data communication * FHIR standards provided by AHRQ * Export and deposit standardized PRO data to a FTP site selected by AHRQ * The application's technical methods and frameworks are sound and appropriate for the given problem and user base * The solution utilizes best-practices and draft FHIR technical specifications in the development of the application * **Usability and functionality (30%)** * The application has a simple design, and is intuitive to use and easy to navigate * The application enables users to easily collect, aggregate, and manage PRO data * The application can administer computerized adaptive test (CAT) * The application is not connected with any social media applications * The application was tested with five users and is designed with an understanding of the needs and goals of providers and patients * The application has the capability to adapt and embed a variety of PRO measures other than physical functioning * **Deployability (30%)** * The application can be easily adopted by a healthcare practice (e.g., use open and flexible industry standards) * The team addresses all risks and mitigation strategies faced in a real-life pilot test * The application has a mechanism to capture and leverage user data/feedback in the pilot test * The application has the capacity to scale to multiple healthcare practices * The application adequately describes sustainability plans beyond the completion of this Challenge
Participants can participate by visiting **[www.challenge.gov/ahrq-step-up-app-challenge](http://www.challenge.gov/ahrq-step-up-app-challenge)** and creating a Challenge.gov account. Once logged in, click “Submit” anytime during the submission period stated above. Submission requirements and Challenge information including the judging criteria will be provided at **[www.ahrq.gov/stepupappchallenge](http://www.ahrq.gov/stepupappchallenge)**.
true

Description

Video Message From AHRQ Director Gopal Khanna About Entering the Step Up App Challenge

<iframe src="https://www.youtube.com/embed/JL69rrgciMs" width="560" height="315" frameborder="0" allowfullscreen=""></iframe>

 

***Miss the Step Up App Challenge: Phase 1 Informational Webinar? Check out the slides presented during the August 23 webinar***

 

The Background

Patient-reported outcomes (PROs) are vital to understanding the complex needs of a patient. For this challenge, PRO data are defined as “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or other medical expert.” In other words, PRO data come from patient self-assessments and look beyond the clinical data to assess how a patient feels about his or her health.

The Problem

While PRO data have proven useful to healthcare providers, they are not widely used in clinical settings. Recently, some digital tools have been developed to streamline the collection of PRO data. However, those digital tools are not widely adopted due to challenges related to clinician workflow and patient ease with the tools.  Also, researchers who want to analyze PRO data across different practices or health systems often encounter the issue that data are not collected in a standardized manner. Creating a digital tool to efficiently collect, aggregate, and share PRO data is critical to advancing the quality of care.

Phase 1 Winners

The Agency for Healthcare Research and Quality (AHRQ) Step Up App Challenge is seeking innovative technologists to develop user-friendly apps that collect standardized PRO data for clinical and research purposes.

Phase 1 winners announced: 

AgilisIT: AgilisITTM's solution focuses on patient engagement, ease of use for physicians, and EHR integration in order to encourage the digital collection of PRO data. The patient centered app offers both mobile and web applications, enabling patients to receive reminders and track their own progress while providers can receive patient health notifications and easily review data to prepare treatment plans based on patient priorities. 

Asymmetrik: Asymmetrik’s Phoenix Integration PlatformTM allows healthcare providers to use FHIR® to leverage existing investments in data infrastructure, simplify data analytics to make informed decisions across health systems, and deploy secure, data-driven applications for researchers, doctors, and patients. Built on the award-winning Phoenix ServerTM, Asymmetrik’s patient engagement solution can help healthcare providers improve patient engagement and care coordination by exchanging PRO data with patients, targeting patients with low response rates or certain medical conditions, and analyzing responses in real time.

Booz Allen Hamilton: DEXi Care MapTM is an application designed to allow patients to interact directly with their providers including inputs from wearables, and has an easy-to-understand interface viewable to both the patient and provider that tracks progress. Using a combination of artificial intelligence (AI), machine intelligence (MI), and natural language processing (NLP), DEXi Care Map can provide tailored information to patients and providers and is able to parse user input and feedback into structured data.

Health Wizz:  The Health WizzTM solution incentivizes patients to provide sought after data through "Campaigns" (e.g., Chronic Disease Management Campaign, Patient Reported Outcome Campaign, etc.). Participants can choose to accept Campaign invitations and can receive rewards for participation. The data can be entered and tracked on the patient's mobile device or wearable and can be transferred into an EHR system accessible to clinicians.

1upHealth: ProUpTM is a secure, patient facing application developed by 1upHealth that facilitates seamless integration between EHR systems and PRO data collected via easy to complete PRO questionnaires.

Qidza Inc: Qidza's ProNogginTM mobile app is designed to improve the collection of PRO data by addressing health literacy gaps via simple designs, increasing patient engagement via personalization, and improving data aggregation and sharing via FHIR integration.

cliexa: Tailored for tracking chronic disease activity, cliexa®’s Platform for Chronic Care Management enables clinicians to fully leverage patient-reported data through allowing assessment customization, EMR integration, facial and voice recognition using AI, FDA linked databases for medication tracking functions, body maps and pain diagrams, and a reimbursement framework.

PEER Technologies, PLLC: PEER Technologies utilizes the Shared Decision Making (SDM) process between patients and providers in a user-friendly application that strives to improve patient knowledge and satisfaction through producing benchmarked PRO scores for patients and risk/benefit evaluation when discussing surgery. Patients easily enter PRO data regarding non-specific low back pain and are presented with comprehendible recommendations including personalized estimates of normative outcomes based on patients’ age, sex and other features. 

PRISM: PRISMTM is an application developed by a multi-disciplinary team that enhances the quality of clinical discussion between healthcare providers and patients by providing the option for patients to fill out PRO questions before or during office visits via a mobile application or web site. Patients can compare their health trends to the overall population while providers and healthcare organizations have access to the PRO data from within the EHR.  Providers have the ability to send individual reminders to patients and they can use dashboards to track population trends and patient response rates.

Social & Scientific Systems PatientPort: PatientPortTM is a secure, patient-centric application leveraging a user-friendly interface to maximize engagement. The application is built to provide interoperability with user devices, regardless of operating system and can be integrated with commercially available EHR products to promote adoption by both health systems and patients. 

The ten Phase 1 winners will now move to Phase 2 Development. The teams will compete for $35,000 for the first place winner, $30,000 for the second place winner, and $25,000 for the third place winner. The applications will be evaluated based on the technical merit, usability and functionality, and deployability of their fully functioning application.

Submission Requirements

Overview of Phase 1 – Proposals

The proposal phase allows participants to present plans and frameworks that demonstrate how they plan to carry out the application development. In this phase, participants should describe the technical, operational, and financial aspects of their proposed approaches. The main goal of Phase 1 is for participants to show the originality, feasibility, and executability of their application development approaches.

Phase 1 Submission Requirements:

  • Applications should be in English and all documents uploaded in PDF format
  • Submit by the deadline of September 24, 2018 using the online platform: https://www.challenge.gov/ahrq-step-up-app-challenge
  • The submission should be made up of, minimally, a title and two uploaded .pdf files. The two files are described in more detail below. You may also optionally include a picture/logo (for instance a logo), and a company’s URL
  • Business case (5-page maximum)
    • Include an executive summary stating the value proposition:
      • Describe the specific problem being solved
      • Analysis of the existing market and market opportunity
      • Describe how the proposed application will solve the problem
      • Describe the customers/users
    • Describe the methods and technologies proposed to develop the application
    • Describe the business model:
      • Describe the specific problem being solved
      • Analysis of the existing market and market opportunity
      • Describe how the proposed application will solve the problem
      • Describe the customers/users
    • Development plan and timeline
      • Describe key activities and resources required to build the application
      • Plan to make the application readily available to patients. For example, to be used on existing mobile platforms or deployed on a public-facing website
    • Define metrics for success (e.g., number of users of the application, money saved by using the application, time saved, increases in patient-reported outcome data exchanges between patients and providers)
    • Show potential risks and mitigation strategies, including security constraints
    • Describe participant roles, responsibilities, and capabilities. A multi-disciplinary team is encouraged.
  • Include a brief deck presentation PDF of the application and use case(s) to visually present the project (10 slides maximum)
    • Content:
      • Briefly describe the proposed approach and how the participants will develop an application using the FHIR technical specifications and PROMIS® physical functioning measures provided by AHRQ
      • Explain the competitive advantage of the approach:
        • Innovative and does not infringe on in-market solutions
        • Includes an advanced application technology that has the potential to improve the collection and use of PRO data
        • Designed for patient engagement
      • Give an example use case
      • Show proposed development workflow and deliverables
      • An architecture diagram or framework
      •  
  • The above documents should discuss how you will comply with the Health Insurance Portability and Accountability Act (HIPAA) if applicable.

Overview of Phase 2 – Development

Phase 1 winners will advance to Phase 2 focused on developing the application using the FHIR technical specifications and PROMIS® resources provided by AHRQ. The application should have the capability to do CATs which requires the device to be connected with a server for real-time question calling and scoring (online administration). Participants can use the Assessment Center Application Programming Interface (API) to build an application that enables CAT. In addition to CAT, participants are highly encouraged to also include the capability to administer the selected PROMIS® short form v1.0 – Physical Function 10a (offline administration). Below are the references to PROMIS® API, CAT, and selected short form. A webinar on how to use the PROMIS® API will be held at the beginning of Phase 2.

Register for credentials to use the API: https://www.assessmentcenter.net/ac_api

Documentation on how to administer PROMIS® items using the API: https://www.assessmentcenter.net/ac_api/2012-01/Docs/API_tech_walkthrough.docx

The goal of Phase 2 is to present a viable application to collect standardized PRO data in ambulatory care settings including both primary care and specialty care practices. Participants need to demonstrate that the app could export standardized data which would facilitate future data integration into an EHR or IT system. Participants are required to undertake user testing of the application during Phase 2. Participants must provide evidence of testing with at least five people. A larger amount of time spent with each tester, greater formal rigor, and the number and diversity of people used for testing will result in a more positive assessment under the judging criteria. Evidence demonstrating user testing could include sample feedback, quotes, or pictures, and should include how it affected the development of the application.

 Phase 2 Submission Requirements

The application must be submitted by the deadline, February 15, 2019, in PDF format. The submission should include the following:

  • Implementation plan (up to 10 pages):
    • Describe key activities and resources required to deploy and pilot the application.
    • Refine and incorporate relevant information from business case submission in Phase 1.
  • Provide evidence of user testing including how it affected the application development (up to 5 pages).
  • Provide completed Attestation Questionnaire along with a trace log.
  • Apps must incorporate the following usability, functionality, and technical requirements:
    • Computerized adaptive testing (CAT).
    • One or two samples of using the Assessment Center API to do CAT.
    • Adaptable to PRO domains other than physical functioning (e.g., pain, mental health).
    • Simple design, easy navigation, and intuitive to use.
    • Personalized content.
    • Not connected with any social media applications.
    • HIPAA compliance if applicable.
  • Apps must employ the following device-based security ONLY if the developer chooses to store ANY patient data on the device (username, password, or Face ID are not considered as patient data):
    • The PRO data must be encrypted at rest, even when the device is unlocked.
    • Follow specifications and current recommendations for AES-256 bit encryption.
    • The key must be saved in the Keychain or SharedPreferences for iOS and Android respectively.
  • Authorization Protocol:
    • Demonstrate the authorization protocol by using the "OAuth 2.0 Username and Password Grant Flow" and HL7 patient-facing app standalone launch sequence.
  • Data communication:
    • Demonstrate the ability to represent the collected PRO data as either JavaScript Object Notation (JSON) or Extensible Markup Language (XML).
  • Technical ability to do video demonstration of the application via video demonstration to display the application’s device and data security, usability, accessibility, ease of data collection and HIPAA compliance if applicable.
  • Export and deposit standardized PRO data to a FTP site selected by AHRQ.

 

Overview of Phase 3 – Pilot Test

The final phase will involve pilot testing the application in “real-life” clinical settings. The grand prize winner in Phase 2 is required to collaborate with MedStar Health to pilot test the application in nine practice settings in the District/Maryland/Virginia area. If the grand prize winner cannot be at the pilot sites physically, MedStar could facilitate remote access for the collaboration. There will be a mix of primary care and specialty care practices. The goal of Phase 3 is to assess the usability of the application. This phase will also test the feasibility of implementation, the scalability of the application, and acceptance by stakeholders.

 Phase 3 Requirements

  • Work with MedStar Health on the pilot test including setting up the IT infrastructure, modifying the application if needed, and troubleshooting issues during data collection
    • Pilot Test Period: Monday, February 25, 2019 – Monday, September 9, 2019
  • Demonstrate achievement of milestones to receive the final prize payment
    • Post an open source version of the application’s code on the GitHub source repository to make the code publicly available
    • Successful data collection from all pilot sites
    • Successful export of standardized data at all pilot sites
  • Submit monthly progress report of the pilot test (25th of every month from Feb – Sept, 2019)
  • Submit a final report by September 13, 2019 on lessons learned, recommendations, and concrete next steps for sustainability including commercialization and/or broadened use and how to attract patients and/or providers to adopt and use the application

Prizes

Phase 1

Up to 10 winners receive up to $12,000 each.

Phase 2

  • 1st place winner receives up to $35,000
  • 2nd place winner receives up to  $30,000
  • 3rd place winner receives up to $25,000

Phase 3

Grand Prize Winner receives $40,000 after successful pilot test.

Rules

Eligibility Rules for Participating in the Challenge

To be eligible to win a prize under this Challenge, an individual or entity:

  1. Shall have registered to participate in the Challenge under the rules promulgated by the Agency for Healthcare Research and Quality.
  2. Shall have complied with all the stated requirements of the Step Up App Challenge
  3. In the case of a private entity, shall be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States.
  4. May not be a Federal entity or Federal employee acting within the scope of their employment.
  5. Shall not be an HHS employee working on their applications or Submissions during assigned duty hours.
  6. Shall not be an employee of the Agency for Healthcare Research and Quality.
  7. Federal grantees may not use Federal funds to develop COMPETES Act Challenge applications unless consistent with the purpose of their grant award.
  8. Federal contractors may not use Federal funds from a contract to develop COMPETES Act Challenge applications or to fund efforts in support of a COMPETES Act Challenge Submission.
  9. A product may be disqualified if it fails to function as expressed in the description provided by the Participant, or if it provides inaccurate or incomplete information.
  10. If applicable, the proposed application must be HIPAA compliant to be eligible for entry into the Challenge.
  11. All individual members of a team must meet the eligibility requirements.
  12. Shall not be currently under contract with the Office of the National Coordinator for Health Information Technology (ONC) to perform work related to electronic capture of patient-reported outcomes or be a subcontractor under such contract.

An individual or entity shall not be deemed ineligible because the individual or entity used Federal facilities or consulted with Federal employees during a Challenge if the facilities and employees are made available to all individuals and entities participating in the Challenge on an equitable basis.

By participating in this Challenge, Participants:

  1. Agree to assume any and all risks and waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from their participation in this prize contest, whether the injury, death, damage, or loss arises through negligence or otherwise.
  2. Agree to obtain liability insurance, or demonstrate financial responsibility, in the amount of $500,000, for claims by (a) a third party for death, bodily injury, or property damage, or loss resulting from an activity carried out in connection with participation in a Challenge, with the Federal Government named as an additional insured under the registered participant’s insurance policy, and (b) the Federal Government for damage or loss to Government property resulting from such activity.
  3. Agree to indemnify the Federal Government against third party claims for damages arising from or related to Challenge activities.

General Submission Requirements

In order for a Submission to be eligible to win this Challenge, it must meet the following requirements:

  1. No HHS or AHRQ logo—They must not use HHS' or AHRQ's logos or official seals and must not claim endorsement.
  2. Functionality/Accuracy—An application may be disqualified if it fails to function as expressed in the description provided by the participant, or if it provides inaccurate or incomplete information.
  3. Security—Submissions must be free of malware. Participant agrees that AHRQ may conduct testing on the application(s) to determine whether malware or other security threats may be present. AHRQ may disqualify the application(s) if, in AHRQ's judgment, the application may damage government or others' equipment or operating environment.

Judging Criteria

Basis Upon Which Winner Will be Selected

Phase 1 Evaluation Criteria:

  • Team/Participant Capabilities (20%)
    • There is appropriate expertise and capability to bring the idea to the development stage. A multi-disciplinary team is encouraged.
    • The team and/or participant has the resources available to carry out the proposed work
  • Impact (30%)
    • The proposed Solution has the potential to improve the state of PRO collection, aggregation, and reporting
    • The proposed Solution describes the value proposition to users
    • There is a clear plan to make the proposed Solution readily available to patients and providers on existing mobile platforms, or customer-facing websites
  • Feasibility (30%)
    • The proposed Solution demonstrate a reasonable and credible approach to accomplish the proposed objectives, tasks, outcomes, and deliverables
    • The proposed Solution addresses a reasonable pathway or timeline to application development
    • The proposed Solution clearly defines potential risks and mitigation strategies including security constraints
    • The proposed Solution includes an architecture diagram or framework that is feasible and executable for deployment
  • Originality (20%)
    • The proposed Solution is innovative and does not infringe on in-market solutions
    • The proposed Solution includes an advanced application technology that has the potential to improve the collection and use of PRO data
    • The proposed solution is designed for patient engagement

Phase 2 Evaluation Criteria:

  • Technical Merit (40%)
    • The application demonstrates technical compliance regarding
      • Device-based security
      • Authentication security
      • Data communication
      • FHIR standards provided by AHRQ
      • Export and deposit standardized PRO data to a FTP site selected by AHRQ
    • The application's technical methods and frameworks are sound and appropriate for the given problem and user base
    • The solution utilizes best-practices and draft FHIR technical specifications in the development of the application
  • Usability and functionality (30%)
    • The application has a simple design, and is intuitive to use and easy to navigate
    • The application enables users to easily collect, aggregate, and manage PRO data
    • The application can administer computerized adaptive test (CAT)
    • The application is not connected with any social media applications
    • The application was tested with five users and is designed with an understanding of the needs and goals of providers and patients
    • The application has the capability to adapt and embed a variety of PRO measures other than physical functioning
  • Deployability (30%)
    • The application can be easily adopted by a healthcare practice (e.g., use open and flexible industry standards)
    • The team addresses all risks and mitigation strategies faced in a real-life pilot test
    • The application has a mechanism to capture and leverage user data/feedback in the pilot test
    • The application has the capacity to scale to multiple healthcare practices
    • The application adequately describes sustainability plans beyond the completion of this Challenge

How To Enter

Participants can participate by visiting www.challenge.gov/ahrq-step-up-app-challenge and creating a Challenge.gov account. Once logged in, click “Submit” anytime during the submission period stated above. Submission requirements and Challenge information including the judging criteria will be provided at www.ahrq.gov/stepupappchallenge.