Skip to content

Latest commit

 

History

History
605 lines (342 loc) · 45.1 KB

1137-gsa-artificial-intelligence-and-machine-learning-eula-challenge-2020.md

File metadata and controls

605 lines (342 loc) · 45.1 KB
title challenge-title layout permalink challenge-id status sidenav card-image agency-logo tagline agency total-prize-offered-cash type-of-challenge submission-start submission-end fiscal-year legal-authority challenge-manager challenge-manager-email point-of-contact description prizes rules judging how-to-enter
GSA Artificial Intelligence and Machine Learning End-User License Agreement Challenge 2020
GSA Artificial Intelligence and Machine Learning End-User License Agreement Challenge 2020
front-matter-data-markdownify-content
/challenge/GSA-artificial-intelligence-AI-machine-learning-ML-challenge/
1137
closed
true
/assets/netlify-uploads/gsa-ai-ml-eula-graphic-2.jpg
/assets/netlify-uploads/1198px-US-GeneralServicesAdministration-Logo.png
Create an artificial intelligence or machine learning solution to aid in the review of end-user license agreements.
General Services Administration
$20,000
Software and apps
07/06/2020 10:00 AM
08/20/2020 5:00 PM
FY20
America COMPETES Act
Cindy Smith
*"Emerging technologies such as artificial intelligence and machine learning are paving the way to automate business processes ... Let us ride the digital wave of transformation together." -- Keith Nakasone, Deputy Assistant Commissioner for Acquisition, General Services Administration (GSA)* *"AI & ML leveraged successfully are force multipliers; transitioning historically resource intensive activities into high-speed-low-drag transactions.  Huge thank you in advance for your participation and contributions!" -- Sean C. Zerges, Director, Office of the Chief Technology Officer, GSA* The goal of this challenge is to develop an artificial intelligence (AI), or machine learning (ML) solution that will review end-user license agreements (EULA) for terms and conditions that are unacceptable to the government. On average it takes all parties involved approximately 7-14 days to review an EULA and ensure that all unacceptable terms and conditions are identified. A EULA details the rights and restrictions which apply to the use of software or services. It can also be known as a software license agreement or acceptable use policy. As part of the acquisition process of software or services, GSA reviews the associated EULAs. This review must be completed prior to awarding new contracts or modifying existing contracts. A GSA contracting officer (CO) reviews applicable EULAs for terms and conditions that are not in accordance with Federal law and regulations. The CO may also coordinate a legal review with the Office of General Counsel if they feel it is warranted. Should EULAs contain language that would conflict with Federal law and regulations, the CO must negotiate changes to the EULA to remove the problematic language. We are looking for a solution that will use AI and/or ML to improve this manual process. The solution will include a user interface that GSA will use to process the documents and identify unacceptable clauses/terms in the EULAs. **[Watch our AI / ML challenge video](https://youtu.be/uOBFIqoOJ8k) to learn more about the desired solution.** This solution will decrease the time spent manually reviewing EULAs and free resources to focus on other aspects of the acquisition process. It will also improve the accuracy and consistency of the review process. #### Data Provided: All Solvers (individuals, teams, or legal entities) shall have access to the following data for use in creating their solutions. All the items below will be available at: <https://github.com/GSA/ai-ml-Challenge-2020> * Training data * Additional reference documents * Sample validation file #### Key Dates: * Registration begins: 10 a.m. ET, July 6, 2020 * Submit solutions before: 5 p.m. ET, August 20, 2020 * Semi-finalists announced: 10 a.m. ET, September 11, 2020 * Winner announced: 10 a.m. ET, September 18, 2020
GSA anticipates awarding cash prizes, for a $20,000.00 total prize purse. After all submissions have been received, our judges will review all submissions and select the top three (3) solutions. Each solution selected will be eligible to win a cash prize of $2,500.00 per team. The top three (3) solutions will again be reviewed by our judges and subject matter experts in accordance with the judging criteria outlined below. One (1) team may be selected as the winner and an additional cash prize of $12,500.00 may be awarded. Prizes will be distributed to the eligible winners via electronic funds transfer (EFT), within 90 business days of the announcement of the winner(s). GSA will distribute the funds evenly to all team members. Eligibility to receive an award is based upon the participant’s adherence to all rules for this Challenge. Funding for this Challenge will come from GSA. * 1st place team: $15,000.00 * 2nd place team: $2,500.00 * 3rd place team: $2,500.00 We are proud to announce the final winners of our AI/ML Challenge 2020. They are: * 1st Place: Dev Technology * 2nd Place: Gaussian Solutions * 3rd Place: Team SoKat
#### Eligibility to Participate: * Each challenge solver (individual, team, or legal entity) is required to register by sending an email to [[email protected]](mailto:[email protected]) with the subject line stating ‘Challenge 2020 Registration’. * There shall be one official representative for each challenge registration. The official representative must provide a name and email address and, by sending the registration, affirms on behalf of the solver (individual, team, or legal entity), that he or she has read and consents to be governed by the challenge rules. * There shall be no more than 10 individuals per team. * At the time of registration, all solvers including the official representative (whether an individual, team, or legal entity) must be age 18 or older. * As determined by GSA, any violation of these rules will be grounds for disqualification from the challenge. * Multiple individuals and/or legal entities may collaborate as a solver team to submit a single entry, in which case the designated Official Representative will be responsible for meeting all entry and evaluation requirements. * Participation is subject to all U.S. federal, state, local, and country of residence laws and regulations. * Solvers must not be suspended, debarred, or otherwise excluded from doing business with the federal government. An individual or entity that is determined to be on the GSA Excluded Parties List (www.sam.gov) is ineligible to participate. Individuals entering on behalf of or representing a company, institution or other legal entity are also responsible for confirming that their entry does not violate any policies of that company, institution or legal entity. * As a condition of participation, all solvers must agree to indemnify the federal government against third-party claims for damages arising from or related to challenge activities. Entrants are not required to obtain liability insurance or demonstrate financial responsibility in order to participate in the challenge. By entering the challenge, solvers agree to hold GSA harmless from all legal and administrative claims to include associated expenses that may arise from any claims related to their challenge solution or its use. * Federal grantees may not use federal funds to develop challenge solutions unless consistent with the purpose of their grant award. * Federal contractors may not use federal funds from a contract to develop challenge solutions or to fund efforts in support of a challenge entry. * Solvers may not be a federal entity or federal employee acting within the scope of their employment. * Federal government employees are not eligible to participate as solvers in this challenge. #### Eligibility to Win a Cash Prize: To be eligible for a cash prize: * A solver (whether an individual, team, or legal entity), through one official representative, must have registered to participate and complied with all of the requirements under section 3719 of title 15, United States Code and the Challenge rules. * At the time of registration, all Solvers including the official representative (individual, team, or legal entity) must be age 18 or older. * An individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States. * In the case of a private entity that is participating as a solver or as part of a solver team, the business must be incorporated in and maintain a primary place of business in the United States or its territories. * Payments to the winning teams will be made via electronic funds transfer (EFT). All EFT forms must be completed and submitted to Cindy Smith ([[email protected]](mailto:[email protected])) by September 18, 2020 for processing. Failure to return the completed EFT form by the date stated above may be a basis for disqualification of a cash prize winning entry. * A solver shall not be deemed ineligible because the solver consulted with federal employees in preparing its submission for this GSA challenge if the federal employees are made available to all solvers on an equitable basis. Solvers may use any public-facing websites or publicly available federal government resources, as well as all resources provided to the challenge solvers on an equitable basis. * The official representative for each challenge entry shall follow the steps listed below under "Submission Guidelines" in advance of the submission end date (08/20/2020 5:00 PM ET). #### Terms and Conditions * This challenge shall be performed in accordance with the America COMPETES Reauthorization Act of 2010, Pub. Law 111-358, Title I, § 105(a), Jan. 4, 2011, codified at 15 U.S.C. § 3719, as amended. * All challenges are subject to all applicable federal laws and regulations. Participation constitutes full and unconditional agreement to these official rules and administrative decisions, which are final and binding in all matters related to the challenge. * Eligibility for a cash prize is contingent upon fulfilling all requirements set forth herein. This notice is not an obligation of funds; the final award of prizes is contingent upon the availability of appropriations and receiving suitable entries. * This GSA challenge is voluntary and open to all entities that meet the eligibility requirements. There may be only one submission (“Submission”) per eligible entity. Submissions must be received before the deadline indicated (August 20, 2020, 5:00 PM ET), if any submissions are received after the deadline they will not be considered. * A Challenge entry constitutes an agreement to adhere to the challenge rules, terms and conditions set forth by the challenge sponsor, GSA. * Solvers must meet the eligibility requirements described in the Eligibility section, to participate and/or win a cash prize. * Any solvers or entry found in violation of any rule will be disqualified at GSA’s sole discretion. * Each individual or solver team certifies, through entry to the contest, that the entry is his/her own original, creative work and does not violate or infringe upon the creative work of others, as protected under applicable intellectual property (IP) law. * Each solver certifies, through entry to the challenge, that any submission by the official representative does not contain any harmful computer code (sometimes referred to as “malware,” “viruses,” or “worms”). * By entering the challenge, the solvers agree to hold GSA harmless from all legal and administrative claims to include associated expenses that may arise from any claims related to their entry or its use. * All evaluation panel decisions are final and may not be appealed. * All cash prizes awarded by GSA to solvers are subject to tax liabilities, and no withholding will be assessed by GSA on behalf of the solver claiming a cash prize. * GSA reserves the right for any reason, including but not limited to an insufficient number of qualified entries, to modify or cancel the challenge at any time during the duration of the challenge. * All solvers agree that they, their heirs and estates agree to assume any and all risks and waive claims against the federal government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from their participation in a prize challenge, whether the injury, death, damage, or loss arises through negligence or otherwise. * By participating in the challenge, each solver agrees to comply with and abide by all GSA challenge rules, terms, and conditions, and the decisions of GSA and/or the individual judges, which shall be final and binding in all respects. #### Ownership Any applicable ownership of IP in the submission will remain with the solver. By participating in the challenge the solver is not transferring any exclusive IP rights in applicable patents, pending patent applications, or copyrights in connection with the submission. However, by entering the submission, the solver agrees to grant the federal government certain license rights, as set forth in this section. Namely, the solver grants the government the right to review the submission, to publicly describe the submission in any materials created in connection with this challenge, to screen and evaluate the submission, and to have the judges, the Challenge administrators, and the designees of any of them review the submission. The government is also granted the right to publicize Solver names and, as applicable, the names of solver team members and/or organizations that participated in the submission following the conclusion of the challenge. In addition, a solver that receives a cash prize from this challenge agrees to grant the government the following license rights in the submission: 1. a non-exclusive, non-transferable, irrevocable, paid-up, royalty-free license to practice or have practiced for or on the government’s behalf, throughout the world, any invention created by the Solver that covers the Submission; and 2. a non-exclusive, non-transferable, irrevocable, paid-up, royalty-free license to reproduce, distribute publicly, prepare derivative works, and publicly perform and display the submission by or on behalf of the government, worldwide in any format, whether electronically or in print. By participating in the challenge, each solver (individual, team, or legal entity) represents and warrants that he or she is the sole author or owner of IP in connection with the submission, or otherwise has the necessary rights to use the submission for purposes of the challenge, including having any and all rights necessary to grant the license rights identified in this section. Each solver further represents and warrants that the Submission does not infringe any copyright or any other rights of any third party of which the solver is aware. To participate in the challenge, each solver warrants that there are no legal obstacles to granting the license rights of this section to the government. Use of open source, unless otherwise stated in the challenge rules above, if open source code is used in the model to generate a solution, then you must only use open source code licensed under an open source initiative-approved license (see [www.opensource.org](http://www.opensource.org)) that in no event limits commercial use of such code or model containing or depending on such code. The winners of a cash prize for the Challenge (collectively, "Winners") may be featured on Federal websites, in newsletters, social media, and in other print and electronic outreach materials. Except where prohibited, participation in the challenge constitutes the consent of each winner to the government’s and its agents’ use of each winner’s name, likeness, photograph, voice, opinions, public summary, and/or hometown and state information for promotional purposes through any form of media, worldwide, without further permission, payment, or consideration. Finally, the government will continue to communicate the resulting winner’s solution only if it continues to deliver mutual benefit to the government in achieving its mission to provide relevant services. In order to measure this and continue to improve programs and services, the Solver or Solver team must share platform use data with the government.
#### Judging Panel Submissions that comply with the challenge requirements will be reviewed by a panel of judges consisting of Federal employees. Entries will be scored by a panel of judges representing a variety of fields, including the Director of AI and Data, Innovation Specialist, AI/ML Acquisition Executive Sponsor and Industry Specialists. #### Judging Criteria Each submission will be judged based on the following criteria with each judge independently scoring the solution on a scale of 1-5 and then calculated into a weighted percentage as detailed below. The winner will be determined by averaging all judges’ scores and calculating the highest scoring solution. Scoring rubric for judges: [AI/ML Challenge Scoring Rubric](<https://github.com/GSA/ai-ml-challenge-2020/blob/master/reference/AI_ML Challenge Scoring Rubric.pdf>) **Technical Evaluation (40%)** * The solution classifies clauses as acceptable or unacceptable using the validation data file and provides a probability of each classification. The solution will be validated based on a combination of: * The self-reported metrics in the ‘Description of Methods’ document. * GSA processing of the Validation File to calculate the Brier Score and F1 Score. * The Description of Methods provides a comprehensive description and demonstration of the data, methods, and software used in the solution. * The Description of Methods clearly explains the reasons for predictions made using the validation data. **Functionality & User Interface (30%)** * The solution allows the user to provide a EULA document in MS Word or PDF format. * The solution successfully parses documents into individual clauses for evaluation. * For each individual clause, the solution reports: * A classification of acceptable or unacceptable. * A prediction probability for each clause. * The user interface of the solution is realistic and usable by a business user. **Creativity and Innovation (10%)** * The solution provides an innovative approach to solve the issues presented. * The solution provides creative solutions to overcome limited training data available, such as: * Transfer learning from additional GSA provided data. * Incorporation of additional exhibits provided by GSA. * Identifying and using additional public sources of data that are relevant to the solution. **Quality of Demonstration (20%)** * The demonstration of the solution clearly provides a high level view of its functionality, purpose and usage. * The demonstration describes the accuracy of the solution. * The demonstration exhibits an understanding of the goals of the user. * The demonstration exhibits an understanding of difficulty caused by limited training data. * The demonstration is attractive and professional in presentation. * The demonstration uses multimedia, graphics, or video to engage the judges. **Bonus: Advanced techniques (Optional 0-5% points)** * Up to 5% bonus points on the total score for exceptional functionality of greater difficulty, including: * Methods of retraining the model based on user review of solution output. * Methods to explain the reason that individual clauses were classified as unacceptable. **Total Possible Score = 105/100 (including bonus points)**
**Open Date:** Beginning July 6th at 10:00 a.m. ET, the identified official representative (individual, team, or legal entity) may register for this challenge by sending an email to [[email protected]](mailto:[email protected]) with the subject line stating ‘**Challenge 2020 Registration**'. There shall be only one official representative for each challenge registration. The official representative must provide a name and email address and, by sending in the registration, and affirms on behalf of the solver (individual, team, or legal entity), that he or she has read and consents to be governed by the challenge rules. **Duration:** This challenge shall be open for 45 days (July 6 through August 20), and is completely virtual. There will not be any in person meetings held for this challenge. **Close Date:** This challenge will close at 5:00 p.m. ET, August 20; any submissions received after 5 p.m. ET, August 20 will not be accepted. #### Submission Guidelines **Submission: Prior to the closing date, email the following deliverables to [[email protected]](mailto:[email protected]):** * The email subject line must state “GSA Artificial Intelligence (AI) or Machine Learning (ML) EULA Challenge 2020 Submission”. * Email must come from the official representative for the team. * All submissions must be in English. * Solvers must not use GSA, or other government logos or official seals in their submissions and must not otherwise give an appearance of federal government endorsement. * Solution description Document ([template](<https://github.com/GSA/ai-ml-challenge-2020/blob/master/reference/AI_ML_Challenge_Solution Description_Document.docx>)) * Solution demonstration * A document or video that provides a high level view of the functionality and purpose of the solution, along with a demonstration of how it would be used by a business user. * Can include links to a working example of the solution if it is hosted on a publicly accessible location. * If a video is used, it must be in MP4 format. Upload the following deliverables to GitHub following these instructions: <https://github.com/GSA/ai-ml-Challenge-2020/blob/master/README.md>. (Submissions will be publicly viewable.) **1. Validation Data File** * Output file in prescribed format. * Includes classification of clauses contained in Validation File, with the following information: * Clause ID * Prediction: 0 - acceptable, 1 - unacceptable * Probability acceptable (percent) **2. Description of Methods Document** * A document in PDF, MS Word, or Jupyter notebook format. * Provides a comprehensive description of the data, methods and software used to complete the solution. * Provides a demonstration of the process used to complete the model used in the solution, including data inputs and visualizations. * Clearly explains the reasons for predictions made in the Validation Data File submission. * Contains self-reported metrics of the solution, by providing: * Brier score * F1 score (also known as F-Measure) **3. Source Code, Input Data, and Compiled Models** * All source code used in creation of the solution. * All input data used in the creation of the solution. * Compiled versions of models used in the solution. #### Questions Questions regarding the GSA Artificial Intelligence (AI) or Machine Learning (ML) EULA Challenge 2020 can be directed to [[email protected]](mailto:[email protected]). GSA will share questions and answers with all registered participants by email during the challenge.

"Emerging technologies such as artificial intelligence and machine learning are paving the way to automate business processes ... Let us ride the digital wave of transformation together." -- Keith Nakasone, Deputy Assistant Commissioner for Acquisition, General Services Administration (GSA)

"AI & ML leveraged successfully are force multipliers; transitioning historically resource intensive activities into high-speed-low-drag transactions.  Huge thank you in advance for your participation and contributions!" -- Sean C. Zerges, Director, Office of the Chief Technology Officer, GSA

Description

The goal of this challenge is to develop an artificial intelligence (AI), or machine learning (ML) solution that will review end-user license agreements (EULA) for terms and conditions that are unacceptable to the government. On average it takes all parties involved approximately 7-14 days to review an EULA and ensure that all unacceptable terms and conditions are identified.

A EULA details the rights and restrictions which apply to the use of software or services. It can also be known as a software license agreement or acceptable use policy. As part of the acquisition process of software or services, GSA reviews the associated EULAs. This review must be completed prior to awarding new contracts or modifying existing contracts. A GSA contracting officer (CO) reviews applicable EULAs for terms and conditions that are not in accordance with Federal law and regulations. The CO may also coordinate a legal review with the Office of General Counsel if they feel it is warranted. Should EULAs contain language that would conflict with Federal law and regulations, the CO must negotiate changes to the EULA to remove the problematic language.

We are looking for a solution that will use AI and/or ML to improve this manual process. The solution will include a user interface that GSA will use to process the documents and identify unacceptable clauses/terms in the EULAs. Watch our AI / ML challenge video to learn more about the desired solution.

This solution will decrease the time spent manually reviewing EULAs and free resources to focus on other aspects of the acquisition process. It will also improve the accuracy and consistency of the review process.

Data Provided:

All Solvers (individuals, teams, or legal entities) shall have access to the following data for use in creating their solutions. All the items below will be available at: https://github.com/GSA/ai-ml-Challenge-2020

  • Training data
  • Additional reference documents
  • Sample validation file

Key Dates:

  • Registration begins: 10 a.m. ET, July 6, 2020
  • Submit solutions before: 5 p.m. ET, August 20, 2020
  • Semi-finalists announced: 10 a.m. ET, September 11, 2020
  • Winner announced: 10 a.m. ET, September 18, 2020

Prizes

GSA anticipates awarding cash prizes, for a $20,000.00 total prize purse. After all submissions have been received, our judges will review all submissions and select the top three (3) solutions. Each solution selected will be eligible to win a cash prize of $2,500.00 per team.

The top three (3) solutions will again be reviewed by our judges and subject matter experts in accordance with the judging criteria outlined below. One (1) team may be selected as the winner and an additional cash prize of $12,500.00 may be awarded.

Prizes will be distributed to the eligible winners via electronic funds transfer (EFT), within 90 business days of the announcement of the winner(s). GSA will distribute the funds evenly to all team members. Eligibility to receive an award is based upon the participant’s adherence to all rules for this Challenge. Funding for this Challenge will come from GSA.

  • 1st place team: $15,000.00
  • 2nd place team: $2,500.00
  • 3rd place team: $2,500.00

We are proud to announce the final winners of our AI/ML Challenge 2020. They are:

  • 1st Place: Dev Technology
  • 2nd Place: Gaussian Solutions
  • 3rd Place: Team SoKat

Rules

Eligibility to Participate:

  • Each challenge solver (individual, team, or legal entity) is required to register by sending an email to [email protected] with the subject line stating ‘Challenge 2020 Registration’.
  • There shall be one official representative for each challenge registration. The official representative must provide a name and email address and, by sending the registration, affirms on behalf of the solver (individual, team, or legal entity), that he or she has read and consents to be governed by the challenge rules.
  • There shall be no more than 10 individuals per team.
  • At the time of registration, all solvers including the official representative (whether an individual, team, or legal entity) must be age 18 or older.
  • As determined by GSA, any violation of these rules will be grounds for disqualification from the challenge.
  • Multiple individuals and/or legal entities may collaborate as a solver team to submit a single entry, in which case the designated Official Representative will be responsible for meeting all entry and evaluation requirements.
  • Participation is subject to all U.S. federal, state, local, and country of residence laws and regulations.
  • Solvers must not be suspended, debarred, or otherwise excluded from doing business with the federal government. An individual or entity that is determined to be on the GSA Excluded Parties List (www.sam.gov) is ineligible to participate. Individuals entering on behalf of or representing a company, institution or other legal entity are also responsible for confirming that their entry does not violate any policies of that company, institution or legal entity.
  • As a condition of participation, all solvers must agree to indemnify the federal government against third-party claims for damages arising from or related to challenge activities. Entrants are not required to obtain liability insurance or demonstrate financial responsibility in order to participate in the challenge. By entering the challenge, solvers agree to hold GSA harmless from all legal and administrative claims to include associated expenses that may arise from any claims related to their challenge solution or its use.
  • Federal grantees may not use federal funds to develop challenge solutions unless consistent with the purpose of their grant award.
  • Federal contractors may not use federal funds from a contract to develop challenge solutions or to fund efforts in support of a challenge entry.
  • Solvers may not be a federal entity or federal employee acting within the scope of their employment.
  • Federal government employees are not eligible to participate as solvers in this challenge.

Eligibility to Win a Cash Prize:

To be eligible for a cash prize:

  • A solver (whether an individual, team, or legal entity), through one official representative, must have registered to participate and complied with all of the requirements under section 3719 of title 15, United States Code and the Challenge rules.
  • At the time of registration, all Solvers including the official representative (individual, team, or legal entity) must be age 18 or older.
  • An individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States.
  • In the case of a private entity that is participating as a solver or as part of a solver team, the business must be incorporated in and maintain a primary place of business in the United States or its territories.
  • Payments to the winning teams will be made via electronic funds transfer (EFT). All EFT forms must be completed and submitted to Cindy Smith ([email protected]) by September 18, 2020 for processing. Failure to return the completed EFT form by the date stated above may be a basis for disqualification of a cash prize winning entry.
  • A solver shall not be deemed ineligible because the solver consulted with federal employees in preparing its submission for this GSA challenge if the federal employees are made available to all solvers on an equitable basis. Solvers may use any public-facing websites or publicly available federal government resources, as well as all resources provided to the challenge solvers on an equitable basis.
  • The official representative for each challenge entry shall follow the steps listed below under "Submission Guidelines" in advance of the submission end date (08/20/2020 5:00 PM ET).

Terms and Conditions

  • This challenge shall be performed in accordance with the America COMPETES Reauthorization Act of 2010, Pub. Law 111-358, Title I, § 105(a), Jan. 4, 2011, codified at 15 U.S.C. § 3719, as amended.
  • All challenges are subject to all applicable federal laws and regulations. Participation constitutes full and unconditional agreement to these official rules and administrative decisions, which are final and binding in all matters related to the challenge.
  • Eligibility for a cash prize is contingent upon fulfilling all requirements set forth herein. This notice is not an obligation of funds; the final award of prizes is contingent upon the availability of appropriations and receiving suitable entries.
  • This GSA challenge is voluntary and open to all entities that meet the eligibility requirements. There may be only one submission (“Submission”) per eligible entity. Submissions must be received before the deadline indicated (August 20, 2020, 5:00 PM ET), if any submissions are received after the deadline they will not be considered.
  • A Challenge entry constitutes an agreement to adhere to the challenge rules, terms and conditions set forth by the challenge sponsor, GSA.
  • Solvers must meet the eligibility requirements described in the Eligibility section, to participate and/or win a cash prize.
  • Any solvers or entry found in violation of any rule will be disqualified at GSA’s sole discretion.
  • Each individual or solver team certifies, through entry to the contest, that the entry is his/her own original, creative work and does not violate or infringe upon the creative work of others, as protected under applicable intellectual property (IP) law.
  • Each solver certifies, through entry to the challenge, that any submission by the official representative does not contain any harmful computer code (sometimes referred to as “malware,” “viruses,” or “worms”).
  • By entering the challenge, the solvers agree to hold GSA harmless from all legal and administrative claims to include associated expenses that may arise from any claims related to their entry or its use.
  • All evaluation panel decisions are final and may not be appealed.
  • All cash prizes awarded by GSA to solvers are subject to tax liabilities, and no withholding will be assessed by GSA on behalf of the solver claiming a cash prize.
  • GSA reserves the right for any reason, including but not limited to an insufficient number of qualified entries, to modify or cancel the challenge at any time during the duration of the challenge.
  • All solvers agree that they, their heirs and estates agree to assume any and all risks and waive claims against the federal government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from their participation in a prize challenge, whether the injury, death, damage, or loss arises through negligence or otherwise.
  • By participating in the challenge, each solver agrees to comply with and abide by all GSA challenge rules, terms, and conditions, and the decisions of GSA and/or the individual judges, which shall be final and binding in all respects.

Ownership

Any applicable ownership of IP in the submission will remain with the solver. By participating in the challenge the solver is not transferring any exclusive IP rights in applicable patents, pending patent applications, or copyrights in connection with the submission. However, by entering the submission, the solver agrees to grant the federal government certain license rights, as set forth in this section.

Namely, the solver grants the government the right to review the submission, to publicly describe the submission in any materials created in connection with this challenge, to screen and evaluate the submission, and to have the judges, the Challenge administrators, and the designees of any of them review the submission. The government is also granted the right to publicize Solver names and, as applicable, the names of solver team members and/or organizations that participated in the submission following the conclusion of the challenge.

In addition, a solver that receives a cash prize from this challenge agrees to grant the government the following license rights in the submission:

  1. a non-exclusive, non-transferable, irrevocable, paid-up, royalty-free license to practice or have practiced for or on the government’s behalf, throughout the world, any invention created by the Solver that covers the Submission; and
  2. a non-exclusive, non-transferable, irrevocable, paid-up, royalty-free license to reproduce, distribute publicly, prepare derivative works, and publicly perform and display the submission by or on behalf of the government, worldwide in any format, whether electronically or in print.

By participating in the challenge, each solver (individual, team, or legal entity) represents and warrants that he or she is the sole author or owner of IP in connection with the submission, or otherwise has the necessary rights to use the submission for purposes of the challenge, including having any and all rights necessary to grant the license rights identified in this section. Each solver further represents and warrants that the Submission does not infringe any copyright or any other rights of any third party of which the solver is aware. To participate in the challenge, each solver warrants that there are no legal obstacles to granting the license rights of this section to the government.

Use of open source, unless otherwise stated in the challenge rules above, if open source code is used in the model to generate a solution, then you must only use open source code licensed under an open source initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code.

The winners of a cash prize for the Challenge (collectively, "Winners") may be featured on Federal websites, in newsletters, social media, and in other print and electronic outreach materials.

Except where prohibited, participation in the challenge constitutes the consent of each winner to the government’s and its agents’ use of each winner’s name, likeness, photograph, voice, opinions, public summary, and/or hometown and state information for promotional purposes through any form of media, worldwide, without further permission, payment, or consideration.

Finally, the government will continue to communicate the resulting winner’s solution only if it continues to deliver mutual benefit to the government in achieving its mission to provide relevant services. In order to measure this and continue to improve programs and services, the Solver or Solver team must share platform use data with the government.

Judging

Judging Panel

Submissions that comply with the challenge requirements will be reviewed by a panel of judges consisting of Federal employees. Entries will be scored by a panel of judges representing a variety of fields, including the Director of AI and Data, Innovation Specialist, AI/ML Acquisition Executive Sponsor and Industry Specialists.

Judging Criteria

Each submission will be judged based on the following criteria with each judge independently scoring the solution on a scale of 1-5 and then calculated into a weighted percentage as detailed below. The winner will be determined by averaging all judges’ scores and calculating the highest scoring solution.

Scoring rubric for judges: AI/ML Challenge Scoring Rubric

Technical Evaluation (40%)

  • The solution classifies clauses as acceptable or unacceptable using the validation data file and provides a probability of each classification. The solution will be validated based on a combination of:

    • The self-reported metrics in the ‘Description of Methods’ document.
    • GSA processing of the Validation File to calculate the Brier Score and F1 Score.

  • The Description of Methods provides a comprehensive description and demonstration of the data, methods, and software used in the solution.

  • The Description of Methods clearly explains the reasons for predictions made using the validation data.

Functionality & User Interface (30%)

  • The solution allows the user to provide a EULA document in MS Word or PDF format.

  • The solution successfully parses documents into individual clauses for evaluation.

  • For each individual clause, the solution reports:

    • A classification of acceptable or unacceptable.
    • A prediction probability for each clause.

  • The user interface of the solution is realistic and usable by a business user.

Creativity and Innovation (10%)

  • The solution provides an innovative approach to solve the issues presented.

  • The solution provides creative solutions to overcome limited training data available, such as:

    • Transfer learning from additional GSA provided data.
    • Incorporation of additional exhibits provided by GSA.
    • Identifying and using additional public sources of data that are relevant to the solution.

Quality of Demonstration (20%)

  • The demonstration of the solution clearly provides a high level view of its functionality, purpose and usage.
  • The demonstration describes the accuracy of the solution.
  • The demonstration exhibits an understanding of the goals of the user.
  • The demonstration exhibits an understanding of difficulty caused by limited training data.
  • The demonstration is attractive and professional in presentation.
  • The demonstration uses multimedia, graphics, or video to engage the judges.

Bonus: Advanced techniques (Optional 0-5% points)

  • Up to 5% bonus points on the total score for exceptional functionality of greater difficulty, including:

    • Methods of retraining the model based on user review of solution output.
    • Methods to explain the reason that individual clauses were classified as unacceptable.

Total Possible Score = 105/100 (including bonus points)

How to Enter

Open Date: Beginning July 6th at 10:00 a.m. ET, the identified official representative (individual, team, or legal entity) may register for this challenge by sending an email to [email protected] with the subject line stating ‘Challenge 2020 Registration'.

There shall be only one official representative for each challenge registration. The official representative must provide a name and email address and, by sending in the registration, and affirms on behalf of the solver (individual, team, or legal entity), that he or she has read and consents to be governed by the challenge rules.

Duration: This challenge shall be open for 45 days (July 6 through August 20), and is completely virtual. There will not be any in person meetings held for this challenge.

Close Date: This challenge will close at 5:00 p.m. ET, August 20; any submissions received after 5 p.m. ET, August 20 will not be accepted.

Submission Guidelines

Submission: Prior to the closing date, email the following deliverables to [email protected]:

  • The email subject line must state “GSA Artificial Intelligence (AI) or Machine Learning (ML) EULA Challenge 2020 Submission”.

  • Email must come from the official representative for the team.

  • All submissions must be in English.

  • Solvers must not use GSA, or other government logos or official seals in their submissions and must not otherwise give an appearance of federal government endorsement.

  • Solution Description Document (template)

  • Solution Demonstration

    • A document or video that provides a high level view of the functionality and purpose of the solution, along with a demonstration of how it would be used by a business user.
    • Can include links to a working example of the solution if it is hosted on a publicly accessible location.
    • If a video is used, it must be in MP4 format.

Upload the following deliverables to GitHub following these instructions: https://github.com/GSA/ai-ml-Challenge-2020/blob/master/README.md. (Submissions will be publicly viewable.)

1. Validation Data File

  • Output file in prescribed format.

  • Includes classification of clauses contained in Validation File, with the following information:

    • Clause ID
    • Prediction: 0 - acceptable, 1 - unacceptable
    • Probability acceptable (percent)

2. Description of Methods Document

  • A document in PDF, MS Word, or Jupyter notebook format.

  • Provides a comprehensive description of the data, methods and software used to complete the solution.

  • Provides a demonstration of the process used to complete the model used in the solution, including data inputs and visualizations.

  • Clearly explains the reasons for predictions made in the Validation Data File submission.

  • Contains self-reported metrics of the solution, by providing:

    • Brier score
    • F1 score (also known as F-Measure)

3. Source Code, Input Data, and Compiled Models

  • All source code used in creation of the solution.
  • All input data used in the creation of the solution.
  • Compiled versions of models used in the solution.

Questions

Questions regarding the GSA Artificial Intelligence (AI) or Machine Learning (ML) EULA Challenge 2020 can be directed to [email protected]. GSA will share questions and answers with all registered participants by email during the challenge.