initial commit

Co-authored-by: Yu Liu <[email protected]>

.gitignore

@@ -0,0 +1,110 @@
+*~
+\#*\#
+.DS_Store
+_downloads/
+/userstudy/miniconda/
+
+
+# IntelliJ
+.idea
+out/
+
+# mpeltonen/sbt-idea plugin
+.idea_modules/
+
+# JIRA plugin
+atlassian-ide-plugin.xml
+
+# Crashlytics plugin (for Android Studio and IntelliJ)
+com_crashlytics_export_strings.xml
+crashlytics.properties
+crashlytics-build.properties
+fabric.properties
+
+
+### https://raw.github.com/github/gitignore/80a8803b004013d17291196825a327b9e871f009/Global/macOS.gitignore
+
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Icon must end with two \r
+Icon
+
+# Thumbnails
+._*
+
+# Files that might appear in the root of a volume
+.DocumentRevisions-V100
+.fseventsd
+.Spotlight-V100
+.TemporaryItems
+.Trashes
+.VolumeIcon.icns
+.com.apple.timemachine.donotpresent
+
+# Directories potentially created on remote AFP share
+.AppleDB
+.AppleDesktop
+Network Trash Folder
+Temporary Items
+.apdisk
+
+### https://raw.github.com/github/gitignore/80a8803b004013d17291196825a327b9e871f009/Global/VisualStudioCode.gitignore
+# .vscode
+workspace.code-workspace
+
+# Compiled class file
+*.class
+
+# Log file
+*.log
+
+# BlueJ files
+*.ctxt
+
+# Mobile Tools for Java (J2ME)
+.mtj.tmp/
+
+# Package Files #
+*.jar
+*.war
+*.nar
+*.ear
+*.zip
+*.tar.gz
+*.rar
+
+# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
+hs_err_pid*
+replay_pid*
+
+target/
+pom.xml.tag
+pom.xml.releaseBackup
+pom.xml.versionsBackup
+pom.xml.next
+release.properties
+dependency-reduced-pom.xml
+buildNumber.properties
+.mvn/timing.properties
+# https://github.com/takari/maven-wrapper#usage-without-binary-jar
+.mvn/wrapper/maven-wrapper.jar
+
+# Eclipse m2e generated files
+# Eclipse Core
+.project
+# JDT-specific (Eclipse Java Development Tools)
+.classpath
+
+*.iml
+
+__pycache__/
+
+python/cov.xml
+python/.coverage
+
+.pytest_cache
+
+.class

INSTALL.md

@@ -0,0 +1,148 @@
+# Installation Guide
+
+I-Test currently supports two programming languages, Python and Java.
+We utilize the package management systems
+([conda][conda-webpage] for Python and
+[sdkman][sdkman-webpage] for Java) to install the necessary
+dependencies for I-Test itself, for our experiment scripts, and for
+running the unit tests of other open-source projects in our integrated
+experiments.
+
+This document will guide you through the following steps (with some
+usage examples to verify if the installation is successful):
+- install the package management systems
+- install the I-Test framework
+- install the environment for running our experiment scripts
+
+
+## System Requirements
+
+The minimum requirements for replicating our experiments are:
+- a Linux operating system (MacOS not guaranteed to work)
+- at least 10GB of free disk space
+
+Note that I-Test framework itself is not limited to Linux; we have
+tested the I-Test framework on a MacOS machine (with MacOS 10.15.7).
+However, a part of our experiment scripts uses Bash and relies on
+Linux-specific grammars.
+
+For your reference, we used the machine with the following specs to
+run experiments:
+- Intel Core i7-11700K @ 3.60GHz (8 cores, 16 threads) CPU
+- 64 GB RAM
+- Ubuntu 20.04 operating system
+
+
+## Installing Package Management Systems
+
+The two package management systems we use, [conda][conda-webpage] and
+[sdkman][sdkman-webpage], can both be installed in user mode (i.e.,
+does not require sudo).  To install each of them, you will need to
+execute some commands AND configure your `.bashrc` appropriately, and
+then restart the terminal for the changes to take effect.  If you
+happend to have existing installations of (a recent version of) either
+package management system, you do not need to install it again.
+
+### conda for Python
+
+1. Download the Miniconda installation script from
+   [here](https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links);
+   pick the correct link according to your CPU architecture (usually
+   "Miniconda3 Linux 64-bit", but for ARM should be "Miniconda3
+   Linux-aarch64 64-bit").
+   For example:
+```
+wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+```
+
+(Installing Anaconda is also fine, which is bundled with some
+libraries that we don't need.)
+
+2. Execute the downloaded script, and follow the prompts on the
+   terminal to install conda at the desired location. For example:
+
+```
+bash Miniconda3-latest-Linux-x86_64.sh
+```
+
+3. If not already done at the end of the last step, run `conda init` which automatically changes your `.bashrc` file to use conda.
+
+4. Restart your terminal. By executing `conda --version`, you should be able to see the version of conda you just installed.
+
+
+### sdkman for Java
+
+1. Execute the following command, and follow the prompts on the
+   terminal to install sdkman.
+
+```
+curl -s "https://get.sdkman.io" | bash
+```
+
+2. Restart your terminal. By executing `sdk version`, you should be able to see the version of sdkman you just installed.
+
+
+## Installing the I-Test Framework
+
+### Python
+
+1. Change directory to "python". Assuming you were at the root of this
+   repository: `cd python`
+
+2. Execute `./prepare_conda_env.sh`
+
+3. Execute `conda activate inline-dev`
+
+4. You should be able to see "(inline-dev) " as the prefix of the
+   prompt in your terminal. Then, to further check if installation is
+   successful, you can run the tests for the I-Test framework in
+   Python: `pytest`
+
+* Trouble shooting: if you get error "CondaEnvironmentError: cannot
+  remove current environment. deactivate and run conda remove again",
+  please run `conda deactivate` to exit the inline-dev environment,
+  then try again.
+
+### Java
+
+1. Change directory to "java". Assuming you were at the root of this
+   repository: `cd java`
+
+2. Execute `./install.sh`
+
+3. During the installation in the previous step, we actually already
+   run the tests for the I-Test framework in Java.  You can also run
+   these tests again to double check if the installation is
+   successful: `sdk use java 8.0.302-open; sdk use maven 3.8.3; mvn test`
+
+
+## Installing the Environment for Experiment Scripts
+
+1. Change directory to "research". Assuming you were at the root of
+   this repository: `cd research`
+
+2. Execute `./prepare_conda_env.sh`
+
+3. Execute `conda activate inline-research`
+
+4. You should be able to see "(inline-dev) " as the prefix of the
+   prompt in your terminal. Then, to further check if installation is
+   successful, you can run the script for running the inline tests in
+   the 50 Python example files and 50 Java example files:
+
+```
+python -m research.exp_standalone run --language=python --requests_file=../data/exp/standalone/python.yaml --run_dir=../data/examples/python --out_dir=/tmp/inlinetest-smoke/python --force=True
+python -m research.exp_standalone run --language=java --requests_file=../data/exp/standalone/java.yaml --run_dir=../data/examples/java --out_dir=/tmp/inlinetest-smoke/java --force=True
+```
+
+For both of the previous commands, you should see a progress bar that ends at "running: 100% ... 50/50 ...", without warning or error messages.
+
+
+* Trouble shooting: if you get error "CondaEnvironmentError: cannot
+  remove current environment. deactivate and run conda remove again",
+  please run `conda deactivate` to exit the inline-dev environment,
+  then try again.
+
+
+[conda-webpage]: https://docs.conda.io/en/latest/
+[sdkman-webpage]: https://sdkman.io/

LICENSE

@@ -0,0 +1,8 @@
+The MIT License (MIT)
+Copyright © 2022 Team I-Test
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -0,0 +1,69 @@
+# I-Test
+
+This repo hosts the code and data for the following ASE 2022 paper:
+
+Title: [Inline Tests][paper-url]
+
+Authors: [Yu Liu](https://sweetstreet.github.io/), [Pengyu Nie](https://pengyunie.github.io/), [Owolabi Legunsen](https://mir.cs.illinois.edu/legunsen/), [Milos Gligoric](http://users.ece.utexas.edu/~gligoric/)
+
+```bibtex
+@inproceedings{LiuASE22InlineTests,
+  title =        {Inline Tests},
+  author =       {Yu Liu and Pengyu Nie and Owolabi Legunsen and Milos Gligoric},
+  pages =        {to appear},
+  booktitle =    {International Conference on Automated Software Engineering},
+  year =         {2022},
+}
+```
+
+## Introduction
+
+This repo contains the code and data for producing the experiments in
+[Inline Tests][paper-url].  In this work, we proposed a new type of
+tests, inline tests, which reside below unit tests in the hierarchy of
+test types.  Inline tests can be used to test the code on the
+statement level. We implemented I-Test framework for developers to
+write inline tests in Java and Python. I-Test framework has been
+integrated with popular testing frameworks pytest and Junit. 
+
+The code includes:
+* I-Test framework for Python
+* I-Test framework for Java
+* scripts for collecting and filtering examples suitable for writing inline tests
+* scripts for evaluating the performance of I-Test
+
+The data includes:
+* the 50 Python examples and 50 Java examples with our written inline tests
+* results of performance evaluation (on our machine)
+* the documents used in our user study
+* anonymized results of our user study
+
+
+**How to...**
+* **install I-Test and the environment for replicating our study**: see detailed steps in [INSTALL.md](/INSTALL.md)
+* **replicate our performance evaluation of I-Test**: see detailed steps in [REPLICATION.md](/REPLICATION.md)
+* **replicate other parts of our paper, e.g., collecting examples and user study**: we already described the steps in the paper, and you may find the documents and intermediate files for those in this repository; see the remainder of this README for more details
+
+## Content of this Repository
+
+- [java](/java): code of I-Test framework for Java
+- [python](/python): code of I-Test framework for Python
+- [research](/research): scripts for our experiments
+- data
+  - [examples](/data/examples): 50 Python and 50 Java examples with our written inline tests
+  - [exp](/data/exp): the configurations for running performance evluation experiments
+  - [patches](/data/patches): the patches used in performance evaluation experiments, to integrated inline tests into open-source projects
+  - [projects](/data/projects) and [projects-used](/data/projects-used): the list of top-100 starred open-source GitHub projects that we used to search for statements under test
+  - [scripts](/data/scripts): the scripts used in performance evaluation experiments, for preparing environment and executing the unit tests or inline tests in open-source projects
+- [results](/results): directory for storing the results of running performance evaluation experiments; used in the [replication guide](/REPLICATION.md)
+- [results-ours](/results-ours): the results of performance evaluation on our machine
+- userstudy
+  - [content](/userstudy/content): the package we send to each participant in our user study
+  - [response](/userstudy/response): anonymized responses collected of our user study
+- [appendix.pdf](/appendix.pdf): an appendix that describes:
+  - A: the details procedure of searching for statements under test in open-source projects
+  - B: API of I-Test framework
+  - C: analysis of user study responses
+
+
+[paper-url]: /README.md