update doc on the runtime test mechanism. change the name of BaseDeb…

…ugParserTestDescriptor to BaseDiagnosticParserTestDescriptor.
speedo-sp7 · Nov 15, 2016 · d534efd · d534efd
1 parent a7ecf03
commit d534efd
Show file tree

Hide file tree

Showing 13 changed files with 165 additions and 157 deletions.
diff --git a/doc/adding-tests.md b/doc/adding-tests.md
@@ -1,119 +1,125 @@
 # Adding unit tests
 
-## Generating Runtime Tests
+## Introduction
+
+Because ANTLR supports multiple target languages, the unit tests are broken into two groups: the unit tests that test the tool itself (in `tool-testsuite`) and the unit tests that test the parser runtimes (in `antlr4/runtime-testsuite`).  The tool tests are straightforward because they are Java code testing Java code; see the section at the bottom of this file.
+
+The runtime tests must be specified in a generic fashion to work across language targets. Furthermore, we must test the various targets from Java. This usually means Java launching processes to compile, say, C++ and run parsers.
+
+As of 4.6, we use [a Java descriptor object](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) to describe each runtime test.  Unit tests are grouped together into categories such as [ParserExecDescriptors](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParserExecDescriptors.java), which has multiple nested descriptor objects, one per test. For example, here is the start of that file:
+
+```java
+public class ParserExecDescriptors {
+    public static class APlus extends BaseParserTestDescriptor {
+        public String input = "a b c";
+        public String output = "abc\n";
+        public String errors = "";
+        public String startRule = "a";
+        public String grammarName = "T";
+
+        /**
+         grammar T;
+         a : ID+ {
+         <writeln("$text")>
+         };
+         ID : 'a'..'z'+;
+         WS : (' '|'\n') -> skip;
+         */
+        @CommentHasStringValue
+        public String grammar;
+    }
+```
 
-Because ANTLR supports multiple target languages, the unit tests are broken into two groups: the unit tests that test the tool itself (in `tool-testsuite`)  and the unit tests that test the parser runtimes (in antlr4/runtime-testsuite). To avoid a lot of cut-and-paste, we generate all **runtime** tests from a set of templates using [runtime-testsuite/src/org/antlr/v4/testgen/TestGenerator.java](../runtime-testsuite/src/org/antlr/v4/testgen/TestGenerator.java). The `mvn` command is simple to use:
+The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allows multi-line strings in Java. This kung fu is required so that we can use Java classes rather than StringTemplate group files to specify runtime tests (the legacy system used those and it was hard to get them right). Here are all the [Runtime test descriptors](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) organized into groups.
 
-```
-$ cd ~/antlr/code/antlr4/runtime-testsuite
-$ mvn -Pgen generate-test-sources
-...
-rootDir = /Users/parrt/antlr/code/antlr4/runtime-testsuite
-outputDir = /Users/parrt/antlr/code/antlr4/runtime-testsuite/test
-templates = /Users/parrt/antlr/code/antlr4/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates
-target = ALL
-browsers = false
-viz = false
-INFO: Generating target Java
-INFO: Generating target Go
-INFO: Generating target CSharp
-INFO: Generating target Python2
-INFO: Generating target Python3
-INFO: Generating target JavaScript/Node
-...
-```
+The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target.  Here are all of the 
+[Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates).
 
-It basically runs the Java program:
+## Running the runtime tests
 
-```bash
-$ java org.antlr.v4.testgen.TestGenerator \
-	-root ~/antlr/code/antlr4/runtime-testsuite \
-	-outdir ~/antlr/code/antlr4/runtime-testsuite/test \
-	-templates ~/antlr/code/antlr4/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates
-```
+A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of
+[Target runtime test rigs](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime) that allow developers such flexibility. For example, here are the Python3 test rigs in intellij:
 
-## Adding a runtime test
+<img src=images/testrigs.png width=300>
 
-For each target, you will find an `Index.stg` file with a dictionary of all test groups. E.g., `runtime-testsuite/resources/org/antlr/v4/test/runtime/templates/Index.stg` looks like:
+And the result of testing the entire subdirectory:
 
-```
-TestFolders ::= [
-        "CompositeLexers": [],
-        "CompositeParsers": [],
-        "FullContextParsing": [],
-        "LeftRecursion": [],
-        "LexerErrors": [],
-        "LexerExec": [],
-        "Listeners": [],
-        "ParserErrors": [],
-        "ParserExec": [],
-        "ParseTrees": [],
-        "Performance": [],
-        "SemPredEvalLexer": [],
-        "SemPredEvalParser": [],
-        "Sets": []
-]
-```
+<img src=images/python3-tests.png width=400>
 
-Then each group has a subdirectory with another index. E.g., `Sets/Index.stg` looks like:
+From `mvn`, on the commandline, you will see:
 
-```
-TestTemplates ::= [
-        "SeqDoesNotBecomeSet": [],
-        "ParserSet": [],
-        "ParserNotSet": [],
-        "ParserNotToken": [],
-        "ParserNotTokenWithLabel": [],
-        "RuleAsSet": [],
-        "NotChar": [],
-        "OptionalSingleElement": [],
+```bash
+-------------------------------------------------------
+ T E S T S
+-------------------------------------------------------
+Running org.antlr.v4.test.runtime.javascript.node.TestCompositeLexers
+Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.581 sec
+Running org.antlr.v4.test.runtime.javascript.node.TestLexerErrors
+Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec
+Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalParser
+Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.084 sec
+Running org.antlr.v4.test.runtime.javascript.node.TestSets
+Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.798 sec
+Running org.antlr.v4.test.runtime.javascript.node.TestPerformance
+Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec
+Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalLexer
+Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.994 sec
+Running org.antlr.v4.test.runtime.javascript.node.TestLexerExec
+Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.433 sec
 ...
 ```
 
-For every name mentioned, you will find a `.stg` file with the actual test template. E.g., `Sets/StarSet.stg`.
+## Adding a runtime test
 
-Each `.stg` file descripes the following mandatory elements for the test:
- - the test type: "Parser" or "Lexer"
- - some ANTLR options, such as "Debug"
- - the grammar 
- - the start rule
- - the input i.e. the text to parse
- - the expected output
- - the expected errors
+To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:
 
-The grammar can itself contain template expressions such as `<something>`.
-The test generator replaces these with the corresponding values from the target language template (see below).
-It then generates a unit test in which the grammar, the input and the expected output and errors are inlined.
- 
-Here is an example test template:
+* [BaseParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseParserTestDescriptor.java); see example [APlus](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParserExecDescriptors.java#L7).
+* [BaseDiagnosticParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseDiagnosticParserTestDescriptor) if you want to test parser diagnostic output; see [example output](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/FullContextParsingDescriptors.java#L16).
+* [BaseCompositeParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeParserTestDescriptor.java); see example [BringInLiteralsFromDelegate](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeParsersDescriptors.java#L11)
+* [BaseLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseLexerTestDescriptor.java); see example [ActionPlacement](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/LexerExecDescriptors.java#L12).
+* [BaseCompositeLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeLexerTestDescriptor.java); see example [LexerDelegatorInvokesDelegateRule](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeLexersDescriptors.java#L11)
 
-```
-TestType() ::= "Parser"
 
-Options ::= [
-	"Debug": false
-]
+Each descriptor object describes the following mandatory elements for the test:
 
-Grammar ::= [
-	"T": {<grammar("T")>}
-]
+ * the test type
+ * the grammar 
+ * the start rule
+ * the input text to parse or lex
+ * the expected output
+ * the expected errors
 
-Input() ::= "abaac"
+Your best bet is to find a similar test in the appropriate group and then copy and paste the descriptor object, creating a new nested class within the test group class. Modify the field definitions to suit your new problem.
 
-Rule() ::= "a"
+If you need to create a whole new group of tests, it requires a new descriptor class; call it `XDescriptors`. Then, in each [target subdirectory](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime), you need to create a new test rig `TestX.java` file:
 
-Output() ::= <<
-abaac<\n>
->>
+```java
+package org.antlr.v4.test.runtime.java;
 
-Errors() ::= ""
+import org.antlr.v4.test.runtime.BaseRuntimeTest;
+import org.antlr.v4.test.runtime.RuntimeTestDescriptor;
+import org.antlr.v4.test.runtime.descriptors.ListenersDescriptors;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
 
-grammar(grammarName) ::= <<
-grammar <grammarName>;
-a : ('a'|'b')* 'c' {<InputText():writeln()>} ;
->>
+@RunWith(Parameterized.class)
+public class TestX extends BaseRuntimeTest {
+	public TestX(RuntimeTestDescriptor descriptor) {
+		super(descriptor,new Base<TARGET>Test());
+	}
+
+	@Parameterized.Parameters(name="{0}")
+	public static RuntimeTestDescriptor[] getAllTestDescriptors() {
+		return BaseRuntimeTest.getRuntimeTestDescriptors(XDescriptors.class, "<TARGET>");
+	}
+}
 ```
 
+where `<TARGET>` is replaced with Java, Cpp, CSharp, Python2, ... in the various subdirectories.
+
+### Ignoring tests
+
+In order to turn off a test for a particular target, we need to use the `ignore` method. Given a target name, a descriptor object can decide whether to ignore the test. This is not always convenient but it is fully general and works well for the one case we have now where we have to ignore `Visitor` tests in all targets except JavaScript.
 
 ### Cross-language actions embedded within grammars
 
@@ -129,14 +135,16 @@ Use instead the language-neutral:
 <writeln("$set.stop")>
 ```
 
-File `runtime-testsuite/resources/org/antlr/v4/test/runtime/java/Java.test.stg` has templates like:
+Template file [runtime-testsuite/resources/org/antlr/v4/test/runtime/templates/Java.test.stg](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates/Java.test.stg) has templates like:
 
 ```
 writeln(s) ::= <<System.out.println(<s>);>>
 ```
 
+that translate generic operations to target-specific language statements or expressions.
+
 ## Adding an ANTLR tool unit test
 
-Just go into the appropriate Java test class in dir `antlr4/tool-testsuite/test/org/antlr/v4/test/tool` and add your unit test.
+Just go into the appropriate Java test class in dir [antlr4/tool-testsuite/test/org/antlr/v4/test/tool](https://github.com/antlr/antlr4/tree/master/tool-testsuite/test/org/antlr/v4/test/tool) and add your unit test.
 
 
diff --git a/doc/creating-a-language-target.md b/doc/creating-a-language-target.md
@@ -9,12 +9,12 @@ Creating a new target involves the following key elements:
 1. For the tool, create class *X*Target as a subclass of class `Target` in package `org.antlr.v4.codegen.target`. This class describes language specific details about escape characters and strings and so on. There is very little to do here typically.
 1. Create *X*.stg in directory tool/resources/org/antlr/v4/tool/templates/codegen/*X*/*X*.stg. This is a [StringTemplate](http://www.stringtemplate.org/) group file (`.stg`) that tells ANTLR how to express all of the parsing elements needed to generate code.  You will see templates called `ParserFile`, `Parser`, `Lexer`, `CodeBlockForAlt`, `AltBlock`, etc... Each of these must be described how to build the indicated chunk of code. Your best bet is to find the closest existing target, copy that template file, and tweak to suit.
 1. Create a runtime library to support the parsers generated by ANTLR. Under directory runtime/*X*, you are in complete control of the directory structure as dictated by common usage of that target language. For example, Java has: `runtime/Java/lib` and `runtime/Java/src` directories. Under `src`, you will find a directory structure for package `org.antlr.v4.runtime` and below.
-1. Create a template file for runtime tests. All you have to do is provide a few simple templates that indicate how to print values and declare variables. Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code in a new target and check the results. All it needs to know is how to generate a test rig (i.e., a `main` program), how to define various class fields, compare members and so on. You must create a *X* directory underneath `runtime-testsuite/resources/org/antlr/v4/test/runtime`. Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit.
+1. Create a template file for runtime tests. All you have to do is provide a few templates that indicate how to print values and declare variables. Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code using these templates for each target and check the test results.  It needs to know how to define various class fields, compare members and so on. You must create a *X*.test.stg file underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime). Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit.
 
 ## Getting started
 
 1. Fork the `antlr/antlr4` repository at github to your own user so that you have repository `username/antlr4`.
-2. Clone `username/antlr4`, forked repository, to your local disk.  Your remote `origin` will be the forked repository on GitHub. Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`). Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/).
+2. Clone `username/antlr4`, the forked repository, to your local disk.  Your remote `origin` will be the forked repository on GitHub. Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`). Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/).
 3. Try to build it before doing anything
 ```bash
 $ mvn compile

diff --git a/doc/images/python3-tests.png b/doc/images/python3-tests.png
diff --git a/doc/images/testrigs.png b/doc/images/testrigs.png
diff --git a/...untime/BaseDebugParserTestDescriptor.java → ...e/BaseDiagnosticParserTestDescriptor.java b/...untime/BaseDebugParserTestDescriptor.java → ...e/BaseDiagnosticParserTestDescriptor.java
@@ -1,6 +1,6 @@
 package org.antlr.v4.test.runtime;
 
-public abstract class BaseDebugParserTestDescriptor extends BaseParserTestDescriptor {
+public abstract class BaseDiagnosticParserTestDescriptor extends BaseParserTestDescriptor {
 	@Override
 	public boolean showDiagnosticErrors() {
 		return true;

diff --git a/...ime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeParsersDescriptors.java b/...ime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeParsersDescriptors.java
@@ -458,7 +458,7 @@ public List<Pair<String, String>> getSlaveGrammars() {
 	 */
 	public static class ImportLexerWithOnlyFragmentRules extends BaseCompositeParserTestDescriptor {
 		public String input = "test test";
-		public String output = "";
+		public String output = null;
 		public String errors = null;
 		public String startRule = "program";
 		public String grammarName = "Test";
@@ -499,7 +499,7 @@ public List<Pair<String, String>> getSlaveGrammars() {
 
 	public static class ImportedGrammarWithEmptyOptions extends BaseCompositeParserTestDescriptor {
 		public String input = "b";
-		public String output = "";
+		public String output = null;
 		public String errors = null;
 		public String startRule = "s";
 		public String grammarName = "M";
@@ -533,7 +533,7 @@ public List<Pair<String, String>> getSlaveGrammars() {
 
 	public static class ImportedRuleWithAction extends BaseCompositeParserTestDescriptor {
 		public String input = "b";
-		public String output = "";
+		public String output = null;
 		public String errors = null;
 		public String startRule = "s";
 		public String grammarName = "M";