Skip to content

Commit

Permalink
Update testing for latest adacore/gpr tests.
Browse files Browse the repository at this point in the history
  • Loading branch information
brownts committed Aug 10, 2024
1 parent caf7fd9 commit aed8f75
Show file tree
Hide file tree
Showing 4 changed files with 38 additions and 28 deletions.
5 changes: 5 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,10 @@ jobs:
- name: Test Project (Corpus)
run: npm run test

# Initialize tree-sitter config to prevent nuisance message
# during tree-sitter parsing, when config doesn't exist.
- name: Create Tree-sitter Config
run: npx tree-sitter init-config

- name: Test Project (GPR)
run: npm run gpr-test
40 changes: 20 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,26 +68,26 @@ npm run test

## Testing (GPR)

The [GPR project] also contains both examples and a test suite which
are used to test this parser. This was helpful originally in
determining that the production rules in the User Manual were
insufficient. However it is also useful in order to test potential
regressions between the tool and the grammar defined here.
The [GPR project] also contains many GPR files (e.g., examples, test
suite, etc.) which are used to test this parser. This was helpful
originally in determining that the production rules in the User Manual
were insufficient. However it is also useful in order to test
potential regressions between the tool and the grammar defined here.

In order to perform this testing, there is an assumption that the [GPR
project] exists adjacent to this project. As such, a relative path is
used to search for and run tests on the set of GPR files found there.

The approach taken in this testing, is to use the set of GPR files in
the examples and test suites directories of that project. The set of
files are determined based on a recursive search of the directories.
However, some of the GPR files used in the test suite are used to
check for invalid syntax, thus they are not syntactically valid. In
order to account for these files, a "skip list" is maintained to omit
these GPR files from the set that are tested. This list is manually
maintained and each file listed there has been inspected to verify
that they do in fact contain invalid syntax. This list will need to
be updated if new GPR files are added which contain invalid syntax.
the directories of that project. The set of files are determined
based on a recursive search of the directories. However, some of the
GPR files used in the test suite are used to check for invalid syntax,
thus they are not syntactically valid. In order to account for these
files, a "skip list" is maintained to omit these GPR files from the
set that are tested. This list is manually maintained and each file
listed there has been inspected to verify that they do in fact contain
invalid syntax. This list will need to be updated if new GPR files
are added which contain invalid syntax.

It was considered to be smarter about this testing and to hook into
the GPR test suite, since those tests know which should fail and which
Expand All @@ -98,12 +98,12 @@ that test suite and failures due to parsing, that approach was not
pursued.

This form of testing does not guarantee that it detects all erroneous
GPR files, but it does at least make sure that valid GPR files in the
examples and test suite do not cause parsing errors. At the heart of
this testing, the "parse" functionality of the Node tree-sitter
package is used to run each GPR file through the parser. Currently
there are 900+ unique GPR files that are parsed using this method, so
this provides a high level of confidence.
GPR files, but it does at least make sure that valid GPR files in that
project do not cause parsing errors. At the heart of this testing,
the "parse" functionality of the Node tree-sitter package is used to
run each GPR file through the parser. Currently there are 1000+
unique GPR files that are parsed using this method, so this provides a
high level of confidence.

Currently the "test harness" consists of a Makefile and some commands
known to exist in a Linux environment (i.e., `make`, `sed`, `find`,
Expand Down
19 changes: 12 additions & 7 deletions test/gpr/Makefile
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
MAKEFILE_DIR = $(dir $(lastword $(MAKEFILE_LIST)))

.PHONY: default examples testsuite build
TEST_DIRS =
TEST_DIRS += examples
TEST_DIRS += gpr_compliance_test_database
TEST_DIRS += testsuite

GPR_BASE_DIR ?= $(MAKEFILE_DIR)../../../gpr
.PHONY: default $(TEST_DIRS)

GPR_DIR_NAME = gpr
GPR_BASE_DIR ?= $(MAKEFILE_DIR)../../../$(GPR_DIR_NAME)
GPR_SKIP_FILE = $(MAKEFILE_DIR)gpr_skip.txt

default: examples testsuite
default: $(TEST_DIRS)

examples: TEST_DIR=$(GPR_BASE_DIR)/examples
testsuite: TEST_DIR=$(GPR_BASE_DIR)/testsuite
examples testsuite:
@sed 's;^;! -path */;' $(GPR_SKIP_FILE) | xargs find $(TEST_DIR) -iname '*.gpr' | sort | xargs npx tree-sitter parse --quiet --stat
$(TEST_DIRS):
@echo Testing against $(GPR_DIR_NAME)/$@...
@sed 's;^;! -path */;' $(GPR_SKIP_FILE) | xargs find $(GPR_BASE_DIR)/$@ -iname '*.gpr' | sort | xargs npx tree-sitter parse --quiet --stat
2 changes: 1 addition & 1 deletion test/gpr/gpr_skip.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ testsuite/tests/file-reader/aggregated.gpr
testsuite/tests/file-reader/aggregating.gpr
testsuite/tests/file-reader/extended.gpr
testsuite/tests/file-reader/withed.gpr
testsuite/tests/gprname-invalid-gpr-file/prj.gpr
testsuite/tests/gprinspect-invalid-project/prj1.gpr
testsuite/tests/invalid-project/p.gpr
testsuite/tests/options/new-current-directory/default/default.gpr
testsuite/tests/options/new-current-directory/default/test.gpr
Expand Down

0 comments on commit aed8f75

Please sign in to comment.