feat: adding sarif file extension validation support #217

arthurflame · 2024-11-17T12:25:22Z

As discussed in this PR #183:

Parsing the report as JSON
Validating that schema is specified
Validating that the content of the .sarif file matches the schema

Content of .sarif files was taken from these samples.

pkg/filetype/file_type.go

pkg/validator/sarif.go

* schemaUrl variable changed to schemaURL * error formatting when schema is absent, and when schema isn't valid * validation that schema is a string * formatError function refactor, typo fix, and return multiple errors (if applicable)

pkg/validator/sarif.go

kehoecj

@arthurflame Looks good! Please take care of the golint-ci findings and resolve conflicts from the dependency changes in main

* moved defer out of the for loop * wrapping error using %w

* golint: truly moved defer out of the for loop * golint: error handling for connection.Close() * timeout for DialTimeout increased to 3 seconds * removed "error -" prefix from error messages to prevent "error: error - " duplicates

* modified Test_FileSystemFinderDuplicateFiles to check for a different length of files in a directory * updated validate_test.go to include sarif files in testData * added loadFile function to read the contents of sample files in test/fixtures/ instead of creating lengthy variables * removed embed package since it wasn't used * added comments

kehoecj

@arthurflame Only at 86% code coverage

arthurflame · 2024-12-17T17:35:08Z

Hey, not sure how to increase test coverage at this point.
Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those.
Any suggestions? @kehoecj

kehoecj · 2024-12-18T18:23:37Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:

go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html

* included a separate directory to cover most cases for .sarif validation * ignored returning an error from schema.Validate() since this error wasn't reachable

arthurflame · 2024-12-19T10:27:16Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html

Hey,

Awesome, thanks for the tip!
Made some changes, please take a look.

arthurflame · 2024-12-20T10:00:19Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html
Hey,

Awesome, thanks for the tip! Made some changes, please take a look.

Running go test -coverprofile=coverage.out ./... locally, can't reproduce the same error that can be seen on the runner.

Referring to this error:
https://github.com/Boeing/config-file-validator/actions/runs/12417060386/job/34667432261#step:6:304

arthurflame · 2024-12-25T12:59:21Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html
Hey,
Awesome, thanks for the tip! Made some changes, please take a look.
Running go test -coverprofile=coverage.out ./... locally, can't reproduce the same error that can be seen on the runner.

Referring to this error: https://github.com/Boeing/config-file-validator/actions/runs/12417060386/job/34667432261#step:6:304

Have been sitting on this for a while.
Still can't figure out the root cause of failing tests.

Prior to Test_CLI in cli_test.go is executed, as far as I understood, reporter_test.go has to generate result.json file and place it under test/output/ directory.
And since it wasn't generated (?), test fails.

Could it be a race condition?

Also, a rather strange behaviour that I've seen while trying to address this problem.

The second test execution generates files mentioned on the screenshot, the first one - didn't.
Not sure if it's expected, jfyi.

Would greatly appreciate any help.
Thank you!
@kehoecj

kehoecj · 2025-01-14T17:13:31Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html
Hey,
Awesome, thanks for the tip! Made some changes, please take a look.
Running go test -coverprofile=coverage.out ./... locally, can't reproduce the same error that can be seen on the runner.
Referring to this error: https://github.com/Boeing/config-file-validator/actions/runs/12417060386/job/34667432261#step:6:304
Have been sitting on this for a while. Still can't figure out the root cause of failing tests.

Prior to Test_CLI in cli_test.go is executed, as far as I understood, reporter_test.go has to generate result.json file and place it under test/output/ directory. And since it wasn't generated (?), test fails.

Could it be a race condition?

Also, a rather strange behaviour that I've seen while trying to address this problem. The second test execution generates files mentioned on the screenshot, the first one - didn't. Not sure if it's expected, jfyi.

Would greatly appreciate any help. Thank you! @kehoecj

Thanks for the analysis @arthurflame - I'll take a look

kehoecj · 2025-01-21T19:27:27Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html
Hey,
Awesome, thanks for the tip! Made some changes, please take a look.
Running go test -coverprofile=coverage.out ./... locally, can't reproduce the same error that can be seen on the runner.
Referring to this error: https://github.com/Boeing/config-file-validator/actions/runs/12417060386/job/34667432261#step:6:304
Have been sitting on this for a while. Still can't figure out the root cause of failing tests.
Prior to Test_CLI in cli_test.go is executed, as far as I understood, reporter_test.go has to generate result.json file and place it under test/output/ directory. And since it wasn't generated (?), test fails.
Could it be a race condition?
Also, a rather strange behaviour that I've seen while trying to address this problem. The second test execution generates files mentioned on the screenshot, the first one - didn't. Not sure if it's expected, jfyi.
Would greatly appreciate any help. Thank you! @kehoecj
Thanks for the analysis @arthurflame - I'll take a look

I was able to fix the tests locally by forcing the tests to run sequentially rather than in parallel. I believe running in parallel with output files was causing the issue. If you update the https://github.com/Boeing/config-file-validator/blob/main/.github/workflows/go.yml#L101 to include -parallel=1 then the tests will succeed.

arthurflame · 2025-01-21T21:47:26Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html
Hey,
Awesome, thanks for the tip! Made some changes, please take a look.
Running go test -coverprofile=coverage.out ./... locally, can't reproduce the same error that can be seen on the runner.
Referring to this error: https://github.com/Boeing/config-file-validator/actions/runs/12417060386/job/34667432261#step:6:304
Have been sitting on this for a while. Still can't figure out the root cause of failing tests.
Prior to Test_CLI in cli_test.go is executed, as far as I understood, reporter_test.go has to generate result.json file and place it under test/output/ directory. And since it wasn't generated (?), test fails.
Could it be a race condition?
Also, a rather strange behaviour that I've seen while trying to address this problem. The second test execution generates files mentioned on the screenshot, the first one - didn't. Not sure if it's expected, jfyi.
Would greatly appreciate any help. Thank you! @kehoecj
Thanks for the analysis @arthurflame - I'll take a look
I was able to fix the tests locally by forcing the tests to run sequentially rather than in parallel. I believe running in parallel with output files was causing the issue. If you update the https://github.com/Boeing/config-file-validator/blob/main/.github/workflows/go.yml#L101 to include -parallel=1 then the tests will succeed.

Perfect, thank you very much for taking a look.
Will check on it tomorrow!

arthurflame · 2025-02-01T15:19:49Z

Hey, not sure how to increase test coverage at this point. Was thinking about covering additional functions from sarif.go file, although, don't want to bloat the validator_test.go file with those. Any suggestions? @kehoecj

Hi @arthurflame - you're just missing several lines of your coverage because you're not testing all the failure conditions. Check the coverage report and see what lines you're missing:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
google-chrome coverage.html
Hey,
Awesome, thanks for the tip! Made some changes, please take a look.
Running go test -coverprofile=coverage.out ./... locally, can't reproduce the same error that can be seen on the runner.
Referring to this error: https://github.com/Boeing/config-file-validator/actions/runs/12417060386/job/34667432261#step:6:304
Have been sitting on this for a while. Still can't figure out the root cause of failing tests.
Prior to Test_CLI in cli_test.go is executed, as far as I understood, reporter_test.go has to generate result.json file and place it under test/output/ directory. And since it wasn't generated (?), test fails.
Could it be a race condition?
Also, a rather strange behaviour that I've seen while trying to address this problem. The second test execution generates files mentioned on the screenshot, the first one - didn't. Not sure if it's expected, jfyi.
Would greatly appreciate any help. Thank you! @kehoecj
Thanks for the analysis @arthurflame - I'll take a look
I was able to fix the tests locally by forcing the tests to run sequentially rather than in parallel. I believe running in parallel with output files was causing the issue. If you update the https://github.com/Boeing/config-file-validator/blob/main/.github/workflows/go.yml#L101 to include -parallel=1 then the tests will succeed.
Perfect, thank you very much for taking a look. Will check on it tomorrow!

Unfortunately, adding -parallel={1,2} doesn't do the trick, tests are still failing on the runners.
Any other ideas how I should approach this?
@kehoecj

Thanks!

Artur Gerashchenko and others added 4 commits November 14, 2024 22:03

initial sarif validation

d0199e8

initial implementation

f3876dc

sarif files added

28a0743

docs and extra lines added

f5fab30

arthurflame mentioned this pull request Nov 17, 2024

Adding .sarif extension for validation. #183

Closed

go mod tidy

2558607

ccoVeille reviewed Nov 17, 2024

View reviewed changes

fixes:

0d9006c

* schemaUrl variable changed to schemaURL * error formatting when schema is absent, and when schema isn't valid * validation that schema is a string * formatError function refactor, typo fix, and return multiple errors (if applicable)

kehoecj added OSS Community Contribution Contributions from the OSS Community waiting-on-maintainer-review PR is waiting to be reviewed and functionally tested by the maintainers labels Dec 3, 2024

kehoecj self-requested a review December 10, 2024 15:31

kehoecj reviewed Dec 10, 2024

View reviewed changes

pkg/validator/sarif.go Show resolved Hide resolved

kehoecj added pr-action-requested PR is awaiting feedback from the submitting developer and removed waiting-on-maintainer-review PR is waiting to be reviewed and functionally tested by the maintainers labels Dec 10, 2024

kehoecj requested changes Dec 10, 2024

View reviewed changes

arthurflame and others added 9 commits December 11, 2024 19:49

initial resolve and connect implementation

019cab5

error placement outside of the loop

4718959

error refactor

43bf9d1

Merge branch 'main' into feature/sarif-validation

65c8788

go mod tidy

433d1fd

golangci-lint warning fix

6fa39db

fixing golint errors

b15a2ee

* moved defer out of the for loop * wrapping error using %w

fixes:

7d3a360

* golint: truly moved defer out of the for loop * golint: error handling for connection.Close() * timeout for DialTimeout increased to 3 seconds * removed "error -" prefix from error messages to prevent "error: error - " duplicates

arthurflame requested review from kehoecj and ccoVeille December 14, 2024 12:36

kehoecj requested changes Dec 16, 2024

View reviewed changes

adding an additional field to testData to validate another sarif file

45f6cce

arthurflame requested a review from kehoecj December 16, 2024 18:00

Merge branch 'main' into feature/sarif-validation

2b6d2e1

Merge branch 'main' into feature/sarif-validation

a17ca81

arthurflame and others added 2 commits December 19, 2024 11:23

fixes:

9082728

* included a separate directory to cover most cases for .sarif validation * ignored returning an error from schema.Validate() since this error wasn't reachable

Merge branch 'main' into feature/sarif-validation

c20be6d

unit test fixes

d26ec3b

fix: adding bad-sarif dir into excludeDirs slice for CLIReportErr

c4587b4

kehoecj added waiting-on-maintainer-review PR is waiting to be reviewed and functionally tested by the maintainers and removed pr-action-requested PR is awaiting feedback from the submitting developer labels Jan 7, 2025

Merge branch 'main' into feature/sarif-validation

3ebf1fe

Merge branch 'main' into feature/sarif-validation

49487a8

arthurflame and others added 3 commits January 22, 2025 18:21

Merge branch 'main' into feature/sarif-validation

a7cdefb

feat: adding -parallel=1 to the go.yml workflow for unit test step

71234e5

wip: setting go test -parallel to 2 for testing purposes

9f29326

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: adding sarif file extension validation support #217

feat: adding sarif file extension validation support #217

arthurflame commented Nov 17, 2024

kehoecj left a comment

kehoecj left a comment

arthurflame commented Dec 17, 2024

kehoecj commented Dec 18, 2024

arthurflame commented Dec 19, 2024

arthurflame commented Dec 20, 2024

arthurflame commented Dec 25, 2024

kehoecj commented Jan 14, 2025

kehoecj commented Jan 21, 2025

arthurflame commented Jan 21, 2025

arthurflame commented Feb 1, 2025

feat: adding sarif file extension validation support #217

Are you sure you want to change the base?

feat: adding sarif file extension validation support #217

Conversation

arthurflame commented Nov 17, 2024

kehoecj left a comment

Choose a reason for hiding this comment

kehoecj left a comment

Choose a reason for hiding this comment

arthurflame commented Dec 17, 2024

kehoecj commented Dec 18, 2024

arthurflame commented Dec 19, 2024

arthurflame commented Dec 20, 2024

arthurflame commented Dec 25, 2024

kehoecj commented Jan 14, 2025

kehoecj commented Jan 21, 2025

arthurflame commented Jan 21, 2025

arthurflame commented Feb 1, 2025