Skip to content
This repository has been archived by the owner on Dec 18, 2019. It is now read-only.

Split pending and skipped tests #34

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Split pending and skipped tests #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3ace3e7
split pending and skipped tests
just-boris Sep 18, 2016
ba76ad1
report pending tests as skipped as well
just-boris Sep 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion lib/reporter.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,8 @@ class CucumberReporter {
case Cucumber.Status.PENDING:
case Cucumber.Status.SKIPPED:
case Cucumber.Status.AMBIGUOUS:
e = 'pending'
e = 'skipped'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@just-boris I still don't know why we change the event name here, can't we handle this in the allure reporter only?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pending status is still there for really pending tests (tests with empty implementation)

According to this change, pending, skipped and ambiguous tests are still treated equally.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of this change is normalizing behavior of Cucumber with other frameworks.

Neither Mocha nor Jasmine doesn't emit test:start for pending tests.
I tried to cancel test:start event for pending test, but Cucumber API is designed in a way that doesn't let us know about pending test in advance.

This is why I came up with the current solution

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that the base reporter does not trigger if the event name is skipped, see https://github.com/webdriverio/webdriverio/blob/master/lib/utils/BaseReporter.js#L114

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the allure reporter is the only reporter that has problems with it why not just fixing it there?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a valid point, but then here is a different problem.
Mocha framework doesn't fire test:start event for pending tests, but Cucumber does.
Allure is the only reporter that suffers from it just because it is the only reporter from the standard set, that uses this event, but there is a general problem in inconsistency, that I am trying to mitigate.

break
}
let error = {}
let stepTitle = step.getName() || step.getKeyword() || 'Undefined Step'
Expand Down
13 changes: 13 additions & 0 deletions test/fixtures/steps-status-step-definitions.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
var assert = require('assert')

module.exports = function () {
this.Given(/Test will fail/, (url) => {
return assert.ok(false, 'expected failure')
})

this.Given('Pending test', () => 'pending')

this.Then(/this step will be skipped/, (selector) => {
throw new Error('unexpected error')
})
}
11 changes: 11 additions & 0 deletions test/fixtures/steps-status.conf.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
export default {
sync: false,
capabilities: {
browserName: 'chrome'
},

cucumberOpts: {
timeout: 5000,
require: [__dirname + '/steps-status-step-definitions.js']
}
}
9 changes: 9 additions & 0 deletions test/fixtures/steps-status.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Feature: Steps status

Scenario: Failing test
Given Test will fail
And this step will be skipped
Then this step will be skipped

Scenario: Pending
Given Pending test
44 changes: 44 additions & 0 deletions test/steps.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
import { CucumberAdapter } from '../lib/adapter'
import CucumberReporter from '../lib/reporter'
import config from './fixtures/steps-status.conf'

const specs = ['./test/fixtures/steps-status.feature']

const WebdriverIO = class {}

describe('steps', () => {
it('should report different status for steps', async () => {
const messages = []
const send = CucumberReporter.prototype.send
CucumberReporter.prototype.send = message => messages.push(message)
global.browser = new WebdriverIO()
global.browser.options = config
const adapter = new CucumberAdapter(0, config, specs, {})

;(await adapter.run()).should.be.equal(1)

messages.map(msg => msg.event).should.be.deepEqual([
'suite:start',

// failed
'suite:start',
'test:start',
'test:fail',
'test:start',
'test:skipped',
'test:start',
'test:skipped',
'suite:end',

// pending
'suite:start',
'test:start',
'test:skipped',
'suite:end',

'suite:end'
])

CucumberReporter.prototype.send = send
})
})