Test Progress Notification - LightBDD/LightBDD GitHub Wiki
Note: By default, the dotnet test command does not print any output for pending or successful tests.
It is possible to force it to print output by enabling console logging with normal verbosity: dotnet test --logger:"console;verbosity=normal", however the scenario output will be displayed only upon test completion not during its execution.
Please check Test Framework Integrations page for more accurate alternatives if they are present.
Progress notifiers
DefaultProgressNotifier
During run, the test progress would be printed on console, like:
FEATURE: [Story-1] Login feature
In order to access personal data
As an user
I want to login into system
SCENARIO: [Ticket-1] Successful login
STEP 1/6: GIVEN the user is about to login...
STEP 1/6: GIVEN the user is about to login (Passed after 9ms)
STEP 2/6: AND the user entered valid login...
STEP 2/6: AND the user entered valid login (Passed after <1ms)
STEP 3/6: AND the user entered valid password...
STEP 3/6: AND the user entered valid password (Passed after <1ms)
STEP 4/6: WHEN the user clicks login button...
STEP 4/6: WHEN the user clicks login button (Passed after 182ms)
STEP 5/6: THEN the login operation should be successful...
STEP 5/6: THEN the login operation should be successful (Passed after 5ms)
STEP 6/6: AND a welcome message containing user name should be returned...
STEP 6/6: AND a welcome message containing user name should be returned (Passed after 14ms)
SCENARIO RESULT: Passed after 233ms
Before each step execution, a step number and name is printed, which allows developers to track long running scenarios. After step execution is finished, a step status and execution time is printed.
The presented representation of progress text is provided by DefaultProgressNotifier class which is used by NUnit and MSTest integration.
ParallelProgressNotifier
A notifier dedicated for parallel test execution. As during the execution, multiple scenarios may be executed at the same time, the notifier adds the additional header to each notification.
The header looks like this: Fi=009,Fa=000,Pe=002 # 11> and provides information about currently FInished, FAiled and PEnding scenarios as well as the ID of currently executing scenario.
Below, there is an example output generated by this notifier:
Fi=000,Fa=000,Pe=000 # > FEATURE: [Story-3] Product spedition feature
In order to deliver products to customer effectively
As a spedition manager
I want to dispatch products to customer as soon as the payment is finalized
Fi=000,Fa=000,Pe=002 # 1> SCENARIO: [Ticket-1] Successful login
Fi=000,Fa=000,Pe=002 # 2> SCENARIO: [Ticket-6] No product in stock
Fi=000,Fa=000,Pe=003 # 3> SCENARIO: [Ticket-10][Ticket-11] Successful payment
Fi=000,Fa=000,Pe=006 # 6> SCENARIO: [Ticket-5] Should dispatch product after payment is finalized
Fi=000,Fa=000,Pe=004 # 4> SCENARIO: [Ticket-9] Contact book should allow me to remove all contacts
Fi=000,Fa=000,Pe=005 # 5> SCENARIO: [Ticket-4] Receiving invoice for products
Fi=000,Fa=000,Pe=006 # 1> STEP 1/6: GIVEN the user is about to login...
Fi=000,Fa=000,Pe=006 # 3> STEP 1/4: GIVEN customer has some products in basket...
Fi=000,Fa=000,Pe=006 # 6> STEP 1/6: GIVEN There is an active customer with id "ABC-123"...
Fi=000,Fa=000,Pe=006 # 4> STEP 1/3: GIVEN my contact book is filled with many contacts...
Fi=000,Fa=000,Pe=006 # 5> STEP 1/7: GIVEN product "wooden desk" is available in product storage...
Fi=000,Fa=000,Pe=006 # 2> STEP 1/4: GIVEN product is out of stock...
Fi=000,Fa=000,Pe=006 # 5> STEP 1/7: GIVEN product "wooden desk" is available in product storage (Passed after 3ms)
Fi=000,Fa=000,Pe=006 # 4> STEP 1/3: GIVEN my contact book is filled with many contacts (Passed after 11ms)
Fi=000,Fa=000,Pe=006 # 4> STEP 2/3: WHEN I clear it...
Fi=000,Fa=000,Pe=006 # 1> STEP 1/6: GIVEN the user is about to login (Passed after 3ms)
Fi=000,Fa=000,Pe=006 # 1> STEP 2/6: AND the user entered valid login...
Fi=000,Fa=000,Pe=006 # 1> STEP 2/6: AND the user entered valid login (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 5> STEP 2/7: AND product "wooden shelf" is available in product storage...
Fi=000,Fa=000,Pe=006 # 5> STEP 2/7: AND product "wooden shelf" is available in product storage (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 1> STEP 3/6: AND the user entered valid password...
Fi=000,Fa=000,Pe=006 # 4> STEP 2/3: WHEN I clear it (Bypassed after 2ms)
Fi=000,Fa=000,Pe=006 # 4> STEP 3/3: THEN the contact book should be empty...
Fi=000,Fa=000,Pe=006 # 2> STEP 1/4: GIVEN product is out of stock (Passed after 3ms)
Fi=000,Fa=000,Pe=006 # 2> STEP 2/4: WHEN customer adds it to the basket...
Fi=000,Fa=000,Pe=006 # 6> STEP 1/6: GIVEN There is an active customer with id "ABC-123" (Passed after 3ms)
Fi=000,Fa=000,Pe=006 # 6> STEP 2/6: AND The customer has product "wooden shelf" in basket...
Fi=000,Fa=000,Pe=006 # 6> STEP 2/6: AND The customer has product "wooden shelf" in basket (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 5> STEP 3/7: WHEN customer buys product "wooden desk"...
Fi=000,Fa=000,Pe=006 # 5> STEP 3/7: WHEN customer buys product "wooden desk" (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 5> STEP 4/7: AND customer buys product "wooden shelf"...
Fi=000,Fa=000,Pe=006 # 5> STEP 4/7: AND customer buys product "wooden shelf" (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 5> STEP 5/7: THEN an invoice should be sent to the customer...
Fi=000,Fa=000,Pe=006 # 2> STEP 2/4: /* Transferring 'product' to the basket */
Fi=000,Fa=000,Pe=006 # 6> STEP 3/6: AND The customer has product "wooden desk" in basket...
Fi=000,Fa=000,Pe=006 # 4> STEP 3/3: THEN the contact book should be empty (Passed after 2ms)
Fi=000,Fa=000,Pe=006 # 5> STEP 5/7: THEN an invoice should be sent to the customer (Ignored after <1ms)
Fi=000,Fa=000,Pe=006 # 6> STEP 3/6: AND The customer has product "wooden desk" in basket (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 6> STEP 4/6: WHEN The customer payment finalizes...
Fi=000,Fa=000,Pe=006 # 1> STEP 3/6: AND the user entered valid password (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 1> STEP 4/6: WHEN the user clicks login button...
Fi=000,Fa=000,Pe=006 # 6> STEP 4/6: WHEN The customer payment finalizes (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 6> STEP 5/6: THEN Product "wooden shelf" should be dispatched to the customer...
Fi=000,Fa=000,Pe=006 # 6> STEP 5/6: THEN Product "wooden shelf" should be dispatched to the customer (Passed after <1ms)
Fi=000,Fa=000,Pe=006 # 6> STEP 6/6: AND Product "wooden desk" should be dispatched to the customer...
Fi=000,Fa=000,Pe=006 # 6> STEP 6/6: AND Product "wooden desk" should be dispatched to the customer (Passed after <1ms)
Fi=003,Fa=000,Pe=003 # 4> SCENARIO RESULT: Bypassed after 46ms
Step 2: Contact book clearing is not implemented yet. Contacts are removed one by one.
Due to the complexity of the notifier, it cannot be instantiated directly, but through ParallelProgressNotifierProvider.Default instance.
By default, LightBDD.XUnit2 uses this notifier.
DelegatingFeatureProgressNotifier and DelegatingScenarioProgressNotifier
Those are notifier allowing to delegate all notifications to many notifier instances. They do not emit any output by themselves.
NoProgressNotifier
This notifier offers no notification capability and it can be used if it is desired to switch off the notification. By default it is used in LightBDD.NUnit3 to disable feature level notifications as NUnit renders them after all scenarios (which is meaningless).
Customizing progress notification
Before LightBDD 3.5.0
A custom implementation of progress notification can be configured in LightBDD initialization. The LightBDD configuration mechanism is described in section LightBDD Configuration. It is possible to customize progress notification with following code:
configuration.Get<FeatureProgressNotifierConfiguration>()
.UpdateNotifier(new CustomFeatureProgressNotifier());
configuration.Get<ScenarioProgressNotifierConfiguration>()
.UpdateNotifierProvider(() => new CustomScenarioProgressNotifier());
There are following methods allowing to customize configuration:
UpdateNotifier/UpdateNotifierProvider- replaces existing notifier configuration with new one,ClearNotifiers/ClearNotifierProviders- clears existing notifier configuration by setting it toNoProgressNotifierAppendNotifiers/AppendNotifierProviders- appends 1 or more notifiers to the existing ones, allowing to combine them in easy manner,AppendFrameworkDefaultProgressNotifiers- appends integration specific default progress notifiers (useful if it is needed to re-apply them after usingClearNotifiers()/ClearNotifierProvidersmethods).
Example usage of new configuration methods:
configuration.Get<FeatureProgressNotifierConfiguration>()
.ClearNotifiers() // remove defaults
.AppendNotifiers(new CustomFeatureProgressNotifier()) // add custom notifiers
.AppendNotifiers(new CustomFeatureProgressNotifier2());
configuration.Get<ScenarioProgressNotifierConfiguration>()
.ClearNotifierProviders() // remove defaults
.AppendNotifierProviders(() => new CustomScenarioProgressNotifier()) // add custom notifiers
.AppendNotifierProviders<FeatureFixture>(
fixture => new CustomScenarioProgressNotifier2(fixture));
LightBDD 3.5.0 onward
With LightBDD 3.5.0 the IScenarioProgressNotifier and IFeatureProgressNotifier have been obsoleted in favor of a unified IProgressNotifier interface which now covers both, scenario and feature progress notification using the new event-based mechanism.
The configuration mechanism now looks as follows:
configuration.ProgressNotifierConfiguration()
.Clear() // remove defaults
.Append(new CustomProgressNotifier(), new CustomProgressNotifier2());
In case the notifier requires to use of test specific object such as ITestOutputProvider for LightBDD.XUnit2, the new ScenarioExecutionContext.GetCurrentScenarioFixtureIfPresent<T>() method can be used to retrieve it just before reporting the progress.
The implementation of XUnit2ProgressNotifier.cs uses this mechanism.
To configure
Continue reading: Generating Reports