Pester returns errors even when no tests fail

[cjfarrelly]

Where are you running it?

• Azure DevOps Service (VSTS)

Version of Extension/Task

10.0.5

Expected behaviour and actual behaviour

Expected: Pester task passes without error when no tests fail

Actual: Pester task fails with error even though all tests pass

Steps to reproduce the problem

Here's a link to a public project where this issue can be repro'd

https://dev.azure.com/cfarrelly/test/_build/results?buildId=4&view=logs&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=f969aa47-2434-555b-2817-6587663ed32a

I thought originally it may be down to tags and skipping tests but I created a simple repro here with 1 test to run and the test runner fails with 'Pester returned errors', appears to be an issue when checking the results here

AzureDevOpsExtension/Extension/PesterTask/PesterV10/Pester.ps1

Line 145 in e2141f2

if ($result.failedCount -ne 0) {

Thanks,

Colin.

[ChrisLGardner]

Thanks for reporting this. I'll have a look at what's going on here and see if I can fix it this week.

[fmad]

Just got bitten by this one too: was starting to add some tests and couldn't figure out why they were passing on my local machine and even showing as passing in the output logs but then the task fails with:

D:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
D:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
+ d:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
 
+ d:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1

[ChrisLGardner]

Thanks for the extra reports. I'll bump this up my list and see if I can get it done tonight or earlier.

[dkalbertson]

Just hit this bug as well...

2020-06-03T12:15:39.5996504Z Tests Passed: 3, 
2020-06-03T12:15:39.6033071Z Failed: 0, 
2020-06-03T12:15:39.6070344Z Skipped: 0 
2020-06-03T12:15:39.6150264Z NotRun: 0
2020-06-03T12:15:39.6180214Z 
2020-06-03T12:15:39.6181136Z 
2020-06-03T12:15:40.2186133Z ##[error]C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

2020-06-03T12:15:40.2198914Z ##[error]C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

2020-06-03T12:15:40.2217307Z ##[error]+ C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
 

2020-06-03T12:15:40.2220096Z ##[error]+ C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
 

2020-06-03T12:15:40.2716171Z Pester Script finished
2020-06-03T12:15:40.2875525Z ##[section]Finishing: Pester Test Runner

[Indrakanth-star]

I too faced the same error and waiting for the solution. Please find below for the raw error log report.

Also the test result is showing as passed even though I intentionally placed the code to fail the test.

2020-06-04T10:22:55.0696725Z ##[error]D:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

• d:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...

2020-06-04T10:22:55.0709967Z ##[error]D:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

• d:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...

2020-06-04T10:22:55.0729293Z ##[error]+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1

2020-06-04T10:22:55.0731667Z ##[error]+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1

2020-06-04T10:22:55.1061073Z Pester Script finished
2020-06-04T10:22:55.1140399Z ##[section]Finishing: Pester Test Runner

[ChrisLGardner]

Thanks for the extra reports, I'm working on a fix but I also want to improve my testing in the pipeline to make sure I catch more of these issues before they hit users. It's going to take me a few hours of work to do that so I'm hoping to get it done this weekend.

[ChrisLGardner]

This should be fixed now in 10.1.x, took a bit longer than expected as my testing pipeline needed work and then wasn't quite doing what it should have.

[Indrakanth-star]

Hi @ChrisLGardner
Thanks for resolving the errors.
Still i am able to see the issue as below
"test result is showing as passed even though I intentionally placed the code to fail the test"

Can you please look into this too.

Thanks
Indra

[ChrisLGardner]

Hi @ChrisLGardner
Thanks for resolving the errors.
Still i am able to see the issue as below
"test result is showing as passed even though I intentionally placed the code to fail the test"

Can you please look into this too.

Thanks
Indra

I saw this a few times when I was fixing the task. What version is it trying to run? If you look in the logs for the task it'll give you an exact version number at the top.

[Indrakanth-star]

Task : Pester Test Runner
Description : Run Pester tests by either installing the latest version of Pester at run time (if possible) or using the version shipped with the task (5.0.0)
Version : 10.1.12
Author : Pester
Help : Version: #{Build.BuildNumber}#. More Information

[Indrakanth-star]

Hi @ChrisLGardner , Any update on the issue ?