类似于此处发布的解决方案TestNG retrying failed tests doesn't output the correct test results，我试图在onFinish（ITestContext上下文）中使用测试侦听器删除（重复）测试结果。
尽管使用context.getFailedTests（）。removeResult（result）删除结果似乎工作正常（结果实际上已被删除），但似乎存在“某些其他位置”，其中结果被拉出，因为构建仍然失败。
还要注意，当我从上面的文章（其中有一个重复的失败被删除和一个通过测试）运行样本测试时，我得到了“测试结果”（没有按预期清理）与“套件结果”的区别（重复故障按预期删除）。
而且，报告从哪里提取结果以决定是否使构建失败？或者只是它在我清理失败的测试之前拉动结果......？
===============================================
Default test
Tests run: 3, Failures: 2, Skips: 0
===============================================
===============================================
Default suite
Total tests run: 2, Failures: 1, Skips: 0
===============================================
编辑：只是为了澄清，我们正在使用maven运行这些测试，他们是IT，因此我们使用failsafe插件运行它们。问题是即使看起来测试被删除，mvn验证仍然无法构建，因为它认为无论如何都会发现构建失败。
而且如果从Eclipse运行这样的测试，即使删除了测试，当套件完成时，仍会在日志中打印失败。
关于RetryAnalyzer：我根本不会考虑使用RetryAnalyzer良好/最佳实践，但如果您发现自己处于需要解决问题的情况，例如你继承了一个依赖RetryAnalyzer的测试套件，你可能会觉得这很有用。
尝试使用此代码：
ListenerApadter：
public class MyTestListenerAdapter extends TestListenerAdapter {
@Override
public void onTestFailure(ITestResult result) {
if (result.getMethod().getRetryAnalyzer() != null) {
MyRetryAnalyzer retryAnalyzer = (MyRetryAnalyzer)result.getMethod().getRetryAnalyzer();
if(retryAnalyzer.isRetryAvailable()) {
result.setStatus(ITestResult.SKIP);
} else {
result.setStatus(ITestResult.FAILURE);
}
Reporter.setCurrentTestResult(result);
}
}
@Overrride
public void onFinish(ITestContext context) {
Iterator<ITestResult> failedTestCases =context.getFailedTests().getAllResults().iterator();
while (failedTestCases.hasNext()) {
System.out.println("failedTestCases");
ITestResult failedTestCase = failedTestCases.next();
ITestNGMethod method = failedTestCase.getMethod();
if (context.getFailedTests().getResults(method).size() > 1) {
System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
} else {
if (context.getPassedTests().getResults(method).size() > 0) {
System.out.println("failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
}
}
}
}
}
RetryAnalizer：
public class MyRetryAnalyzer implements IRetryAnalyzer {
private static int MAX_RETRY_COUNT = 3;
AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);
public boolean isRetryAvailable() {
return (count.intValue() > 0);
}
@Override
public boolean retry(ITestResult result) {
boolean retry = false;
if (isRetryAvailable()) {
System.out.println("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT);
retry = true;
count.decrementAndGet();
}
return retry;
}
}
POM.xml - > Surefire配置：
这是你应该配置“覆盖”surefire监听器的地方，它有自己的计数器。
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<suiteXmlFiles><suiteXmlFile>${basedir}/testng.xml</suiteXmlFile></suiteXmlFiles>
<properties>
<property>
<name>listener</name>
<value>Utils.MyTestListenerAdapter,Utils.MyRetryAnalizer</value>
</property>
</properties>
我最终选择了一个使用套件监听器的解决方案。
该解决方案可能无法完全清理TestNG /您的测试记录到控制台的内容，但如果您使用TestNG Jenkins插件，并且每次测试都先失败，然后成功，那么测试运行最终会变为绿色，即I猜是最重要的事情。
是的，我们运行mvn integration-test（不是mvn verify），让TestNG插件处理pass / fail。 The solution is quite similar / builds on what was posted here。
import java.util.Map;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
/**
* {@link ISuiteListener} implementation to clean up duplicate test results caused by retrying tests using the
* {@link RetryAnalyzer}
*/
public class SuiteResultListener implements ISuiteListener {
private static final Logger LOG = LogManager.getLogger();
@Override
public void onStart(ISuite suite) {
}
@Override
public void onFinish(ISuite suite) {
LOG.info("Cleaning up duplicate test failures in suite '" + suite.getName() + "' ...");
final Map<String, ISuiteResult> results = suite.getResults();
int removedFailures = 0;
for (ISuiteResult result : results.values()) {
final ITestContext testContext = result.getTestContext();
removedFailures += TestListenerUtil.cleanUpDuplicateFailures(testContext);
}
LOG.info("Removed " + removedFailures + " duplicate test failure(s) from suite '" + suite.getName() + "'");
}
}
这是TestListenerUtil类中发生的魔力：
public static int cleanUpDuplicateFailures(ITestContext testContext) {
final String testContextName = testContext.getName();
int removedFailures = 0;
LOG.info("Cleaning up failures in test context '" + testContextName + "' ...");
final Set<ITestResult> failedTests = testContext.getFailedTests().getAllResults();
if (failedTests.isEmpty()) {
LOG.info("There are no failures in test context '" + testContextName + "'\n");
} else {
// collect all id's from passed test
final Set<Integer> passedTestIds = new HashSet<>();
final Set<ITestResult> passedTests = testContext.getPassedTests().getAllResults();
LOG.info("Analyzing " + passedTests.size() + " passed test(s)");
for (ITestResult result : passedTests) {
final int testId = TestListenerUtil.getId(result);
passedTestIds.add(testId);
LOG.info(" Passed test " + TestListenerUtil.getName(result) + ": #" + testId + " @ "
+ getStartTime(result));
}
// check which failed test results should be removed
final List<Integer> resultsToBeRemoved = new ArrayList<>();
final Set<Integer> failedTestIds = new HashSet<>();
LOG.info("Analyzing " + failedTests.size() + " failed test(s)");
for (ITestResult result : failedTests) {
final int testId = TestListenerUtil.getId(result);
final String name = TestListenerUtil.getName(result);
// if we saw this test pass or fail before we mark the result for deletion
if (failedTestIds.contains(testId) || passedTestIds.contains(testId)) {
LOG.info(" Adding test " + name + " to be removed: #" + testId + " @ " + getStartTime(result));
resultsToBeRemoved.add(testId);
} else {
LOG.info(" Remembering failed test " + name + ": #" + testId + " @ " + getStartTime(result));
failedTestIds.add(testId);
}
}
// finally delete all duplicate failures (if any)
final int duplicateFailures = resultsToBeRemoved.size();
if (duplicateFailures > 0) {
LOG.info("Cleaning up failed tests (expecting to remove " + resultsToBeRemoved.size()
+ " result(s)) ...");
for (ITestResult result : testContext.getFailedTests().getAllResults()) {
final int testId = TestListenerUtil.getId(result);
final String info = TestListenerUtil.getName(result) + ": #" + testId + " @ "
+ getStartTime(result);
if (resultsToBeRemoved.contains(testId)) {
LOG.info(" Removing failed test result " + info);
testContext.getFailedTests().removeResult(result);
resultsToBeRemoved.remove((Integer) testId);
removedFailures++;
} else {
LOG.info(" Not removing failed test result " + info);
}
}
}
if (removedFailures == duplicateFailures) {
LOG.info("Removed " + removedFailures + " failed test result(s) in '" + testContextName + "'\n");
} else {
LOG.warn("Removed " + removedFailures + " failed test result(s) in '" + testContextName
+ "' (expected to remove " + duplicateFailures + ")\n");
}
}
return removedFailures;
}
使用这两个额外的utils方法：
public static String getName(ITestResult result) {
final List<String> parameters = new ArrayList<>();
if (result.getParameters() != null) {
for (Object parameter : result.getParameters()) {
if (parameter instanceof TestResult && ((TestResult) parameter).getStatus() < 0) {
// TestResult.toString() will explode with status < 0, can't use the toString() method
parameters.add(parameter.getClass().getName() + "@" + parameter.hashCode());
} else {
parameters.add(parameter == null ? "null" : parameter.toString());
}
}
}
return result.getTestClass().getRealClass().getSimpleName() + "." + result.getMethod().getMethodName() + "("
+ StringUtils.join(parameters, ",") + ")";
}
public static int getId(ITestResult result) {
final HashCodeBuilder builder = new HashCodeBuilder();
builder.append(result.getTestClass().getRealClass());
builder.append(result.getMethod().getMethodName());
builder.append(result.getParameters());
return builder.toHashCode();
}
此外，如果您对我们的RetryAnalyzer如何工作感兴趣，请参阅下文。
需要理解的一件事是，我们在RetryAnalyzer和重复结果清理中都考虑了测试方法的参数。这些是相关的原因我们经常使用DataProviders。
import java.util.HashMap;
import java.util.Map;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryAnalyzer implements IRetryAnalyzer {
private static final Logger LOG = LogManager.getLogger();
private static Integer maxRetries;
private final Map<Integer, Integer> retryCount = new HashMap<>();
@Override
public boolean retry(ITestResult result) {
// only re-try failures
if (result.getStatus() == ITestResult.FAILURE) {
final String testName = TestListenerUtil.getName(result);
final int count = getRetryCount(result);
final int maxRetriesAllowed = getMaxRetriesAllowed();
if (count < maxRetriesAllowed) {
retryCount.put(TestListenerUtil.getId(result), count + 1);
LOG.info("Retrying test (attempt " + (count + 1) + "/" + maxRetriesAllowed + "): " + testName);
return true;
} else {
LOG.error("Failing test after " + count + " retries: " + testName);
}
}
return false;
}
public boolean canRetry(ITestResult result) {
return result.getStatus() == ITestResult.FAILURE && getRetryCount(result) < getMaxRetriesAllowed();
}
private int getRetryCount(ITestResult result) {
final int testId = TestListenerUtil.getId(result);
return retryCount.containsKey(testId) ? retryCount.get(testId) : 0;
}
public static int getMaxRetriesAllowed() {
return maxRetries == null ? Config.MAX_TEST_RETRIES : maxRetries;
}
}