Skip to content

Detecting Order-Dependent flaky tests using iDFlakies tools.

Notifications You must be signed in to change notification settings

kotlarmilos/OD-flaky-tests

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Flaky tests detection

Description

This repository contains nodejs script for detecting Order-Dependent flaky tests using iDFlakies tool. This repository contains projects directory for cloning github repositories. Directory unfixedFlakyTests contains list of unfixed flaky tests according to the http://mir.cs.illinois.edu/flakytests/unfixed.html. Directory mockedLogs contains mocked logs of command mvn testrunner:testplugin -Ddetector.detector_type=random-class-method -Ddt.randomize.rounds=10 -Ddt.detector.original_order.all_must_pass=false for faster development. Script downloads https://github.com/idflakies/iDFlakies repository and using pom-modify/modify-project.sh injects dependency for flaky tests detection.

Prerequests

  • Node v9.11.2
  • Java SDK 8

Usage

In order to run script change projectName and projectURL in script.js file and run the following command.

node script.js

Example output log

/usr/local/bin/node /Users/miloskotlar/Desktop/ODflakytests/script.js
=================================================================
             Running iDFlakies script for OD tests
=================================================================
               Project Name: openhtmltopdf
               Project URL: https://github.com/danfickle/openhtmltopdf
=================================================================

Latest commit from master branch is 73731543655c75604837a3725883b513938cc33b
Current commit from master branch is 4a0612fab7b3a55e3e11d485233b063006d03b0f

Running iDFlakies script for ID tests. This may take up to several hours.
Analysing results, this may take up to several minutes.

There are still 1 old unfixed flaky tests, 0 new flaky tests, and 34 fixed flaky tests.
Old flaky tests:
com.openhtmltopdf.slf4j.Slf4JLoggerTest.testLogger

New flaky tests:


Fixed flaky tests:
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testBookmarkHeadAfterOverflowPage
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testMetaInformation
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaMultipleBoxes
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaMultipleLine
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAfterOverflowTarget
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkTransformTarget
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testPr489DiagnosticConsumer
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaTransformRotate
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaAfterOverflowPage
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkOnOverflowTarget
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkSimpleBlock
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testBookmarkHeadNested
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testBookmarkHeadSimple
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testIssue458PageContentRepeatedInMargin
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaTransformNested
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaOverflowPage
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testFormControlText
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testPr492InfiniteLoopBugsInLineBreakingFuzz
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testFormControlOverflowPage
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testInputWithoutNameAttribute
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaPageMarginTransform
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testBookmarkHeadTransform
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaTransformTranslateY
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testBookmarkBodySimple
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaMultiplePage
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkInlineTarget
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testFormControlAfterOverflowPage
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkAreaPageMargin
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testPR480LinkShapes
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testBookmarkHeadOnOverflowPage
com.openhtmltopdf.nonvisualregressiontests.NonVisualRegressionTest.testLinkExternalUrl
com.openhtmltopdf.testcases.ConcateOutputTest.testConcateOutput
com.openhtmltopdf.testcases.TestcaseRunnerTest.runTestcaseRunner
com.openhtmltopdf.freemarker.FreeMarkerGeneratorTest.testFreeMarkerGenerator

Process finished with exit code 0

Further development

This script could be further expanded to iterate through projects and automatically detect changes for flaky tests.

About

Detecting Order-Dependent flaky tests using iDFlakies tools.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published