Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Java ITs: Enable parallelization #7570

Closed
wants to merge 2 commits into from
Closed

Java ITs: Enable parallelization #7570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
764d575
Enable Java IT parallelization
gregory-paidis-sonarsource Jul 10, 2023
68329a7
Try one thread per CPU core
gregory-paidis-sonarsource Jul 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions its/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>classes</parallel>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you testing this with C#-only run to get local data? While excluding the time needed to download SQ zip at the 1st local SQ-cache populating run?

Because from what I saw, surefire is stupid, JUnit is stupid, and they are not any smarter together in test discovery. So "parallel" might not play nicely with "Suite", only with the usual "run all at once" scaffolding

<threadCount>1</threadCount>
<perCoreThreadCount>true</perCoreThreadCount>
<includes>
<include>com/sonar/it/csharp/Tests.java</include>
<include>com/sonar/it/vbnet/Tests.java</include>
Expand Down
15 changes: 8 additions & 7 deletions its/src/test/java/com/sonar/it/csharp/MetricsIncludeHeaderCommentTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,10 @@ public class MetricsIncludeHeaderCommentTest {

public static TemporaryFolder temp = TestUtils.createTempFolder();

private static final String PROJECT = "MetricsTest";
private static final String DIRECTORY = "MetricsTest:foo";
private static final String FILE = "MetricsTest:foo/Class1.cs";
private static final String PROJECT_DIR = "MetricsTest";
pavel-mikula-sonarsource marked this conversation as resolved.
Show resolved Hide resolved
private static final String PROJECT_NAME = "MetricsTestIncludeHeaderComment";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private static final String PROJECT_NAME = "MetricsTestIncludeHeaderComment";
private static final String PROJECT_KEY = "MetricsTestIncludeHeaderComment";

private static final String DIRECTORY = "MetricsTestIncludeHeaderComment:foo";
private static final String FILE = "MetricsTestIncludeHeaderComment:foo/Class1.cs";
private static final int NUMBER_OF_HEADER_COMMENT_LINES = 2;

// This is where the analysis is done.
Expand All @@ -63,7 +64,7 @@ public class MetricsIncludeHeaderCommentTest {

@Test
public void projectIsAnalyzed() {
assertThat(getComponent(PROJECT).getName()).isEqualTo("MetricsTest");
assertThat(getComponent(PROJECT_NAME).getName()).isEqualTo("MetricsTestIncludeHeaderComment");
assertThat(getComponent(DIRECTORY).getName()).isEqualTo("foo");
assertThat(getComponent(FILE).getName()).isEqualTo("Class1.cs");
}
Expand Down Expand Up @@ -122,7 +123,7 @@ public void commentLinesAtFileLevel() {
/* Helper methods */

private Integer getProjectMeasureAsInt(String metricKey) {
return getMeasureAsInt(PROJECT, metricKey);
return getMeasureAsInt(PROJECT_NAME, metricKey);
}

private Integer getDirectoryMeasureAsInt(String metricKey) {
Expand All @@ -144,9 +145,9 @@ private static RuleChain getRuleChain() {
protected void before() throws Throwable {
TestUtils.initLocal(ORCHESTRATOR);

Path projectDir = Tests.projectDir(temp, "MetricsTest");
Path projectDir = Tests.projectDir(temp, PROJECT_DIR);

ScannerForMSBuild beginStep = TestUtils.createBeginStep("MetricsTest", projectDir)
ScannerForMSBuild beginStep = TestUtils.createBeginStep(PROJECT_NAME, projectDir)
.setProfile("no_rule")
// Without that, the MetricsTest project is considered as a Test project :)
.setProperty("sonar.msbuild.testProjectPattern", "noTests");
Expand Down
15 changes: 10 additions & 5 deletions its/src/test/java/com/sonar/it/csharp/Tests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,16 @@ public class Tests {
.restoreProfileAtStartup(FileLocation.of("profiles/custom_complexity.xml"))
.build();

@ClassRule
public static final TemporaryFolder temp = TestUtils.createTempFolder();

@BeforeClass
public static void init() throws IOException {
TestUtils.deleteLocalCache();
// Analyze a project to avoid race conditions
analyzeProject(temp, "Empty");
}

public static Path projectDir(TemporaryFolder temp, String projectName) throws IOException {
Path projectDir = Paths.get("projects").resolve(projectName);
FileUtils.deleteDirectory(new File(temp.getRoot(), projectName));
Expand All @@ -94,11 +104,6 @@ public static Path projectDir(TemporaryFolder temp, String projectName) throws I
return tmpProjectDir;
}

@BeforeClass
public static void deleteLocalCache() {
TestUtils.deleteLocalCache();
}

static BuildResult analyzeProject(TemporaryFolder temp, String projectDir) throws IOException {
return analyzeProject(projectDir, temp, projectDir);
}
Expand Down
11 changes: 11 additions & 0 deletions its/src/test/java/com/sonar/it/shared/Tests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import javax.annotation.CheckForNull;
import javax.annotation.Nullable;
import org.apache.commons.io.FileUtils;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
Expand All @@ -54,6 +55,16 @@ public class Tests {
.addPlugin(MavenLocation.of("org.sonarsource.html", "sonar-html-plugin", "3.2.0.2082"))
.build();

@ClassRule
public static final TemporaryFolder temp = TestUtils.createTempFolder();

@BeforeClass
public static void init() throws IOException {
TestUtils.deleteLocalCache();
// Analyze a project to avoid race conditions
analyzeProject(temp, "Empty");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory, you can skip the msbuild part. And just run "begin" and "end", where "end" will emit some useless warning that sarif report and other stuff was not configured for C# analysis.

You would just get project dir, execute begin step, execute end step. You can make it a method in TestUtils that

  • would deleteLocalCache() - could be made private then
  • would do the scanner thing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There would also be no more accidental cache deletions

}

public static Path projectDir(TemporaryFolder temp, String projectName) throws IOException {
Path projectDir = Paths.get("projects").resolve(projectName);
FileUtils.deleteDirectory(new File(temp.getRoot(), projectName));
Expand Down
15 changes: 10 additions & 5 deletions its/src/test/java/com/sonar/it/vbnet/Tests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@
import org.junit.runners.Suite.SuiteClasses;
import org.sonarqube.ws.Components;
import org.sonarqube.ws.Hotspots;
import org.sonarqube.ws.Issues;
import org.sonarqube.ws.Measures;

@RunWith(Suite.class)
Expand All @@ -64,6 +63,16 @@ public class Tests {
.restoreProfileAtStartup(FileLocation.of("profiles/vbnet_class_name.xml"))
.build();

@ClassRule
public static final TemporaryFolder temp = TestUtils.createTempFolder();

@BeforeClass
public static void init() throws IOException {
TestUtils.deleteLocalCache();
// Analyze a project to avoid race conditions
analyzeProject(temp, "Empty");
}

public static Path projectDir(TemporaryFolder temp, String projectName) throws IOException {
Path projectDir = Paths.get("projects").resolve(projectName);
FileUtils.deleteDirectory(new File(temp.getRoot(), projectName));
Expand Down Expand Up @@ -109,10 +118,6 @@ static Measures.Measure getMeasure(String componentKey, String metricKey) {
return TestUtils.getMeasure(ORCHESTRATOR, componentKey, metricKey);
}

static List<Issues.Issue> getIssues(String componentKey) {
return TestUtils.getIssues(ORCHESTRATOR, componentKey);
}

static List<Hotspots.SearchWsResponse.Hotspot> getHotspots(String projectKey) {
return TestUtils.getHotspots(ORCHESTRATOR, projectKey);
}
Expand Down