Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed 35 different flaky tests with ID category #4783

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

xinying-7
Copy link

@xinying-7 xinying-7 commented Nov 15, 2024

These tests are failing NonDex since they're directly comparing JSON Strings which can have different key order when run multiple times.

Click on to see more details on the error message
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   NumericFormatTest.testFormatOfDecimal:48 expected [{
  "ModelWithDecimalFields" : {
    "type" : "object",
    "properties" : {
      "id" : {
        "minimum" : 3.3,
        "exclusiveMinimum" : false,
        "type" : "number",
        "format" : "double"
      }
    }
  }
}] but found [{
  "ModelWithDecimalFields" : {
    "properties" : {
      "id" : {
        "minimum" : 3.3,
        "format" : "double",
        "exclusiveMinimum" : false,
        "type" : "number"
      }
    },
    "type" : "object"
  }
}]

All the tests I fixed have similar issues, so reproducing them requires similar steps. For example, for the testFormatOfDecimal function located inio/swagger/v3/core/converting/NumericFormatTest.java , Here's the steps to reproduce:

Run this at the root directory:
mvn -pl modules/swagger-core edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=io.swagger.v3.core.converting.NumericFormatTest#testFormatOfDecimal -DnondexRuns=1

(Note: The failing test might not be seen if every test happens to have the JSON string in the correct order. Try running it several times or increase the the number of runs with -DnondexRuns= to reproduce the issue.)

To fix this, I converted the actual and expected JSON String into JSON objects and compare with JSONAssert.assertEquals, which ensures correct equality checks regardless of key order.

@xinying-7 xinying-7 changed the title fixing flaky test testFormatOfBigDecimal fixing flaky test testFormatOfDecimal Nov 16, 2024
@xinying-7 xinying-7 changed the title fixing flaky test testFormatOfDecimal fixing 14 different flaky tests with ID category Nov 16, 2024
@xinying-7 xinying-7 changed the title fixing 14 different flaky tests with ID category Fixed14 different flaky tests with ID category Nov 16, 2024
fixing flaky test serializeDoubleProperty

fix flaky test testFormatOfBigDecimal

fix flaky test testExtensionObjectWithProperties

fix flaky test serializeReadOnlyStringProperty

fix flaky test serializeArrayModel

fix flaky test deserializeArrayModel

fix flaky test serializeArrayStringProperty

fix flaky test deserializeArrayStringProperty

fix flaky test serializeDateTimeProperty

fix flaky test deserializeDateTimeProperty

fix flaky test serializeObjectPropertyWithRequiredProperties

fix flaky test serializeLongMapProperty

fix flaky test deserializeLongMapProperty

fixed flaky test deserializeIntegerProperty

fixed 9 flaky tests in PropertySerializationTest.java: 'deserializeNotReadOnlyStringProperty', 'deserializeIntegerMapProperty', 'serializeFloatProperty', 'deserializeDoubleProperty', 'serializeIntegerProperty', 'serializeLongProperty', 'deserializeLongProperty', 'deserializeFloatProperty', 'serializeIntegerMapProperty'
@xinying-7 xinying-7 changed the title Fixed14 different flaky tests with ID category Fixed 31 different flaky tests with ID category Dec 6, 2024
@xinying-7 xinying-7 changed the title Fixed 31 different flaky tests with ID category Fixed 35 different flaky tests with ID category Dec 6, 2024
@xinying-7
Copy link
Author

The previous comment is for the majority of test cases. Here's some specialized tests which I used different approach to fix:

For test case deserialization.JsonDeserializationTest.testNullExampleAndValues, it was comparing serialized strings , which caused it to fail NonDex due to potential variations in key order. So I used map-based comparison to ensure logical equality of the YAML structure, regardless of the key order in the serialized output.

For test case converting.ModelConverterTest.serializeParameterizedType, it fails NonDex due to the non-deterministic key order in class schema, the iterator et.next() doesn't always return id as the first key and the following keys in the same order.
Since the main prupose is to check if each key has the correct schema type, I fixed the flakiness by directly access them by name instead of iterating, for example, using props.get("id").

For test case resolving.JsonPropertyTest.testTicket2845, it fails NonDex because of variations in key order as well. So I split each lines in the YAML string and store them into HashSet objects, and directly comparing the HashSet objects, which would make sure the comparison passes regardless of order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant