Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Usage explosion in parse command in version 0.2.37 #196

Open
hernancorigliano opened this issue Dec 11, 2023 · 8 comments
Open

Memory Usage explosion in parse command in version 0.2.37 #196

hernancorigliano opened this issue Dec 11, 2023 · 8 comments

Comments

@hernancorigliano
Copy link

hernancorigliano commented Dec 11, 2023

Hi, I'd like to report an issue after upgrading from 0.2.36 to 0.2.37.

The parse command started having an issue in which it starts taking up massive amounts of system memory (RAM) indefinitely until the OS collapses.

The issue is not happening for previous versions where the parse command finishes instantly.

I have tried using the mentioned flags --omit_notes --omit_warnings to simplify the parsing of a large log, but this won't stop the issue.

Steps to reproduce:

  • Build your app in Xcode 15.0.1
  • Run parse command as xclogparser parse --project MyProject --reporter flatJson --output log.json
  • The xclogparser process will slowly start taking up all available system memory. (Check the growing amount in Activity Monitor Memory tab)
  • (Manually kill the process before the OS crashes)

Context:

  • MacOS: Sonoma 14.1.2
  • Device: MacbookPro M1
  • Xcode: 15.0.1
  • XCLogParser installation: 0.2.37 (Homebrew)
@Parabak
Copy link

Parabak commented Dec 15, 2023

+1

1 similar comment
@seyfeddin
Copy link

+1

@BalestraPatrick
Copy link
Collaborator

Hello, unfortunately I don't have time to debug exactly what's wrong, but it must be one of the recently added contributions between the last two versions: v0.2.36...v0.2.37

I have a feeling it must be one of these 3 changes:

If you can locally try to revert each of those commits and see which one fixes it, that would be helpful.

@Parabak
Copy link

Parabak commented Dec 18, 2023

I can confirm that rolling back to version 0.2.36 worked for me

@bcylin
Copy link

bcylin commented Dec 20, 2023

Tested locally. Reverting #182 avoids the massive RAM usage. But the underlying issue seems to be the BuildStep struct that has too many subSteps.


With SwiftCompile as a detail type, it includes each file's compilation, such as:

SwiftCompile normal arm64 /opt/homebrew/var/agent/builds/ip-123-45-67-890-internal/project/pipeline/.../Sources/.../file.swift (in target '...' from project '...') ... etc

I tried it in a project with ~7,000 Swift files, it adds more than 400,000 sub steps into the parsed result, nested in 5 levels down.

In both use cases of --reporter json and --reporter flatJson, the struct becomes too large that JSONEncoder can't handle. The memory usage shoots up on line 42:

private func report<T: Encodable>(encodable: T, output: ReporterOutput) throws {
let encoder = JSONEncoder()
encoder.outputFormatting = .prettyPrinted
let json = try encoder.encode(encodable)
try output.write(report: json)
}

An alternative workaround is to use --reporter summaryJson as it drops all the sub steps in the SummaryJsonReporter.

@BalestraPatrick
Copy link
Collaborator

Reverting in #199 and then will cut a new release. Thanks for reporting!

@InspektorKek
Copy link

Same problem with JSON, but not only on last version. Actually on each version I face that issue(

Context:
MacOS: Sonoma 14.2.1
Device: MacbookPro M3 Pro
Xcode: 15.1
XCLogParser installation: rake build v 0.2.36

@daltonclaybrook
Copy link

Any update on this? It's unfortunate that SwiftCompile is not supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants