ConcurrentException

[anon767]

Hey,
I receive a concurrentException on some occurrences when enabling useParallelFrontends(true).
I can't tell what the root cause is. However, this is my TranslationConfiguration:

        val translationConfiguration = TranslationConfiguration.builder()
            .sourceLocations(filePaths)
            .defaultPasses()
            .defaultLanguages()
            .processAnnotations(true)
            .codeInNodes(true)
            .debugParser(false)
            .failOnError(false)
            .symbols(symbols)
            .useParallelFrontends(true)
            .typeSystemActiveInFrontend(true)
        return TranslationManager.builder().config(translationConfiguration.build()).build()

filePaths is a list of completely independent .c files from different projects. Might that be the problem?

Here is the stack trace:

Caused by: java.util.ConcurrentModificationException
	at java.base/java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex(IdentityHashMap.java:737)
	at java.base/java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:845)
	at java.base/java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:839)
	at java.base/java.util.IdentityHashMap.putAll(IdentityHashMap.java:508)
	at de.fraunhofer.aisec.cpg.passes.scopes.ScopeManager.mergeFrom(ScopeManager.kt:183)
	at de.fraunhofer.aisec.cpg.TranslationManager.parseParallel(TranslationManager.kt:267)
	at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.kt:203)
	at de.fraunhofer.aisec.cpg.TranslationManager.analyze$lambda-2(TranslationManager.kt:86)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1764)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1756)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Thanks in advance

[oxisto]

You need to set typeSystemActiveInFrontend to false, there should be a warning about this in the log. I know, the log is kind of full of warnings ;) The types will be set / activated after all frontends have been run successfully. In the future we want to remove this little hack and make the type parsing in parallel frontends more parallel

[anon767]

Ah, thanks!
I remember I added that because I sometimes ran into NullPointerExceptions when concurrently calling TranslationManager.analyze().
I can't reproduce it anymore, but if I recall correctly the frontend wasn't set in TypeManager.

Edit::

java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "de.fraunhofer.aisec.cpg.frontends.LanguageFrontend.getScopeManager()" because "this.frontend" is null

[oxisto]

Do you have a snippet to reproduce this?

[anon767]

Sure, it boils down to something like this:

fun getTranslationResult(paths: List<File>): TranslationManager {
    val translationConfiguration = TranslationConfiguration
        .builder()
        .sourceLocations(paths)
        .defaultPasses()
        .defaultLanguages()
        .processAnnotations(true)
        .codeInNodes(true)
        .debugParser(false)
        .failOnError(false)
        .useParallelFrontends(true)
        .typeSystemActiveInFrontend(false)
        .registerPass(IdentifierPass())
        .build()
    return TranslationManager.builder().config(translationConfiguration).build()
}
fun main(){
  val files = listOf(File("bla.c"), File("blub.c"), ...)
  files.parallelStream().forEach {
    val translationManager = getTranslationResult(listOf(it))
    try {
      translationResult = translationManager.analyze().get(5, TimeUnit.MINUTES)
    }catch(e: TimeoutException){}
  }
}

Unfortunately, I have to wrap it inside this try-catch, since sometimes the parser doesn't terminate, and I'd rather ignore that file then.
The downside is, of course, I have to calculate the Call Graph by myself and currently, it's slower since I'm using "parallelStream()" and receive the aforementioned NullPointerException.

Is there maybe a more sophisticated way to parse all files in a single analyze() pass, respecting a time limit for each file respectively?
Thanks in advance :)

[oxisto]

I meant the snippet for the source code file that seems to crash the analysis :) A time limit per file is indeed something that we do not have :( If the analysis does not finish in adequate time it seems to hint to another error.

If you are using C/C++, you might also benefit from the unity builds (without parallel), since they combine all the code into a single translation unit, removing double header includes, etc.

[anon767]

I meant the snippet for the source code file that seems to crash the analysis :) A time limit per file is indeed something that we do not have :( If the analysis does not finish in adequate time it seems to hint to another error.

The null pointer exceptions look kind of random to me (different files per different runs causing a crash).
This file for example is not finishing in reasonable time whatsoever: ianatables.c. It's not super important for me that this file gets parsed, since it yields little information of interest, however, the entire parsing should terminate at least. :D

If you are using C/C++, you might also benefit from the unity builds (without parallel), since they combine all the code into a single translation unit, removing double header includes, etc.

I tried unity builds, but it's not working for me. The graphs are just empty :( ( using those snippets https://gist.github.com/anon767/d47295dae6c3e63b157655c708c7585e )

Another more or less related question:
Is there a way to merge translationUnitDeclarations from different analysis passes? E.g. each file is analyzed and parsed separately (like in the example from my last post) and then merged into a single TranslationResult?
Then it would be possible to run a CallResolver pass over the merged translationDeclarations. When I tried that, I wasn't too sure how to populate the Scopes correctly.

Edit://
I think the unity build problem is my fault. I'm reading the files without specifying topLevel in the configuration. I guess that's the issue here right?