Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Program Dependence Graph #1227

Merged
merged 66 commits into from
Aug 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
19ef696
Try to use a more formal worklist EOG worklist iteration for the CFSe…
KuechA Mar 17, 2023
50e1d48
Add new SplitsControlFlow interface
KuechA Mar 18, 2023
67086bc
New unreachable EOG edges pass
KuechA Mar 19, 2023
d2dc925
Handle AssignExpressions in ControlFlowSensitiveDFGPass
KuechA Mar 21, 2023
56c8a55
Mark statements which should be irrelevant because the same edges are…
KuechA Mar 21, 2023
f1cd62b
Remove setting unnecessary (already existing) edges from CFSensitiveD…
KuechA Mar 23, 2023
3229f54
ConditionalExpression splits control flow
KuechA Mar 27, 2023
f5e6342
Fix issue 1141
KuechA Mar 27, 2023
3f58534
Fix issue 1141
KuechA Mar 27, 2023
a2ab3aa
Merge branch 'main' into worklist-eog-iteration
KuechA Mar 27, 2023
9a57f2b
Consistency
KuechA Mar 28, 2023
68d97fe
Rename class and add fields
KuechA May 12, 2023
71fc6f7
Generate CDG
KuechA May 16, 2023
3e50dd6
Merge main into worklist-eog-iteration
KuechA May 16, 2023
cb035c8
Fix problem with very strange EOG self reference of literals
KuechA May 17, 2023
011ab5d
Test if statements and cdg
KuechA May 19, 2023
3396b0a
Add test for foreach loop
KuechA May 19, 2023
8560256
Reduce sonar warnings
KuechA May 22, 2023
1356f94
Fix failing tests
KuechA May 22, 2023
6262559
Merge branch 'main' into worklist-eog-iteration
KuechA May 22, 2023
ee8e2ed
Use HasShortcircuitOperators to generate ShortcircuitOperator objects…
KuechA May 22, 2023
26d25d5
Merge branch 'main' into worklist-eog-iteration
KuechA May 23, 2023
8f5e0c4
Little fix
KuechA May 23, 2023
0edfa6e
Integrate review feedback
KuechA May 31, 2023
b6090dd
Merge main into worklist-eog-iteration
KuechA May 31, 2023
648fa65
Merge main into worklist-eog-iteration
KuechA May 31, 2023
871530a
More review feedback
KuechA May 31, 2023
73b741b
Update cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/helpers/EOGWo…
KuechA Jun 7, 2023
cb9f2a2
Merge two iterateEOG functions
KuechA Jun 7, 2023
4c54266
Merge branch 'main' into worklist-eog-iteration
KuechA Jun 7, 2023
4ecc621
Fluent DSL for loops
oxisto Jun 11, 2023
c79d7cb
Fix redundant stuff
KuechA Jun 13, 2023
6ce7340
Revert "Fluent DSL for loops"
KuechA Jun 13, 2023
5a237eb
Get rid of "expectedUpdate"
KuechA Jun 13, 2023
2e6f440
Change method signatures
KuechA Jun 13, 2023
83971cf
Merge branch 'main' into worklist-eog-iteration
KuechA Jun 13, 2023
e67a75c
Small cleanup
KuechA Jun 14, 2023
6b1ff1b
Merge branch 'main' into worklist-eog-iteration
KuechA Jun 14, 2023
2f5ec72
Rename lattice to latticeelement
KuechA Jun 16, 2023
faa7d70
Remove nullable properties
KuechA Jun 16, 2023
9455a71
Nicer code
KuechA Jun 16, 2023
39b0bbd
Documentation
KuechA Jun 16, 2023
f2377fa
Merge branch 'main' into worklist-eog-iteration
KuechA Jun 16, 2023
94496dd
first draft for generating pdg property edges
seelchen Jun 16, 2023
0279df3
change delegate constructor parameter to KProperty1 for consistency
seelchen Jun 19, 2023
19e6b02
change PDGEdges properties from using delegates to custom getters
seelchen Jun 19, 2023
339bbec
add EventListener to set pdg edges before neo4j serialization
seelchen Jun 22, 2023
998d1b3
change to adding the PDG edges through a new Pass
seelchen Jun 23, 2023
cc9dd65
fix typos
seelchen Jun 26, 2023
d55fbc4
add test for the ProgramDependenceGraphPass
seelchen Jun 26, 2023
25f7995
add more tests
seelchen Jun 29, 2023
7423ab5
add documentation
seelchen Jun 30, 2023
777da64
fix format violation
seelchen Jun 30, 2023
5a543fb
remove unused delegate class
seelchen Jun 30, 2023
3e47441
rename pass to ProgramDependenceGraphPass and add documentation
seelchen Jul 3, 2023
35a7d73
rename pdgSetter to visitor
seelchen Jul 3, 2023
3466320
add new edge property DEPENDENCE
seelchen Jul 7, 2023
104fa00
use new DEPENDENCE property for PDGEdges and change addAllPrevPDGEdge…
seelchen Jul 7, 2023
ae1c173
change test to not rely on hashCode for comparing the lists
seelchen Jul 7, 2023
f9d2a8c
Merge branch 'main' into add-pdg-properties
KuechA Jul 28, 2023
01d4922
Fix problems of merge
KuechA Jul 28, 2023
93aec2b
change to use DFG PropertyEdges instead of the nodes
seelchen Aug 2, 2023
e193aa1
Merge remote-tracking branch 'origin/main' into add-pdg-properties
seelchen Aug 2, 2023
ed0255d
change to use sets for storing PDG and adjust PropertyEdgeSetDelegate…
seelchen Aug 2, 2023
4aa9c7f
fix types in test
seelchen Aug 2, 2023
f666f0c
Merge branch 'main' into add-pdg-properties
seelchen Aug 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 35 additions & 5 deletions cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/Node.kt
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import de.fraunhofer.aisec.cpg.graph.declarations.MethodDeclaration
import de.fraunhofer.aisec.cpg.graph.declarations.RecordDeclaration
import de.fraunhofer.aisec.cpg.graph.declarations.TranslationUnitDeclaration
import de.fraunhofer.aisec.cpg.graph.declarations.TypedefDeclaration
import de.fraunhofer.aisec.cpg.graph.edge.DependenceType
import de.fraunhofer.aisec.cpg.graph.edge.Properties
import de.fraunhofer.aisec.cpg.graph.edge.PropertyEdge
import de.fraunhofer.aisec.cpg.graph.edge.PropertyEdge.Companion.unwrap
Expand All @@ -46,14 +47,11 @@ import de.fraunhofer.aisec.cpg.graph.scopes.Scope
import de.fraunhofer.aisec.cpg.helpers.SubgraphWalker
import de.fraunhofer.aisec.cpg.helpers.neo4j.LocationConverter
import de.fraunhofer.aisec.cpg.helpers.neo4j.NameConverter
import de.fraunhofer.aisec.cpg.passes.ControlDependenceGraphPass
import de.fraunhofer.aisec.cpg.passes.ControlFlowSensitiveDFGPass
import de.fraunhofer.aisec.cpg.passes.DFGPass
import de.fraunhofer.aisec.cpg.passes.EvaluationOrderGraphPass
import de.fraunhofer.aisec.cpg.passes.FilenameMapper
import de.fraunhofer.aisec.cpg.passes.*
import de.fraunhofer.aisec.cpg.processing.IVisitable
import de.fraunhofer.aisec.cpg.sarif.PhysicalLocation
import java.util.*
import kotlin.collections.ArrayList
import org.apache.commons.lang3.builder.ToStringBuilder
import org.apache.commons.lang3.builder.ToStringStyle
import org.neo4j.ogm.annotation.*
Expand Down Expand Up @@ -208,6 +206,24 @@ open class Node : IVisitable<Node>, Persistable, LanguageProvider, ScopeProvider
@PopulatedByPass(DFGPass::class, ControlFlowSensitiveDFGPass::class)
var nextDFG: MutableSet<Node> by PropertyEdgeSetDelegate(Node::nextDFGEdges, true)

/** Outgoing Program Dependence Edges. */
@PopulatedByPass(ProgramDependenceGraphPass::class)
@Relationship(value = "PDG", direction = Relationship.Direction.OUTGOING)
var nextPDGEdges: MutableSet<PropertyEdge<Node>> = mutableSetOf()
protected set

/** Virtual property for accessing the children of the Program Dependence Graph (PDG). */
var nextPDG: MutableSet<Node> by PropertyEdgeSetDelegate(Node::nextPDGEdges, true)

/** Incoming Program Dependence Edges. */
@PopulatedByPass(ProgramDependenceGraphPass::class)
@Relationship(value = "PDG", direction = Relationship.Direction.INCOMING)
var prevPDGEdges: MutableSet<PropertyEdge<Node>> = mutableSetOf()
protected set

/** Virtual property for accessing the parents of the Program Dependence Graph (PDG). */
var prevPDG: MutableSet<Node> by PropertyEdgeSetDelegate(Node::prevPDGEdges, false)

var typedefs: MutableSet<TypedefDeclaration> = HashSet()

/**
Expand Down Expand Up @@ -300,6 +316,20 @@ open class Node : IVisitable<Node>, Persistable, LanguageProvider, ScopeProvider
prev.forEach { addPrevDFG(it, properties.toMutableMap()) }
}

fun addAllPrevPDG(prev: Collection<Node>, dependenceType: DependenceType) {
addAllPrevPDGEdges(prev.map { PropertyEdge(it, this) }, dependenceType)
}

fun addAllPrevPDGEdges(prev: Collection<PropertyEdge<Node>>, dependenceType: DependenceType) {

prev.forEach {
val edge = PropertyEdge(it).apply { addProperty(Properties.DEPENDENCE, dependenceType) }
this.prevPDGEdges.add(edge)
val other = if (it.start != this) it.start else it.end
other.nextPDGEdges.add(edge)
}
}

fun removePrevDFG(prev: Node?) {
if (prev != null) {
val thisRemove =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,22 @@ package de.fraunhofer.aisec.cpg.graph.edge
*
* [UNREACHABLE]:(boolean) True if the edge flows into unreachable code i.e. a branch condition
* which is always false.
*
* [DEPENDENCE]: ([DependenceType] Specifies the type of dependence the property edge might
* represent
*/
enum class Properties {
INDEX,
BRANCH,
NAME,
INSTANTIATION,
UNREACHABLE,
ACCESS
ACCESS,
DEPENDENCE
}

/** The types of dependencies that might be represented in the CPG */
enum class DependenceType {
CONTROL,
DATA
}
Original file line number Diff line number Diff line change
Expand Up @@ -377,11 +377,11 @@ class PropertyEdgeDelegate<T : Node, S : Node>(
/** Similar to a [PropertyEdgeDelegate], but with a [Set] instead of [List]. */
@Transient
class PropertyEdgeSetDelegate<T : Node, S : Node>(
val edge: KProperty1<S, List<PropertyEdge<T>>>,
val edge: KProperty1<S, Collection<PropertyEdge<T>>>,
val outgoing: Boolean = true
) {
operator fun getValue(thisRef: S, property: KProperty<*>): MutableSet<T> {
return PropertyEdge.unwrap(edge.get(thisRef), outgoing).toMutableSet()
return PropertyEdge.unwrap(edge.get(thisRef).toList(), outgoing).toMutableSet()
}

operator fun setValue(thisRef: S, property: KProperty<*>, value: MutableSet<T>) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
/*
* Copyright (c) 2023, Fraunhofer AISEC. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* $$$$$$\ $$$$$$$\ $$$$$$\
* $$ __$$\ $$ __$$\ $$ __$$\
* $$ / \__|$$ | $$ |$$ / \__|
* $$ | $$$$$$$ |$$ |$$$$\
* $$ | $$ ____/ $$ |\_$$ |
* $$ | $$\ $$ | $$ | $$ |
* \$$$$$ |$$ | \$$$$$ |
* \______/ \__| \______/
*
*/
package de.fraunhofer.aisec.cpg.passes

import de.fraunhofer.aisec.cpg.TranslationContext
import de.fraunhofer.aisec.cpg.graph.*
import de.fraunhofer.aisec.cpg.graph.declarations.TranslationUnitDeclaration
import de.fraunhofer.aisec.cpg.graph.edge.DependenceType
import de.fraunhofer.aisec.cpg.passes.order.DependsOn
import de.fraunhofer.aisec.cpg.processing.IVisitor
import de.fraunhofer.aisec.cpg.processing.strategy.Strategy

/**
* This pass collects the dependence information of each node into a Program Dependence Graph (PDG)
* by traversing through the AST.
*/
@DependsOn(ControlDependenceGraphPass::class)
@DependsOn(DFGPass::class)
@DependsOn(ControlFlowSensitiveDFGPass::class, softDependency = true)
class ProgramDependenceGraphPass(ctx: TranslationContext) : TranslationUnitPass(ctx) {
private val visitor =
object : IVisitor<Node>() {
/**
* Collects the data and control dependence edges of a node and adds them to the program
* dependence edges
*/
override fun visit(t: Node) {
t.addAllPrevPDGEdges(t.prevDFGEdges, DependenceType.DATA)
t.addAllPrevPDGEdges(t.prevCDGEdges, DependenceType.CONTROL)
}
}

override fun accept(tu: TranslationUnitDeclaration) {
tu.statements.forEach(::handle)
tu.namespaces.forEach(::handle)
tu.declarations.forEach(::handle)
}

override fun cleanup() {
// Nothing to do
}

private fun handle(node: Node) {
node.accept(Strategy::AST_FORWARD, visitor)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
/*
* Copyright (c) 2023, Fraunhofer AISEC. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* $$$$$$\ $$$$$$$\ $$$$$$\
* $$ __$$\ $$ __$$\ $$ __$$\
* $$ / \__|$$ | $$ |$$ / \__|
* $$ | $$$$$$$ |$$ |$$$$\
* $$ | $$ ____/ $$ |\_$$ |
* $$ | $$\ $$ | $$ | $$ |
* \$$$$$ |$$ | \$$$$$ |
* \______/ \__| \______/
*
*/
package de.fraunhofer.aisec.cpg.passes

import de.fraunhofer.aisec.cpg.*
import de.fraunhofer.aisec.cpg.frontends.TestLanguage
import de.fraunhofer.aisec.cpg.frontends.TestLanguageFrontend
import de.fraunhofer.aisec.cpg.graph.Node
import de.fraunhofer.aisec.cpg.graph.builder.*
import de.fraunhofer.aisec.cpg.graph.edge.DependenceType
import de.fraunhofer.aisec.cpg.graph.edge.Properties
import de.fraunhofer.aisec.cpg.graph.edge.PropertyEdge
import de.fraunhofer.aisec.cpg.graph.functions
import de.fraunhofer.aisec.cpg.graph.get
import de.fraunhofer.aisec.cpg.processing.IVisitor
import de.fraunhofer.aisec.cpg.processing.strategy.Strategy
import java.util.*
import java.util.stream.Stream
import kotlin.test.assertNotNull
import kotlin.test.assertTrue
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource

class ProgramDependenceGraphPassTest {

@ParameterizedTest(name = "test if pdg of {1} is equal to the union of cdg and dfg")
@MethodSource("provideTranslationResultForPDGTest")
fun `test if pdg is equal to union of cdg and dfg`(result: TranslationResult, name: String) {
assertNotNull(result)
val main = result.functions["main"]
assertNotNull(main)

main.accept(
Strategy::AST_FORWARD,
object : IVisitor<Node>() {
override fun visit(t: Node) {
val expectedPrevEdges =
t.prevCDGEdges.map {
it.apply { addProperty(Properties.DEPENDENCE, DependenceType.CONTROL) }
} +
t.prevDFG.map {
PropertyEdge(it, t).apply {
addProperty(Properties.DEPENDENCE, DependenceType.DATA)
}
}
assertTrue(
"prevPDGEdges did not contain all prevCDGEdges and edges to all prevDFG.\n" +
"expectedPrevEdges: ${expectedPrevEdges.sortedBy { it.hashCode() }}\n" +
"actualPrevEdges: ${t.prevPDGEdges.sortedBy { it.hashCode() }}"
) {
compareCollectionWithoutOrder(expectedPrevEdges, t.prevPDGEdges)
}

val expectedNextEdges =
t.nextCDGEdges.map {
it.apply { addProperty(Properties.DEPENDENCE, DependenceType.CONTROL) }
} +
t.nextDFG.map {
PropertyEdge(t, it).apply {
addProperty(Properties.DEPENDENCE, DependenceType.DATA)
}
}
assertTrue(
"nextPDGEdges did not contain all nextCDGEdges and edges to all nextDFG." +
"\nexpectedNextEdges: ${expectedNextEdges.sortedBy { it.hashCode() }}" +
"\nactualNextEdges: ${t.nextPDGEdges.sortedBy { it.hashCode() }}"
) {
compareCollectionWithoutOrder(expectedNextEdges, t.nextPDGEdges)
}
}
}
)
}

private fun <T> compareCollectionWithoutOrder(
expected: Collection<T>,
actual: Collection<T>
): Boolean {
val expectedWithDuplicatesGrouped = expected.groupingBy { it }.eachCount()
val actualWithDuplicatesGrouped = actual.groupingBy { it }.eachCount()

return expected.size == actual.size &&
expectedWithDuplicatesGrouped == actualWithDuplicatesGrouped
}

companion object {
fun testFrontend(config: TranslationConfiguration): TestLanguageFrontend {
val ctx = TranslationContext(config, ScopeManager(), TypeManager())
val language = config.languages.filterIsInstance<TestLanguage>().first()
return TestLanguageFrontend(language.namespaceDelimiter, language, ctx)
}

@JvmStatic
fun provideTranslationResultForPDGTest() =
Stream.of(
Arguments.of(getIfTest(), "if statement"),
Arguments.of(getWhileLoopTest(), "while loop")
)

private fun getIfTest() =
testFrontend(
TranslationConfiguration.builder()
.registerLanguage(TestLanguage("::"))
.defaultPasses()
.registerPass<ControlDependenceGraphPass>()
.registerPass<ProgramDependenceGraphPass>()
.build()
)
.build {
translationResult {
translationUnit("if.cpp") {
// The main method
function("main", t("int")) {
body {
declare { variable("i", t("int")) { call("rand") } }
ifStmt {
condition { ref("i") lt literal(0, t("int")) }
thenStmt {
ref("i") assign (ref("i") * literal(-1, t("int")))
}
}
returnStmt { ref("i") }
}
}
}
}
}

private fun getWhileLoopTest() =
testFrontend(
TranslationConfiguration.builder()
.registerLanguage(TestLanguage("::"))
.defaultPasses()
.registerPass<ControlDependenceGraphPass>()
.registerPass<ProgramDependenceGraphPass>()
.build()
)
.build {
translationResult {
translationUnit("loop.cpp") {
// The main method
function("main", t("int")) {
body {
declare { variable("i", t("int")) { call("rand") } }
whileStmt {
whileCondition { ref("i") gt literal(0, t("int")) }
loopBody {
call("printf") { literal("#", t("string")) }
ref("i").dec()
}
}
call("printf") { literal("\n", t("string")) }
returnStmt { literal(0, t("int")) }
}
}
}
}
}
}
}
Loading