Feature: Added creating of Directed Acyclic Graphs (DAG) to existing DAG Driver #433

NotAnAddictz · 2024-05-22T16:10:19Z

Summary

Previously, in PR #383 , we added functionality for sequential invocation for each function. This commit adds on to this existing feature, making every function act as an entry point and create a DAG structure for each function based on the width and depth distributions in data/traces/example/dag_structure.xlsx and invoke it according to the frequency of the entry functions.

Implementation Notes ⚒️

Added various helper functions to create and manage the DAG structure.
Added functionality to download the sampled_150 folder containing the folders for each group of functions if required.
Tweaked the generation of specifications for each function to cover the highest possible invocation frequencies/min
Wrapped the functions into Nodes to facilitate DAG generation
Added parameter entriesWritten to functionsDriver to ensure all invocations are written in the output file
Added a retry limit of 1 for each function in the DAG.
All DAG Structures will not have duplicate functions inside it, and is populated by randomly chosen functions in the function list.

External Dependencies 🍀

N/A

Breaking API Changes ⚠️

N/A

Simply specify none (N/A) if not applicable.

cvetkovic · 2024-07-16T10:34:17Z

@leokondrashov: Is this still relevant or we close this PR?

wanghanchengchn · 2024-07-16T11:23:34Z

@leokondrashov: Is this still relevant or we close this PR?

I apologize for the late reply. I will review this pull request!

pkg/config/parser.go

pkg/driver/trace_driver.go

wanghanchengchn · 2024-08-30T04:08:42Z

Dear @cvetkovic,

The current version looks good to me. However, due to my limited experience in reviewing pull requests, I would greatly appreciate it if you could provide us with some feedback when you have a moment. Thank you very much!

cmd/loader.go

cvetkovic · 2024-08-30T12:07:22Z

cmd/loader.go

+func getTrace() bool {
+	_, err := exec.Command("git", "lfs", "pull").CombinedOutput()
+	if err != nil {
+		log.Warnf("Failed to get traces")
+		return false
+	}
+	_, err = exec.Command("tar", "-xzf", "data/traces/reference/sampled_150.tar.gz", "-C", "data/traces").CombinedOutput()
+	if err != nil {
+		log.Warnf("Failed to extract sample")
+		return false
+	}
+	return true
+}


I don't think this should be git and tar commands should be part of the the loader. The user should be responsible for making sure these files exist and are in proper format.

cvetkovic · 2024-08-30T12:07:40Z

data/traces/example/dag_structure.xlsx

Please no proprietary formats. Use CSV.

cvetkovic · 2024-08-30T12:08:11Z

pkg/config/parser.go

+	DAGTracePath                 string `json:"DAGTracePath"`
+	EnableDAGDataset			 bool `json:"EnableDAGDataset"`
+	Width 						 int `json:"Width"`
+	Depth 						 int `json:"Depth"`


Indent properly.

pkg/driver/trace_driver.go

Signed-off-by: Kway Yi Shen <[email protected]> Signed-off-by: Kway Yi Shen <SamKway@localhost> Signed-off-by: Kway Yi Shen <[email protected]>

cvetkovic

LGTM. @leokondrashov If good, you can proceed with merging.

cvetkovic · 2024-10-04T15:43:18Z

@wanghanchengchn Just fix the errors linter reports.

wanghanchengchn · 2024-10-04T19:19:48Z

Thank you! Dear @NotAnAddictz, could you please address the failed checks? Additionally, this branch is out-of-date with the base branch. Kindly rebase on the main branch and verify that all checks are passing. Thank you!

leokondrashov

Looks good to me codewise. But I'd like to have documentation for the feature in the repo, not only in the PR description.

I think the linter problem is not caused by you, but it's easy to fix by changing Fatalf to Fatal.

leokondrashov · 2024-10-05T04:24:46Z

cmd/config.json

@@ -21,5 +21,9 @@

  "GRPCConnectionTimeoutSeconds": 15,
  "GRPCFunctionTimeoutSeconds": 900,
-  "DAGMode": false
+  "DAGMode": false,
+  "DAGTracePath": "data/traces/sampled_150/20",


Does this file exist by default?

leokondrashov · 2024-10-05T04:31:28Z

docs/configuration.md

@@ -19,7 +19,12 @@
 | MetricScrapingPeriodSeconds  | int       | > 0                                                                 | 15                  | Period of Prometheus metrics scrapping                                               |
 | GRPCConnectionTimeoutSeconds | int       | > 0                                                                 | 60                  | Timeout for establishing a gRPC connection                                           |
 | GRPCFunctionTimeoutSeconds   | int       | > 0                                                                 | 90                  | Maximum time given to function to execute[^4]                                        |
-| DAGMode             | bool      | true/false                                                          | false               | Sequential invocation of all functions one after another                                                    |
+| DAGMode                      | bool      | true/false                                                          | false               | Parallel invocation of DAG with each function acting as entry function                                                                                                   |


This is okay for the description of the config option, but we need proper documentation of this feature in the docs. Please consider writing down the generation and invocation processes of the DAG so the user can understand the output results without looking at the code.

leokondrashov · 2024-10-05T04:32:49Z

docs/configuration.md

@@ -19,7 +19,12 @@
 | MetricScrapingPeriodSeconds  | int       | > 0                                                                 | 15                  | Period of Prometheus metrics scrapping                                               |
 | GRPCConnectionTimeoutSeconds | int       | > 0                                                                 | 60                  | Timeout for establishing a gRPC connection                                           |
 | GRPCFunctionTimeoutSeconds   | int       | > 0                                                                 | 90                  | Maximum time given to function to execute[^4]                                        |
-| DAGMode             | bool      | true/false                                                          | false               | Sequential invocation of all functions one after another                                                    |
+| DAGMode                      | bool      | true/false                                                          | false               | Parallel invocation of DAG with each function acting as entry function                                                                                                   |
+| DAGTracePath                 | string    | string                                                              | data/traces/sampled_150/20               | Folder with Azure trace dimensions used for DAG Invocation.                                                                                                |


Please add a reference to the trace. It can be located with the rest of the docs (see other comment).

NotAnAddictz force-pushed the main branch 2 times, most recently from d9e9ef0 to 9161a77 Compare May 22, 2024 16:25

wanghanchengchn reviewed Jul 17, 2024

View reviewed changes

pkg/config/parser.go Outdated Show resolved Hide resolved

wanghanchengchn reviewed Jul 17, 2024

View reviewed changes

pkg/driver/trace_driver.go Outdated Show resolved Hide resolved

wanghanchengchn reviewed Jul 17, 2024

View reviewed changes

pkg/driver/trace_driver.go Outdated Show resolved Hide resolved

NotAnAddictz force-pushed the main branch 2 times, most recently from d75890a to fca5d95 Compare July 19, 2024 12:10

wanghanchengchn reviewed Jul 22, 2024

View reviewed changes

pkg/driver/trace_driver.go Outdated Show resolved Hide resolved

NotAnAddictz force-pushed the main branch 2 times, most recently from a8f5730 to 3357247 Compare August 1, 2024 13:33

NotAnAddictz force-pushed the main branch from 3357247 to 486319d Compare August 3, 2024 06:40

wanghanchengchn reviewed Aug 4, 2024

View reviewed changes

pkg/driver/trace_driver.go Show resolved Hide resolved

wanghanchengchn reviewed Aug 4, 2024

View reviewed changes

pkg/driver/trace_driver.go Show resolved Hide resolved

wanghanchengchn reviewed Aug 4, 2024

View reviewed changes

pkg/driver/trace_driver.go Outdated Show resolved Hide resolved

wanghanchengchn reviewed Aug 4, 2024

View reviewed changes

pkg/driver/trace_driver.go Outdated Show resolved Hide resolved

NotAnAddictz force-pushed the main branch from 486319d to aa5ba32 Compare August 4, 2024 16:27

NotAnAddictz requested a review from wanghanchengchn August 5, 2024 01:14

NotAnAddictz force-pushed the main branch 2 times, most recently from cbc9203 to e35a93e Compare August 12, 2024 07:59

wanghanchengchn requested a review from cvetkovic August 30, 2024 03:59

cvetkovic requested changes Aug 30, 2024

View reviewed changes

Adding logic for creating and managing DAG invocations

a0c869f

Signed-off-by: Kway Yi Shen <[email protected]> Signed-off-by: Kway Yi Shen <SamKway@localhost> Signed-off-by: Kway Yi Shen <[email protected]>

NotAnAddictz force-pushed the main branch from e35a93e to a0c869f Compare September 3, 2024 10:28

NotAnAddictz requested a review from cvetkovic September 3, 2024 10:53

cvetkovic approved these changes Oct 4, 2024

View reviewed changes

leokondrashov reviewed Oct 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Added creating of Directed Acyclic Graphs (DAG) to existing DAG Driver #433

Feature: Added creating of Directed Acyclic Graphs (DAG) to existing DAG Driver #433

NotAnAddictz commented May 22, 2024

cvetkovic commented Jul 16, 2024

wanghanchengchn commented Jul 16, 2024

wanghanchengchn commented Aug 30, 2024

cvetkovic Aug 30, 2024

cvetkovic Aug 30, 2024

cvetkovic Aug 30, 2024

cvetkovic left a comment

cvetkovic commented Oct 4, 2024

wanghanchengchn commented Oct 4, 2024

leokondrashov left a comment

leokondrashov Oct 5, 2024

leokondrashov Oct 5, 2024

leokondrashov Oct 5, 2024

Feature: Added creating of Directed Acyclic Graphs (DAG) to existing DAG Driver #433

Are you sure you want to change the base?

Feature: Added creating of Directed Acyclic Graphs (DAG) to existing DAG Driver #433

Conversation

NotAnAddictz commented May 22, 2024

Summary

Implementation Notes ⚒️

External Dependencies 🍀

Breaking API Changes ⚠️

cvetkovic commented Jul 16, 2024

wanghanchengchn commented Jul 16, 2024

wanghanchengchn commented Aug 30, 2024

cvetkovic Aug 30, 2024

Choose a reason for hiding this comment

cvetkovic Aug 30, 2024

Choose a reason for hiding this comment

cvetkovic Aug 30, 2024

Choose a reason for hiding this comment

cvetkovic left a comment

Choose a reason for hiding this comment

cvetkovic commented Oct 4, 2024

wanghanchengchn commented Oct 4, 2024

leokondrashov left a comment

Choose a reason for hiding this comment

leokondrashov Oct 5, 2024

Choose a reason for hiding this comment

leokondrashov Oct 5, 2024

Choose a reason for hiding this comment

leokondrashov Oct 5, 2024

Choose a reason for hiding this comment