This repository has been archived by the owner on May 10, 2024. It is now read-only.
artifacts pipeline and sample plots #147
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
…parsed by ddlog-sql Changes: - char(N) --> varchar(N) because the DDlog language doesn't support char(N) - Remove FK constraints, as neither Calcite nor Presto support this - Add types to arrays so they are parseable by Calcite ---------- All uncommented views in DBSchema pass both H2 and Presto translation Edits required: - select --> select distinct, necessary for both Calcite and Presto - remove camelcase column names, as Presto make everything case-insensitive - DDlog cannot support LIMIT, which will need to be addressed in the future - Remove ';' at the end of some statements, as Presto doesn't like this TODO: - Fix commented-out views in DBSchema, which are a combination of sql->DDlog compiler not supporting some statements, or Calcite doesn't like some of the complex SQL ---------- All views other than affinity/anti-affinity pass - Add all columns that are using in the HAVING clause to GROUP BY clause - Add array_length functionality to DDlog - DDlog only supports array_agg for now, instead of array_agg_distinct TODO: - array_agg_distinct - Affinity views ----------- Move all DDlog SQL updates to separate files, so that we can still run our H2 tests ----------- Signed-off-by: Amy Tai <[email protected]> Changes to make SchedulerTest::testSchedulerNodePredicates pass Summary of changes: - New IConnectionPool interface so that we can seamlessly swap DBConnectionPool out for DDlogDBConnectionPool - Temporarily comment out OnDuplicateKeyUpdate queries in PodEventsToDatabase and NodeResourceEventHandler, as DDlog backend cannot handle MERGE SQL queries at the moment - Inserts into DDlog must be in order of the column declarations in the CREATE TABLE statement - Any SQL queries that will eventually hit DDlog must use unqualified names - Wherever there is a dslContext.meta.getTables() call (or any dslContext.meta.* function call), for the DDlog backend this must be called against the dslContext that is a class variable within the DDlogJooqProvider. Every other dslContext call (i.e. ones that are used to execute select/insert/etc. queries), should be made against the DSLContext held inside the DDlogDBConnectionPool. These two use cases are differentiated by the two different accessor functions inside IConnectionPool. Signed-off-by: Amy Tai <[email protected]> Migrate modifications to scheduler_tables.sql and DBView.java to DDlog versions testSchedulerNodePredicates and testSpareCapacity both pass. Signed-off-by: Amy Tai <[email protected]> testHostPorts passes Goes with corresponding DDlog patch on vectors (array_agg column type) Signed-off-by: Amy Tai <[email protected]>
…spare capacity views that Lalith has Signed-off-by: Amy Tai <[email protected]>
…ables on the DDlog side Signed-off-by: Amy Tai <[email protected]>
… calcite packages
…ts seem to work at this commit Signed-off-by: Amy Tai <[email protected]>
- "_augmented" views still don't work because they require preemption views from DDlogDBViews, which were commented out hitherto Signed-off-by: Amy Tai <[email protected]>
… to work Signed-off-by: Amy Tai <[email protected]>
… use a stale DDlog API. Instead, don't compileAndLoad until the very last time (after Autoscope) Signed-off-by: Amy Tai <[email protected]>
Signed-off-by: Amy Tai <[email protected]>
Signed-off-by: Amy Tai <[email protected]>
…ptimized DDlog program file. In that case, /tmp/program.dl is not compiled using sql-ddlog compiler. Instead, we just copy over contents of the hand-optimized file into /tmp/program.dl. Signed-off-by: Amy Tai <[email protected]>
Signed-off-by: Amy Tai <[email protected]>
Signed-off-by: Amy Tai <[email protected]>
…kend Signed-off-by: Amy Tai <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
… Views once, so that we never have to compile it again with each of the CartesianProductTests. We also temporarily change the Scheduler constructor to add AutoScoped views to DDlog, even if the Scheduler isn't running in AutoScoped mode. This is so that we can use the same DDlog program (with scoping) for non-scoped tests as well. Need to fix this eventually, as for real tests/workloads, we might want to turn off Autoscoping. Signed-off-by: Amy Tai <[email protected]>
|
@askiad could you remove the plots and the generated data? They needn't go into version control. |
Signed-off-by: Amy Tai <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
…t-joins are brewing Signed-off-by: Lalith Suresh <[email protected]>
|
@askiad, you must sign every commit in this pull request acknowledging our Developer Certificate of Origin before your changes are merged. This can be done by adding
|
…th identity views Signed-off-by: Lalith Suresh <[email protected]>
…passes. Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Amy Tai <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Lalith Suresh <[email protected]>
Signed-off-by: Athinagoras Skiadopoulos <[email protected]>
Signed-off-by: Athinagoras Skiadopoulos <[email protected]>
Signed-off-by: Athinagoras Skiadopoulos <[email protected]>
Signed-off-by: Athinagoras Skiadopoulos <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Artifacts pipeline (tested on delta branch)
Contains sample outputs in sample_results
Run with ./run_emulation [long]
Currently runs for generated traces (generated-data.txt from generate_trace.py) and small cluster sizes.
Signed-off-by: Athinagoras Skiadopoulos [email protected]