[15721] Blocking Schema Changes #1341

sxzh93 · 2018-05-05T02:55:12Z

Overview:
This PR contains all blocking schema changes implementation for Add/Drop column, rename column, and change column type. Alter table for Add/Drop column and change column type operations could be aggregated in one single SQL statement; rename column is separated alone.

Implementation Changes:
1.Functional code: We construct a new schema object for target table in parser, this object includes all changes related to this alter statement. Executor will execute the alter plan and invoke catalog to replace old table with new table using the new schema object. Catalog will use a scan to apply the changes to all tuples.
2.JUnit test: add unit test for add/drop column, change column type.

Known issues:
1.Multiple transactions bugs.
2.We don't handle drop column contains foreign key in this PR.

This reverts commit 6704d0e, reversing changes made to 39fa518.

- added alter and rename plan and statements - modified optimizor to bind db name and make plans - modified parsenodes - added node to string support and vice versa - added nodetransform of alter table

This reverts commit 6704d0e, reversing changes made to 39fa518.

- added alter and rename plan and statements - modified optimizor to bind db name and make plans - modified parsenodes - added node to string support and vice versa - added nodetransform of alter table

This reverts commit 6704d0e, reversing changes made to 39fa518.

add alter executor

… suppport

added change type, changed logic in alter_executor, varchar now has default length.

… a statement

basic test for alterTable

addressed bug

added alter varchar length support, changed the plan to use schema

star013 · 2018-05-11T19:23:44Z

src/catalog/catalog.cpp

+      // txn->RecordDrop(database_oid, old_table->GetOid(), INVALID_OID);
+
+      // Final step of physical change should be moved to commit time
+      database->ReplaceTableWithOid(table_oid, new_table);


[memory leak]
old table object is not explicitly freed.
database->ReplaceTableWithOid just replace the table pointer to new_table but old_table still exists.

Thanks for pointing this out! We left a TODO here, and we are still looking for the correct way to handle this.

star013 · 2018-05-11T19:32:52Z

src/catalog/catalog.cpp

+        new_table->AddIndex(new_index);
+
+        // reinsert record into pg_index
+        pg_index->InsertIndex(


[race condition]
It is risky to manipulate index directly under multiple transactions, because it is not protected by any consistency control mechanism like locks. It is better to do such operations through an executor just like your step 4(use a SeqScanPlan). The index executor with lock mechanism will take care of consistency.

Actually InsertIndex uses a InsertPlan to insert a new tuple into pg_attribute so we can get concurrency control for free.

star013 · 2018-05-11T19:55:54Z

src/catalog/catalog.cpp

+      for (oid_t old_column_id = 0;
+           old_column_id < old_schema->GetColumnCount(); old_column_id++) {
+        old_column_ids.push_back(old_column_id);
+        for (oid_t new_column_id = 0;


[inefficient implementation]
It is not efficient to get a matched name in a for loop. It is better to use a hashmap to do such a thing.

Sure, thanks for your suggestion.

star013 · 2018-05-11T20:13:08Z

src/executor/alter_executor.cpp

+  auto new_column_name = node.GetNewName();
+  auto old_column_name = node.GetOldName();
+
+  ResultType result = catalog::Catalog::GetInstance()->RenameColumn(


[race condition]
If there are two transactions trying to rename a column in a table at the same time, it is not safe to do operations on catalog instance without locks.

We planed to add lock logic here and in the alterTable, should we merge your lock manager and executor implementation?

Sure.
Although our current implementation of lock manager is correct, we are still working on it to support more functions. Most interface may not change in future, but I suggest that you keep an eye on our future pull request to guarantee your codes can run on our latest version of lock manager and index executors.
If you find any bugs in our lock manager or index executor, please feel free to contact us. We are pleasure to fix those problems.

star013 · 2018-05-11T20:16:39Z

src/include/parser/alter_statement.h

+        oldName(nullptr),
+        newName(nullptr) {
+    dropped_names =
+        type == AlterTableType::RENAME ? nullptr : (new std::vector<char *>);


[code style]
It seems better to bracket the judgement in the clause.

Of course, thanks for pointing out.

star013

General Questions:

The codes seem to work. They modified postgres parser, planner, executor, and catelog, which is the typical information flow to change schema. Some places need to add locks to protect consistency under multiple transactions.
All the codes are easily understood because of detailed comments and clear names.
There are a few redundant codes like using for loop to find a match string, but most of codes are neat.
The codes are modular and they re-use other modules properly.
There exist some commented out codes, like some test codes, but I believe they retain those codes to debug in future.
They use debug log functions properly and most of them are at TRACE level, which is suitable to avoid too many debug info.

Documentation Questions:

Comments describe intents of codes correctly.
All important functions are commented. Although some functions are not commented, their names are good enough suggestions.
No unusual behavior observed.
No other third-party libraries.
Garbage collection for the old table is not completed.

Testing Questions:

They have tests and they are comprehensive on single transaction operations, but they do not test concurrent situations.
Their test is valid because they use JDBC to send SQL instructions and the parser in peloton will convert it into plans to execute.
No hardcoded answers.
They tests cover all their basic functions like rename, type change etc.

alphalzh

[Major]: Major issues need to notice
[Minor]: Minor issues need to notice
[Trivial]: Suggestions

Improvements:
+ Much better use of comments
+ Much more test cases
+ Overall LGTM

Documentation:
+ Proper use of TODO
+ No incomplete code
+ Good comments

Tests:
+ Test exists and comprehensive
- Some tests still fails

Generally better coding style, keep it up!

alphalzh · 2018-05-11T22:52:31Z

script/testing/junit/AlterTableTest.java

+    public void test_RenameCol_Base() throws SQLException {
+        conn.createStatement().execute(SQL_RENAME_COLUMN);
+        ResultSet rs = conn.createStatement().executeQuery(SQL_SELECT_STAR);
+        rs.next();


[suggestion] If rs doesn't have next, this will through an exception that is not easy to catch. Maybe add some more handling here?

Sure, we should use try-catch here.

alphalzh · 2018-05-11T22:53:50Z

script/testing/junit/AlterTableTest.java

+        conn.commit();
+        conn2.commit();
+    }
+


[minor] Still commented out code, need to address this in the next PR

alphalzh · 2018-05-11T23:02:11Z

script/testing/junit/AlterTableTest.java

+                new int [] {5, 400});
+        assertNoMoreRows(rs);
+    }
+}


[comment] great tests!

alphalzh · 2018-05-11T23:03:03Z

src/catalog/catalog.cpp

+// ALTER TABLE
+//===--------------------------------------------------------------------===//
+
+/**


[good] Good comments!

alphalzh · 2018-05-11T23:08:57Z

src/catalog/catalog.cpp

+
+  if (txn == nullptr)
+    throw CatalogException("Alter table requires transaction");
+  try {


[minor] Why double try catch block?

Sure, we'll fix this.

alphalzh · 2018-05-11T23:40:15Z

src/include/common/logger.h

@@ -26,8 +26,8 @@

 // Fix for PRId64 (See https://stackoverflow.com/a/18719205)
 #if defined(__cplusplus) && !defined(__STDC_FORMAT_MACROS)
-#define __STDC_FORMAT_MACROS 1 // Not sure where to put this
-#endif 
+#define __STDC_FORMAT_MACROS 1  // Not sure where to put this


[trivial] Is this a auto-formatting change?

alphalzh · 2018-05-11T23:40:56Z

src/include/executor/alter_executor.h

+
+class AlterExecutor : public AbstractExecutor {
+ public:
+  AlterExecutor(const AlterExecutor &) = delete;


[good] Nice use of delete

alphalzh · 2018-05-11T23:42:04Z

src/include/parser/alter_statement.h

+/**
+ * @class AlterTableStatement
+ * @brief Represents "ALTER TABLE add column COLUMN_NAME COLUMN_TYPE"
+ * TODO: add implementation of AlterTableStatement


[trivial] maybe you forgot to remove todo...

Thanks for pointing out!

alphalzh · 2018-05-11T23:43:13Z

src/include/planner/alter_plan.h

+                              this->table_name.c_str(),
+                              this->database_name.c_str());
+  }
+


[trivial] Maybe putting longer functions to .cpp file

alphalzh · 2018-05-11T23:46:34Z

src/planner/alter_plan.cpp

+        if ((*parse_tree->added_columns)[i].get()->not_null) {
+          catalog::Constraint constraint(ConstraintType::NOTNULL,
+                                         "con_not_null");
+          column.AddConstraint(constraint);


[trivial] Easy-to-mixup variable names column and columns

eloiseh · 2018-05-11T22:26:16Z

script/testing/junit/AlterTableTest.java

+        conn2.commit();
+    }
+
+//    The following tests are currently broken.


What will happen when running these test cases?

Currently the system will crash due to infinite retry in the binder.

eloiseh · 2018-05-11T22:28:14Z

script/testing/junit/AlterTableTest.java

+        conn.createStatement().execute(sql);
+        ResultSet rs = conn.createStatement().executeQuery(SQL_SELECT_STAR);
+        rs.next();
+        checkRow(rs,


I think this check can't ensure there is only one column left.

eloiseh · 2018-05-11T22:42:54Z

src/catalog/catalog.cpp

+      for (auto index_oid_pair : old_index_oids) {
+        oid_t index_oid = index_oid_pair.first;
+        // delete record in pg_index
+        pg_index->DeleteIndex(index_oid, txn);


Why not delete index in pg_index only if not all indexed columns still exists?

eloiseh · 2018-05-11T23:54:51Z

src/catalog/catalog.cpp

+            txn);
+        column_offset++;
+      }
+      // TODO: Add gc logic


Have you guys finished this?

We're still working on this, sorry about that.

eloiseh · 2018-05-12T00:24:27Z

src/executor/alter_executor.cpp

+  auto old_schema = old_table->GetSchema();
+  std::vector<oid_t> column_ids;
+
+  // Step 1: remove drop columns from old schema


Good strategy to remove dropped columns.

eloiseh · 2018-05-12T00:31:50Z

src/include/parser/parsenodes.h

+  bool missing_ok;    /* skip error if table missing */
+} AlterTableStmt;
+
+typedef enum AlterTableType {


Very comprehensive types.

eloiseh · 2018-05-12T00:46:26Z

src/include/planner/alter_plan.h

+      const std::unique_ptr<catalog::Schema> &added_columns,
+      const std::unique_ptr<catalog::Schema> &changed_type_columns,
+      AlterType a_type);
+  explicit AlterPlan(const std::string &database_name,


I believe you guys have changed rename columns from using vector to string, why still vector here?

Sorry, will correct it. Thanks for pointing out.

DeanChensj and others added 30 commits April 28, 2018 13:53

add alter table catalog

0d83528

add change column catalog

f015e94

Revert "Merge pull request #1 from DeanChensj/catlog"

583ec6a

This reverts commit 6704d0e, reversing changes made to 39fa518.

This commit added parser support of alter table

469474b

- added alter and rename plan and statements - modified optimizor to bind db name and make plans - modified parsenodes - added node to string support and vice versa - added nodetransform of alter table

add rename column executor

7b2ae56

Catalog for change column name

f255f78

resolve comments, add header, TODO, remove context in subclass

fd0cb0b

fix

a3a7cfa

add alter table catalog

9e07a95

add change column catalog

e695ac2

Revert "Merge pull request #1 from DeanChensj/catlog"

d843ae3

This reverts commit 6704d0e, reversing changes made to 39fa518.

This commit added parser support of alter table

59e5982

- added alter and rename plan and statements - modified optimizor to bind db name and make plans - modified parsenodes - added node to string support and vice versa - added nodetransform of alter table

add rename column executor

dd42a3b

Catalog for change column name

a845135

resolve comments, add header, TODO, remove context in subclass

57e9a6e

fix

3575048

AlterTable init test

6b5da72

added headers, comments and TODO

b08fa9e

remove comments

0e46c90

add alter table catalog

d07920e

add change column catalog

88a397c

Revert "Merge pull request #1 from DeanChensj/catlog"

f0ecc68

This reverts commit 6704d0e, reversing changes made to 39fa518.

add rename column executor

a2f73d9

Catalog for change column name

ed7a2a9

fix

397cf24

add alter table catalog

e61451e

add change column catalog

098a35a

Revert "Merge pull request #1 from DeanChensj/catlog"

f344a55

This reverts commit 6704d0e, reversing changes made to 39fa518.

add rename column executor

e930b11

Catalog for change column name

c31ef3a

Dingshilun and others added 15 commits May 2, 2018 20:14

added change type into alter tabe

21ee68d

add alter executor

68f492f

resolve pr comments

9904788

Merge pull request #16 from sxzh93/alter_executor

a1e7265

add alter executor

modified alter plan, fixed bugs

fdce5cf

added change type, changed logic in alter_executor, varchar still not…

ae65f67

… suppport

addressed comments and fix varchar

4e4dfdc

Merge pull request #17 from sxzh93/change_type

4d00de1

added change type, changed logic in alter_executor, varchar now has default length.

basic test for alterTable

5e454d4

add alter table tests, change unsupported type, multiple operation in…

bc26411

… a statement

Merge pull request #18 from sxzh93/alter_test

32a260f

basic test for alterTable

added alter varchar length support, changed the plan to use schema

72c35ac

addressed bug

Merge pull request #19 from sxzh93/varchar_length

733fa9b

added alter varchar length support, changed the plan to use schema

rebase master

a57a5df

addressed the modification from cmu-db master

5b9818c

saatviks added do not merge class-project labels May 6, 2018

star013 reviewed May 11, 2018

View reviewed changes

alphalzh reviewed May 11, 2018

View reviewed changes

eloiseh reviewed May 12, 2018

View reviewed changes

change type quick fix

939df35

DeanChensj force-pushed the review2 branch from e3c75c3 to 939df35 Compare May 13, 2018 17:04

DeanChensj and others added 3 commits May 14, 2018 04:28

add gc logic

20f5d27

added alter benchmark, leave all workload of benchmark to be empty

dbf6d7d

cache-only set to true when get db object

fc3e951

[15721] Blocking Schema Changes #1341

Are you sure you want to change the base?

[15721] Blocking Schema Changes #1341

Conversation

sxzh93 commented May 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

star013 May 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

star013 May 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

star013 left a comment • edited Loading

Choose a reason for hiding this comment

alphalzh left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sxzh93 commented May 5, 2018 •

edited

Loading

star013 May 11, 2018 •

edited

Loading

star013 May 12, 2018 •

edited

Loading

star013 left a comment •

edited

Loading

alphalzh left a comment •

edited

Loading