Flav/int to bigint migration #9065

flvndvd · 2024-12-02T17:00:49Z

Integer to BigInt Migration Tool

See migration details here.

This PR introduces a PostgreSQL migration tool designed to safely convert integer primary/foreign keys to BigInt while maintaining zero-downtime capability and full rollback support.

Key Features

Zero-downtime migration: Uses concurrent index creation and maintains dual columns during transition
Phased approach: Splits migration into distinct steps (setup, backfill, pre-cutover, cutover, rollback)
Data integrity: Maintains referential integrity throughout the process using triggers
Rollback support: Full rollback capability at any stage
Safety measures:
- Dry-run mode for validation
- Progress tracking
- Transaction safety
- Concurrent index creation
- NOT VALID + VALIDATE pattern for constraints

Key Implementation Details

Constraint Handling

Standardized Constraint Behavior
- All foreign key constraints are enforced to use ON UPDATE RESTRICT ON DELETE RESTRICT
- This applies to both new constraints and rollback scenarios
- Even when rolling back, we don't restore the original CASCADE behaviors
- This is an intentional design decision to maintain consistent referential integrity rules
Primary Key Dependencies
- PostgreSQL requires dropping foreign key constraints before modifying primary key constraints
- This is because foreign keys create dependencies that prevent primary key modifications
- The original foreign keys will get restored when using the rollback step

Rollback Considerations

Constraint Behavior Changes
- Original: Mixed ON UPDATE/DELETE behaviors (CASCADE/RESTRICT)
- After rollback: Standardized to RESTRICT for both
- This is the only permanent change after rollback
- Assuming we properly delete each resources individually and don't rely on the CASCADE effect.
Data Integrity During Rollback
- Maintains both integer and BigInt columns
- Preserves data synchronization via triggers
- Ensures zero data loss during rollback

Migration Steps

1. Setup

Adds new BigInt columns (_new suffix)
Creates sync triggers to maintain data consistency (id -> id_new)
Preserves existing constraints

2. Backfill

Copies data from integer to BigInt columns
Batched processing to minimize impact
Progress tracking with percentage completion

3. Pre-cutover

Creates necessary indexes concurrently
Sets up constraints for new columns
Validates data consistency
Don't require any transaction

4. Cutover

Switches primary key to BigInt column
Updates foreign key references
Maintains rollback capability via legacy columns

5. Rollback (if needed)

Restores integer primary keys
Rebuilds original foreign key relationships
Maintains data sync for safety

Usage Example

// Setup phase
npx tsx migration.ts --database=postgresql://... --table=users --step=setup

// Backfill phase
npx tsx migration.ts --database=postgresql://... --table=users --step=backfill --batchSize=5000

// Pre-cutover phase
npx tsx migration.ts --database=postgresql://... --table=users --step=pre-cutover

// Cutover phase
npx tsx migration.ts --database=postgresql://... --table=users --step=cutover

// Rollback (if needed)
npx tsx migration.ts --database=postgresql://... --table=users --step=rollback


## Risk

<!-- Discuss potential risks and how they will be mitigated. Consider the impact and whether the changes are safe to rollback. -->

## Deploy Plan

<!-- Outline the deployment steps. Specify the order of operations and any considerations that should be made before, during, and after deployment/ -->

flvndvd · 2024-12-02T17:09:40Z

front/lib/resources/storage/index.ts

+// prevents silent precision loss when handling large integers from the database.
+// Throws an assertion error if a BIGINT value exceeds JavaScript's safe integer
+// limits.
+types.setTypeParser(builtins.INT8, function (val) {


Will extract to a dedicated PR.

Shipped here.

spolu

LGTM. Very nice \o/

I presume final clean-up is still missing.

Let's start with connectors 👍

spolu · 2024-12-02T17:21:36Z