Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(joins): support MySQL 8.0 #4639

Merged
merged 35 commits into from
Feb 6, 2024
Merged

feat(joins): support MySQL 8.0 #4639

merged 35 commits into from
Feb 6, 2024

Conversation

Weakky
Copy link
Contributor

@Weakky Weakky commented Jan 11, 2024

Overview

closes https://github.com/prisma/team-orm/issues/455

Adds support for joins on MySQL 8.0, through correlated sub-queries. Also works with PlanetScale. Does not work with MySQL 5.6, MySQL 5.7 and MariaDB.

Here are examples for 1-1, 1-m and m-n relations between a User and Post model:

1-1

model User {
  id   Int     @id @default(autoincrement())
  name String?

  posts Post[]
}

model Post {
  id    Int     @id @default(autoincrement())
  title String?

  user   User? @relation(fields: [userId], references: [id])
  userId Int?
}
{
  findManyPost {
    id
    title
    user {
      id
      name
    }
  }
}
SELECT
  `t1`.`id`,
  `t1`.`title`,
  `t1`.`userId`,
  (
    SELECT
      JSON_OBJECT('id', `t2`.`id`, 'name', `t2`.`name`)
    FROM
      `User` AS `t2`
    WHERE
      `t1`.`userId` = `t2`.`id`
    LIMIT
      1
  ) AS `user`
FROM
  `Post` AS `t1`

1-m

model User {
  id   Int     @id @default(autoincrement())
  name String?

  posts Post[]
}

model Post {
  id    Int     @id @default(autoincrement())
  title String?

  user   User? @relation(fields: [userId], references: [id])
  userId Int?
}
{
  findManyUser {
    id
    name
    posts(orderBy: { id: asc }) {
      id
      title
    }
  }
}
SELECT
  `t1`.`id`,
  `t1`.`name`,
  (
    SELECT
      COALESCE(
        JSON_ARRAYAGG(`__prisma_data__`),
        CONVERT('[]', JSON)
      ) AS `__prisma_data__`
    FROM
      (
        SELECT
          `t4`.`__prisma_data__`
        FROM
          (
            SELECT
              JSON_OBJECT('id', `t3`.`id`, 'title', `t3`.`title`) AS `__prisma_data__`,
              `t3`.`id`
            FROM
              (
                SELECT
                  `t2`.*
                FROM
                  `Post` AS `t2`
                WHERE
                  `t1`.`id` = `t2`.`userId`
                  /* root select */
              ) AS `t3`
              /* inner select */
          ) AS `t4`
        ORDER BY
          `t4`.`id` ASC
        LIMIT 9223372036854775807 -- large limit to enable ordering to work
          /* middle select */
      ) AS `t5`
      /* outer select */
  ) AS `posts`
FROM
  `User` AS `t1`

m-n

model User {
  id   Int     @id @default(autoincrement())
  name String?

  posts Post[]
}

model Post {
  id    Int     @id @default(autoincrement())
  title String?

  users User[]
}
{
  findManyUser {
    id
    name
    posts(orderBy: { id: asc }) {
      id
      title
    }
  }
}
SELECT
  `t1`.`id`,
  `t1`.`name`,
  (
    SELECT
      COALESCE(
        JSON_ARRAYAGG(`__prisma_data__`),
        CONVERT('[]', JSON)
      ) AS `__prisma_data__`
    FROM
      (
        SELECT
          JSON_OBJECT('id', `t3`.`id`, 'title', `t3`.`title`) AS `__prisma_data__`
        FROM
          (
            SELECT
              `Post`.*
            FROM
              `_PostToUser` AS `t2`
              INNER JOIN `Post` ON `t2`.`A` = `Post`.`id`
            WHERE
              `t2`.`B` = `t1`.`id`
            ORDER BY
              `Post`.`id` ASC
              /* root */
          ) AS `t3`
        LIMIT
          9223372036854775807
          /* inner */
      ) AS `t4`
      /* outer */
  ) AS `posts`
FROM
  `User` AS `t1`

Note: When fetching to-many relations, if any ordering is present, we add a limit of i64::MAX which inexplicably forces MySQL to order the aggregated rows.

@Weakky Weakky requested a review from a team as a code owner January 11, 2024 16:01
@Weakky Weakky requested review from miguelff and jkomyno and removed request for a team January 11, 2024 16:01
@Weakky Weakky force-pushed the feat/mysql-8-joins branch from 85f4380 to 29525e7 Compare January 11, 2024 17:09
@Weakky Weakky added this to the 5.9.0 milestone Jan 11, 2024
@Weakky Weakky self-assigned this Jan 11, 2024
Copy link
Contributor

github-actions bot commented Jan 11, 2024

WASM Size

Engine This PR Base branch Diff
WASM 2.176MiB 2.163MiB 14.127KiB
WASM (gzip) 841.617KiB 836.581KiB 5.037KiB

Copy link

codspeed-hq bot commented Jan 11, 2024

CodSpeed Performance Report

Merging #4639 will not alter performance

Comparing feat/mysql-8-joins (a58db5f) with main (16a6fe5)

Summary

✅ 11 untouched benchmarks

Copy link
Contributor

github-actions bot commented Jan 11, 2024

✅ WASM query-engine performance won't change substantially (0.996x)

Full benchmark report
DATABASE_URL="postgresql://postgres:postgres@localhost:5432/bench?schema=imdb_bench&sslmode=disable" \
node --experimental-wasm-modules query-engine/driver-adapters/executor/dist/bench.mjs
cpu: AMD EPYC 7763 64-Core Processor
runtime: node v18.19.0 (x64-linux)

benchmark                   time (avg)             (min … max)       p75       p99      p999
-------------------------------------------------------------- -----------------------------
• movies.findMany() (all - 25000)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  302.47 ms/iter (298.29 ms … 307.15 ms) 306.13 ms 307.15 ms 307.15 ms
Web Assembly: Latest    383.66 ms/iter (380.69 ms … 391.52 ms) 388.76 ms 391.52 ms 391.52 ms
Web Assembly: Current   387.45 ms/iter (385.77 ms … 390.75 ms) 388.38 ms 390.75 ms 390.75 ms
Node API: Current       229.74 ms/iter (221.14 ms … 238.71 ms) 236.52 ms 238.71 ms 238.71 ms

summary for movies.findMany() (all - 25000)
  Web Assembly: Current
   1.69x slower than Node API: Current
   1.28x slower than Web Assembly: Baseline
   1.01x slower than Web Assembly: Latest

• movies.findMany({ take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   12.18 ms/iter      (12 ms … 14.04 ms)  12.13 ms  14.04 ms  14.04 ms
Web Assembly: Latest     16.17 ms/iter   (15.63 ms … 26.39 ms)  15.81 ms  26.39 ms  26.39 ms
Web Assembly: Current    15.84 ms/iter    (15.6 ms … 17.51 ms)  15.78 ms  17.51 ms  17.51 ms
Node API: Current        8,932 µs/iter   (8,682 µs … 9,662 µs)  9,037 µs  9,662 µs  9,662 µs

summary for movies.findMany({ take: 2000 })
  Web Assembly: Current
   1.77x slower than Node API: Current
   1.3x slower than Web Assembly: Baseline
   1.02x faster than Web Assembly: Latest

• movies.findMany({ where: {...}, take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   1,924 µs/iter   (1,818 µs … 3,340 µs)  1,904 µs  3,091 µs  3,340 µs
Web Assembly: Latest     2,498 µs/iter   (2,401 µs … 3,893 µs)  2,478 µs  3,480 µs  3,893 µs
Web Assembly: Current    2,510 µs/iter   (2,400 µs … 4,258 µs)  2,479 µs  3,917 µs  4,258 µs
Node API: Current        1,516 µs/iter   (1,435 µs … 1,702 µs)  1,531 µs  1,689 µs  1,702 µs

summary for movies.findMany({ where: {...}, take: 2000 })
  Web Assembly: Current
   1.66x slower than Node API: Current
   1.3x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline    12.1 ms/iter   (11.97 ms … 12.44 ms)  12.13 ms  12.44 ms  12.44 ms
Web Assembly: Latest     15.68 ms/iter   (15.58 ms … 15.88 ms)  15.72 ms  15.88 ms  15.88 ms
Web Assembly: Current    15.77 ms/iter   (15.56 ms … 16.25 ms)  15.86 ms  16.25 ms  16.25 ms
Node API: Current        8,976 µs/iter   (8,732 µs … 9,284 µs)  9,078 µs  9,284 µs  9,284 µs

summary for movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.76x slower than Node API: Current
   1.3x slower than Web Assembly: Baseline
   1.01x slower than Web Assembly: Latest

• movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   1,884 µs/iter   (1,800 µs … 2,718 µs)  1,882 µs  2,432 µs  2,718 µs
Web Assembly: Latest     2,637 µs/iter   (2,404 µs … 4,418 µs)  2,537 µs  4,356 µs  4,418 µs
Web Assembly: Current    2,476 µs/iter   (2,381 µs … 3,067 µs)  2,466 µs  3,008 µs  3,067 µs
Node API: Current        1,475 µs/iter   (1,401 µs … 2,006 µs)  1,497 µs  1,671 µs  2,006 µs

summary for movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.68x slower than Node API: Current
   1.31x slower than Web Assembly: Baseline
   1.06x faster than Web Assembly: Latest

• movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   12.03 ms/iter   (11.96 ms … 12.15 ms)  12.07 ms  12.15 ms  12.15 ms
Web Assembly: Latest     15.75 ms/iter   (15.65 ms … 15.91 ms)   15.8 ms  15.91 ms  15.91 ms
Web Assembly: Current    15.74 ms/iter   (15.62 ms … 16.02 ms)  15.77 ms  16.02 ms  16.02 ms
Node API: Current        8,860 µs/iter   (8,739 µs … 9,329 µs)  8,888 µs  9,329 µs  9,329 µs

summary for movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.78x slower than Node API: Current
   1.31x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   1,873 µs/iter   (1,799 µs … 2,851 µs)  1,867 µs  2,480 µs  2,851 µs
Web Assembly: Latest     2,452 µs/iter   (2,377 µs … 2,938 µs)  2,450 µs  2,856 µs  2,938 µs
Web Assembly: Current    2,487 µs/iter   (2,378 µs … 4,453 µs)  2,457 µs  3,911 µs  4,453 µs
Node API: Current        1,503 µs/iter   (1,426 µs … 1,736 µs)  1,531 µs  1,690 µs  1,736 µs

summary for movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.65x slower than Node API: Current
   1.33x slower than Web Assembly: Baseline
   1.01x slower than Web Assembly: Latest

• movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  933.42 µs/iter  (843.39 µs … 1,953 µs) 904.61 µs  1,675 µs  1,953 µs
Web Assembly: Latest     1,201 µs/iter   (1,136 µs … 2,047 µs)  1,201 µs  1,720 µs  2,047 µs
Web Assembly: Current    1,200 µs/iter   (1,153 µs … 1,804 µs)  1,207 µs  1,488 µs  1,804 µs
Node API: Current       811.59 µs/iter (744.89 µs … 961.52 µs) 838.02 µs 907.95 µs 961.52 µs

summary for movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
  Web Assembly: Current
   1.48x slower than Node API: Current
   1.29x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  908.04 µs/iter  (871.17 µs … 1,263 µs)  912.6 µs  1,141 µs  1,263 µs
Web Assembly: Latest     1,208 µs/iter   (1,143 µs … 2,029 µs)  1,210 µs  1,685 µs  2,029 µs
Web Assembly: Current    1,234 µs/iter   (1,187 µs … 1,553 µs)  1,239 µs  1,471 µs  1,553 µs
Node API: Current       826.22 µs/iter  (740.07 µs … 1,641 µs) 837.18 µs  1,277 µs  1,641 µs

summary for movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
  Web Assembly: Current
   1.49x slower than Node API: Current
   1.36x slower than Web Assembly: Baseline
   1.02x slower than Web Assembly: Latest

After changes in a58db5f

@Weakky Weakky force-pushed the feat/mysql-8-joins branch from e7af6ad to 8bf285c Compare January 17, 2024 13:23
Comment on lines +98 to +107
(
Some(TypeFamily::Text(_)),
Some("LONGBLOB") | Some("BLOB") | Some("MEDIUMBLOB") | Some("SMALLBLOB") | Some("TINYBLOB")
| Some("VARBINARY") | Some("BINARY") | Some("BIT"),
) => {
self.write("to_base64")?;
self.surround_with("(", ")", |s| s.visit_expression(expr))?;

Ok(())
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We convert binary columns to base64 to ease coercion in the Query Engine

Comment on lines +109 to +117
(_, Some("FLOAT")) => {
self.write("CONVERT")?;
self.surround_with("(", ")", |s| {
s.visit_expression(expr)?;
s.write(", ")?;
s.write("CHAR")
})?;
Ok(())
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We convert floats to string to avoid losing precision

ScalarFieldId::InCompositeType(id) => self.dm.walk(id).scalar_type(),
};

let nt = psl_nt.or_else(|| scalar_type.and_then(|st| connector.default_native_type_for_scalar_type(&st)))?;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To enable applying special conversion in quaint for specific native types, I updated the ScalarField::native_type function to return the default native type in case none are specified. Previously, this function would only return something if a native type was explicitly specified.

@@ -39,7 +39,7 @@ static DEFAULT_MAPPING: Lazy<HashMap<ScalarType, MongoDbType>> = Lazy::new(|| {
(ScalarType::Float, MongoDbType::Double),
(ScalarType::Boolean, MongoDbType::Bool),
(ScalarType::String, MongoDbType::String),
(ScalarType::DateTime, MongoDbType::Timestamp),
(ScalarType::DateTime, MongoDbType::Date),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change mentioned above 👆 has rippled in unintended ways. Notably, it surfaced an incorrect default native type for the DateTime prisma type. This is why I've changed this.

Comment on lines +69 to 71
fn default_native_type_for_scalar_type(&self, _scalar_type: &ScalarType) -> Option<NativeTypeInstance> {
None
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, now that this method is called for all connectors when constructing quaint columns, the function's signature was changed to better signal when a scalar type has no default native type.

Comment on lines +89 to +91
let actual = logs
.iter()
.any(|l| l.contains("LEFT JOIN LATERAL") || (l.contains("JSON_ARRAYAGG") && l.contains("JSON_OBJECT")));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're not using JOINs to resolve relations on MySQL, this had to be changed with a loose and worse heuristic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine. Should we maybe now rename the function to something else? Same problem as #4639 (comment) I guess.

Comment on lines 384 to 392
let exclusions = exclude
.iter()
.filter_map(|c| ConnectorVersion::try_from(*c).ok())
.map(|c| ConnectorVersion::try_from(*c).unwrap())
.collect::<Vec<_>>();

let inclusions = only
.iter()
.filter_map(|c| ConnectorVersion::try_from(*c).ok())
.map(|c| ConnectorVersion::try_from(*c).unwrap())
.collect::<Vec<_>>();
Copy link
Contributor Author

@Weakky Weakky Feb 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated change to make sure that when a version is specified for the only / exclude attributes in the test suite and this version is incorrectly specified, it errors instead of silently ignoring it. Got bitten by it multiple times when working on this PR.

Comment on lines +98 to +100
if version.is_mysql() && !matches!(version, ConnectorVersion::MySql(Some(MySqlVersion::V8))) {
excluded_features.push(format!(r#""{}""#, PreviewFeature::RelationJoins));
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid having to exclude hundreds of tests for MySQL 5.6 & 5.7, we simply do not render the relationJoins preview feature for now when rendering the datamodel of each tests. This ensure the relationLoadStrategy: join is never used.

@@ -152,7 +152,7 @@ impl IntoBson for (&MongoDbType, PrismaValue) {

// Double
(MongoDbType::Double, PrismaValue::Int(i)) => Bson::Double(i as f64),
(MongoDbType::Double, PrismaValue::Float(f)) => Bson::Double(f.to_f64().convert(expl::MONGO_DOUBLE)?),
(MongoDbType::Double, PrismaValue::Float(f)) => bigdecimal_to_bson_double(f)?,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side-effect of ScalarField::native_type always returning a native type now. This is just a bug-fix that had not been caught before.

Comment on lines +192 to +199
(MongoDbType::Json, PrismaValue::Json(json)) => {
let val: Value = serde_json::from_str(&json)?;

Bson::try_from(val).map_err(|_| MongoError::ConversionError {
from: "Stringified JSON".to_owned(),
to: "Mongo BSON (extJSON)".to_owned(),
})?
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Comment on lines +144 to +154
TypeIdentifier::Boolean => {
let err =
|| build_conversion_error(sf, &format!("Number({n})"), &format!("{:?}", sf.type_identifier()));
let i = n.as_i64().ok_or_else(err)?;

match i {
0 => Ok(PrismaValue::Boolean(false)),
1 => Ok(PrismaValue::Boolean(true)),
_ => Err(err()),
}
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MySQL-specific coercion for tinyint (IIRC)


pub(crate) fn build(
pub(crate) trait JoinSelectBuilder {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To deal with the differences of PG & MySQL (lateral join vs correlated sub-queries), I've went for a trait design with a bunch of unimplemented methods that handle the few differences that we have between both connectors.

It has grown a bit too much to my taste already with the addition of _count support, but I can't think of a cleaner way to avoid huge code duplication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the downside, it made the code harder to follow.

Comment on lines +196 to +202
let middle_take = match connector_flavour(&rs.args) {
// On MySQL, using LIMIT makes the ordering of the JSON_AGG working. Beware, this is undocumented behavior.
// Note: Ideally, this should live in the MySQL select builder, but it's currently the only implementation difference
// between MySQL and Postgres, so we keep it here for now to avoid code duplication.
Flavour::Mysql if !rs.args.order_by.is_empty() => rs.args.take_abs().or(Some(i64::MAX)),
_ => rs.args.take_abs(),
};
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫠

Copy link
Member

@aqrln aqrln left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review, I'll finish reading tomorrow (haven't gotten to the most important part yet)

Copy link
Member

@aqrln aqrln left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@laplab laplab self-requested a review February 6, 2024 13:44
Copy link
Contributor

@laplab laplab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a quick call with Flavian to go over the key parts. LGTM, great work!

@Weakky Weakky merged commit 6ba3e3d into main Feb 6, 2024
138 of 140 checks passed
@Weakky Weakky deleted the feat/mysql-8-joins branch February 6, 2024 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants