Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(deps): update sqllogictest requirement from 0.23.0 to 0.24.0 #13902

Merged
merged 1 commit into from
Dec 26, 2024
+21 −11
Merged

chore(deps): update sqllogictest requirement from 0.23.0 to 0.24.0 #13902

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion datafusion/sqllogictest/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ object_store = { workspace = true }
postgres-protocol = { version = "0.6.7", optional = true }
postgres-types = { version = "0.2.8", features = ["derive", "with-chrono-0_4"], optional = true }
rust_decimal = { version = "1.36.0", features = ["tokio-pg"] }
sqllogictest = "0.23.0"
sqllogictest = "0.24.0"
sqlparser = { workspace = true }
tempfile = { workspace = true }
thiserror = "2.0.0"
Expand Down
30 changes: 20 additions & 10 deletions datafusion/sqllogictest/bin/sqllogictests.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ use datafusion_sqllogictest::{DataFusion, TestContext};
use futures::stream::StreamExt;
use itertools::Itertools;
use log::info;
use sqllogictest::strict_column_validator;
use sqllogictest::{strict_column_validator, Normalizer};
use std::ffi::OsStr;
use std::fs;
use std::path::{Path, PathBuf};
Expand All @@ -40,19 +40,25 @@ pub fn main() -> Result<()> {
.block_on(run_tests())
}

fn value_validator(actual: &[Vec<String>], expected: &[String]) -> bool {
let expected = expected
.iter()
// Trailing whitespace from lines in SLT will typically be removed, but do not fail if it is not
// If particular test wants to cover trailing whitespace on a value,
// it should project additional non-whitespace column on the right.
.map(|s| s.trim_end().to_owned())
.collect::<Vec<_>>();
#[allow(clippy::ptr_arg)]
fn normalizer(s: &String) -> String {
// Trailing whitespace from lines in SLT will typically be removed, but do not fail if it is not
// If particular test wants to cover trailing whitespace on a value,
// it should project additional non-whitespace column on the right.
s.trim_end().to_owned()
}

fn value_validator(
normalizer: Normalizer,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to this PR, I'm a bit confused about why sqllogictests need a Normalizer🤔
The value validator can already be customized, and specifically, the validator will call the normalizer within its proces. Normalizer seems like an unnecessary design to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @Omega359 added this in

He may have more backstory as well

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to this PR, I'm a bit confused about why sqllogictests need a Normalizer🤔 The value validator can already be customized, and specifically, the validator will call the normalizer within its proces. Normalizer seems like an unnecessary design to me.

That was not entirely true unfortunately, the normalization was hardcoded in places.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was not entirely true unfortunately, the normalization was hardcoded in places.

normalize_string was an internal part of the default validator. Although it was hard-coded, the outer validator can be user-defined. A user-defined validator like the one provided by DataFusion can offer more flexibility to meet specific row comparison needs, and might not require normalization.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. However, in my sidebranch for the sqlite tests it made things a bit cleaner, and in the branch I used to generate the 'cleansed' .slt files it was necessary as normalization was required where validation was not. I have no desire to spend any more time working on that dependency tbh so if you feel strongly about it I'll point you at that project :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Omega359 for the explanation. I don't have a strong opinion on this, just curious about its use cases. Since it can make some things cleaner, its existence is reasonable.

actual: &[Vec<String>],
expected: &[String],
) -> bool {
let expected = expected.iter().map(normalizer).collect::<Vec<_>>();
let actual = actual
.iter()
.map(|strs| strs.iter().join(" "))
// Editors do not preserve trailing whitespace, so expected may or may not lack it included
.map(|s| s.trim_end().to_owned())
.map(|str| normalizer(&str))
.collect::<Vec<_>>();
actual == expected
}
Expand Down Expand Up @@ -159,6 +165,7 @@ async fn run_test_file(test_file: TestFile) -> Result<()> {
))
});
runner.with_column_validator(strict_column_validator);
runner.with_normalizer(normalizer);
runner.with_validator(value_validator);
runner
.run_file_async(path)
Expand All @@ -178,6 +185,7 @@ async fn run_test_file_with_postgres(test_file: TestFile) -> Result<()> {
let mut runner =
sqllogictest::Runner::new(|| Postgres::connect(relative_path.clone()));
runner.with_column_validator(strict_column_validator);
runner.with_normalizer(normalizer);
runner.with_validator(value_validator);
runner
.run_file_async(path)
Expand Down Expand Up @@ -217,6 +225,7 @@ async fn run_complete_file(test_file: TestFile) -> Result<()> {
path,
col_separator,
value_validator,
normalizer,
strict_column_validator,
)
.await
Expand Down Expand Up @@ -246,6 +255,7 @@ async fn run_complete_file_with_postgres(test_file: TestFile) -> Result<()> {
path,
col_separator,
value_validator,
normalizer,
strict_column_validator,
)
.await
Expand Down
Loading