Skip to content

Commit

Permalink
sources: remove newlines from inferredSchemaIsNotAvailable
Browse files Browse the repository at this point in the history
The sentinel schema that's used when the inferred schema is not available
contained newline characters, which make their way into the connector
protocols. This breaks some connectors, which expect each message to be on its
own line, since the schema value in, for example, a `Validate` request could
contain newlines. This changes the handling of that sentinel schema to use
plain `serde_json::Value`s instead of `RawValue`s, so that re-encoding it as
JSON will cause the newlines to be removed.
  • Loading branch information
psFried committed Oct 10, 2023
1 parent 73d8608 commit b9ee14a
Showing 1 changed file with 7 additions and 2 deletions.
9 changes: 7 additions & 2 deletions crates/models/src/schemas.rs
Original file line number Diff line number Diff line change
Expand Up @@ -137,12 +137,17 @@ impl Schema {
}
"###,
);
let mut inferred_schema: Skim = serde_json::from_str(inferred_bundle).unwrap();
// We don't use `Skim` here because we want the serde round trip to
// transform the sentinel schema from pretty-printed to dense. This
// is important because newlines in the schema could otherwise break
// connectors using the airbyte protocol.
let mut inferred_schema: BTreeMap<String, serde_json::Value> =
serde_json::from_str(inferred_bundle).unwrap();

// Set $id to "flow://inferred-schema".
_ = inferred_schema.insert(
KEYWORD_ID.to_string(),
RawValue::from_value(&Value::String(Self::REF_INFERRED_SCHEMA_URL.to_string())),
Value::String(Self::REF_INFERRED_SCHEMA_URL.to_string()),
);
// Add as a definition within the read schema.
read_defs.insert(
Expand Down

0 comments on commit b9ee14a

Please sign in to comment.