Skip to content

Commit

Permalink
Use encodingErrorPolicy replace, even when set to error
Browse files Browse the repository at this point in the history
If encodingErrorPolicy="error" we print a warning letting the user know
it is not supported and that we will use "replace" instead. This is so
that we can support schemas written for IBM DFDL (which only supports
"error") and have the exact same behavior as IBM DFDL except for when
data has encoding errors, which is usually rare.

However, even though we warn that we will use "replace", we do not
actually do so, which leads to an assertion exception if there are
encoding errors. This fixes that so our behavior matches the warning
message and we always use "replace".

DAFFODIL-2861
  • Loading branch information
stevedlawrence committed Dec 12, 2023
1 parent c58434c commit 37d6746
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -59,15 +59,19 @@ trait TermEncodingMixin extends KnownEncodingMixin { self: Term =>
)
optionEncodingErrorPolicy.getOrElse(EncodingErrorPolicy.Replace)
}
if (policy == EncodingErrorPolicy.Error) {
// DFDL-935 to enable
SDW(
WarnID.EncodingErrorPolicyError,
"dfdl:encodingErrorPolicy=\"error\" is not yet implemented. The 'replace' value will be used.",
)
EncodingErrorPolicy.Replace
}
policy

// DFDL-935 to enable
schemaDefinitionWarningWhen(
WarnID.EncodingErrorPolicyError,
policy == EncodingErrorPolicy.Error,
"dfdl:encodingErrorPolicy=\"error\" is not yet implemented. The 'replace' value will be used.",
)

// The encodingErrorPolicy property is essentially ignored. We always use "replace", but
// warn if it is "error". This allows for compatibility with schemas written for IBM DFDL
// (which only supports "error") and has the same behavior as IBM DFDL as long as data does
// not contain encoding errors
EncodingErrorPolicy.Replace
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -678,6 +678,7 @@

<tdml:document>
<tdml:documentPart type="text">1234</tdml:documentPart>
<tdml:documentPart type="byte">80</tdml:documentPart> <!-- invalid in US-ASCII -->
</tdml:document>

<tdml:warnings>
Expand All @@ -693,6 +694,7 @@
<ex:char>2</ex:char>
<ex:char>3</ex:char>
<ex:char>4</ex:char>
<ex:char>&#xFFFD;</ex:char> <!-- unicode replacement character -->
</ex:root>
</tdml:dfdlInfoset>
</tdml:infoset>
Expand Down

0 comments on commit 37d6746

Please sign in to comment.