Skip to content

Commit

Permalink
Introducing repetition_delimiter to EDI schema. (#215)
Browse files Browse the repository at this point in the history 
Issue: #212

`repetition_delimiter`: delimiter to separate multiple data instances for an element. For example,
if `^` is the repetition delimiter for a segment `DMG*D8*19690815*M**A^B^C^D~`, then the last
element has 4 pieces of data: `A`, `B`, `C`, and `D`. Any element without `repetition_delimiter`
present has essentially one piece of data; similarly, if `^` is the repetition delimiter for a
segment `CLM*A37YH556*500***11:B:1^12:B:2~`, the last element has 2 pieces of data: `11:B:1` and
`12:B:2`, each of which is further delimited by a `component_delimiter` `:`. Note, since
`repetition_delimiter` creates multiple pieces of data under the same element name in the schema,
in most cases the suitable construct type in `transform_declarations` is `array`.

Currently we read in all the elements and their components in serial in `NonValidatingReader` into
a slice: `[]RawSegElem`, each of which contains the element value, the element index, and component
index if there are more than 1 component. When `repetition_delimiter` is added, we continue down
the same pattern: `NonValidatingReader` still reads everything into the slice, except now, there
potentially can be multiple `RawSegElem` share the same `ElemIndex` and `CompIndex`.

Using the example above: `^` is the rep delim and seg is `CLM*A37YH556*500***11:B:1^12:B:2~`. After
`NonValidatingReader.Read()` is done, we'll have the following `[]RawSegElem` (simplified):

```
{
   {'CLM', ElemIndex: 0, CompIndex: 1},
   {'A37YH556', ElemIndex: 1, CompIndex: 1},
   {'500', ElemIndex: 2, CompIndex: 1},
   {'', ElemIndex: 3, CompIndex: 1},
   {'', ElemIndex: 4, CompIndex: 1},
   {'', ElemIndex: 4, CompIndex: 1},
   {'11', ElemIndex: 5, CompIndex: 1},
   {'B', ElemIndex: 5, CompIndex: 2},
   {'1', ElemIndex: 5, CompIndex: 3},
   {'12', ElemIndex: 5, CompIndex: 1},
   {'B', ElemIndex: 5, CompIndex: 2},
   {'2', ElemIndex: 5, CompIndex: 3},
}
```

Note the last 3 elements have the same `ElemIndex` and `CompIndex` as the previous 3 elements.
This behavior is new and introduced in this PR.

Now on the EDI reader side (reader.go), previously when we match element decl against the raw element
slice, we only do one way scan, because `ElemIndex` and `CompIndex` are always increase, thus we
never need to back-scan. With introduction of potentially duplicate `ElemIndex` and `CompIndex`, now
for each of the element decl, we simply do a full `[]RawSegElem` scan. Yes, it is a bit more expensive
but given usually the number of total elements and components in a seg is really really small (around
20), we feel this trade-off is acceptable without making the already-complex code even more so.

With this reader change, the IDR produced will potentially contain child element nodes with the same
element name. Thus in schema writing, it's practically required that the user of the
`repetition_delimiter` feature needs to use `array` type in the `transform_declarations`.
  • Loading branch information
@jf-tech
jf-tech authored Jul 25, 2023
1 parent 9e0c8da commit 79a540b
Show file tree
Hide file tree
Showing 21 changed files with 7,025 additions and 105 deletions.
12 changes: 11 additions & 1 deletion doc/edi_in_depth.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ A full EDI schema `file_declaration` is as follows:
"segment_delimiter": "<segment delimiter>", <== required
"element_delimiter": "<element delimiter>", <== required
"component_delimiter": "<component delimiter>", <== optional
"repetition_delimiter": "<repetition delimiter>", <== optional
"release_character": "<release character>", <== optional
"ignore_crlf": true/false, <== optional
"segment_declarations": [
Expand Down Expand Up @@ -126,6 +127,15 @@ standards call for a single ASCII character as `element_delimiter`, omniparser a
`component_delimiter` in omniparser allows UTF-8 string. This is optional, and if not specified, you
can treat each element as of a single component.

- `repetition_delimiter`: delimiter to separate multiple data instances for an element. For example,
if `^` is the repetition delimiter for a segment `DMG*D8*19690815*M**A^B^C^D~`, then the last
element has 4 pieces of data: `A`, `B`, `C`, and `D`. Any element without `repetition_delimiter`
present has essentially one piece of data; similarly, if `^` is the repetition delimiter for a
segment `CLM*A37YH556*500***11:B:1^12:B:2~`, the last element has 2 pieces of data: `11:B:1` and
`12:B:2`, each of which is further delimited by a `component_delimiter` `:`. Note, since
`repetition_delimiter` creates multiple pieces of data under the same element name in the schema,
in most cases the suitable construct type in `transform_declarations` is `array`.

- `release_character`: an optional escape character for delimiters. Imagine a piece of element data
contains a `*` which happens to be `element_delimiter`. Without escaping, parser would treat that `*`
as a real delimiter. Any character preceded by `release_character` will be treated literally.
Expand Down Expand Up @@ -550,7 +560,7 @@ And we can add the transform reference into the `FINAL_OUTPUT` directly:
```
Run cli we have:
```
$ cli.sh transform -i 2_ups_edi_210.input.txt -s test.schema.json
$ cli.sh transform -i 2_ups_edi_210.input.txt -s test.schema.json
[
{
"invoice_number": "0000001808WW308"
Expand Down
155 changes: 155 additions & 0 deletions extensions/omniv21/fileformat/edi/.snapshots/TestRead-1_seg_decl_with_rep-delim,_success
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
{
"Records": [
{
"Children": [
{
"Children": [
{
"Children": null,
"Data": "D8",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode d8)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "d8",
"FirstChild": "(TextNode 'D8')",
"FormatSpecific": null,
"LastChild": "(TextNode 'D8')",
"NextSibling": "(ElementNode d_date)",
"Parent": "(ElementNode DMG)",
"PrevSibling": null,
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "19910512",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode d_date)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "d_date",
"FirstChild": "(TextNode '19910512')",
"FormatSpecific": null,
"LastChild": "(TextNode '19910512')",
"NextSibling": "(ElementNode d_cat)",
"Parent": "(ElementNode DMG)",
"PrevSibling": "(ElementNode d8)",
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "RET",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode d_cat)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "d_cat",
"FirstChild": "(TextNode 'RET')",
"FormatSpecific": null,
"LastChild": "(TextNode 'RET')",
"NextSibling": "(ElementNode d_cat)",
"Parent": "(ElementNode DMG)",
"PrevSibling": "(ElementNode d_date)",
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "RET",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode d_cat)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "d_cat",
"FirstChild": "(TextNode 'RET')",
"FormatSpecific": null,
"LastChild": "(TextNode 'RET')",
"NextSibling": "(ElementNode d_code)",
"Parent": "(ElementNode DMG)",
"PrevSibling": "(ElementNode d_cat)",
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "2135-2",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode d_code)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "d_code",
"FirstChild": "(TextNode '2135-2')",
"FormatSpecific": null,
"LastChild": "(TextNode '2135-2')",
"NextSibling": "(ElementNode d_code)",
"Parent": "(ElementNode DMG)",
"PrevSibling": "(ElementNode d_cat)",
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "2106-3",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode d_code)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "d_code",
"FirstChild": "(TextNode '2106-3')",
"FormatSpecific": null,
"LastChild": "(TextNode '2106-3')",
"NextSibling": null,
"Parent": "(ElementNode DMG)",
"PrevSibling": "(ElementNode d_code)",
"Type": "ElementNode"
}
],
"Data": "DMG",
"FirstChild": "(ElementNode d8)",
"FormatSpecific": null,
"LastChild": "(ElementNode d_code)",
"NextSibling": null,
"Parent": "(DocumentNode)",
"PrevSibling": null,
"Type": "ElementNode"
}
],
"FinalErr": "EOF"
}
164 changes: 162 additions & 2 deletions extensions/omniv21/fileformat/edi/.snapshots/TestRead-2_seg_decls,_success
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,167 @@
{
"Records": [
"{'e1':'0','e2':'1','e3':'2'}",
"{'e1':'3','e2':'4','e3':'5'}"
{
"Children": [
{
"Children": [
{
"Children": null,
"Data": "0",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode e1)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "e1",
"FirstChild": "(TextNode '0')",
"FormatSpecific": null,
"LastChild": "(TextNode '0')",
"NextSibling": "(ElementNode e2)",
"Parent": "(ElementNode ISA)",
"PrevSibling": null,
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "1",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode e2)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "e2",
"FirstChild": "(TextNode '1')",
"FormatSpecific": null,
"LastChild": "(TextNode '1')",
"NextSibling": "(ElementNode e3)",
"Parent": "(ElementNode ISA)",
"PrevSibling": "(ElementNode e1)",
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "2",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode e3)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "e3",
"FirstChild": "(TextNode '2')",
"FormatSpecific": null,
"LastChild": "(TextNode '2')",
"NextSibling": null,
"Parent": "(ElementNode ISA)",
"PrevSibling": "(ElementNode e2)",
"Type": "ElementNode"
}
],
"Data": "ISA",
"FirstChild": "(ElementNode e1)",
"FormatSpecific": null,
"LastChild": "(ElementNode e3)",
"NextSibling": null,
"Parent": "(DocumentNode)",
"PrevSibling": null,
"Type": "ElementNode"
},
{
"Children": [
{
"Children": [
{
"Children": null,
"Data": "3",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode e1)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "e1",
"FirstChild": "(TextNode '3')",
"FormatSpecific": null,
"LastChild": "(TextNode '3')",
"NextSibling": "(ElementNode e2)",
"Parent": "(ElementNode ISA)",
"PrevSibling": null,
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "4",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode e2)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "e2",
"FirstChild": "(TextNode '4')",
"FormatSpecific": null,
"LastChild": "(TextNode '4')",
"NextSibling": "(ElementNode e3)",
"Parent": "(ElementNode ISA)",
"PrevSibling": "(ElementNode e1)",
"Type": "ElementNode"
},
{
"Children": [
{
"Children": null,
"Data": "5",
"FirstChild": null,
"FormatSpecific": null,
"LastChild": null,
"NextSibling": null,
"Parent": "(ElementNode e3)",
"PrevSibling": null,
"Type": "TextNode"
}
],
"Data": "e3",
"FirstChild": "(TextNode '5')",
"FormatSpecific": null,
"LastChild": "(TextNode '5')",
"NextSibling": null,
"Parent": "(ElementNode ISA)",
"PrevSibling": "(ElementNode e2)",
"Type": "ElementNode"
}
],
"Data": "ISA",
"FirstChild": "(ElementNode e1)",
"FormatSpecific": null,
"LastChild": "(ElementNode e3)",
"NextSibling": null,
"Parent": "(DocumentNode)",
"PrevSibling": null,
"Type": "ElementNode"
}
],
"FinalErr": "EOF"
}
Loading

0 comments on commit 79a540b

Please sign in to comment.