Ion 1.1 Implementation Roadmap

[toddjonker]

Master issue tracking work to do across various repositories.

• Integrate with conformance test suite
  • Java
  • Rust

System macro table and invocation

Simple system macros are available for invocation by E-expressions in both text and binary documents.

• System macros void and values (implicit-rest only)
  • 💼 Add tests for E-expressions using simple system macros. #100
  • ☕
  • 🦀
• Text Eexp can resolve system macros by local reference (:values)
  • 💼 Add tests for E-expressions using simple system macros. #100
  • ☕
  • 🦀
• Text Eexp can resolve system macros by qualified reference (:$ion:values)
  • 💼 Add tests for E-expressions using simple system macros. #100
  • ☕
  • 🦀
• System macros make_list and make_sexp (implicit-rest only)
  • 💼 Add tests for E-expressions using simple system macros. #100
  • ☕
  • 🦀
• System macros make_string and make_symbol (implicit-rest only)
  • 💼 Add tests for E-expressions using simple system macros. #100
  • ☕
  • 🦀
• System macro annotate with exactly-one annotation (to avoid groups)
  • 💼
  • ☕
  • 🦀

Relevant roadmap items:

• Java:
  • Binary Parsing of E-Expressions
  • Binary Evaluation of Built-in E-Expressions
  • Text Parsing of E-Expressions
  • Text Evaluation of Built-in E-Expressions
  • Binary Write Support for E-Expressions
  • Text Write Support for E-Expressions
• Rust:
  • Binary Write Support for E-Expressions
  • Text Write Support for E-Expressions

Basic Directives; symbol tables

The text and binary parsers detect simple 1.1 directives. Most symbol table features are supported.

• $ion_literal
  • 💼 Add tests for $ion_literal. #97
  • ☕
  • 🦀
• Do-nothing $ion_encoding
  • 💼 Add tests for trivial $ion_encoding forms. #101
  • ☕
  • 🦀
• $ion_encoding symbol_table with local symbols
  • 💼 Add tests for simple symbol_table forms in $ion_encoding. #102
  • ☕
  • 🦀
• load clause, exact-match, with module referenced into symbol_table
  • 💼 Test encoding directive that loads shared module into symbol_table #112
  • ☕
  • 🦀
• Inline module symbol_table and installation
  • 💼 Test simple inline modules with symbol tables. #113
  • ☕
  • 🦀
• Inline module with load
  • 💼 Test simple inline modules with symbol tables. #113
  • ☕
  • 🦀
• $ion_encoding (retain *)
  • 💼 Test encoding directive that loads shared module into symbol_table #112
  • ☕
  • 🦀
• Retain individual modules by name
  • 💼 Test encoding directive that loads shared module into symbol_table #112
  • ☕
  • 🦀

Relevant roadmap items:

• Java:
  • Text Write Support for Encoding Directives
  • Binary Write Support for Encoding Directives
  • Text Read Support for Encoding Directives
  • Binary Read Support for Encoding Directives
• Rust:
  • Text Write Support for Encoding Directives
  • Binary Write Support for Encoding Directives
  • Text Read Support for Encoding Directives
  • Binary Read Support for Encoding Directives

User macro table and invocation

Macro tables can be defined by composing from other modules. E-expressions in both text and binary documents can invoke macros from user-defined modules. (We don't actually define macros yet; we just pass-through system macros.)

• $ion_encoding macro_table; can only meaningfully export $ion at this point
  • 💼 Test simple macro_table in encoding directive. #114
  • ☕
  • 🦀
• module macro_table should be able to pass-through from $ion
  • 💼 Test inline modules with simple macro tables. #115
  • ☕
  • 🦀
• Text/binary Eexp can resolve user macros by local address (:0)
  • 💼 Test simple macro_table in encoding directive. #114
  • ☕
  • 🦀
• Text Eexp can resolve user macros by qualified address (:M:0)
  • 💼 Test inline modules with simple macro tables. #115
  • ☕
  • 🦀
• Text Eexp can resolve user macros by qualified name (:M:values)
  • 💼 Test inline modules with simple macro tables. #115
  • ☕
  • 🦀

Relevant roadmap items:

• Java:
  • Binary Evaluation of User-Defined Macros
  • Text Evaluation of User-Defined Macros

Simple Macro Definitions and Template Language expressions

Macros with single-value parameters can be defined and invoked.

• Constant macro definitions; TL support for literal-value expressions
  • 💼 Test trivial macros in local modules. #116
  • ☕
  • 🦀
• TL literal special form
  • 💼 Test fundamental templates and macro arguments. #117
  • ☕
  • 🦀
• TL invocation of no-argument macros in the same module
  • 💼 Test trivial macros in local modules. #116
  • ☕
  • 🦀
• TL invocation of no-argument macros in the system module eg (values)
  • 💼 Test trivial macros in local modules. #116
  • ☕
  • 🦀
• Macro name validation; duplicate macro-name rejection
  • 💼
  • ☕
  • 🦀
• Macros with one argument (implicitly !); TL parameter references
  • 💼 Test fundamental templates and macro arguments. #117
  • ☕
  • 🦀
• Multi-parameter macros (all implicitly !)
  • 💼 Test fundamental templates and macro arguments. #117
  • ☕
  • 🦀
• TL List quasi-literals; null.list is literal
  • 💼 Test fundamental templates and macro arguments. #117
  • ☕
  • 🦀
• TL struct quasi-literals; null.struct is literal; field fanout {n:(values 1 2)}
  • 💼 Test fundamental templates and macro arguments. #117
  • ☕
  • 🦀

Relevant roadmap items:

• Java:
  • Binary Evaluation of User-Defined Macros
  • Text Evaluation of User-Defined Macros

Cardinality and Rest invocations

Macro definitions can declare non-singleton parameters. Invocations can use argument groups and implicit-rest syntax.

• Explicit ! cardinality declaration in signature
  • 💼
  • ☕
  • 🦀
• ? cardinality: declaration, TL empty-group invocation, Eexp empty-groups, stream-length check
  • 💼
  • ☕
  • 🦀
• Omitting trailing optional arguments
  • 💼
  • ☕
  • 🦀
• * and + cardinality: declaration, TL group invocation, Eexp groups, stream-length check
  • 💼
  • ☕
  • 🦀
• Validate parameter stream-length against declared cardinality
  • 💼
  • ☕
  • 🦀
• Implicit-rest invocation in TL and Text Eexp; now values, make_string, etc are usable
  • 💼
  • ☕
  • 🦀
• Update system macros (esp annotate) with proper cardinalities
  • 💼
  • ☕
  • 🦀
• TL macro invocation using qualified references to system module: (':$ion:values') (':$ion:1')
  • 💼
  • ☕
  • 🦀

Relevant roadmap items:

• Java:
  • Binary Read Support for Tagless Parameter Encodings
  • Text Read Support for Tagless Parameter Encodings
  • Binary Write Support for Tagless Parameter Encodings
  • Text Write Support for Tagless Parameter Encodings
• Rust:
  • Text Read Support for Tagless Parameter Encodings
  • Binary Write Support for Tagless Parameter Encodings
  • Text Write Support for Tagless Parameter Encodings

Importing into modules

Modules can reference user macros from preceding inline modules.

• module use another local module
• TL macro invocation using qualified references to used module: (':M:mac' ...)
• TL macro invocation using bare name: (':mac' ...)
• TL macro invocation using local address: (':3' ...)

Encoding declarations

Macro parameters can be constrained to primitive and macro-shaped encodings.

• Support uint8 encoding; range checks at invocation site
• Parse signatures with macro-type annotations
• Invocation with ! and ? macro-shaped arguments in TL and Eexp
• Invocation with * and + macro-shaped arguments in TL and Eexp (incl arg groups)
• Implicit-rest invocation with macro-shapes

TODO other primitive encodings

Advanced Template Language expressions

• TL if_void, if_single, if_multi special forms
• TL for special form

Convenience Forms

• Anonymous macros
  • 💼
  • ☕
  • 🦀
• Encoding-level use
• import clauses

TODO Shared modules; tunneled modules; more advanced system macros

Other roadmap items:

• Java:
  • Binary Shared Encoding Modules
  • Text Shared Encoding Modules
  • Testing Runner for Ion 1.1 Test Vectors
  • Automatic Macro Generation for Jackson
  • Testing Performance Regression Detection for Ion 1.1
• Rust:
  • Shared Encoding Modules
  • Built-In Macro Generation System
  • Procedural Macro Generation System
  • Ion-Rust Benchmark CLI and Ion 1.1 Test Runner
  • Ion Performance Regression Detection Workflow Integration
  • Testing Runner for Ion 1.1 Test Vectors