New scalar-unit grammar proposal

[tliron]

Following up on our ad hoc discussion, I propose the following tweaks:

To create a scalar-unit type you need to do two things at minimum:

1. Derive the type from either integer or float or any derivative of them (Note: does it make any sense to derive from another scalar-unit type? Could that be useful in any way, e.g. to add additional unit symbols? Probably an unnecessary complication, but if we do support it we have to explain how the units are "refined" in inheritance.)
2. Add a base_units keyname (Proposed rename from unit_symbol_map; rationale: shorter, and also the rows are actually both unit symbols and factors and unit_symbol_factor_map would be way too long)

The base_units keyname is a map of unit symbols to factors (multipliers). Depending on the base type, the factors must be either integers or floats. (Note that factors could potentially be negative. Hard to think of real-world use cases, but no reason to forbid it. Zero would also be weird, but possible.)

One (and only one) of these entries must have a 1 factor (or 1.0 for float). Otherwise it will be an error. (Implied: base_units can never be an empty map, it needs at the very least a 1 or 1.0 unit symbol). For other entries, there could be more than one unit symbol mapped to the same factor, allowing for notation variations.

This required 1 (or 1.0) factor unit symbol is called the "canonical unit". It can optionally be used by TOSCA implementations to convert all data to the canonical unit for calculations (preserving precision), uniform storage, representation, etc.

Rules for unit symbols (must be validated by implementations):

1. They cannot start with a numeric digit or the decimal .. (Otherwise parsing a scalar-unit string could break in crazy ways. A numeric digit in the middle or the end shouldn't cause any problems. Weird, but unproblematic.)
2. They cannot start or end with whitespace. (Same rationale as above, but different parsing challenge.)
3. Cannot be an empty string. (No strong reason why not in terms of parsing, but it could just be inconsistent and confusing.)

Also note that unit symbols are case sensitive, if relevant (they don't have to include letters). To support different case variations, just add all the variants to base_units and map them to the same factor.

Minimal example:

data_types:
  time:
    derived_from: float
    base_units: { s: 1.0 }

Note that you can add a validation keyname just like with any other type, which is often a good idea for scalar-units for quantities:

data_types:
  length:
    derived_from: float
    validation: { $greater_or_equal: [ $value, 0.0 ] }
    base_units: { feet: 1.0 }

AUTO-GENERATING SYMBOLS VIA PREFIXES

One can optionally add a prefixes keyname to automatically fill in the units map with many entries. The syntax for prefixes is identical to that of base_units (but without the requirement of having a canonical symbol).

The way it works is that the actual units will actually be every prefix with every base unit combination (written consecutively, with no whitespace between), with both factors multiplied.

Important! If you want to have entries with no prefix, you explicitly have to add an empty string prefix with a factor of 1 or 1.0!

Simple example:

data_types:
  length:
    derived_from: float
    base_units: { m: 1.0 }
    prefixes:
      '': 1.0, # allows for no-prefix units
      k: 1000.0
      μ: 0.0001

The actual units for this example would be all the combinations. If we wrote them explicitly without prefixes it would be equivalent to this:

data_types:
  length:
    derived_from: float
    base_units:
      m: 1.0 # '' + 'm' to 1.0 * 1.0
      km: 1000.0 # 'k' + 'm' to 1000.0 * 1.0
      μm: 0.0001 # 'μ' + 'm' to 0.0001 * 1.0

Important little implementation note: When prefixes is present, the "canonical unit symbol" must be checked for after generating all possible units. This is to make sure the final result indeed has a single entry with a 1 or 1.0 factor.

The current example in the spec of using YAML anchors to reuse prefixes is excellent!

[tliron]

Variation on the proposal:

Instead of requiring the canonical unit symbol to be the "one and only one 1 factor" in units, we could require it to be specified with a mandatory keyname. The advantage would be allowing for multiple 1-factor units. The disadvantage is that, well, it's an extra keyname.

I propose canonical_unit, which must to be equal to one of the unit symbols in units (its factor does not have to be 1).

Real-world example:

data_types:
  colonial-length:
    derived_from: float
    base_units:
      inch: 1.0
      inches: 1.0
      '"': 1.0
      foot: 12.0
      feet: 12.0
      "'": 12.0
      yard: 36.0
      yards: 36.0
    canonical_unit: '"'

MORE!

Note that there is another possibility for grammar that has the advantages of both approaches:

1. If there is a canonical_unit keyname (non-mandatory), use it.
2. If there isn't, then look for the the canonical unit from among the base_units using the original heuristic.
3. Only if the heuristic doesn't apply (there is not one and only one 1-factor unit) do we emit an error in grammar.

It's a bit more complicated to explain but balances ease of use while still allowing edge cases. If users get it wrong they will get a clear error and can decide how to fix it, either by adding a canonical_unit keyname or changing the units to make sure there is only one 1-factor unit.

[pmjordan]

The above broadly accords with my understanding of our discussion in the meeting and I can see that it is a reasonable simpilifaction of the syntax. However:

1. I don't like having both 'units' and 'real units', or having the keyword 'unit' means something slightly different depending on whether 'prefixes' is present. Better to rename the map to some other keyword (e.g. 'symbols') and then state that unit either equals symbol or equals prefix+symbol depending on whether 'prefixes' is present.
2. It should be possible to derive scalar from derivatives of integer and float as well as integer and float themselves.
3. Nice idea having a single entry map rather than a new keyword when using 'prefixes'. That would need to be a stated rrequirement when the 'prefix' keyword is used.
4. The null string line looks a bit odd but I can see that it makes implementation easier
5. Would we allow canonical_symbol if 'prefixes' is present? I would prefer not as there is a difficulty in knowing whether to name it as unit or symbol. I think it is reasonable to assume and implement that the one entry in the map currently named 'units' is taken as the cannonical entry when 'prefixes' are used.
6. I think you stated on the call that you would like to be able to enter a value with no unit. That is not allowed in the current text. The default unit in that case would be the connonical. I don't much like that idea as a user could get confused as to what default unit applies to the parameter. Less errors if the unit has to be specified.

[tliron]

All good points, I'm basically good with all your suggestions.

As for 1, I think symbols might not be clear enough as a keyname. For one, it's actually a table of symbols and factors. But also, if you just look at a definition that has symbols it's not immediately clear that this a scalar-unit definition. Having the word "unit" (or maybe also "scalar"?) there is important, I think. But this is a petty point.

As for 4, I agree, but it's also a very "mechanical" algorithm that joins the two tables together. If people choose to use this feature I think it's best to be very mechanical about its usage. A clear example in the spec, like the one I provided, should dispel any confusion.

As for 6, I merely proposed it as an option, but I think we collectively agreed that it's more confusing than helpful and we prefer to force users to explicitly specify a unit symbol, even if it is technically possible to default to the canonical symbol.

Any opinions on the variants I proposed in the follow up comment?

[pmjordan]

point 1. My main concern is to have a word which is not 'unit' for that map as in some cases the indexes in teh map are in fact only a substring of the final unit string. How about calling the map 'multipliers'?
point 4. OK
point 6. OK, Let's not allow it

I am OK with the concept of canonical_unit_symbol but it is quite a long keyname and you prefer short names, It should only be allowed where 'prefixes' is not used (my point 5) in which case 'canoical_unit' is enough.

[tliron]

I do prefer short but also clarity is a high priority. :) Agree with canonical_unit.

Actually, maybe we can take this a step further: we use the word "symbol" a lot in the spec, and I followed through here. But maybe we can avoid using the word "symbol". Just use the word "unit" to refer to the symbol for the unit, too.

[pmjordan]

When 'prefixes' is used then unit is derived from the key associated with the mulitplier and the prefix key. When 'prefixes' is not used then unit is equal to the key associted with the muliplier. i.e. the unit for kilometers is Km. So it is incorrect to call either the K or the m a unit. Therefore table containing the multipliers must have a name other than unit.

[pmjordan]

base_units is good enough to address my concern. So I think you and I are now agreed on all the issues.

[tliron]

Thanks! As the original author, to you want to rework the text of the spec or should I? I'm happy either way. I'll be doing a lot of work on the spec anyway.

[tliron]

Also, I guess we should wait for the other folk to opine, too. :)

[tliron]

I have updated the proposal based on the feedback.

[pmjordan]

I will make a start.

[pmjordan]

I've drafted the required changes in my repo,

[lauwers]

I know this is late, but I have two comments:

1. Based on the description, I'm unclear about the purpose of canonical_base_unit. Wouldn't the multiplier be used to convert all values to a "canonical" representation? I recommend removing the canonical_base_unit keyword.
2. I find the base_units name slightly confusing. How about the following alternative:
   • If no prefixes are defined, then the units are expressed simply using a units keyword which defines a map of "units" and associated multipliers.
   • If prefixes are used, then we support two keynames: a unit keyword which is a string representing the unit, and a prefixes keyword that defines a map of prefixes and associated multipliers.

[tliron]

1. Multiplying by factor gives you a number, but with what unit symbol would you represent it? My first proposal was that it can be determined with a heuristic -- the unit would be whatever is the "1" factor. This is indeed fairly easy, but as you can see in my addendum it does have limitations. There I also propose supporting both: the keyword being optional. If the keyword is not there, the heuristic would be used. By the way, I proposed called it canonical_unit (not canonical_base_unit) because I expect a complete "actual" unit to be written there (prefix, if used, plus base).
2. Honestly, I think units would be OK, that was my original proposal. But @pmjordan does have a point that it's confusing when used in combination with prefixes, because these are then only half of the "actual" units. But ... is it really that confusing? The documentation can make the meaning clear enough.

[lauwers]

1. OK, so the canonical_unit would only be used for presentation purposes?
2. I wasn't proposing using units in combination with prefixes. I suggested using the singular unit instead when prefixes are used. One would either use units by themselves or prefixes + unit.

[tliron]

1. For storage and transfer, too. If you are moving a number around, wouldn't you have to know what it represents? "Length" is not enough.
2. Ah, I thought that was a typo. But look closely: when you have prefixes, there could still be many units, so I don't think just unit would be clear. My example was minimal with just one unit, but there could be more than one, no?

[lauwers]

1. In that case, aren't we breaking inheritance (i.e. scalar units are not really integers or floats)?
2. Paul's language explicitly states that when prefixes are used there can only be one base unit.

[pmjordan]

Yes I confirm that when we use prefixes there can only be exactly one base-unit.
I included the canonical keyword at the request of @tliron , it can only be used where prefix is not used. I would be ok for it to be removed and implictly set to the unit with a multiplier of one.
I would also be ok to have a syntax which alllowed either a) a single keyword of unit or b) two keywords of unit_prefixes and unit_suffix but I thought that one of our design aims was to minimise the number of keywords. I am still not keen on using a combination of unit_prefixes and unit for the reasons already stated.

[tliron]

I think having a canonical unit is necessary for any real-world usage. It seems you two are not convinced, but I haven't heard any specific response to the rationales I provided: representation, data storage, data transfer, calculation precision. Consider a specific "transfer" use case of updating a scalar-unit attribute. In what unit should the number be?

I actually proposed 3 different ways to determine the canonical unit: an automatic heuristic, a mandatory keyname, or a combination of both (non-mandatory keyname). I was hoping to hear from you folk what seems best, because I didn't have a strong opinion. However, I think I'm leaning on the "both" proposal variant. For most uses the heuristic will just work. When it doesn't work, there will be a clear error message and designers can just add the keyname. The only real disadvantage of this proposal variant is that it's a bit harder to explain in the spec.

Somehow I missed the point that prefixes only apply to one single base unit, but it does make sense! I tried to think of a real world use case in which there might be more than one, but couldn't think of any. (Although I wonder if the fact that I can't think of any is just due to my limited knowledge...)

Here is my new proposal.

There would be two ways to define a scalar unit type:

1. units keyname. or
2. base_unit and prefixes (these two keynames must appear together and cannot appear together with units)

And there is an additional non-mandatory keyname:

canonical_unit: must be a possible unit. So if you're using prefixes, it must be a complete unit (prefix + base unit). If this keyname is not present, the heuristic will be used (look for a one-and-only 1-factor unit). If the heuristic fails, there will be an error message.

[lauwers]

If a value of scalar unit type IS A integer (as specified by derived_from) then it needs to be stored as an integer, which implies that the conversion from <value><unit> must be done immediately.

[tliron]

Right, but what is that integer? Is it the number of seconds or number of milliseconds? Users can define their scalar-units in any way they want.

[pmjordan]

If the integer is converted immediately using the multipler then the stored value must be a float?

[tliron]

If the base type is integer, all the factors have to be integers, too, and the result would be an integer. When the base type is a float everything will be floats. That's how I understand it. That keeps precision intact through all the math.

[pmjordan]

My error - my draft does include that restriction.

[tliron]

New proposal based on ad hoc discussion.

A new TOSCA type named scalar. Scalars are represented as YAML strings (and never as floats or integers):

{optional whitespace} + {a number in YAML notation} + {whitespace} + {a unit symbol} + {optional whitespace}

Note that we should support all ways that numbers can be represented in YAML. This includes floating points and integer representations, scientific notations, hex, octal, etc. So implementations should best use their YAML parser to parse the number string.

The scalar type cannot be used directly, instead usable types must be derived from it. Since these are custom types (and not primitives) we will prefer the camel-case naming convention.

When you derive from scalar, you must specify the units keyname, which is a table of unit symbols to unit factors. (See original proposal above for rules for valid symbols.)

For maintaining calculation precision, scalar data must be either float (the default), or integer, or a derivative of them. This can be specified via the data_type keyname. The unit factors must match the data type (again, to maintain precision in the multiplication).

Simple examples:

data_types:
  Frequency:
    derived_from: scalar
    # default to data_type: float
    units:
      hz: 1.0
      khz: 1000.0
      mhz: 1000000.0

  DataSize:
    derived_from: scalar
    data_type: integer
    units:
      b: 1
      kib: 1024
      mib: 1048576

At least one of the units must have a factor of 1.0 (or 1), called the canonical unit. This is used for translating raw numbers back to the scalar representation (which might have to happen in TOSCA as well as in cloud and storage systems). For example, if you subtract "1 b" from "1 kib", the result would be represented as "1023 b", and indeed be stored as such.

Note that for integer-based scalars, we round down the result of multiplying a floating point value by its unit factor. Thus "0.3 mib" = 0.3 * 1048576 = 314572.8 = canonical "314572 b".

If there is more than one unit with a factor of 1.0, then the canonical_unit keyname must be added to specify which of the units to use. It cannot be left as an arbitrary choice. Otherwise representations would be inconsistent, for example breaking string-based equality comparisons. Example:

data_types:
  ColonialLength:
    derived_from: scalar
    units:
      inch: 1.0
      inches: 1.0
      '"': 1.0
      foot: 12.0
      feet: 12.0
      "'": 12.0
      yard: 36.0
      yards: 36.0
    canonical_unit: '"'

It is also possible to auto-generate the units via the addition of prefixes to all units. Note that an empty-string prefix must be explicitly specified if appropriate:

data_types:
  Frequency:
    derived_from: scalar
    units:
      hz: 1.0
      Hz: 1.0
    prefixes:
      '': 1.0
      μ: 0.0001
      m: 0.001
      k: 1000
      M: 1000000
      g: 1000000000
    canonical_unit: Hz

Note that canonical_unit must refer to the complete unit, with the prefix. (In this example it was the empty prefix "Hz".)

[lauwers]

I'm fine with this proposal, except that we need to rename scalar to scalarunit since we already use the word scalar in the document to refer to all data types that can be considered scalars, including integers, floats, strings, etc.

[tliron]

So, I looked at the spec and you're right, we do call these basic types "scalars". However, I would argue that we shouldn't. We should call them primitives, and call this type just scalar.

1. "Primitive" is how I referred to them in the text I wrote for their definitions.
2. It's not very common in computer programming to call a "string" a scalar, or in general to use the term. (Counterpoint: It's true that the YAML spec uses that language. But I would argue that it shouldn't.)
3. "Scalar" could be confused with "scalar-unit", too.
4. I've always disliked the term "scalar-unit". It sounds like "scalar" is an adjective describing the unit, which doesn't make sense and also is not the intent.

If you agree I am happy to find uses of the word "scalar" in the spec and change them to "primitive" when appropriate.

EDIT: Actually, looking closely it's a bit worse than that. I've noticed places where we use the word "scalar" when we specifically mean numerical. Substitute "string" in those cases and it's clearly wrong. So even we get confused, and we need to edit those out anyway. I think by just not using the word "scalar" there we can improve clarity and remove ambiguity.

[pmjordan]

I agree in particular with Tal's point 4 and have no objection to Tal's proposed solution to naming of scalars etc.

[pmjordan]

I'm in general agreement with the summary proposal but would make the following comments:

As written it is not clear to the reader why we have the prefixes keyword. I would like to keep an example which re-uses the same prefix map across more than one scalar derivation. The current spec references such an example from the dsl_defintions section so it servers two purposes.

The examples provided by Tal are look a bit like SI but are not quite right:
hz should be Hz, mhz should be MHz, mib should be MiB, g should be G

The given example for datasize could use the built-in datatype of bytes

The current spec needs to adapt the range example as it currently uses scalar-unit.frequency which no longer exists.

We discussed including a warning that implementor may want to guard against word overflow when performing calculations on multipliers with many digits.

Mulitpliers for scalars with a data_type value of float may be given as integers because of the TOSCA rule that integers can be converted to floats (see float section).

Time will need to be editted to this new syntax.

Do we want to reinstate time as an in-built data type but specified using this syntax?

As an aside I was interested to see the term 'Colonial length' which I've not come across before. In the UK we would call this Imperial length. Still OASIS uses American English so it can remain :-)

[tliron]

Oh, I did not intend to write the spec language here, this was just for us to discuss. I just assumed that you would be writing it clearly! Please do use real-world examples, I just spat things out. Ignore my examples.

Good point about auto conversion of integers to floats.

[tliron]

Rereading and realized I missed some important points:

The given example for datasize could use the built-in datatype of bytes

I'm not sure what you mean here. It's true that bytes has a size (number of bytes), but it's not a defined part of that value. We do have the $length function that can find it for us, but that returns an integer, not a scalar.

Which unfortunately leads me to a gap in our plan. We probably do need to specify a way to cast scalar values to their data type and back. Because it makes sense that someone would create a DataSize type and then want to compare it to a $length.

I will humbly suggest that, at the very least, we allow comparisons of scalar values to float and integer values, and that when this happens the canonical unit is used. So the integer 100 would be "100 b", and on the flip side the scalar "1 kb" would be the integer 1024.

With calculations it's more difficult. Adding an integer and a scalar could make sense (we use the canonical unit for the scalar), but should the result be an integer or a scalar?

Or do we need explicit functions to cast between scalars and primitives?

This is something we can postpone to TOSCA 2.1, but it's important to note the limitations of our current solution.

We discussed including a warning that implementor may want to guard against word overflow when performing calculations on multipliers with many digits.

Yes. And this is true for all our arithmetic functions, too.

It would be good to get feedback from TOSCA users as to the suitability of our data model. We've at least standardized it in 2.0 (64 bits), but there may be room for improvement. E.g. precision specification (128 bit integers? unsigned integers?), support for fixed point, and probably we also need a BigInt type.

Do we want to reinstate time as an in-built data type but specified using this syntax?

It's a slippery slope. If we allow for time we should also do byte-size, frequency, etc.

I would say that time types, especially, vary dramatically in the canonical unit and even data type. Some count seconds, some nanoseconds. Some do it in float, some in integer. So I would say that time especially should be a custom type in profiles and not part of TOSCA.

I think we're right in making all of the scalars custom. However, as I pointed out above, we probably need to allow for casting/conversion for them to be fully usable.

[pmjordan]

I've added the warning about word overflow to an additional pull request.
I've left time as an example of scalar, not a built-in
We are not using the datasize example so let's not get hung up on it now.
I can't think of a need to compare a scalar with a float or integer but you might want to multiply or divide but I'm content to leave that to version 2.x

Note that I've also disallowed validation on scalars, validation can of course be added to the data_type of the number part and I show that in an example.

[tliron]

Why disallow the validation keyname? It should work like on any data type. Just note that unless we allow for automatic casting then comparisons would have to be via the string representation, e.g.:

validation: { $greater_or_equal: [ $value, '0 hz' ] }

I think it could be useful to include that in one of the examples.

[tliron]

Another thought: I think we should provide examples for all TOSCA 1.3 scalars to make it easier to migrate.

[lauwers]

I support making all of these changes, but I also would like to get a final draft of the spec done before the weekend. Any chance we can make that happen?

[tliron]

I will submit a lot of papercut fixes tomorrow, including what we discussed here. I'm leaving the text of scalar to @pmjordan .

[pmjordan]

I'll try

[pmjordan]

@tliron You will need to make the changes to reword misuse of the word scalar elsewhere in the spec.

[lauwers]

Hi Tal, any progress on your edits?

[tliron]

Some progress, my goal now is to finish this weekend.

[pmjordan]

Regarding validation. The working branch currently has a conflict - I disallowed validation of scalars because it seemed to hard to address the complexities at this late stage but failed to remove one from the example - see line 4722. That has a format rather different to your suggestion but closer to what I think a user would like.

[pmjordan]

I've now resolved that - I've corrected the validation example to be the correct syntax as mentioned by @tliron. Type derivation is disallowed but validation of scalars is allowed.

[pmjordan]

I think that we should allow scalar type derivation - but only for canonical_unit. e.g. allow TimeInSeconds and TimeInYears to be derived from Time. That would sensible presentation of say ProtocolTimeout and PersonAge BUT ALSO allow comparision of any unit derived from Time by refernce to the canonical unit of the common parent. More implementation complexity though.
What do you think?

[tliron]

I think it's possible but

1. Not sure how useful it really is
2. Ability to cast to integers/floats can solve such use cases in a different way (users can do their own math)
3. We unfortunately are out of time to dig into this for 2.0, though our benevolent chair should have the final say here

[lauwers]

1. Scalar type derivation should be possible just like it is for any other type. This could be useful, for example, to add validations in a derived type.
2. At the same time, we need to clearly define derivation rules. For example, I don't think it should be allowed to redefine the canonical_unit if it is already defined in a parent type.

[tliron]

Nor should the data type change.

But things get complicated when deciding how to refine prefixes+units. I feel like postponing this to 2.1 is a good idea. We can also take time to get feedback from users about the whole idea of custom scalars.

[lauwers]

Yes, it seems to me that the derivation rules could be very simple. The following should be allowed:

1. Add validations
2. Add to the prefixes map if it is defined
3. If there are no prefixes, then add to the units map.

[pmjordan]

Yes, it seems to me that the derivation rules could be very simple. The following should be allowed:

1. Add validations

2. Add to the `prefixes` map if it is defined

3. If there are no `prefixes`, then add to the `units` map.

We did discuss derivation of scalars once before, in particular adding more 'units' to the existing mao, but at that time decided against it as it was not considered useful enough to bother with.
The 'canonical_unit' already has a implict default so there is no difference between adding an explict value for the first time and overwriting an existing explict value.

We have probably reached the point where publishing is more important than a further debate but happy to add theses simple rules or for @lauwers to do so if that is the consensus.