Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow milliseconds for all timestamps #333

Open
peterdesmet opened this issue Jun 20, 2023 · 2 comments
Open

Allow milliseconds for all timestamps #333

peterdesmet opened this issue Jun 20, 2023 · 2 comments

Comments

@peterdesmet
Copy link
Member

peterdesmet commented Jun 20, 2023

Some terms (especially media timestamp and eventStart and eventEnd) could benefit from expressing milliseconds. Ideally we support 4 ways of writing timestamps (all ISO):

2013-11-23T08:30:00Z         # seconds UTC
2013-11-23T06:30:00-0200     # seconds + offset
2013-11-23T08:30:00.300Z     # milliseconds + UTC
2013-11-23T06:30:00.300-0200 # milliseconds + offset

All of these seem to pass Frictionless Framework validation for format=default, but according to the specs, default implies:

An ISO8601 format string e.g. YYYY-MM-DDThh:mm:ssZ in UTC time

It looks like Frictionless Framework also allows timezone offsets (i.e. more than the specs allow). @roll is this intentional?

For Camtrap DP we therefor opted to explicitly define the format:

"format": "%Y-%m-%dT%H:%M:%S%z",

That pattern will however fail for milliseconds, so we should adapt it. Or we should always require milliseconds.

Note that %z correctly allows -0200, +0200, Z and correctly forbids empty and -0280.

@peterdesmet
Copy link
Member Author

peterdesmet commented Jun 20, 2023

Explicitly defining the format (i.e. not relying on default) and having milliseconds optional is not possible. format relies on strptime, which does not have a format code for optional milliseconds (there is a feature request for it). So we have two options:

  1. Always requiring milliseconds and using format %Y-%m-%dT%H:%M:%S.%f%z. Note that e.g. Movebank exports always have milliseconds.
  2. Relying on format = default, which allows (in Frictionless Framework) the four formats above, even though the specs say it should forbid non UTC. It (annoyingly) also allows timestamps without timezone information.

@peterdesmet
Copy link
Member Author

peterdesmet commented Oct 31, 2023

Discussed with @kbubnicki, moved to 1.1. Camtrap DP 1.0 will thus not allow milliseconds, but we keep the explicit format that warns for missing timezone (which is likely a more common use case). We'll have to see how this evolves at Frictionless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant