Underflow in timing code on aarch64

[manopapad]

Originally reported by @1193749292 on nv-legate/legate#667. Bug occurs on an Ubuntu 20.04.5 LTS aarch64 system, on a debug build.

The following assertion is failing: https://github.com/StanfordLegion/legion/blob/stable/runtime/realm/timers.inl#L72. I asked @1193749292 to add some instrumentation around that code:

native = 125710472513989694 nanoseconds = 1645055 absolute = 0 zero_time = 1681439899479082722 LLONG_MAX = 9223372036854775807 uint64_t(LLONG_MAX) = 9223372036854775807
native = 125710472514051510 nanoseconds = 1665654 absolute = 0 zero_time = 1681439899479082722 LLONG_MAX = 9223372036854775807 uint64_t(LLONG_MAX) = 9223372036854775807
...(Several similar messages)

Signal 6legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:80: static long long int Realm::Clock::current_time_in_nanoseconds(bool): Assertion `nanoseconds <= uint64_t(LLONG_MAX)' failed.
 uint64_t(LLONG_MAX) = 9223372036854775807
9223372036854775807
legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:80: static long long int Realm::Clock::current_time_in_nanoseconds(bool): Assertion `nanoseconds <= uint64_t(LLONG_MAX)' failed.
 received by node 0, process 

It appears as if the code computing nanoseconds from native is producing a much smaller number than expected.

[lightsighter]

Does this happen deterministically or randomly?

[streichler]

None of those messages have a value of nanoseconds that would trigger the assert. Can we get the contents of the ... there to see if maybe one of the snipped messages is not actually similar?)

[1193749292]

Thank you for your reply.I checked the content of all the ellipses The content in the ```Downloading...``` ellipsis only includes downloading Python modules such as ninja, cmake, wheel, omli, setuptools, packaging, distro, and scikit build from the network, just like this ```Downloading https://files.pythonhosted.org/packages/61/86/cc8d1ff2ca31a312a25a708c891cf9facbad4eae493b3872638db6785eb5/wheel-0.40.0-py3-none-any.whl  (1.3MB)``` ```http://xxxx... ```Of course, it means“ https://files.pythonhosted.org/ ”These websites are available. The path contains a name and is replaced with 'xxx'. Apart from this, no other content is hidden.

[streichler]

Can you attach the entire output? There's at least one partial logging message tangled up in the assert messages, so the other half of it should be somewhere in the blob of text.

[1193749292]

@streichler

All the output of the legion build, legate.core installation and running is here, especially the legate.core installation, which does end in

/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/setuptools/command/egg_info.py:643: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
Please extend command classes from setuptools instead of distutils.
  warnings.warn(

And then there's no