Observation.weather.reference_time() is not consistent across queries.

[julj]

When running successive call to mgr.weather_at_id (and other requests) I get inconsistent results (see below).
Using directly the OWM API, the 'dt' value changes only every few minutes...

In [63]: for _ in range(5):
    ...:     obs = owm_mgr.weather_at_id(3143244)
    ...:     print(obs.weather.reference_time())
    ...: 
    ...: 

1608211374
1608211374
1608211448
1608211394
1608211374

[csparpa]

Hello @julj

the dt says when the actual measurement has been taken on the OWM API side.

This means that if that measurement is the only available one during your for loop, the value of obs.weather.reference_time()) will never change.

This of course depends on how frequently the measurements change for the place you're asking for

So, in conclusion: PyOWM always returns the dt that is embedded in the actual OWM API response, if that dt doesn't change over time this means that either the datapoint hasn't changed since or the OWM API is not updating it correctly

[julj]

Hi,

there seems to be 2 different issues.

A difference between the dt from a call with your module VS directly to the API:

In [35]: for i in range(5):
    ...:     obs = owm_mgr.weather_at_id(city_id)
    ...:     print(f"{'pyowm reference_time':<25}: {obs.weather.reference_time()}")
    ...:     r = requests.get(f'https://api.openweathermap.org/data/2.5/weather?id={city_id}&appid={OWM_KEY}')
    ...:     print(f"{'owm dt':<25}: {json.loads(r.content.decode('utf-8'))['dt']}")
    ...: 
pyowm reference_time     : 1608734979
owm dt              : 1608734675
pyowm reference_time     : 1608734979
owm dt              : 1608734675
pyowm reference_time     : 1608734979
owm dt              : 1608734675
pyowm reference_time     : 1608734979
owm dt              : 1608734675
pyowm reference_time     : 1608734979
owm dt              : 1608734675

And, when called from within a Docker container, some weird dt variations from both (I'll investigate that):

In [24]: for _ in range(5):
    ...:     obs = owm_mgr.weather_at_id(city_id)
    ...:     print(f"{'pyowm reference_time':<25}: {obs.weather.reference_time()}")
    ...:     r = requests.get(f"https://api.openweathermap.org/data/2.5/weather?id={city_id}&appid={os.getenv('OWM_API_KEY')}")
    ...:     print(f"{'own dt':<25}: {json.loads(r.content.decode('utf-8'))['dt']}")
    ...: 
pyowm reference_time     : 1608734913
own dt              : 1608734675
pyowm reference_time     : 1608734979
own dt              : 1608734974
pyowm reference_time     : 1608734913
own dt              : 1608734675
pyowm reference_time     : 1608734979
own dt              : 1608734974
pyowm reference_time     : 1608734913
own dt              : 1608734675

[csparpa]

Ok this deserves further investigation

[csparpa]

@julj Long answer, take time to read

ISSUE N.2 (DOCKER)

This is very likely tied to how Docker containers time systems work - they were known to have jitters with respect to the host machine's timeframe

ISSUE N.1 (PYOWM vs OWM API TIME DRIFT)

Spoiler: I don't have an explanation for that. But I've been able to come up with some further evidence.

I've modified your original code so that the call you make with requests to the OWM API is exactly equal to the one made by PyOWM (in the example, on city 3143244):

r = requests.get(f'https://api.openweathermap.org/data/2.5/weather',
                            verify=True, stream=True,
                            params={'APPID': OWM_KEY, 'lang': 'en', 'id': 3143244})

pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868

It basically seems like that the dt returned by the server depends on the parameters in the request.

Provided that PyOWM provides the timestamp value as provided by the API (no rounding is performed), I have NO IDEA why this happens.

It is worth to note that the params for the PyOWM GET request depend on PyOWM configuration (eg. proxies, language, etc.)

Any thoughts?

[julj]

@julj Long answer, take time to read

ISSUE N.2 (DOCKER)

This is very likely tied to how Docker containers time systems work - they were known to have jitters with respect to the host machine's timeframe

But those timestamps are from the API, not from the system clock, right ?

ISSUE N.1 (PYOWM vs OWM API TIME DRIFT)

Spoiler: I don't have an explanation for that. But I've been able to come up with some further evidence.

I've modified your original code so that the call you make with requests to the OWM API is exactly equal to the one made by PyOWM (in the example, on city 3143244):

r = requests.get(f'https://api.openweathermap.org/data/2.5/weather',
                            verify=True, stream=True,
                            params={'APPID': OWM_KEY, 'lang': 'en', 'id': 3143244})

pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868
pyowm reference_time     : 1609250868
owm dt                   : 1609250868

It basically seems like that the dt returned by the server depends on the parameters in the request.

Provided that PyOWM provides the timestamp value as provided by the API (no rounding is performed), I have NO IDEA why this happens.

It is worth to note that the params for the PyOWM GET request depend on PyOWM configuration (eg. proxies, language, etc.)

Any thoughts?

Not really, I don't see why the parameters you set in the request, verify or stream, would change the response. I don't have much time to investigate, but if I come up with any idea I will let you know here. Thank you for your help !