Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loss of Energy Controls - Error fetching Teslemetry Energy Site Info XXXXXX data: Multiple 5xx failures #151

Open
3 tasks done
purcell-lab opened this issue Nov 8, 2024 · 10 comments

Comments

@purcell-lab
Copy link
Contributor

Checklist

  • I have filled out the issue template to the best of my ability.
  • This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
  • This issue is not a duplicate issue of any previous issues..

Describe the issue

As reported in Discord the Tesla site is returning 500 status errors, which results in the Energy controls going unavailable.

Screenshot 2024-11-08 16 05 12

I'm wondering if a delay between calling for live_status & site_info may smooth things out a bit?

Reproduction steps

Config log attached.

config_entry-teslemetry-9148783ba548ef27e70b73108cd5fd97 (9).json

Debug logs

2024-11-08 16:04:29.800 DEBUG (MainThread) [tesla_fleet_api] Requested: https://na.teslemetry.com/api/1/energy_sites/2252097574301386/live_status
2024-11-08 16:04:29.801 DEBUG (MainThread) [tesla_fleet_api] Response Status: 200
2024-11-08 16:04:29.801 DEBUG (MainThread) [tesla_fleet_api] Response JSON: {'response': {'solar_power': 1961, 'percentage_charged': 100, 'battery_power': -200, 'load_power': 6130, 'grid_status': 'Active', 'grid
_power': 4369, 'generator_power': 0, 'island_status': 'on_grid', 'storm_mode_active': False, 'timestamp': '2024-11-08T16:04:29+10:00', 'wall_connectors': [{'din': '1529455-02-E--PGT22325087207', 'wall_connector_
state': 2, 'wall_connector_fault_state': 2, 'wall_connector_power': 0, 'ocpp_status': 1, 'powershare_session_state': 1}, {'din': '1529455-02-E--PGT22325086977', 'vin': '5YJ3F7EC5LF698201', 'wall_connector_state'
: 4, 'wall_connector_fault_state': 2, 'wall_connector_power': 0, 'ocpp_status': 1, 'powershare_session_state': 1}]}}
2024-11-08 16:04:29.802 DEBUG (MainThread) [custom_components.teslemetry] Finished fetching Teslemetry Energy Site Live 2252097574301386 data in 0.672 seconds (success: True)
2024-11-08 16:04:43.126 DEBUG (MainThread) [tesla_fleet_api] Requested: https://na.teslemetry.com/api/1/energy_sites/2252097574301386/site_info
2024-11-08 16:04:43.126 DEBUG (MainThread) [tesla_fleet_api] Response Status: 500
2024-11-08 16:04:43.128 DEBUG (MainThread) [tesla_fleet_api] An error occurred while processing the request.
2024-11-08 16:04:43.129 ERROR (MainThread) [custom_components.teslemetry] Error fetching Teslemetry Energy Site Info 2252097574301386 data: Multiple 5xx failures
2024-11-08 16:04:43.130 DEBUG (MainThread) [custom_components.teslemetry] Finished fetching Teslemetry Energy Site Info 2252097574301386 data in 5.240 seconds (success: False)
2024-11-08 16:04:45.335 DEBUG (MainThread) [custom_components.teslemetry] Streaming received vehicle_data from 5YJ3F7EC5LF698201
2024-11-08 16:04:45.335 DEBUG (MainThread) [custom_components.teslemetry] Manually updated Teslemetry Vehicle 5YJ3F7EC5LF698201 data
2024-11-08 16:04:47.385 DEBUG (MainThread) [custom_components.teslemetry] Streaming received state from LRWYHCFS9PC851970
2024-11-08 16:04:47.385 DEBUG (MainThread) [custom_components.teslemetry] Manually updated Teslemetry Vehicle LRWYHCFS9PC851970 data
@Bre77
Copy link
Member

Bre77 commented Nov 10, 2024

Hey Mark do you have any evidence that other services work when Teslemetry doesnt? I am thinking I'll need to start running a test with Tessie, Teslemetry, and the Fleet API directly to see if there is a correlation.

Here is the number of status 500 messages for energy sites over the last 7 days and the number of users it impacted. A small number of people constantly have issues, but there are also spikes.

image

@purcell-lab
Copy link
Contributor Author

Ongoing issue. I mainly use teslemetry for my controls exclusively so haven't noticed the effects on any other platforms.

Is there user ability to reduce or back off the numbers of requests being sent in my behalf?

This error originated from a custom integration.

Logger: custom_components.teslemetry
Source: helpers/update_coordinator.py:414
integration: Teslemetry
First occurred: November 14, 2024 at 7:29:18 PM (77 occurrences)
Last logged: 5:30:43 AM

Error fetching Teslemetry Energy Site Info 2252097574301386 data: Multiple 5xx failures

@purcell-lab
Copy link
Contributor Author

This isdue is only for energy sites, so Tessie is not relevant.

Here are some comparisons, both are going unavailable at similar times.

Entity display from Teslemetry:
Screenshot_20241115-054237

Entity display from Tesla Fleet:

Screenshot_20241115-054435

@Bre77
Copy link
Member

Bre77 commented Nov 14, 2024

Hey Mark,

Updates are run every 30 seconds, you can take control of this as such:

On the Integration entries page, you can click the three dots besides your Teslemetry config entry and select system options. Here you can disable polling for updates. You would then need to setup your own automation to update an entity from each coordinator on whatever schedule you want.

There are three energy coordinators (info, live and history) so if you need help identifying a entity from each let me know.

I am now running a test between Teslemetry, Tessie, and Tesla Fleet API directly to try and identify if I can replicate your issue and isolated it to Teslemetry or not. (Tessie does support the whole Fleet API including energy)

image

This is your requests over the last 24 hours:
image

And this is mine:
image

You are making more POST requests to change things than me, so I may need to start changing things randomly to see if that contributes.

I will also turn up the logging on 5xx errors to see if we can learn something new.

@Bre77
Copy link
Member

Bre77 commented Nov 15, 2024

Its probably no help, but you have more errors on the energy APIs than any other Teslemetry user. Today you have been 45% of all 5xx errors on the energy APIs. Every single one, was a response from Tesla saying upstream internal error.

My only two theories are: there is a connectivity issue to your Powerwall, or because you are frequenty changing your configuration, the powerwall itself is unable to respond to API requests at times.

I will continue to monitor.

@purcell-lab
Copy link
Contributor Author

purcell-lab commented Nov 15, 2024

I do seem to have a lot of devices and entities, but I would image that isn't part of the root cause..:

George (Model Y) was a hire car I haven't had access to for 6 months, but I don't seem to be able to delete..

image

image

@Bre77
Copy link
Member

Bre77 commented Nov 15, 2024

George (Model Y) was a hire car I haven't had access to for 6 months, but I don't seem to be able to delete..

Does it still show up in your Tesla app or products endpoint? There is a seperate issue where I need to do device cleanups.

@purcell-lab
Copy link
Contributor Author

Ok, I have connected my gateway via Ethernet, with 1Gb fibre connection to NBN. Should of done a while ago..

Let's see if this improves connectivity...

Screenshot_20241118-115219

@purcell-lab
Copy link
Contributor Author

Looks like NetZero is seeing increased 5xx errors as well:
netzero-labs/netzero#55 (reply in thread)

@purcell-lab
Copy link
Contributor Author

Looks like some correlation when my sensors are unavailable and the Tesla app reports 3rd party updates.

Screenshot_20241206-055754
Screenshot_20241206-055807

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants