[linky] Yet another website underlaying API modification #17538

clinique · 2024-10-10T14:39:54Z

Resolves #17513

Signed-off-by: Gaël L'hopital <[email protected]>

clinique · 2024-10-10T14:51:19Z

This gives a first answer to the initial connection issue but I must investigate a bit, data may have changed leading to UNDEF items.

Signed-off-by: Gaël L'hopital <[email protected]>

lolodomo · 2024-10-11T11:46:47Z

I will test this evening and finish the review.

...nhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/LinkyHandlerFactory.java

...rg.openhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/dto/PrmDetail.java

...hab.binding.linky/src/main/java/org/openhab/binding/linky/internal/handler/LinkyHandler.java

lolodomo · 2024-10-11T21:05:47Z

@clinique : is your upgrade of jsoup really necessary ? I can't start the binding with your new version.

If I update my bundle with command bundle:update:

22:59:24.572 [ERROR] [Events.Framework                     ] - FrameworkEvent ERROR
org.osgi.framework.BundleException: Could not resolve module: org.openhab.binding.linky [278]
  Unresolved requirement: Import-Package: org.jsoup; version="[1.15.0,2.0.0)"

        at org.eclipse.osgi.container.Module.start(Module.java:463) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.container.ModuleContainer.update(ModuleContainer.java:419) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.storage.Storage.update(Storage.java:969) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.storage.Storage.update(Storage.java:935) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle.update(EquinoxBundle.java:490) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.apache.karaf.bundle.command.Update.update(Update.java:74) ~[?:?]
        at org.apache.karaf.bundle.command.Update.doExecute(Update.java:51) ~[?:?]
        at org.apache.karaf.bundle.command.BundleCommand.execute(BundleCommand.java:49) ~[?:?]
        at org.apache.karaf.shell.impl.action.command.ActionCommand.execute(ActionCommand.java:88) ~[?:?]
        at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:68) ~[?:?]
        at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:86) ~[?:?]
        at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:599) ~[?:?]
        at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:526) ~[?:?]
        at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:415) ~[?:?]
        at org.apache.felix.gogo.runtime.Pipe.doCall(Pipe.java:416) ~[?:?]
        at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:229) ~[?:?]
        at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:59) ~[?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]

If I uninstall the bundle and then put the jar file in folder addons:


23:02:16.166 [WARN ] [org.apache.felix.fileinstall         ] - Error while starting bundle: file:/xxxxxxx/openhab/addons/linky.jar
org.osgi.framework.BundleException: Could not resolve module: org.openhab.binding.linky [299]
  Unresolved requirement: Import-Package: org.jsoup; version="[1.15.0,2.0.0)"

        at org.eclipse.osgi.container.Module.start(Module.java:463) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:445) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1260) [bundleFile:3.7.4]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1233) [bundleFile:3.7.4]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:520) [bundleFile:3.7.4]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:365) [bundleFile:3.7.4]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:316) [bundleFile:3.7.4]

lolodomo · 2024-10-11T21:07:50Z

Please upgrade jsoup in a separate PR.

I am now trying to remove jsoup upgrade to be able to test the fix!

lolodomo · 2024-10-11T21:13:22Z

Please upgrade jsoup in a separate PR.

This will also facilitate backport to branch 4.2.x.

...hab.binding.linky/src/main/java/org/openhab/binding/linky/internal/handler/LinkyHandler.java

lolodomo · 2024-10-11T21:21:05Z

If I remove jsoup upgrade, I am able to use the binding again and it is alive (ONLINE) again, bravo.
You made a mistake with channels for years but except this mistake the channels are apparently set again with values.
I will check tomorrow if the values are consistent with what I get from the web site.

clinique · 2024-10-12T08:10:19Z

If I remove jsoup upgrade, I am able to use the binding again and it is alive (ONLINE) again, bravo. You made a mistake with channels for years but except this mistake the channels are apparently set again with values. I will check tomorrow if the values are consistent with what I get from the web site.

When I switch my dev environment to linky I always have a hard time (mvn clean install, update project, clean project...) resolving dependencies (especially jsoup). I had a look at what was used by other binding and saw 1.14.3, 1.15.3 and 1.15.4. On maven rep I saw that 1.14.3 had 3 security issues. Reason why I made this move but I can revert to 1.14.3

Signed-off-by: Gaël L'hopital <[email protected]>

lolodomo · 2024-10-12T08:44:17Z

When I switch my dev environment to linky I always have a hard time (mvn clean install, update project, clean project...) resolving dependencies (especially jsoup). I had a look at what was used by other binding and saw 1.14.3, 1.15.3 and 1.15.4. On maven rep I saw that 1.14.3 had 3 security issues. Reason why I made this move but I can revert to 1.14.3

Changing this dependency seems to prevent the update of the binding in an existing OH setup.
But I guess that it might work if I build and install a full new distribution.
So it is better to have this PR without jsoup upgrade, we could even distribute the jar file and it could be installed by users in their current OH version.
But you can create a separate PR for jsoup upgrade and this change will be integrated in our next OH versions.

lolodomo · 2024-10-12T08:47:31Z

I just installed the very last version and the internal server error is back ! Let's wait the retries during the day, it occurs that the service is not available before the end of the morning. I will see with the retry ar 11 am.

Edit: Still in error at 11 am.

lolodomo · 2024-10-12T08:52:58Z

...hab.binding.linky/src/main/java/org/openhab/binding/linky/internal/handler/LinkyHandler.java

-                int idxLast = days.periodes.get(days.periodes.size() - 1).dateDebut.get(weekFields.dayOfWeek()) == 7 ? 2
-                        : 1;
-                Aggregate weeks = values.aggregats.weeks;
-                if (weeks.datas.size() > idxLast) {
-                    updateKwhChannel(LAST_WEEK, weeks.datas.get(idxLast));
-                }
-                if (weeks.datas.size() > (idxLast + 1)) {
-                    updateKwhChannel(THIS_WEEK, weeks.datas.get(idxLast + 1));
-                } else {
-                    updateKwhChannel(THIS_WEEK, 0.0);
-                }
+                setCurrentAndPrevious(values.aggregats.weeks, THIS_WEEK, LAST_WEEK);


I would like to be sure that the proper week values are delivered every day of the week. I previously implemented it to have a proper 0 value each Monday as current week value and not the value from last week.
So before merging, I will check if the week values are correct Saturday (today), Sunday, Monday and Tuesday.

No problem, I'm not in a hurry.

Values for yesterday, current week, last week, current month and last month are correct today.
Let's see tomorrow.

clinique · 2024-10-12T10:03:47Z

When I switch my dev environment to linky I always have a hard time (mvn clean install, update project, clean project...) resolving dependencies (especially jsoup). I had a look at what was used by other binding and saw 1.14.3, 1.15.3 and 1.15.4. On maven rep I saw that 1.14.3 had 3 security issues. Reason why I made this move but I can revert to 1.14.3

Changing this dependency seems to prevent the update of the binding in an existing OH setup. But I guess that it might work if I build and install a full new distribution. So it is better to have this PR without jsoup upgrade, we could even distribute the jar file and it could be installed by users in their current OH version. But you can create a separate PR for jsoup upgrade and this change will be integrated in our next OH versions.

I'll make a separate PR then. This will reduce a bit the number of bindings being on 1.14.x (we are far behind main branch of jsoup 1.18.1) and there are known vulnerabilities up to 1.15.2

lolodomo · 2024-10-12T10:10:52Z

I just installed the very last version and the internal server error is back ! Let's wait the retries during the day, it occurs that the service is not available before the end of the morning. I will see with the retry ar 11 am.

Edit: Still in error at 11 am.

Still not working for me at Midday.. Can you try on your side ?

...penhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java

clinique · 2024-10-12T14:40:29Z

I just installed the very last version and the internal server error is back ! Let's wait the retries during the day, it occurs that the service is not available before the end of the morning. I will see with the retry ar 11 am.
Edit: Still in error at 11 am.

Still not working for me at Midday.. Can you try on your side ?

Tested on my side, the same today.
Errors:
Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/HLC699EAZ/prms/21105933323207/donnees-pmax?dateDebut=10-10-2024&dateFin=12-10-2024&mesuretypecode=CONS'

Weird thing is the same query in the browser works without issue.

clinique · 2024-10-12T14:53:36Z

I just installed the very last version and the internal server error is back ! Let's wait the retries during the day, it occurs that the service is not available before the end of the morning. I will see with the retry ar 11 am.
Edit: Still in error at 11 am.

Still not working for me at Midday.. Can you try on your side ?

Now it works...
BtW I think I spotted an error : "Pic de Consommation" should be kVA, not W. I'll add a unitHint

clinique · 2024-10-13T16:27:34Z

Still KO today.

You may try to tace it in debug mode to understand where it's blocked.

lolodomo · 2024-10-13T18:43:25Z

Here are my logs:

20:31:29.713 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'linky:linky:local' changed from UNINITIALIZED (NOT_YET_READY) to INITIALIZING
20:31:29.721 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Initializing Linky handler.
20:31:29.729 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'linky:linky:local' changed from INITIALIZING to UNKNOWN
20:31:29.739 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Starting login process for user: [email protected]
20:31:29.740 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Step 1: getting authentification
20:31:30.272 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Reception request SAML
20:31:30.276 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Step 2: send SSO SAMLRequest
20:31:30.425 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Get the location and the ReqID
20:31:30.435 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Step 3: auth1 - retrieve the template, thanks to cookie internalAuthId user is already set
20:31:30.894 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Step 4: auth2 - send the auth data
20:31:31.140 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Add the tokenId cookie
20:31:31.141 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Step 5: retrieve the SAMLresponse
20:31:31.240 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Step 6: post the SAMLresponse to finish the authentication
20:31:31.299 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'linky:linky:local' changed from UNKNOWN to ONLINE
20:31:31.299 [DEBUG] [g.linky.internal.api.ExpiringDayCache] - getValue from cache "daily cache" is requiring a fresh value
20:31:31.300 [DEBUG] [g.linky.internal.handler.LinkyHandler] - getConsumptionData from 2024-09-28 to 2024-10-13
20:31:31.394 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Exception when getting consumption data: Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=28-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
org.openhab.binding.linky.internal.LinkyException: Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=28-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getContent(EnedisHttpApi.java:214) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getData(EnedisHttpApi.java:228) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getMeasures(EnedisHttpApi.java:263) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getEnergyData(EnedisHttpApi.java:268) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.getConsumptionData(LinkyHandler.java:363) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.lambda$0(LinkyHandler.java:101) ~[?:?]
        at org.openhab.binding.linky.internal.api.ExpiringDayCache.refreshValue(ExpiringDayCache.java:86) ~[?:?]
        at org.openhab.binding.linky.internal.api.ExpiringDayCache.getValue(ExpiringDayCache.java:73) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.updateDailyWeeklyData(LinkyHandler.java:241) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.updateData(LinkyHandler.java:202) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.lambda$4(LinkyHandler.java:175) ~[?:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
20:31:31.401 [DEBUG] [g.linky.internal.api.ExpiringDayCache] - calcNextExpiresAt result = 2024-10-14T01:00:00
20:31:31.402 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'linky:linky:local' changed from ONLINE to OFFLINE (COMMUNICATION_ERROR): Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=28-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
20:31:31.407 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Update channel daily#yesterday with NaN
20:31:31.410 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Update channel weekly#thisWeek with NaN
20:31:31.411 [DEBUG] [g.linky.internal.api.ExpiringDayCache] - getValue from cache "monthly cache" is requiring a fresh value
20:31:31.412 [DEBUG] [g.linky.internal.handler.LinkyHandler] - getConsumptionData from 2024-09-01 to 2024-10-13
20:31:31.557 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Exception when getting consumption data: Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=01-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
org.openhab.binding.linky.internal.LinkyException: Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=01-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getContent(EnedisHttpApi.java:214) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getData(EnedisHttpApi.java:228) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getMeasures(EnedisHttpApi.java:263) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getEnergyData(EnedisHttpApi.java:268) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.getConsumptionData(LinkyHandler.java:363) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.lambda$2(LinkyHandler.java:129) ~[?:?]
        at org.openhab.binding.linky.internal.api.ExpiringDayCache.refreshValue(ExpiringDayCache.java:86) ~[?:?]
        at org.openhab.binding.linky.internal.api.ExpiringDayCache.getValue(ExpiringDayCache.java:73) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.updateMonthlyData(LinkyHandler.java:262) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.updateData(LinkyHandler.java:203) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.lambda$4(LinkyHandler.java:175) ~[?:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
20:31:31.563 [DEBUG] [g.linky.internal.api.ExpiringDayCache] - calcNextExpiresAt result = 2024-10-14T01:00:00
20:31:31.564 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'linky:linky:local' changed from OFFLINE (COMMUNICATION_ERROR): Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=28-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"} to OFFLINE (COMMUNICATION_ERROR): Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=01-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
20:31:31.567 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Update channel monthly#thisMonth with NaN
20:31:31.569 [DEBUG] [g.linky.internal.api.ExpiringDayCache] - getValue from cache "yearly cache" is requiring a fresh value
20:31:31.570 [DEBUG] [g.linky.internal.handler.LinkyHandler] - getConsumptionData from 2023-01-01 to 2024-10-13
20:31:31.719 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Exception when getting consumption data: Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=01-01-2023&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
org.openhab.binding.linky.internal.LinkyException: Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=01-01-2023&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getContent(EnedisHttpApi.java:214) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getData(EnedisHttpApi.java:228) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getMeasures(EnedisHttpApi.java:263) ~[?:?]
        at org.openhab.binding.linky.internal.api.EnedisHttpApi.getEnergyData(EnedisHttpApi.java:268) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.getConsumptionData(LinkyHandler.java:363) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.lambda$3(LinkyHandler.java:139) ~[?:?]
        at org.openhab.binding.linky.internal.api.ExpiringDayCache.refreshValue(ExpiringDayCache.java:86) ~[?:?]
        at org.openhab.binding.linky.internal.api.ExpiringDayCache.getValue(ExpiringDayCache.java:73) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.updateYearlyData(LinkyHandler.java:279) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.updateData(LinkyHandler.java:204) ~[?:?]
        at org.openhab.binding.linky.internal.handler.LinkyHandler.lambda$4(LinkyHandler.java:175) ~[?:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
20:31:31.727 [DEBUG] [g.linky.internal.api.ExpiringDayCache] - calcNextExpiresAt result = 2024-10-14T01:00:00
20:31:31.729 [INFO ] [hab.event.ThingStatusInfoChangedEvent] - Thing 'linky:linky:local' changed from OFFLINE (COMMUNICATION_ERROR): Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=01-09-2024&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"} to OFFLINE (COMMUNICATION_ERROR): Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/NGYYYY/prms/xxxxxx/donnees-energie?dateDebut=01-01-2023&dateFin=13-10-2024&mesuretypecode=CONS': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}
20:31:31.732 [DEBUG] [g.linky.internal.handler.LinkyHandler] - Update channel yearly#thisYear with NaN
20:31:31.733 [DEBUG] [ding.linky.internal.api.EnedisHttpApi] - Logout process

lolodomo · 2024-10-13T19:04:57Z

Checking the request done by the website, I agree that it is the same (except the parameter dateDebut).
We are setting no headers in the request while several are set by the website including Accept, Accept-Encoding, Accept-Language, Host, User-Agent, ... Maybe some of them are important. By the way, it would be better to provide them. Let's try it now...

lolodomo · 2024-10-13T19:40:52Z

I tried with this code but same result, internal server error:

            ContentResponse result = httpClient.newRequest(url).method(HttpMethod.GET)
                    .header(HttpHeader.ACCEPT, "application/json")
                    .header(HttpHeader.ACCEPT_ENCODING, "gzip, deflate, br, zstd")
                    .header(HttpHeader.ACCEPT_LANGUAGE, "fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3")
                    .header("Host", "alex.microapplications.enedis.fr")
                    .agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0").send();

Same without the Host header and same without user agent.

lolodomo · 2024-10-20T08:31:30Z

Hello Gaël , how can we explain that it works for you and not for me ? How can we progress ?

clinique · 2024-10-20T14:06:22Z

Hello Gaël , how can we explain that it works for you and not for me ? How can we progress ?

Sorry Laurent, right now I don't have a clue. Maybe we can ask some other users to test the jar and give us their feedback ?

lolodomo · 2024-10-20T14:26:23Z

Maybe we can ask some other users to test the jar and give us their feedback ?

Ok. Can you prepare that and communicate on the community forum so that we have other feedbacks?

clinique · 2024-10-20T17:53:00Z

Maybe we can ask some other users to test the jar and give us their feedback ?

Ok. Can you prepare that and communicate on the community forum so that we have other feedbacks?

Done here

jlaur · 2024-10-21T08:02:11Z

...penhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java

-            logger.debug("Step 4 : auth2 - send the auth data");
-            result = httpClient.POST(authenticateUrl).header(HttpHeader.CONTENT_TYPE, "application/json")
+            logger.debug("Step 4: auth2 - send the auth data");
+            result = httpClient.POST(authenticateUrl).header(HttpHeader.CONTENT_TYPE, MediaType.APPLICATION_JSON)


I'm not sure if this could cause more than one Content-Type header, which could be rejected by some webservers. The safer approach might be to provide the content-type below instead of manually adding the header:

.content(new StringContentProvider(gson.toJson(authData), MediaType.APPLICATION_JSON))

Thank you for your suggestion but the binding does not break at this step.

lolodomo · 2024-10-27T08:04:49Z

@clinique : very strange but very good news. I just installed the last snapshot and as soon as I updated the linky bundle, it started to work. Even after an OH restart, it is still OK. It is as if your fix requires a recent change in core... Strange...
I will restart a mini review and merge your fix.

lolodomo · 2024-10-28T12:53:39Z

It is again KO today!

clinique · 2024-11-01T09:50:24Z

It is again KO today!

On my side, one day working one day not... :-(

lo92fr · 2024-11-08T16:55:05Z

Hello @clinique, Hello @lolodomo,

Sorry for it, but I’ve don't see you were working on your side on the same issue.

I've hit the same internal server error as you.
I'm not sure if I’m on the correct path, but I was able to fix it on my side.

I've found two different things:

1/ The Enedis site seems now to check / be dependent of the user agent, I've to set a fixed user agent to correct the error.

2/ It's seeming that we need a second cookie to be there to make sure we don't it the 500 error. This cookie can be obtains making a call to the URL: /mon-compte-client/api/private/v1/userinfos. I've made this on the authentication part, as a step 7.

This changes is implements on the linkyv2 branch.
Perhaps you can take a look to see if it helps on your side.

@clinique : I've not fully read your PR so far. Specially your modification on the URL: /mes-prms/api/private/v2/personnes/%s/prms that I've not have on my side.
I will need to realign my Linkyv2 branch to get your modification for it.

One note about linkyv2 current state : It's still WIP.
The matter with it is that I find that the Dataconnect API is far more unstable that the current Website implementation.
I have multiple issue with Enedis in the last past few months, and currently there still an issue on the api that is not fixed since around 5 weeks :(

Also, I still have some initialization error with 4.3 milestone on Dev Environment that I've don't understand so far, and that make my development effort hardier.

What I find currently:

Addon initialization is ok on first time.
When you stop and restart Openhab, Linky thing stay uninitialized.
It seems that the state is like that because of missing prerequisites in ThingManagerImpl (missing Xml resources).
To fix this, I'm currently working with a modified Openhab core where I comment out the missingPrerequisites.put(thing.getUID(), thingPrerequisites) in thingAdded function
For some reasons I've don't understand, the onReadyMarkerAdded function of ThingManagerImpl is never called. And so, the prerequisiteCheckerJob is not scheduled. If so, it will certainly fix the issue?

Also, nothing to go with Linky binding, but I’ve also realized that on my production environment, that javascript rule take forever to load: around 5 minutes after the start of openhab. I think that javascript rules now are load only after all binding are initialized? and on my setup, I have a lot of bindings, and somes take a long time to come online.

Laurent.

lolodomo · 2024-11-09T07:23:55Z

This changes is implements on the linkyv2 branch.
Perhaps you can take a look to see if it helps on your side.

@lo92fr bravo 👍
@clinique : are you available to inject these changes in your PR ?
I am going to test myself in the next hour but to merge the PR myself, it is better that no code is coming from myself.
If it works, we have to commit today or max tomorrow morning.

Signed-off-by: Gaël L'hopital <[email protected]>

clinique · 2024-11-09T09:21:43Z

@lo92fr , @lolodomo : I added your modifications proposal but still facing issue 500 :-(

lolodomo · 2024-11-09T09:35:11Z

I just tried with the following changes but for me the result is still the same: error 500.

$ git diff src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java
diff --git a/bundles/org.openhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java b/bundles/org.openhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java
index 7cbfa1d3b5..0a6269985e 100644
--- a/bundles/org.openhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java
+++ b/bundles/org.openhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java
@@ -16,7 +16,10 @@ import java.net.HttpCookie;
 import java.net.URI;
 import java.time.LocalDate;
 import java.time.format.DateTimeFormatter;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Objects;
+import java.util.Optional;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeoutException;
 import java.util.regex.Matcher;
@@ -27,9 +30,11 @@ import javax.ws.rs.core.MediaType;
 import org.eclipse.jdt.annotation.NonNullByDefault;
 import org.eclipse.jetty.client.HttpClient;
 import org.eclipse.jetty.client.api.ContentResponse;
+import org.eclipse.jetty.client.api.Request;
 import org.eclipse.jetty.client.util.FormContentProvider;
 import org.eclipse.jetty.client.util.StringContentProvider;
 import org.eclipse.jetty.http.HttpHeader;
+import org.eclipse.jetty.http.HttpMethod;
 import org.eclipse.jetty.http.HttpStatus;
 import org.eclipse.jetty.util.Fields;
 import org.jsoup.Jsoup;
@@ -63,6 +68,7 @@ public class EnedisHttpApi {
     private static final String URL_MON_COMPTE = "https://mon-compte" + ENEDIS_DOMAIN;
     private static final String URL_COMPTE_PART = URL_MON_COMPTE.replace("compte", "compte-particulier");
     private static final String URL_ENEDIS_AUTHENTICATE = URL_APPS_LINCS + "/authenticate?target=" + URL_COMPTE_PART;
+    private static final String USER_INFO_CONTRACT_URL = URL_APPS_LINCS + "/mon-compte-client/api/private/v1/userinfos";
     private static final String USER_INFO_URL = URL_APPS_LINCS + "/userinfos";
     private static final String PRM_INFO_BASE_URL = URL_APPS_LINCS + "/mes-mesures/api/private/v1/personnes/";
     private static final String PRM_INFO_URL = URL_APPS_LINCS + "/mes-prms/api/private/v2/personnes/%s/prms";
@@ -161,6 +167,26 @@ public class EnedisHttpApi {
             if (result.getStatus() != HttpStatus.FOUND_302) {
                 throw new LinkyException("Connection failed step 6");
             }
+
+            logger.debug("Step 7 : retrieve a cookie");
+            result = httpClient.GET(USER_INFO_CONTRACT_URL);
+            HashMap hashRes = gson.fromJson(result.getContentAsString(), HashMap.class);
+            String cookieKey;
+            if (hashRes != null && hashRes.containsKey("cnAlex")) {
+                cookieKey = "personne_for_" + (String) hashRes.get("cnAlex");
+            } else {
+                throw new LinkyException("Connection failed step 7, missing cookieKey");
+            }
+            List<HttpCookie> cookies = httpClient.getCookieStore().getCookies();
+            Optional<HttpCookie> cookie = cookies.stream().filter(it -> it.getName().contains(cookieKey)).findFirst();
+            String cookieVal;
+            if (cookie.isPresent()) {
+                cookieVal = cookie.get().getValue();
+            } else {
+                throw new LinkyException("Connection failed step 7, missing cookieVal");
+            }
+            addCookie(cookieKey, cookieVal);
+
             connected = true;
         } catch (InterruptedException | TimeoutException | ExecutionException | JsonSyntaxException e) {
             throw new LinkyException(e, "Error opening connection with Enedis webservice");
@@ -209,7 +235,9 @@ public class EnedisHttpApi {

     private String getContent(String url) throws LinkyException {
         try {
-            ContentResponse result = httpClient.GET(url);
+            Request request = httpClient.newRequest(url).method(HttpMethod.GET)
+                    .agent("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0");
+            ContentResponse result = request.send();
             if (result.getStatus() != HttpStatus.OK_200) {
                 throw new LinkyException("Error requesting '%s': %s", url, result.getContentAsString());
             }

lolodomo · 2024-11-09T09:37:30Z

@lo92fr , @lolodomo : I added your modifications proposal but still facing issue 500 :-(

Ok, we did it in parallel.
I agree with you @clinique , it still does not work :(

lolodomo · 2024-11-09T09:39:24Z

Let see if the retries at 11h and 12h will succeed...

lo92fr · 2024-11-09T10:24:21Z

Let see if the retries at 11h and 12h will succeed...

It's strange, was working on my side this morning.
I've made some test on my dev environment and production environment, and everything was ok.

Laurent.

lo92fr · 2024-11-09T10:28:02Z

Let see if the retries at 11h and 12h will succeed...

It's strange, was working on my side this morning. I've made some test on my dev environment and production environment, and everything was ok.

Laurent.

Let me know if you can't make it work.
I will try to switch back to main branch to see if I can reproduce it on my side !

Laurent.

lo92fr · 2024-11-09T10:39:00Z

Let see if the retries at 11h and 12h will succeed...

It's strange, was working on my side this morning. I've made some test on my dev environment and production environment, and everything was ok.
Laurent.

Let me know if you can't make it work. I will try to switch back to main branch to see if I can reproduce it on my side !

Laurent.

@lolodomo, @clinique : hum, was working this morning, but not work anymore now !

I've hit the error on the URI : https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/undefined/prms/xxxx/courbe-de-charge?dateDebut=03-11-2024&dateFin=09-11-2024&mesuretypecode=CONS.
Not work as well in a standard web browser, so there's something wrong on their side ...

Laurent.

lo92fr · 2024-11-09T16:14:08Z

Let see if the retries at 11h and 12h will succeed...

It's strange, was working on my side this morning. I've made some test on my dev environment and production environment, and everything was ok.
Laurent.

Let me know if you can't make it work. I will try to switch back to main branch to see if I can reproduce it on my side !
Laurent.

@lolodomo, @clinique : hum, was working this morning, but not work anymore now !

I've hit the error on the URI : https://alex.microapplications.enedis.fr/mes-mesures/api/private/v1/personnes/undefined/prms/xxxx/courbe-de-charge?dateDebut=03-11-2024&dateFin=09-11-2024&mesuretypecode=CONS. Not work as well in a standard web browser, so there's something wrong on their side ...

Laurent.

The site seems to came back online.

Laurent.

lolodomo · 2024-11-09T16:45:18Z

Yes, my thing is now ONLINE.

@clinique : same for you ?

lolodomo · 2024-11-09T16:47:52Z

Thanks you Laurent

clinique · 2024-11-09T17:25:56Z

@lo92fr , @lolodomo : yes, online on my side also

Yet another website underlaying API modification

94cba60

Signed-off-by: Gaël L'hopital <[email protected]>

clinique requested a review from lolodomo as a code owner October 10, 2024 14:39

clinique self-assigned this Oct 10, 2024

clinique added the bug An unexpected problem or unintended behavior of an add-on label Oct 10, 2024

clinique added 2 commits October 11, 2024 10:54

Correction for current and previous week, month, year

5c0c9f9

Signed-off-by: Gaël L'hopital <[email protected]>

Correction of pom

d8b7b5d

Signed-off-by: Gaël L'hopital <[email protected]>

lolodomo reviewed Oct 11, 2024

View reviewed changes

...nhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/LinkyHandlerFactory.java Show resolved Hide resolved

lolodomo reviewed Oct 11, 2024

View reviewed changes

...hab.binding.linky/src/main/java/org/openhab/binding/linky/internal/handler/LinkyHandler.java Outdated Show resolved Hide resolved

clinique added 3 commits October 12, 2024 10:11

Code review n°1 corrections, revert jsoup upgrade

4fa8450

Signed-off-by: Gaël L'hopital <[email protected]>

Forgot feature.xml

1d1cc21

Signed-off-by: Gaël L'hopital <[email protected]>

Ongoing code review

952616a

Signed-off-by: Gaël L'hopital <[email protected]>

clinique requested review from lolodomo and lsiepel October 12, 2024 08:25

Reverting

fe8ff60

Signed-off-by: Gaël L'hopital <[email protected]>

lolodomo reviewed Oct 12, 2024

View reviewed changes

...penhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/api/EnedisHttpApi.java Show resolved Hide resolved

lsiepel removed their request for review October 18, 2024 13:02

jlaur reviewed Oct 21, 2024

View reviewed changes

Adding new cookie and user agent

98f9317

Signed-off-by: Gaël L'hopital <[email protected]>

lolodomo approved these changes Nov 9, 2024

View reviewed changes

lolodomo merged commit 7cab153 into openhab:main Nov 9, 2024
5 checks passed

lolodomo added this to the 4.3 milestone Nov 9, 2024

[linky] Yet another website underlaying API modification #17538

[linky] Yet another website underlaying API modification #17538

Conversation

clinique commented Oct 10, 2024 • edited by jlaur Loading

clinique commented Oct 10, 2024

lolodomo commented Oct 11, 2024

lolodomo commented Oct 11, 2024 • edited Loading

lolodomo commented Oct 11, 2024 • edited Loading

lolodomo commented Oct 11, 2024

lolodomo commented Oct 11, 2024 • edited Loading

clinique commented Oct 12, 2024

lolodomo commented Oct 12, 2024

lolodomo commented Oct 12, 2024 • edited Loading

lolodomo Oct 12, 2024

Choose a reason for hiding this comment

clinique Oct 12, 2024

Choose a reason for hiding this comment

lolodomo Oct 27, 2024

Choose a reason for hiding this comment

clinique commented Oct 12, 2024

lolodomo commented Oct 12, 2024 • edited Loading

clinique commented Oct 12, 2024 • edited Loading

clinique commented Oct 12, 2024 • edited Loading

clinique commented Oct 13, 2024

lolodomo commented Oct 13, 2024

lolodomo commented Oct 13, 2024 • edited Loading

lolodomo commented Oct 13, 2024

lolodomo commented Oct 20, 2024

clinique commented Oct 20, 2024

lolodomo commented Oct 20, 2024

clinique commented Oct 20, 2024

jlaur Oct 21, 2024

Choose a reason for hiding this comment

clinique Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

lolodomo commented Oct 27, 2024

lolodomo commented Oct 28, 2024

clinique commented Nov 1, 2024

lo92fr commented Nov 8, 2024

lolodomo commented Nov 9, 2024

clinique commented Nov 9, 2024

lolodomo commented Nov 9, 2024

lolodomo commented Nov 9, 2024

lolodomo commented Nov 9, 2024 • edited Loading

lo92fr commented Nov 9, 2024

lo92fr commented Nov 9, 2024

lo92fr commented Nov 9, 2024

lo92fr commented Nov 9, 2024

lolodomo commented Nov 9, 2024

lolodomo commented Nov 9, 2024

clinique commented Nov 9, 2024

clinique commented Oct 10, 2024 •

edited by jlaur

Loading

lolodomo commented Oct 11, 2024 •

edited

Loading

lolodomo commented Oct 11, 2024 •

edited

Loading

lolodomo commented Oct 11, 2024 •

edited

Loading

lolodomo commented Oct 12, 2024 •

edited

Loading

lolodomo commented Oct 12, 2024 •

edited

Loading

clinique commented Oct 12, 2024 •

edited

Loading

clinique commented Oct 12, 2024 •

edited

Loading

lolodomo commented Oct 13, 2024 •

edited

Loading

clinique Oct 21, 2024 •

edited

Loading

lolodomo commented Nov 9, 2024 •

edited

Loading