Loosing CoAP Client/Server connections after some time

[ksathara]

Hi,
We have our temperature sensors as CoAP client based SED devices and a CoAP server on a FTD collecting temperature data. In most installations this works like a charm. But very few instances, we see client server connection seemingly dropping. As these are on remote sites, we dont have the capability to run wireshark or any other deep diagnosis. All efforts to recreate the problem in our lab has failed.

We are using Nordic nRF52840 chipset handling both radio stack and the application.

Anyone else have seen such behaviors in the field? Any ideas?

Thanks,
Kaushalya

[jwhui]

Can you provide any more details on what "client server connection seemingly dropping" means?

Without more visibility, it's difficult to pinpoint the root cause of your issue.

[ksathara]

Hi Jonathan, Thanks. What we see remotely is that communication between client and server suddenly stops. What we can think is either 1. the connection between the FTD and the SED drops or 2. some issue in our code which causes this (most likely) Since we cant recreate this in the lab, we are closely reviewing our code to see if we can spot any bug on our application side. What I want to know is are there any similar behaviors reported elsewhere in the openthread community, which could mean some issue in the stack. Cheers, Kaushalya

…

------ Original Message ------ From "Jonathan Hui" ***@***.***> To "openthread/openthread" ***@***.***> Cc "Kaushalya Satharasinghe" ***@***.***>; "Author" ***@***.***> Date 2/15/2024 8:41:57 AM Subject Re: [openthread/openthread] Loosing CoAP Client/Server connections after some time (Discussion #9844)

Can you provide any more details on what "client server connection seemingly dropping" means? Without more visibility, it's difficult to pinpoint the root cause of your issue. — Reply to this email directly, view it on GitHub <#9844 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE3WSE3356WAINTFJEQYHPLYTUVSLAVCNFSM6AAAAABDEDS33WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DINZSHAYDG>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ksathara]

Hi Jonathan, We have detected a test system showing this behavior. What has happened is the coap clients (SED) have got disconnected from the server (Leader). When I check child table from the Leader, I can see the SEDs have disconnected. Now our SEDs broadcast their heartbeat every 10 sec. Child timeout is 240sec. We can see the child is operational as far as responding to the push button press and LEDs. What can we do to dig deep into this? Really need some help here. Thanks, Kaushalya

…

On Fri, Feb 16, 2024 at 8:16 AM Kaushalya Satharasinghe ***@***.***> wrote: Hi Jonathan, Thanks. What we see remotely is that communication between client and server suddenly stops. What we can think is either 1. the connection between the FTD and the SED drops or 2. some issue in our code which causes this (most likely) Since we cant recreate this in the lab, we are closely reviewing our code to see if we can spot any bug on our application side. What I want to know is are there any similar behaviors reported elsewhere in the openthread community, which could mean some issue in the stack. Cheers, Kaushalya ------ Original Message ------ From "Jonathan Hui" ***@***.***> To "openthread/openthread" ***@***.***> Cc "Kaushalya Satharasinghe" ***@***.***>; "Author" < ***@***.***> Date 2/15/2024 8:41:57 AM Subject Re: [openthread/openthread] Loosing CoAP Client/Server connections after some time (Discussion #9844) Can you provide any more details on what "client server connection seemingly dropping" means? Without more visibility, it's difficult to pinpoint the root cause of your issue. — Reply to this email directly, view it on GitHub <#9844 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE3WSE3356WAINTFJEQYHPLYTUVSLAVCNFSM6AAAAABDEDS33WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DINZSHAYDG> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jwhui]

Any logs and packet sniffer traces will help provide more visibility to help diagnose.

[ksathara]

Hi Jonathan, Thanks. I have attached a wireshark capture during this time. There were 6 SEDs and one FTD. Out of the 6 SEDs, 4 have been disconnected from the child table. It would be great if you can see any cause from the wireshark sniff. Thanks again, Kaushalya

…

On Tue, Mar 5, 2024 at 8:16 AM Jonathan Hui ***@***.***> wrote: Any logs and packet sniffer traces will help provide more visibility to help diagnose. — Reply to this email directly, view it on GitHub <#9844 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE3WSE6L42FSHZWHPBQSLRDYWTQDBAVCNFSM6AAAAABDEDS33WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMNZSGA4DE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ksathara]

Network Key : 00112233445566778899aabbccdd0011

[jwhui]

Unfortunately, Wireshark does not seem to be able to decrypt frames with that key.

It would also help if you could provide more information about the devices that appear in the Wireshark trace. I see different PAN IDs and Extended Addresses in use.

[ksathara]

Sorry Jonathan, the network key should be 00112233445566778899aabbccdd0001. My apologies.

[ksathara]

There should be only one PAN ID, 0xabcd. What other PAN IDs you detect? what time?

There are six SEDs. Following are the Ext addr and MAC of each.

Ext addr | MAC
da84f912dcd67dba | f4ce3654db20b48e
f6c777d4de877c1c | f4ce3602648ac0d2
aee13cdccab6e500 | f4ce36957ae6e111
0ae42c135ae5e66b | f4ce360c6ae11ce3
8635b00b3dc73212 | f4ce3651d2e8bc7b
76fc42d55c5b0f5f | f4ce364c1cf6bebe

Thanks,
Kaushalya

[jwhui]

In the sniffer trace from https://github.com/orgs/openthread/discussions/9844#discussioncomment-8686654, I could not find any frames that have an Extended Source Address matching any of the addresses you provided in the table above.