Pending Notifications

[tsengia]

Hi Andreas,

I am wondering if pendingNotifications are supported in ACME?
I noticed that it is on your Roadmap, but I'm not sure if that means you are planning to add them, or if you're planning on improving the already existing functionality.

I am doing some testing and getting some odd results, so just wanted to check.

[ankraft]

Tyler,

Sorry, no, pending notifications are not yet implemented. But I have initial ideas, and I plan (so far) to implement them for the next release.

There are two possible situations when a CSE can decide that a notification can become pending: The target is either not reachable (either by getting information from the network or by reachability schedule),or a notification happens outside a notification schedule.

It is very hard to determine whether a target is e2e reachable without access to underlying network functions. That leaves us with schedules, and this is something I plan for the next release. So far ACME does not support the <schedule> resource type, but that is something that is way up on my todo list.

Unfortunately, just reacting on the communication result is not enough here. A notification should be marked as pending before it is sent, not after an attempt failed.

May I ask about your scenario? Do you need pending notifications because your devices are not always reachable?

[tsengia]

I have two AEs that can both update the value of a flex container.
Both AEs create subscriptions to the flex containers on start up, and both create their own polling channels and perform long polling on the channel.
Sometimes, everything works fine and I am able to update the value of a flex container, resulting in the other AE receiving a notification on its polling channel URI.
Other times it seems as if the notification gets lost before making it to the other AE.
Sometimes the PUT request to update a flex container takes a very long time to process, and seems to only be processed once the polling channel times out (which seems as if the polling channel takes all of ACME's CPU resources).

I was asking about pending notifications because I wonder if there is a race condition between the long poll request expiring and the AE creating a new long poll request.

[ankraft]

It could be that there might be something broken, but requests should not just get lost, but they might time out.
One fact is that by default requests to a CSE are blocking. That means that a request only returns when the resource was successfully updated AND all the resulting requests, like notifications, are processed (or something times out).

When a PUT/UPDATE request takes a long time (until a timeout) then this could mean that the notification is perhaps not retrieved from the polling channel. BTW, you need to respond to that notification as well by sending a notification with the response to the polling channel as well! Otherwise the CSE regards that request exchange as unsuccessful.

If you don't want to wait for the PUT/UPDATE then you should consider to use non-blocking requests. They come in two forms:

• nonBlockingRequestSynch : The AE has to actively RETRIEVE a <request> object that contains the result of a previous operation, or
• nonBlockingRequestAsynch : The AE receives a notification with the <request> object.

If you are not interested in the result at all you can perhaps just ignore the <request> objects. They will be removed after the request's timeout.

[tsengia]

Thank you for this information, I had no idea that we had to acknowledge the notifications (which does make sense). I have since fixed my AE's and reverted my ACME installation to its previous state. Thank you!

[ankraft]

Great, that is good news.
In general all communication with and within oneM2M follows the request-response pattern. This sometimes seems to be an overkill but it totally makes sense in a distributed system when requests are routed between the entities. Also, with notifications the responses may transport other information as well.

Regarding the busy waiting: I think at least when waiting for a response from the polling channel after sending a request to an entity, I can replace the waitFor() with an threading.Event object. I will check this.

[tsengia]

I have fixed the symptoms of the issue by doing the following:

• On line 644 of acme/services/RequestManager.py, I removed the call to DateUtils.waitFor(). I instead had it instantly evaluate if there were any requests on the polling channel
• I added a small delay (200ms) between the time when my AE's make their next long poll request after the current one returns/expires

I believe the problem lies in how DateUtils.waitFor() works. Because it calls time.sleep(), it hogs the CPU cycles doing nothing for the allotted time. This keeps ACME from handling other incoming requests.

I realize that my fix is definitely not the correct one (it completely destroys the entire purpose behind long polling).
My fix is more of a hack to get around how ACME is currently written, which is a result of Python's lack of proper multithreading. A proper fix would probably involve using something like select() or poll()

[ankraft]

Thanks for the patch idea. I also changed the waitFor() function in the "development" branch recently, but that doesn't change the sleep() function.

But, what time.sleep() actually does is giving up the process time for a thread, and not using it at all. Perhaps change the sleep time to a bigger value than "0.0.1"? But waitFor() is actually used in a lot of other places and it doesn't use up the CPU there.

select() and poll() are not portable, and don't really work here because at this point there is no network connection that we could use. But perhaps threading.Event() can be used here.

Did you check that you response to notifications received via the pollingChannel?

I will also looking into the CPU problems (tomorrow). Perhaps this happens with multiple parallel PCH requests. There is some necessary internal locking happening.