Synchronization error on noBufferedMessages

[fpagliughi]

Got a bug report from @elzbieta-nowak-kubat on the Paho C++ repo where publishing from two threads cause the application to hang:

I have investigated it a bit and it seems that the problem is caused by wrong synchronization of noBufferedMessages member of MQTTAsync on pahoc side. In can happen that two threads can simultaneously accesses the check on this member, both commands are added to the list and increase this number, so in effect in MQTTAsync_add function processing enters condition

/* delete oldest message if buffer is full.  We wouldn't be here if delete newest was in operation */
if (command->client->createOptions && (command->client->noBufferedMessages >= command->client->createOptions->maxBufferedMessages))

where it should never enter for our create options (delete_oldest_messages(false)) and then ListDetach is called which causes removing the first publish command from the processing. So on pahocpp side wait corresponding to the publish never gets released.

eclipse-paho/paho.mqtt.cpp#499

[icraggs]

I don't think it was exactly at the place pinpointed, but a similar scenario. I would say that setting the maximum no of buffered messages to 1 isn't useful, but serves in this case to highlight an issue.