FI_REMOTE_CQ_DATA flag behavior with for fi_msg, fi_tagged, and fi_rma

[colliej]

I believe that libfabric should add strong words to the documentation with regard to the FI_REMOTE_CQ_DATA flag. If a provider supports passing remote CQ data over its transport then any application calls to fi_senddata, fi_injectdata, fi_tsenddata, fi_tinjectdata, fi_writedata, or fi_inject_writedata along with fi_sendmsg or fi_writemsg when the flag FI_REMOTE_CQ_DATA is set should send CQ data.

Currently the man pages indicate that this flag also applies to the fi_senddata, fi_tsenddata, and fi_writedata calls besides the fi_sendmsg, fi_tsendmsg, and fi_writemsg calls where it is needed. I believe this flag should not apply to fi_senddata, fi_injectdata, fi_tsenddata, fi_tinjectdata, fi_writedata, or fi_inject_writedata (note the inject calls are currently absent from the flag definition anyway). If an application does not want to use CQ data (such as not specifying the flag as part of the op_flags in the fi_getinfo hints) or a provider is known to not support CQ data then it should not make those calls as adding conditional code in a provider to modify the behavior of the call goes against the desire for libfabric to be a high performance communication API.

The man pages should be updated to reflect that an application should not call fi_senddata, fi_injectdata, fi_tsenddata, fi_tinjectdata, fi_writedata, or fi_inject_writedata if application does not intend to actually send CQ data regardless if the provider is known to support sending CQ data or not and that FI_REMOTE_CQ_DATA flag only applies to fi_sendmsg, fi_tsendmsg, and fi_writemsg.

[j-xiong]

This has been discussed in today's (11/28/2023) OFIWG meeting.

Most of the providers today always send data with the fi_*data() calls. I think that's actually how it should be. The confusion comes from the fi_msg.3, fi_tagged.3 and fi_rma.3 man pages. For example, in fi_msg.3:

FI_REMOTE_CQ_DATA
Applies to fi_sendmsg and fi_senddata. Indicates that remote CQ data is available and should be sent as part
of the request. See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.

The term fi_senddata should be removed from the description.

[j-xiong]

@colliej See #9617