Tribblix: manual page: dat_srq_post

DAT_SRQ_POST_RECV(3DAT) Direct Access Transport Library Functions

NAME

dat_srq_post_recv - add receive buffers to shared receive queue

SYNOPSIS

cc [ flag... ] file... -ldat [ library... ]
#include <dat/udat.h>

DAT_RETURN
dat_srq_post_recv (
IN DAT_SRQ_HANDLE srq_handle,
IN DAT_COUNT num_segments,
IN DAT_LMR_TRIPLET *local_iov,
IN DAT_DTO_COOKIE user_cookie
)

PARAMETERS

srq_handle
A handle for an instance of the SRQ.

num_segments
The number of lmr_triplets in local_iov. Can be 0 for
receiving a zero-size message.

local_iov
An I/O Vector that specifies the local buffer to be
filled. Can be NULL for receiving a zero-size
message.

user_cookie
A user-provided cookie that is returned to the
Consumer at the completion of the Receive DTO. Can be
NULL.

DESCRIPTION

The dat_srq_post_recv() function posts the receive buffer that can be
used for the incoming message into the local_iov by any connected EP
that uses SRQ.

The num_segments argument specifies the number of segments in the
local_iov. The local_iov segments are filled in the I/O Vector order
until the whole message is received. This ensures that all the front
segments of the local_iov I/O Vector are completely filled, only one
segment is partially filled, if needed, and all segments that follow
it are not filled at all. The actual order of segment fillings is
left to the implementation.

The user_cookie argument allows Consumers to have unique identifiers
for each DTO. These identifiers are completely under user control and
are opaque to the Provider. There is no requirement on the Consumer
that the value user_cookie should be unique for each DTO. The
user_cookie is returned to the Consumer in the Completion event for
the posted Receive.

The completion of the posted Receive is reported to the Consumer
asynchronously through a DTO Completion event based on the
configuration of the EP that dequeues the posted buffer and the
specified completion_flags value for Solicited Wait for the matching
Send. If EP Recv Completion Flag is DAT_COMPLETION_UNSIGNALLED_FLAG,
which is the default value for SRQ EP, then all posted Recvs will
generate completions with Signal Notifications.

A Consumer should not modify the local_iov or its content until the
DTO is completed. When a Consumer does not adhere to this rule, the
behavior of the Provider and the underlying Transport is not defined.
Providers that allow Consumers to get ownership of the local_iov but
not the memory it specified back after the dat_srq_post_recv()
returns should document this behavior and also specify its support in
Provider attributes. This behavior allows Consumer full control of
the local_iov content after dat_srq_post_recv() returns. Because this
behavior is not guaranteed by all Providers, portable Consumers shall
not rely on this behavior. Consumers shall not rely on the Provider
copying local_iov information.

The DAT_SUCCESS return of the dat_srq_post_recv() is at least the
equivalent of posting a Receive operation directly by native
Transport. Providers shall avoid resource allocation as part of
dat_srq_post_recv() to ensure that this operation is nonblocking.

The completion of the Receive posted to the SRQ is equivalent to what
happened to the Receive posted to the Endpoint for the Endpoint that
dequeued the Receive buffer from the Shared Receive queue.

The posted Recv DTO will complete with signal, equivalently to the
completion of Recv posted directly to the Endpoint that dequeued the
Recv buffer from SRQ with DAT_COMPLETION_UNSIGNALLED_FLAG value not
set for it.

The posted Recv DTOs will complete in the order of Send postings to
the other endpoint of each connection whose local EP uses SRQ. There
is no ordering among different connections regardless if they share
SRQ and recv_evd or not.

If the reported status of the Completion DTO event corresponding to
the posted RDMA Read DTO is not DAT_DTO_SUCCESS, the content of the
local_iov is not defined and the transfered_length in the DTO
Completion event is not defined.

The operation is valid for all states of the Shared Receive Queue.

The dat_srq_post_recv() function is asynchronous, nonblocking, and
its thread safety is Provider-dependent.

RETURN VALUES

DAT_SUCCESS
The operation was successful.

DAT_INVALID_HANDLE
The srq_handle argument is an invalid
DAT handle.

DAT_INSUFFICIENT_RESOURCES
The operation failed due to resource
limitations.

DAT_INVALID_PARAMETER
Invalid parameter. For example, one of
the IOV segments pointed to a memory
outside its LMR.

DAT_PROTECTION_VIOLATION
Protection violation for local or
remote memory access.

Protection Zone mismatch between an LMR
of one of the local_iov segments and
the SRQ.

DAT_PRIVILEGES_VIOLATION
Privileges violation for local or
remote memory access. One of the LMRs
used in local_iov was either invalid or
did not have the local write
privileges.

USAGE

For the best Recv operation performance, the Consumer should align
each buffer segment of local_iov to the Optimal Buffer Alignment
attribute of the Provider. For portable applications, the Consumer
should align each buffer segment of local_iov to the
DAT_OPTIMAL_ALIGNMENT.

Since any of the Endpoints that use the SRQ can dequeue the posted
buffer from SRQ, Consumers should post a buffer large enough to
handle incoming message on any of these Endpoint connections.

The buffer posted to SRQ does not have a DTO completion flag value.
Posting Recv buffer to SRQ is semantically equivalent to posting to
EP with DAT_COMPLETION_UNSIGNALLED_FLAG is not set. The configuration
of the Recv Completion flag of an Endpoint that dequeues the posted
buffer defines how DTO completion is generated. If the Endpoint Recv
Completion flag is DAT_COMPLETION_SOLICITED_WAIT_FLAG then matching
Send DTO completion flag value for Solicited Wait determines if the
completion will be Signalled or not. If the Endpoint Recv Completion
flag is not DAT_COMPLETION_SOLICITED_WAIT_FLAG, the posted Recv
completion will be generated with Signal. If the Endpoint Recv
Completion flag is DAT_COMPLETION_EVD_THRESHOLD_FLAG, the posted Recv
completion will be generated with Signal and dat_evd_wait threshold
value controls if the waiter will be unblocked or not.

Only the Endpoint that is in Connected or Disconnect Pending states
can dequeue buffers from SRQ. When an Endpoint is transitioned into
Disconnected state, all the buffers that it dequeued from SRQ are
queued on the Endpoint recv_evd. All the buffers that the Endpoint
has not completed by the time of transition into Disconnected state
and that have not completed message reception will be flushed.

ATTRIBUTES