Tribblix: manual page: dat_ep_create_with

DAT_EP_CREATE_WITH_SRQ(3DAT) Direct Access Transport Library Functions

NAME

dat_ep_create_with_srq - create an instance of End Point with Shared
Receive Queue

SYNOPSIS

cc [ flag... ] file... -ldat [ library... ]
#include <dat/udat.h>

DAT_RETURN
dat_ep_create_with_srq (
IN DAT_IA_HANDLE ia_handle,
IN DAT_PZ_HANDLE pz_handle,
IN DAT_EVD_HANDLE recv_evd_handle,
IN DAT_EVD_HANDLE request_evd_handle,
IN DAT_EVD_HANDLE connect_evd_handle,
IN DAT_SRQ_HANDLE srq_handle,
IN DAT_EP_ATTR *ep_attributes,
OUT DAT_EP_HANDLE *ep_handle
)

PARAMETERS

ia_handle
Handle for an open instance of the IA to which
the created Endpoint belongs.

pz_handle
Handle for an instance of the Protection Zone.

recv_evd_handle
Handle for the Event Dispatcher where events
for completions of incoming (receive) DTOs are
reported. DAT_HANDLE_NULL specifies that the
Consumer is not interested in events for
completions of receives.

request_evd_handle
Handle for the Event Dispatcher where events
for completions of outgoing (Send, RDMA Write,
RDMA Read, and RMR Bind) DTOs are reported.
DAT_HANDLE_NULL specifies that the Consumer is
not interested in events for completions of
requests.

connect_evd_handle
Handle for the Event Dispatcher where
Connection events are reported.
DAT_HANDLE_NULL specifies that the Consumer is
not interested in connection events for now.

srq_handle
Handle for an instance of the Shared Receive
Queue.

ep_attributes
Pointer to a structure that contains Consumer-
requested Endpoint attributes. Cannot be NULL.

ep_handle
Handle for the created instance of an Endpoint.

DESCRIPTION

The dat_ep_create_with_srq() function creates an instance of an
Endpoint that is using SRQ for Recv buffers is provided to the
Consumer as ep_handle. The value of ep_handle is not defined if the
DAT_RETURN is not DAT_SUCCESS.

The Endpoint is created in the Unconnected state.

Protection Zone pz_handle allows Consumers to control what local
memory the Endpoint can access for DTOs except Recv and what memory
remote RDMA operations can access over the connection of a created
Endpoint. Only memory referred to by LMRs and RMRs that match the
Endpoint Protection Zone can be accessed by the Endpoint. The Recv
DTO buffers PZ must match the SRQ PZ. The SRQ PZ might or might not
be the same as the EP one. Check Provider attribute for the support
of different PZs between SRQ and its EPs.

The recv_evd_handle and request_evd_handle arguments are Event
Dispatcher instances where the Consumer collects completion
notifications of DTOs. Completions of Receive DTOs are reported in
recv_evd_handle Event Dispatcher, and completions of Send, RDMA Read,
and RDMA Write DTOs are reported in request_evd_handle Event
Dispatcher. All completion notifications of RMR bindings are reported
to a Consumer in request_evd_handle Event Dispatcher.

All Connection events for the connected Endpoint are reported to the
Consumer through connect_evd_handle Event Dispatcher.

Shared Receive Queue srq_handle specifies where the EP will dequeue
Recv DTO buffers.

The created EP can be reset. The relationship between SRQ and EP is
not effected by dat_ep_reset(3DAT).

SRQ can not be disassociated or replaced from created EP. The only
way to disassociate SRQ from EP is to destroy EP.

Receive buffers cannot be posted to the created Endpoint. Receive
buffers must be posted to the SRQ to be used for the created
Endpoint.

The ep_attributes parameter specifies the initial attributes of the
created Endpoint. Consumer can not specify NULL for ep_attributes but
can specify values only for the parameters needed and default for the
rest.

For max_request_dtos and max_request_iov, the created Endpoint will
have at least the Consumer requested values but might have larger
values. Consumer can query the created Endpoint to find out the
actual values for these attributes. Created Endpoint has the exact
Consumer requested values for max_recv_dtos, max_message_size,
max_rdma_size, max_rdma_read_in, and max_rdma_read_out. For all other
attributes, except max_recv_iov that is ignored, the created Endpoint
has the exact values requested by Consumer. If Provider cannot
satisfy the Consumer requested attribute values the operation fails.

RETURN VALUES

DAT_SUCCESS
The operation was successful.

DAT_INSUFFICIENT_RESOURCES
The operation failed due to resource
limitations.

DAT_INVALID_HANDLE
Invalid DAT handle.

DAT_INVALID_PARAMETER
Invalid parameter. One of the requested
EP parameters or attributes was invalid
or a combination of attributes or
parameters is invalid. For example,
pz_handle specified does not match the
one for SRQ or the requested maximum
RDMA Read IOV exceeds IA capabilities.

DAT_MODEL_NOT_SUPPORTED
The requested Provider Model was not
supported.

USAGE

The Consumer creates an Endpoint prior to the establishment of a
connection. The created Endpoint is in DAT_EP_STATE_UNCONNECTED.
Consumers can do the following:

1. Request a connection on the Endpoint through
dat_ep_connect(3DAT) or dat_ep_dup_connect(3DAT) for the
active side of the connection model.

2. Associate the Endpoint with the Pending Connection Request
that does not have an associated local Endpoint for
accepting the Pending Connection Request for the
passive/server side of the connection model.

3. Create a Reserved Service Point with the Endpoint for the
passive/server side of the connection model. Upon arrival
of a Connection Request on the Service Point, the Consumer
accepts the Pending Connection Request that has the
Endpoint associated with it.

The Consumer cannot specify a request_evd_handle (recv_evd_handle)
with Request Completion Flags (Recv Completion Flags) that do not
match the other Endpoint Completion Flags for the DTO/RMR completion
streams that use the same EVD. If request_evd_handle
(recv_evd_handle) is used for request (recv) completions of an
Endpoint whose associated Request (Recv) Completion Flag attribute is
DAT_COMPLETION_UNSIGNALLED_FLAG, the Request Completion Flags and
Recv Completion Flags for all Endpoint completion streams that use
the EVD must specify the same. By definition, completions of all Recv
DTO posted to SRQ complete with Signal. Analogously, if
recv_evd_handle is used for recv completions of an Endpoint whose
associated Recv Completion Flag attribute is
DAT_COMPLETION_SOLICITED_WAIT, the Recv Completion Flags for all
Endpoint Recv completion streams that use the same EVD must specify
the same Recv Completion Flags attribute value and the EVD cannot be
used for any other event stream types. If recv_evd_handle is used for
Recv completions of an Endpoint that uses SRQ and whose Recv
Completion Flag attribute is DAT_COMPLETION_EVD_THRESHOLD then all
Endpoint DTO completion streams (request and/or recv completion
streams) that use that recv_evd_handle must specify
DAT_COMPLETION_EVD_THRESHOLD. Other event stream types can also use
the same EVD.

Consumers might want to use DAT_COMPLETION_UNSIGNALLED_FLAG for
Request and/or Recv completions when they control locally with posted
DTO/RMR completion flag (not needed for Recv posted to SRQ) whether
posted DTO/RMR completes with Signal or not. Consumers might want to
use DAT_COMPLETION_SOLICITED_WAIT for Recv completions when the
remote sender side control whether posted Recv competes with Signal
or not or not. uDAPL Consumers might want to use
DAT_COMPLETION_EVD_THRESHOLD for Request and/or Recv completions when
they control waiter unblocking with the threshold parameter of the
dat_evd_wait(3DAT).

Some Providers might restrict whether multiple EPs that share a SRQ
can have different Protection Zones. Check the
srq_ep_pz_difference_support Provider attribute for it.

Consumers might want to have a different PZ between EP and SRQ. This
allows incoming RDMA operations to be specific to this EP PZ and not
the same for all EPs that share SRQ. This is critical for servers
that supports multiple independent clients.

The Provider is strongly encouraged to create an EP that is ready to
be connected. Any effects of previous connections or connection
establishment attempts on the underlying Transport-specific Endpoint
to which the DAT Endpoint is mapped to should be hidden from the
Consumer. The methods described below are examples:

o The Provider does not create an underlying Transport
Endpoint until the Consumer is connecting the Endpoint or
accepting a connection request on it. This allows the
Provider to accumulate Consumer requests for attribute
settings even for attributes that the underlying transport
does not allow to change after the Transport Endpoint is
created.

o The Provider creates the underlying Transport Endpoint or
chooses one from a pool of Provider-controlled Transport
Endpoints when the Consumer creates the Endpoint. The
Provider chooses the Transport Endpoint that is free from
any underlying internal attributes that might prevent the
Endpoint from being connected. For IB and IP, that means
that the Endpoint is not in the TimeWait state. Changing
of some of the Endpoint attributes becomes hard and might
potentially require mapping the Endpoint to another
underlying Transport Endpoint that might not be feasible
for all transports.

o The Provider allocates a Transport-specific Endpoint
without worrying about impact on it from previous
connections or connection establishment attempts. Hide
the Transport-specific TimeWait state or CM timeout of the
underlying transport Endpoint within dat_ep_connect(3DAT),
dat_ep_dup_connect(3DAT), or dat_cr_accept(3DAT). On the
Active side of the connection establishment, if the
remnants of a previous connection for Transport-specific
Endpoint can be hidden within the Timeout parameter, do
so. If not, generating
DAT_CONNECTION_EVENT_NON_PEER_REJECTED is an option. For
the Passive side, generating a
DAT_CONNECTION_COMPLETION_ERROR event locally, while
sending a non-peer-reject message to the active side, is a
way of handling it.

Any transitions of an Endpoint into an Unconnected state can be
handled similarly. One transition from a Disconnected to an
Unconnected state is a special case.

For dat_ep_reset(3DAT), the Provider can hide any remnants of the
previous connection or failed connection establishment in the
operation itself. Because the operation is synchronous, the Provider
can block in it until the TimeWait state effect of the previous
connection or connection setup is expired, or until the Connection
Manager timeout of an unsuccessful connection establishment attempt
is expired. Alternatively, the Provider can create a new Endpoint for
the Consumer that uses the same handle.

DAT Providers are required not to change any Consumer-specified
Endpoint attributes during connection establishment. If the Consumer
does not specify an attribute, the Provider can set it to its own
default. Some EP attributes, like outstanding RDMA Read incoming or
outgoing, if not set up by the Consumer, can be changed by Providers
to establish connection. It is recommended that the Provider pick the
default for outstanding RDMA Read attributes as 0 if the Consumer has
not specified them. This ensures that connection establishment does
not fail due to insufficient outstanding RDMA Read resources, which
is a requirement for the Provider.

The Provider is not required to check for a mismatch between the
maximum RDMA Read IOV and maximum RDMA Read outgoing attributes, but
is allowed to do so. In the latter case it is allowed to return
DAT_INVALID_PARAMETER when a mismatch is detected. Provider must
allocate resources to satisfy the combination of these two EP
attributes for local RDMA Read DTOs.

ATTRIBUTES