当前位置：网站首页>RDMA Verbs API

RDMA Verbs API

2022-06-09 11:16:00 【raindayinrain】

5.1.Protection Domain Operations

5.1.1.rdma_reg_msgs

Template:

struct ibv_mr *rdma_reg_msgs(struct rdma_cm_id *id, void *addr, size_t length)

Input Parameters:

A reference to the communication identifier where the message buffer(s) will be used

addr

The address of the memory buffer(s) to register

length

The total length of the memory to register

Output Parameters:

ibv_mr

A reference to an ibv_mr struct of the registered memory region

Return Value:

A reference to the registered memory region on success or NULL on failure

Description:
        rdma_reg_msgs registers an array of memory buffers for sending or receiving messages or for RDMA operations. The registered memory buffers may then be posted to an rdma_cm_id using rdma_post_send or rdma_post_recv. They may also be specified as the target of an RDMA read operation or the source of an RDMA write request.
        The memory buffers are registered with the protection domain associated with the rdma_cm_id. The start of the data buffer array is specified through the addr parameter and the total size of the array is given by the length.
        All data buffers must be registered before being posted as a work request. They must be deregistered by calling rdma_dereg_mr.

5.1.2.rdma_reg_read

Template:

struct ibv_mr * rdma_reg_read(struct rdma_cm_id *id, void *addr, size_t length)

Input Parameters:

A reference to the communication identifier where the message buffer(s) will be used

addr

The address of the memory buffer(s) to register

length

The total length of the memory to register

Output Parameters:

ibv_mr

A reference to an ibv_mr struct of the registered memory region

Return Value:

A reference to the registered memory region on success or NULL on failure. If an error occurs, errno will be set to indicate the failure reason.

Description:
rdma_reg_read Registers a memory buffer that will be accessed by a remote RDMA read operation. Memory buffers registered using rdma_reg_read may be targeted in an RDMA read request, allowing the buffer to be specified on the remote side of an RDMA connection as the remote_addr of rdma_post_read, or similar call.
rdma_reg_read is used to register a data buffer that will be the target of an RDMA read operation on a queue pair associated with an rdma_cm_id. The memory buffer is registered with the protection domain associated with the identifier. The start of the data buffer is specified through the addr parameter, and the total size of the buffer is given by length.
All data buffers must be registered before being posted as work requests. Users must deregister all registered memory by calling the rdma_dereg_mr.
See Also
rdma_cm(7), rdma_create_id(3), rdma_create_ep(3), rdma_reg_msgs(3), rdma_reg_write(3), ibv_reg_mr(3), ibv_dereg_mr(3), rdma_post_read(3)

5.1.3.rdma_reg_write

Template:

struct ibv_mr *rdma_reg_write(struct rdma_cm_id *id, void *addr, size_t length)

Input Parameters:

A reference to the communication identifier where the message buffer(s) will be used

addr

The address of the memory buffer(s) to register

length

The total length of the memory to register

Output Parameters:

ibv_mr

A reference to an ibv_mr struct of the registered memory region

Return Value:

A reference to the registered memory region on success or NULL on failure. If an error occurs, errno will be set to indicate the failure reason.

Description:
        rdma_reg_write registers a memory buffer which will be accessed by a remote RDMA write operation. Memory buffers registered using this routine may be targeted in an RDMA write request, allowing the buffer to be specified on the remote side of an RDMA connection as the remote_addr of an rdma_post_write or similar call.
        The memory buffer is registered with the protection domain associated with the rdma_cm_id. The start of the data buffer is specified through the addr parameter, and the total size of the buffer is given by the length.
        All data buffers must be registered before being posted as work requests. Users must deregister all registered memory by calling the rdma_dereg_mr.
See Also
        rdma_cm(7), rdma_create_id(3), rdma_create_ep(3), rdma_reg_msgs(3), rdma_reg_read(3), ibv_reg_mr(3), ibv_dereg_mr(3), rdma_post_write(3)

5.1.4.rdma_dereg_mr

Template:

int rdma_dereg_mr(struct ibv_mr *mr)

Input Parameters:

A reference to a registered memory buffer

Output Parameters:

None

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description:
rdma_dereg_mr deregisters a memory buffer which has been registered for RDMA or message operations. This routine must be called for all registered memory associated with a given rdma_cm_id before destroying the rdma_cm_id.

5.1.5.rdma_create_srq

Template:

int rdma_create_srq(struct rdma_cm_id *id, struct ibv_pd *pd, struct ibv_srq_init_attr *attr)

Input Parameters:

The RDMA communication identifier

Optional protection domain for the shared request queue (SRQ)

attr

Initial SRQ attributes

Output Parameters:

attr

The actual capabilities and properties of the created SRQ are returned through this structure

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description:
rdma_create_srq allocates a shared request queue associated with the rdma_cm_id, id. The id must be bound to a local RMDA device before calling this routine. If the protection domain, pd,is provided, it must be for that same device. After being allocated, the SRQ will be ready to handle posting of receives. If a pd is NULL, then the rdma_cm_id will be created using a default protection domain. One default protection domain is allocated per RDMA device. The initial SRQ attributes are specified by the attr parameter.
If a completion queue, CQ, is not specified for the XRC SRQ, then a CQ will be allocated by the rdma_cm for the SRQ, along with corresponding completion channels. Completion channels and CQ data created by the rdma_cm are exposed to the user through the rdma_cm_id structure. The actual capabilities and properties of the created SRQ will be returned to the user through the attr parameter.
An rdma_cm_id may only be associated with a single SRQ.

5.1.6.rdma_destroy_srq

Template:

void rdma_destroy_srq(struct rdma_cm_id *id)

Input Parameters:

The RDMA communication identifier whose associated SRQ we wish to destroy.

Output Parameters:

None

Return Value:

none

Description:
rdma_destroy_srq destroys an SRQ allocated on the rdma_cm_id, id. Any SRQ associated with an rdma_cm_id must be destroyed before destroying the rdma_cm_id, id.

5.2.Active Queue Pair Operations

5.2.1.rdma_post_recvv

Template:

int rdma_post_recvv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge)

Input Parameters:

A reference to the communication identifier where the message buffer(s) will be posted

context

A user-defined context associated with the request

sgl

A scatter-gather list of memory buffers posted as a single request

nsge

The number of scatter-gather entries in the sgl array

Output Parameters:

None

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description:
        rdma_post_recvv posts a single work request to the receive queue of the queue pair associated with the rdma_cm_id, id. The posted buffers will be queued to receive an incoming message sent by the remote peer.
        Please note that this routine supports multiple scatter-gather entries. The user is responsible for ensuring that the receive is posted, and the total buffer space is large enough to contain all sent data before the peer posts the corresponding send message.         The message buffers must have been registered before being posted, and the buffers must remain registered until the receive completes.
        Messages may be posted to an rdma_cm_id only after a queue pair has been associated with it. A queue pair is bound to an rdma_cm_id after calling rdma_create_ep or rdma_create_qp, if the rdma_cm_id is allocated using rdma_create_id.
        The user-defined context associated with the receive request will be returned to the user through the work completion work request identifier (wr_id) field.

5.2.2.rdma_post_sendv

Template:

int rdma_post_sendv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge, int

flags)

Input Parameters:

A reference to the communication identifier where the message buffer will be posted

context

A user-defined context associated with the request

sgl

A scatter-gather list of memory buffers posted as a single request

nsge

The number of scatter-gather entries in the sgl array

flags

Optional flags used to control the send operation

Output Parameters:

None

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description:
        rdma_post_sendv posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the posted buffers will be sent to the remote peer of the connection.
        The user is responsible for ensuring that the remote peer has queued a receive request before issuing the send operations. Also, unless the send request is using inline data, the message buffers must already have been registered before being posted. The buffers must remain registered until the send completes.
        This routine supports multiple scatter-gather entries. Send operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
        The user-defined context associated with the send request will be returned to the user through the work completion work request identifier (wr_id) field.

5.2.3.rdma_post_readv

Template:

int rdma_post_readv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge, int

flags, uint64_t remote_addr, uint32_t rkey)

Input Parameters:

A reference to the communication identifier where the request will be posted

context

A user-defined context associated with the request

sgl

A scatter-gather list of the destination buffers of the read

nsge

The number of scatter-gather entries in the sgl array

flags

Optional flags used to control the read operation

remote_addr

The address of the remote registered memory to read from

rkey

The registered memory key associated with the remote address

Output Parameters:

None

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description:
        rdma_post_readv posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the remote memory region at remote_addr will be read into the local data buffers given in the sgl array.
        The user must ensure that both the remote and local data buffers have been registered before the read is issued. The buffers must remain registered until the read completes. Read operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
        The user-defined context associated with the read request will be returned to the user through the work completion work request identifier (wr_id) field.

5.2.4.rdma_post_writev

Template:

int rdma_post_writev(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge, int

flags, uint64_t remote_addr, uint32_t rkey)

Input Parameters:

A reference to the communication identifier where the request will be posted

context

A user-defined context associated with the request

sgl

A scatter-gather list of the source buffers of the write

nsge

The number of scatter-gather entries in the sgl array

flags

Optional flags used to control the write operation

remote_addr

The address of the remote registered memory to write into

rkey

The registered memory key associated with the remote address

Output Parameters:

None

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description:
        rdma_post_writev posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the local data buffers in the sgl array will be written to the remote memory region at remote_addr.
        Unless inline data is specified, the local data buffers must have been registered before the write is issued, and the buffers must remain registered until the write completes. The remote buffers must always be registered.
        Write operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
        The user-defined context associated with the write request will be returned to the user through the work completion work request identifier (wr_id) field.

5.2.5.rdma_post_recv

Template:

int rdma_post_recv(struct rdma_cm_id *id, void *context, void *addr, size_t length, struct

ibv_mr *mr)

Input Parameters:

A reference to the communication identifier where the message buffer will be posted

context

A user-defined context associated with the request

addr

The address of the memory buffer to post

length

The length of the memory buffer

A registered memory region associated with the posted buffer

Output Parameters:

None

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description:
        rdma_post_recv posts a work request to the receive queue of the queue pair associated with the rdma_cm_id, id. The posted buffer will be queued to receive an incoming message sent by the remote peer.
        The user is responsible for ensuring that receive buffer is posted and is large enough to contain all sent data before the peer posts the corresponding send message. The buffer must have already been registered before being posted, with the mr parameter referencing the registration. The buffer must remain registered until the receive completes.
        Messages may be posted to an rdma_cm_id only after a queue pair has been associated with it. A queue pair is bound to an rdma_cm_id after calling rdma_create_ep or rdma_create_qp, if the rdma_cm_id is allocated using rdma_create_id.
        The user-defined context associated with the receive request will be returned to the user through the work completion request identifier (wr_id) field.
        Please note that this is a simple receive call. There are no scatter-gather lists involved here.

5.2.6.rdma_post_send

Template:

int rdma_post_send(struct rdma_cm_id *id, void *context, void *addr, size_t length, struct

ibv_mr *mr, int flags)

Input Parameters:

id A reference to the communication identifier where the

message buffer will be posted

context A user-defined context associated with the request

addr The address of the memory buffer to post

length The length of the memory buffer

mr Optional registered memory region associated with the posted

buffer

flags Optional flags used to control the send operation

Output Parameters:

None

Return Value:

0 on success, -1 on error. If the call fails, errno will be set to indicate

the reason for the failure

Description:
rdma_post_send posts a work request to the send queue of the queue pair associated with the
rdma_cm_id, id. The contents of the posted buffer will be sent to the remote peer of the connec
tion.
The user is responsible for ensuring that the remote peer has queued a receive request before
issuing the send operations. Also, unless the send request is using inline data, the message buffer
must already have been registered before being posted, with the mr parameter referencing the
registration. The buffer must remain registered until the send completes.
Send operations may not be posted to an rdma_cm_id or the corresponding queue pair until a
connection has been established.
The user-defined context associated with the send request will be returned to the user through the
work completion work request identifier (wr_id) field.