If you have done work on Exadata, you probably hear many buzz words, like Storage Index, Smart Scan, Offloading, and etc. Many of these features are based on InfiniBand Architecture, which is high-speed interconnect architecture with high throughput and low latency. Talking about InfiniBand, many of us know iDB and RDS on Exadata. But not many people know about SDP. In this post, I discuss more in detail about among iDB, RDS and SDP.
Oracle Exadata uses the Intelligent Database protocol (iDB) to transfer data between Database Node and Storage Cell Node. It is implemented in the database kernel and work as funtion shipping architecture to transparently maps database operations to Exadata operations. iDB can be used to transfer SQL operation from Database Node to Cell node, and get query result back or full data blocks back from Cell Node.
iDB is built on Reliable Datagram Sockets (RDS v3) protocol and runs over InfiniBand ZDP (Zero-loss Zero-copy Datagram Protocol). The objective of ZDP is to eliminate unnessary copying of blocks. RDS is based on Socket API with low overhead, low latency, high bandwidth. Exadata Cell Node can send/receive large transfer using Remote Direct Memory Access (RDMA).
RDMA is a direct memory access from the memory of one computer into another computer without involving either’s operating system. The transfer require no work to be done by CPUs, caches, or context switches, and transfers continue in parallel with other system operations. It is quite useful in massively parallel processing environment.
RDS is highly used on Oracle Exadata. RDS can deliver high available and low overhead of datagrams, which is like UDP but more reliable and zero copy. It accesses to InfiniBand via the Socket API. RDS v3 supports both RDMA read and write and can allow large data transfer up to 8MB. It also supports the control messages for asynchronous operation for submit and completion notifications.
If you want to optimize communications between Oracle Engineered System, like Exadata, Big Data Appliance, and Exlatics, you can use Sockets Direct Protocol (SDP) networking protocol. SDP only deals with stream sockets.
SDP allows high-performance zero-copy data transfers via RDMA network fabrics and uses a standard wire protocol over an RDMA fabric to support stream sockets (SOCK_STREAM). The goal of SDP is to provide an RDMA-accelerated alternative to the TCP protocol on IP, at the same time transparent to the application.
It bypasses the OS resident TCP stack for stream connections between any endpoints on the RDMA fabric. All other socket types (such as datagram, raw, packet, etc.) are supported by the IP stack and operate over standard IP interfaces (i.e., IPoIB on InfiniBand fabrics). The IP stack has no dependency on the SDP stack; however, the SDP stack depends on IP drivers for local IP assignments and for IP address resolution for endpoint identifications.
In a future post, I will discuss about some commands useful to check out Infiniband traffic, RDS and SDP.