ExpEther

ExpEther is a System Hardware Virtualization Technology that expands standard PCI Express beyond 1 km having thousands of roots and endpoint devices together on a single network connected through the standard Ethernet. Abundant commodities of PCI Express-based software and hardware can be utilized without any modification. It also provides software-defined re-configurability to make a disaggregated computing system with device-level.

Overview

ExpEther enables a unique "PCI Express switch over Ethernet" architecture that distributes functional blocks of a PCI Express switch over Ethernet, maintaining a logical equivalency with the standard PCI Express switch.

By leveraging Ethernet connectivity, ExpEther can expand the number of port-count to more than 1000, and the working distances to over 1000m. Ethernet is transparent to the OSes, driver, software, thus ExpEther can work with enormous commodity standard OS, drivers, PCIe devices, Ethernet switches without any modification.

It can also control the logical connection of PCI Express among a root and endpoint devices remotely by a management software. I/O devices can be attached/detached with software-defined manner with following PCI Express standard Hot-Plug sequence.

By adding PCI Express device with a certain function, the computing system can be scale-up for the function required for the user application run on the system.

From the data communication point of view, ExpEther realizes reliable communication over standard Ethernet with original congestion control and multi-path communication mechanism. Then DMA (Direct Memory Access) can be directly executed over Ethernet. It can transfer a large amount of data with low latency and high throughput without software stacks of TCP/IP.

Specification

The following implementation forms are provided as products.

implementation	PCIe Logical Standard	Maximum Lane Speed	The number of Lanes	Ethernet X1	Ethernet X2	Multi Route Share of SR-IOV device
ASIC	PCIe　Gen2	2Gbps	1	1Gbps	2Gbps	NA
FPGA	PCIe　Gen2	2Gbps	8	10Gbps	20Gbps	NA
FPGA	PCIe　Gen2	2Gbps	8	10Gbps	20Gbps	Available
FPGA	PCIe　Gen3	8Gbps	8	40Gbps	80Gbps	NA

History

ExpEther was firstly announced in December 2006 by a research and development group of NEC as a computer scale-up technology that virtualizes PCI Express over Ethernet.

In November 2008, the ExpEther Consortium has been established aiming at promotional activities to expand the market of ExpEther for both academic and industrial area, bringing researchers, developers, resellers of computing system software, device, chip together. (Chair: Professor Hideharu Amano of Keio University. Member: 26 organizations are participated (as of July 2017).)

In December 2008, one of the implementation of the ExpEther technology as a PCI Add-In card was passed the PCI-SIG (PCI Express standardizing body) compliant test and interoperability tests, and then posted on the integrator’s list. After that Iventure (acquired by Synopsy) released an evaluation kit.

In 2009, Nethra Imaging announced an ASIC development of Expether.

In 2009, the QCN method proposed by Professor Balaji Prabhakar of Stanford University for standardization of IEEE 802.1Qau was proved by the experiment using the ExpEther card.

In 2009, interview articles on ExpEther were published in Nkkei PC (1/21/2009 p17) and Nikkei Network (Jan.2009 p49). "ExpEther" is listed in the Technical Terms selected by ITPro.

In May 2012, an ExpEther Bridge chip was released with the following commercial products included the chip.

・ExpEther Board (1G) (Host Bus Adapter card for server/workstation)

・ExpEther I/O Expansion Box (1G) (PCI Express expansion box connected through ExpEther).

・ExpEther Client (CPU less thin client connected through ExpEther)

In 2012, the world’s first computing system that enables reconfiguration by attaching/detaching commodity standard PCI Express devices such as CPU (server), storage, GPGPU was deployed in the Osaka University and it has been in-service until now (as of July 2017).

In 2013, ExpEther's end-to-end delay measurement function was provided as a network monitoring tool in SHOWNET of Interop Tokyo 2013.

In October 2013, "ExpEther Board" and "ExpEther I/O Expansion Box" with the transmission performance enhanced from 2x1 GE to 2x10 GE, were released.

In December 2014, "ExpEther I/O Expansion Box" with a new function, sharing PCI Express SR-IOV compliant device among multi host over Ethernet, was released.

In February 2016, "ExpEther Board" and "ExpEther I/O Expansion Box" with transmission performance enhanced to 2x 40GE were released in USA. At the same time Xilinx announced collaboration for developing it.

A few USA system vendors released ExpEther products as OEM.

NEC and IBM are collaborating for developing a Service Acceleration Platform for Power Systems with ExpEther.

Roadmap

ExpEther Bridge chip has been upgraded by only replacing the PHY layer along with the PCI Express and Ethernet standard. Accordingly, the communication speed is expected to be upgraded along with the market growth of PCI Express Gen 4 and 100G-Ethernet.