To understand how ØMQ (version 0.1) scales to 10 gigabit Ethernet we've run couple of tests in Intel's Low Latency Lab in London.
We've run the same test over 1GbE (baseline) and 10GbE. The test consisted of sending 1M messages over the network as fast as possible. The test was run for different messages size (1, 2, 4, 8, 16, 32, 64, 128, 256, 512 and 1024 bytes). The whole test was performed 3 times to capture possible differences between the runs and averages are presented below.
This graph shows the throughput in messages per second for different message sizes:
Here are the same results converted into megabits per second:
With 1Gb Ethernet, ØMQ is able to saturate the network with messages approximately 120 bytes long. As shown on the graph, the throughput increases until it reaches approximately 900 Mb/second with 120 byte messages. After we reach this limit any further increasing of the message size yields no better results. Keep in mind that 900 Mb/second is 900 Mb of raw message data. All the overhead of TCP and IP is not included in the figure. Therefore, we assume that the saturation of the link is pretty close to 1Gb/second for messages larger than 120 bytes.
The situation with 10Gb Ethernet is different. We aren't able to exhaust more than ~2 Gb/second for message sizes up to 1024 bytes.
There may be several reasons for the observed behaviour:
- The hardware itself may be able to reach 10Gb only by transporting huge batches of data (tens/hundreds of kilobyte chunks)
- The operating system (sockets implementation) may be restrictive in the same way.
- Less than linear throughput growth for larger messages may be caused by non-zero-copy property of ØMQ.
We've seen ØMQ performing quite well on 1Gb Ethernet (full link saturation for messages 120 bytes long and above), however, on 10Gb Ethernet we weren't able to saturate more than 20% of the link. It should be taken into consideration though, that we haven't had tested with messages larger than 1024 bytes beacuse ØMQ's primary focus is on small messages (market data).