Test platform Supermicro NVMe 1029U-TN10RT + Intel Virtual RAID on CPU (VROC)
| TestsNVMe technology is becoming increasingly popular, the following tests were conducted in a test environment based on the SYS-1029U-TN10RT platform, using Intel VROC capabilities. The tests were conducted on Linux Centos, Linux Red Hat 7.4 and Linux Red Hat 7.5 distributions.
Check out the Supermicro SYS-1029U-TN10RT server in our store.
Configuration 1.1
Serwer SYS-1029U-TN10RT-SKL4116-256GB-NVMe
|
Lp |
Name |
Descriptipn |
Unit |
Quantity |
|---|---|---|---|---|
|
1 |
SYS-1029U-TN10RT |
Dual Xeon Scalable LGA3647, Up to 3TB DDR4; 2xPCI-E x16; 2x10GBase-T Intel X540; 10x2.5" NVMe, PWS-R 1000W Titanium |
PC |
1 |
|
2 |
P4X-SKL4116-SR3HQ |
Intel® Xeon® Scalable Skylake Silver 4116 Processor 12C/24T 16.5M Cache, 2.10 GHz |
PC |
2 |
|
3 |
MEM-DR432L-SL01-ER29 |
32GB DDR4-2933 2Rx4 LP ECC RDIMM,HF,RoHS |
PC |
8 |
|
4 |
HDS-IUN2-SSDPE2KX010T8 |
Intel DC P4510 1TB NVMe PCIe 3.0 3D TLC 2.5" 15mm 1DWPD, FW VDV10131 |
PC |
10 |
|
5 |
AOC-VROCPREMOD |
INTEL VROC PREMIUM, RAID0,1,5,10,HF,ROHS not only for Intel SSDs, doing raid 0,1,5,10 |
PC |
1 |
Detailed configuration of 1029U-TN10RT
- Procesor - Intel Xeon Scalable 4116 – 2 pieces, each:
- of Cores 12
- of Threads 24
- Processor Base Frequency 2.10 GHz
- Max Turbo Frequency 3.00 GHz
- Cache 16.5 MB L3
Pamięć RAM - MEM-DR432L-SL01-ER29
8x 32GB DDR4 2933MHz ECC REG DIMM – 256GB
Dyski NVMe - HDS-IUN2-SSDPE2KX010T8 – 10 pieces
- Sequential Read (up to) 2850 MB/s
- Sequential Write (up to) 1100 MB/s
- Random Read (100% Span) 465000 IOPS
- Random Write (100% Span) 70000 IOPS
Intel VROC Premium – RAID 0,1,5,10
Test results before optimization
Linux Centos 7 10xNVMe RAID5
RAID 5 with 10 NVMe drives WRITE
[root@centos]# fio --name=writefile --size=100G --filesize=100G --filename=disk_test.bin --bs=1M --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 --iodepth=200 --ioengine=libaiowritefile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200fio-3.1WRITE: bw=1511MiB/s (1585MB/s), 1511MiB/s-1511MiB/s (1585MB/s-1585MB/s), io=100GiB (107GB), run=67750-67750msec
RAID 5 z 10 drivers NVMe READ
[root@centos]# fio --time_based --name=benchmark --size=100G --runtime=30 --filename=disk_test.bin --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randread --blocksize=4k --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
READ: bw=1723MiB/s (1807MB/s), 1723MiB/s-1723MiB/s (1807MB/s-1807MB/s), io=50.5GiB (54.2GB), run=30001-30001mse
Red Hat 7.5 6xNVMe RAID 5, 4xNVMe RAID 0 test results
md124 : active raid0 nvme6n1[3] nvme7n1[2] nvme8n1[1] nvme9n1[0]
md126 : active raid5 nvme0n1[5] nvme1n1[4] nvme2n1[3] nvme3n1[2] nvme4n1[1] nvme5n1[0]
RAID 5 6xNVMe WRITE
[root@rh7.5]# fio --name=writefile --size=100G --filesize=100G --filename=disk_test.bin --bs=1M --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 --iodepth=200 --ioengine=libaio
writefile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200
WRITE: bw=1572MiB/s (1648MB/s), 1572MiB/s-1572MiB/s (1648MB/s-1648MB/s), io=100GiB (107GB), run=65149-65149msec
RAID 0 4xNVMe WRITE
[root@rh7.5 raid0]# fio --name=writefile --size=100G --filesize=100G --filename=disk_test.bin --bs=1M --nrfiles=1 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 --iodepth=200 --ioengine=libaio
writefile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200
WRITE: bw=2411MiB/s (2528MB/s), 2411MiB/s-2411MiB/s (2528MB/s-2528MB/s), io=100GiB (107GB), run=42467-42467msec
RAID 5 6xNVMe READ
[root@rh7.5]# fio --time_based --name=benchmark --size=100G --runtime=30 --filename=disk_test.bin --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randread --blocksize=4k –group_reporting
READ: bw=1621MiB/s (1700MB/s), 1621MiB/s-1621MiB/s (1700MB/s-1700MB/s), io=47.5GiB (51.0GB), run=30001-30001msec
RAID 0 4xNVMe READ
[root@rh7.5 raid0]# fio --time_based --name=benchmark --size=100G --runtime=30 --filename=disk_test.bin --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randread --blocksize=4k –group_reporting
READ: bw=1882MiB/s (1974MB/s), 1882MiB/s-1882MiB/s (1974MB/s-1974MB/s), io=55.1GiB (59.2GB), run=30001-30001msec
RAID 5 6xNVMe WRITE
[root@rh7.5]# fio --time_based --name=benchmark --size=100G --runtime=30 --filename=disk_test.bin --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --blocksize=4k –group_reporting
WRITE: bw=207MiB/s (218MB/s), 207MiB/s-207MiB/s (218MB/s-218MB/s), io=6225MiB (6527MB), run=30006-30006msec
RAID 0 4xNVMe WRITE
[root@rh7.5 raid0]# fio --time_based --name=benchmark --size=100G --runtime=30 --filename=disk_test.bin --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --blocksize=4k --group_reporting
WRITE: bw=441MiB/s (463MB/s), 441MiB/s-441MiB/s (463MB/s-463MB/s), io=12.9GiB (13.9GB), run=30001-30001msec
Summary
|
Test |
Parameters |
RAID 5 |
RAID 0 |
|||
|
|
Block size |
# of jobs |
IOD |
6 NVMe |
10 NVMe |
4 NVMe |
|
Seq write (GB/s) |
1024KiB |
1 |
200 |
1.648 |
1.029 |
2.528 |
|
Rand read (KIOPS) |
4kB |
4 |
128 |
417 |
378 |
483 |
|
Rand write (KIOPS) |
4kB |
4 |
128 |
60.1 |
37.1 |
114 |
Test results after optimization
Red Hat 7.5 6xNVMe RAID 5, 4xNVMe RAID 0 test results.
For a single disk:
[root@rh7.5]# fio --name=dummy --size=50G --runtime=20 --filename=/dev/nvme0n1 --ioengine=libaio --direct=1 --rw=randread -bs=4k** --iodepth=64 --numjobs=8 --group_reporting
dummy: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
READ: bw=2517MiB/s (2640MB/s), 2517MiB/s-2517MiB/s (2640MB/s-2640MB/s), io=49.2GiB (52.8GB), run=20003-20003msec
RAID 0 4xNVMe READ
[root@rh7.5]# fio --name=dummy --size=50G --runtime=20 --filename=/dev/md124p1 --ioengine=libaio --direct=1 --rw=randread -bs=4k** --iodepth=64 --numjobs=48 --group_reporting
dummy: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
READ: bw=7208MiB/s (7558MB/s), 7208MiB/s-7208MiB/s (7558MB/s-7558MB/s), io=141GiB (151GB), run=20007-20007msec
RAID 5 6xNVMe RAID 5
[root@rh7.5]# fio --name=dummy --size=50G --runtime=20 --filename=/dev/md126p1 --ioengine=libaio --direct=1 --rw=randread -bs=4k** --iodepth=64 --numjobs=48 --group_reporting
dummy: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
READ: bw=6116MiB/s (6413MB/s), 6116MiB/s-6116MiB/s (6413MB/s-6413MB/s), io=47.0GiB (51.5GB), run=8035-8035msec
Summary
|
Test |
Parameters |
RAID 5 |
RAID 0 |
||
|
|
Block Size |
# of jobs |
IOD |
6 NVMe |
4 NVMe |
|
Rand Read (KIOPS) |
4kB |
48 |
64 |
1566 |
1845 |
|
Rand Read (KIOPS) |
4kB |
16 |
64 |
1220 |
1690 |
|
Rand Read (KIOPS) |
4kB |
8 |
64 |
735 |
1098 |
CPU load
Linux 3.10.0-957.21.3.el7.x86_64 18.06.2019 _x86_64_ (48 CPU)
|
|
CPU |
%usr |
%nice |
%sys |
%iowait |
%irq |
%soft |
%steal |
%guest |
%gnice |
%idle |
|
21:28:46 |
all |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
100,00 |
|
21:28:47 |
all |
0,00 |
0,00 |
0,04 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
99,96 |
|
21:28:48 |
all |
0,00 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
99,98 |
|
21:28:49 |
all |
0,06 |
0,00 |
0,21 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
99,73 |
|
21:28:50 |
all |
9,01 |
0,00 |
21,70 |
0,00 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
69,27 |
|
21:28:51 |
all |
17,91 |
0,00 |
46,79 |
1,01 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
34,30 |
|
21:28:52 |
all |
18,04 |
0,00 |
47,28 |
1,01 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
33,68 |
|
21:28:53 |
all |
18,19 |
0,00 |
47,19 |
0,02 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
34,59 |
|
21:28:54 |
all |
18,33 |
0,00 |
46,53 |
0,94 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
34,21 |
|
21:28:55 |
all |
18,15 |
0,00 |
46,39 |
0,50 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
34,93 |
|
21:28:56 |
all |
17,76 |
0,00 |
46,38 |
0,45 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
35,40 |
|
21:28:57 |
all |
18,08 |
0,00 |
45,98 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
35,94 |
|
21:28:58 |
all |
17,99 |
0,00 |
45,82 |
0,14 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
36,03 |
|
21:28:59 |
all |
17,96 |
0,00 |
45,27 |
0,02 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
36,74 |
|
21:29:00 |
all |
17,95 |
0,00 |
45,26 |
0,39 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
36,38 |
|
21:29:01 |
all |
18,13 |
0,00 |
46,16 |
0,14 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
35,57 |
|
21:29:02 |
all |
17,93 |
0,00 |
45,57 |
0,23 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
36,27 |
|
21:29:03 |
all |
17,66 |
0,00 |
45,08 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
37,26 |
|
21:29:04 |
all |
17,92 |
0,00 |
44,68 |
0,00 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
37,39 |
|
21:29:05 |
all |
17,58 |
0,00 |
44,78 |
0,14 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
37,50 |
|
21:29:06 |
all |
17,47 |
0,00 |
45,00 |
0,16 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
37,37 |
|
21:29:07 |
all |
17,49 |
0,00 |
44,39 |
0,16 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
37,97 |
|
21:29:08 |
all |
17,80 |
0,00 |
43,95 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
38,25 |
|
21:29:09 |
all |
17,70 |
0,00 |
44,37 |
0,02 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
37,88 |
|
21:29:10 |
all |
8,98 |
0,00 |
22,02 |
0,04 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
68,93 |
|
21:29:11 |
all |
0,00 |
0,00 |
0,02 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
0,00 |
99,98 |
|
Średnia: |
all |
13,50 |
0,00 |
34,35 |
0,20 |
0,00 |
0,01 |
0,00 |
0,00 |
0,00 |
51,94 |
Configuration 1.2
Serwer SYS-1029U-TN10RT-SKL8176-768GB-NVMe
|
Lp |
Name |
Description |
Unit |
Quantity |
|---|---|---|---|---|
|
1 |
SYS-1029U-TN10RT |
Dual Xeon Scalable LGA3647, Up to 3TB DDR4; 2xPCI-E x16; 2x10GBase-T Intel X540; 10x2.5" NVMe, PWS-R 1000W Titanium |
PC |
1 |
|
2 |
P4X-SKL8176-SR37A |
SKL-SP 8176 28C/56T 2.1G 38.5M 10.4GT UPI Pokaż |
PC |
2 |
|
3 |
MEM-DR432L-SL03-ER26 |
32GB DDR4-2666 2Rx4 LP ECC RDIMM,HF,RoHS |
PC |
24 |
|
4 |
HDS-IUN2-SSDPE2KX040T8 |
Intel DC P4510 4TB NVMe PCIe 3.1 3D TLC 2.5" 15mm 1DWPD |
PC |
10 |
|
5 |
AOC-VROCPREMOD |
INTEL VROC PREMIUM, RAID0,1,5,10,HF,ROHS not only for Intel SSDs, doing raid 0,1,5,10 |
PC |
1 |
Detailed configuration of 1029U-TN10RT
Procesor - Intel Xeon Scalable 4116 –2 pieces each:
- of Cores 28
- of Threads 56
- Processor Base Frequency 2.10 GHz
- Max Turbo Frequency 3.80 GHz
- Cache 38.5 MB L3
Pamięć RAM - MEM-DR432L-SL01-ER29
8x 32GB DDR4 2933MHz ECC REG DIMM - 256GB
Dyski NVMe - HDS-IUN2-SSDPE2KX040T8 – 10 sztuk
- Sequential Read (up to) 3000 MB/s
- Sequential Write (up to) 2900 MB/s
- Random Read (100% Span) 636500 IOPS
- Random Write (100% Span) 111500 IOPS
- Intel VROC Premium – RAID 0,1,5,10
Summary (after optimization)
Red Hat 7.4 3xNVMe RAID 5, 8xNVMe RAID 5, 4xNVMe RAID 0
|
Test |
Parameters |
RAID 5 |
RAID 0 |
|||
|
|
Block size |
# of jobs |
IOD |
3 NVMe |
8 Nvme |
4 NVMe |
|
Seq write (GB/s) |
128KB |
1 |
128 |
1.267 |
1.23 |
3.517 |
|
8 |
128 |
1 |
1.347 |
3.254 |
||
|
Rand read (KIOPS) |
4KB |
16 |
128 |
1530 |
1631 |
2739 |
|
Rand write (KIOPS) |
4kB |
16 |
128 |
74.8 |
40.1 |
297 |
|
8 |
128 |
71.9 |
41.9 |
484 |
||
Test results indicate that Intel VROC proves to be an efficient alternative to traditional RAID controllers. However, it should be noted that with 10 NVMe drives, CPU usage can be quite high (up to 18%), even for a processor that was considered efficient at the time—the 12-core Xeon 4116 from 2019.
This suggests that if redundancy is not required, VROC software RAID could be useful in storage servers or in setups with a smaller number of drives.
Related Pages
- Performance of PCIe Gen 4.0 NVMe Drives on the Supermicro AOC-S3816L-L8IR Controller in JBOD, mdadm, and RAID 10 Configurations
- Intel Virtual RAID CPU (KEY VROC) - AOC-VROCINTMOD & AOC-VROCSTNMOD & AOC-VROCPREMOD
- RAID Types 0, 1, 5, 6, 10, 50, 60 & RAID Calculator (What RAID level to choose for a server?)
- RAID solution with GPU acceleration - GRAID SupremeRAID™ (RAID 0/1/10/5/6 on NVMe drives)
- "Phoronix Test Suite - a platform for benchmark testing."
- NVMe optimized Supermicro Chassis (Native NVMe)(Matrix)
- Tests of a Budget 1U AMD Epyc Server for 4 NVMe Gen 4.0 Drives (Will the BPN-SAS3-815TQ-N4 Support NVMe Gen 4.0 Drives?)
- Savings from using new CPUs in hybrid installations using renewable energy sources (RES)