Areca to Areca Raid5 array speed drops to 0

I have got two machines with Areca 1882i (8 ports) cards. The machines are connected by Mellanox Connectx-3 (10Gbps fiber) cards.

Both machines have got a RAID5 array (8x2TB and 8x3TB).

Alle four cards (Areca en Mellanox) are on PCI-E 3.0 x8 slots.

My problem is when i copy large amount of data (multiple TB's) from Server1 to Server 2(or the other way around), the speed drops to zero.

It goes like this:
The speed goes up to around 900 MB/s. Stays there for a few seconds and then drops down to zero in about 5-10 steps. It stays around zero a few seconds and then goes up to 900MB/s again.

Write Back of Write Through does not make a difference. No BBU's but the machines are hooked up to a UPS. HDD Cache and Disk Write Cache mode both enable.

The machines are no were near 100% (cpu/memory). Machines are both Xeon machines (8c 16t en 10c 20t) with lots of memory.

Crystal disk Benchmark (1GB) shows 800-900 MB/s (SEQ Q32T1).

I thought it could be cache related, but then "Write Through" should make a difference... right?

Any ideas?

Do you have either array set to sleep after lack of use? Does it constantly pulse between 900MB/s and 0 like a wave, or does it just happen the first time after you start the copy. What does the task manager performance tab show for Disk during the copy? Please post the log from both cards.

It is like a wave. Task manager for disk i will have to look at. Which log would you like to see? Where can i find it?

No sleep mode, i created a folder first on both arrays to rule out anything sleep related.

Operations said:

It is like a wave. Task manager for disk i will have to look at. Which log would you like to see? Where can i find it?

No sleep mode, i created a folder first on both arrays to rule out anything sleep related.

Click to expand...

If you go into the card via Ethernet or locally via archttp, you can see the log file from the menu system or on the local system with archttp at %ProgramFiles%\Areca\config\areca.log. Wave shaped stutters (especially of the are sine wave sized) tend to be buffer flush problems. Does it make a difference which side of the link initiates the copy? Lets see if there is anything in the log, I also want you to do a local benchmark diag on each end with crystaldiskmark or HD Sentinel to remove explorer from the equation as the conduit. Do both your mellanox's have the latest firmware? Are you running copper or fiber for the link and is it point to point or a switch? Please post the log from both sides.

I will post the log asap.

Your other questions:
- no it doesn't matter which side initiates the copy
- Crystal diskmark shows between 800-900 MB/s (SEQ32, the first number)
- The machines are connected directly by a SFP+ DAC Twinax copper cable. I have tried a different (same type) cable to rule out a cable malfunction
- Mellanox cards have got the newest FW

What are the motherboards on each end, are they the latest bios and what other pcie cards are in the machines?

Supermicro X9SRL-F and X9SCM-F. And yes most recent BIOS. other card in both servers are an Asus USB3.0 pci-e card.

Does the problem occur if you copy from/to any non-Areca array member volume to/from any non-Areca array member volume?

mwroobel said:

Does the problem occur if you copy from/to any non-Areca array member volume to/from any non-Areca array member volume?

Click to expand...

Hi, I also have this issue with a RAID6 array on a 1882 card except it's copying large files (say 40 GB's) from a SSD on Windows 10 to the array on Server 2016. Both machines have Mellanox ConnectX-3 cards running at 10Gbps. I always thought it was how the cache buffering works so never investigated. Sounds like this is not normal...hope it can be resolved!

No more ideas?