iSCSI initiator crashing Server 2008

A

awesomo

Gawd
Joined
Mar 20, 2010
Messages
528
I have a Server 2008 server and a Synology DS1511+ Nas.

The goal is to make an iSCSI connection from the Synology to the Server 2008 server. However, I am having an issue. I set this all up two weeks ago. For two weeks it was all running fine. Now, my iSCSI drive is dropping daily. And everytime I go into the iSCSI initiator and click anything, the server still allows me to do some things, but it basically locks up. I can't restart, services stop responding, etc... The only was I can break it out of that is a hard reset. Does anyone have any idea's why in the hell iSCSI initiator keeps taking down my entire server to the point of hard restart? Once this is solved, I bet the iSCSI dropping daily will be solved as well. I have already talked with Synology, the log is producing nothing and they are suspicious that it has been running fine the past two weeks. Windows update is off (I manually do it during scheduled maintenance), and there is no av or firewall running, so rule those out.
 
awesomo said:
I have a Server 2008 server and a Synology DS1511+ Nas.

The goal is to make an iSCSI connection from the Synology to the Server 2008 server. However, I am having an issue. I set this all up two weeks ago. For two weeks it was all running fine. Now, my iSCSI drive is dropping daily. And everytime I go into the iSCSI initiator and click anything, the server still allows me to do some things, but it basically locks up. I can't restart, services stop responding, etc... The only was I can break it out of that is a hard reset. Does anyone have any idea's why in the hell iSCSI initiator keeps taking down my entire server to the point of hard restart? Once this is solved, I bet the iSCSI dropping daily will be solved as well. I have already talked with Synology, the log is producing nothing and they are suspicious that it has been running fine the past two weeks. Windows update is off (I manually do it during scheduled maintenance), and there is no av or firewall running, so rule those out.
Click to expand...

can you check the cpu log on the synology ? see if it's being taxed ? Have you looked in the event viewer to see errors ?
 
I am starting to lean towards a network problem. I will be able to dig into it deeper tomorrow morning.

I don't get any specific errors before or while the system locks up. But when the volume drops I get the following spammed in my event log for hours. No CPU spike on the Synology.

"Target sent an invalid iSCSI PDU. Dump data contains the entire iSCSI header."
Eventid 23

"Target failed to respond in time to a Task Management request."
Eventid 49

"The description for Event ID 129 from source iScsiPrt cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

\Device\RaidPort0

the message resource is present but the message is not found in the string/message table"
Eventid 129
 
One thing I found with iSCSI is that you really want a separate NIC for your SAN. So your server should have two NICs, one for your data network, one for your SAN. Separate switches also helps with this, or at the very least separate VLANs.

This way you aren't overloading the NIC or switches.
 
Try to isolate the issue replacing MS iSCSI initiator on your server with some third-party one. Say the one from StarWind. If the issue would be still present I guess it's Synology firmware one. Start with checking health status, SMART numbers and so on. I've seen something like this one when one of my ReadyNAS units started developing bad blocks on RAID mirror set. Worked fine under moderate load but started crashing under heavy load.

awesomo said:
I have a Server 2008 server and a Synology DS1511+ Nas.

The goal is to make an iSCSI connection from the Synology to the Server 2008 server. However, I am having an issue. I set this all up two weeks ago. For two weeks it was all running fine. Now, my iSCSI drive is dropping daily. And everytime I go into the iSCSI initiator and click anything, the server still allows me to do some things, but it basically locks up. I can't restart, services stop responding, etc... The only was I can break it out of that is a hard reset. Does anyone have any idea's why in the hell iSCSI initiator keeps taking down my entire server to the point of hard restart? Once this is solved, I bet the iSCSI dropping daily will be solved as well. I have already talked with Synology, the log is producing nothing and they are suspicious that it has been running fine the past two weeks. Windows update is off (I manually do it during scheduled maintenance), and there is no av or firewall running, so rule those out.
Click to expand...
 
Thanks for the great comments. When I first started having issues, No errors appeared in the Synology logs, not even in the secret support only log. No smart errors or raid errors. I pulled each drive and ran a thorough test on them to make sure. They are completely fine. However, I did track it down (Hopefully). The HP switch I think was the culprit. It was causing wierd issues under load. It was dropping packets occasionally on the heavier ports and stopping broadcasts, but there was not a loop. I updated the firmware and all has been fine for a few days now. I will know for sure in a few weeks.
 
awesomo said:
Thanks for the great comments. When I first started having issues, No errors appeared in the Synology logs, not even in the secret support only log. No smart errors or raid errors. I pulled each drive and ran a thorough test on them to make sure. They are completely fine. However, I did track it down (Hopefully). The HP switch I think was the culprit. It was causing wierd issues under load. It was dropping packets occasionally on the heavier ports and stopping broadcasts, but there was not a loop. I updated the firmware and all has been fine for a few days now. I will know for sure in a few weeks.
Click to expand...

on your hp switch, what port are you using for your uplink port ?
 
@dashpuppy I was using port 49 to connect the NAS. It's been a few weeks and nothing has locked up, I am going to call this one solved.
 
You must log in or register to reply here.
Back
Top