It's a load generating applicanceIt's on the support page.
I installed it, but haven't had a chance to read about how to use it. I logged into the web interface and didn't see anything to start load generation.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Tintri on its way!
- Thread starter KapsZ28
- Start date
I installed it, but haven't had a chance to read about how to use it. I logged into the web interface and didn't see anything to start load generation.
So I have a question about the networking. Ever since the Tintri was setup, I am getting a high number of outbound packets being discarded even though there are no errors. I seem to get more when doing a vMotion, but right now with no VMs running on the Tintri SAN, the outbound packets are still being discarded.
Before you ask, yes the networking is setup correctly. At least on the switch side. There are 2x 10Gb Arista switches setup in MLAG. Controller A has 1 NIC going to switch A and the other NIC going to switch B. The ports are setup in port channel with LACP active and jumbo frames. Controller B is setup the same way. On the Tintri LACP and jumbo frames are enable which appear to be the only two settings even available when it comes to networking.
The FreeNAS server I setup a few weeks ago and our NetApp are on the same switches configured the same way with no outbound discards.
Before you ask, yes the networking is setup correctly. At least on the switch side. There are 2x 10Gb Arista switches setup in MLAG. Controller A has 1 NIC going to switch A and the other NIC going to switch B. The ports are setup in port channel with LACP active and jumbo frames. Controller B is setup the same way. On the Tintri LACP and jumbo frames are enable which appear to be the only two settings even available when it comes to networking.
The FreeNAS server I setup a few weeks ago and our NetApp are on the same switches configured the same way with no outbound discards.
A. Yes, all our switches have the highest MTU setting enabled when they are deployed to avoid such issues. B. The switches the Tintri is connected to, are the same switches the ESXi hosts are connected to.
What is interesting is most of the discards are coming from the second NIC on controller A and B. I cleared the statistics a little while ago on the switch. The discards show up on the ports and port-channel. Here are the statistics from the same port-channel on each switch.
Switch A - Port-Channel for Controller A - NIC1
Code:
Port-Channel13 is up, line protocol is up (connected)
Hardware is Port-Channel, address is 001c.734c.d0c3
Description: INF:TINTRISANa
Ethernet MTU 9214 bytes , BW 20000000 kbit
Full-duplex, 20Gb/s
Active members in this channel: 2
... Ethernet42 , Full-duplex, 10Gb/s
... PeerEthernet42 , Full-duplex, 10Gb/s
Fallback mode is: off
Up 20 minutes
2 link status changes since last clear
Last clearing of "show interface" counters 1:05:39 ago
30 seconds input rate 20 bps (0.0% with framing overhead), 0 packets/sec
30 seconds output rate 8.86 kbps (0.0% with framing overhead), 0 packets/sec
125 packets input, 16000 bytes
Received 0 broadcasts, 125 multicast
0 input errors, 0 input discards
847 packets output, 3994227 bytes
Sent 684 broadcasts, 127 multicast
0 output errors, 1543 output discardsSwitch B - Port-Channel for Controller A - NIC2
Code:
Port-Channel13 is up, line protocol is up (connected)
Hardware is Port-Channel, address is 001c.734c.a639
Description: INF:TINTRISANa
Ethernet MTU 9214 bytes , BW 20000000 kbit
Full-duplex, 20Gb/s
Active members in this channel: 2
... Ethernet42 , Full-duplex, 10Gb/s
... PeerEthernet42 , Full-duplex, 10Gb/s
Fallback mode is: off
Up 20 minutes, 32 seconds
2 link status changes since last clear
Last clearing of "show interface" counters 1:04:53 ago
30 seconds input rate 179 bps (0.0% with framing overhead), 0 packets/sec
30 seconds output rate 263 kbps (0.0% with framing overhead), 13 packets/sec
434 packets input, 35648 bytes
Received 311 broadcasts, 123 multicast
0 input errors, 3 input discards
49329 packets output, 106710450 bytes
Sent 34992 broadcasts, 13843 multicast
0 output errors, 61658 output discardsSo that is 1,543 output discards from NIC1 vs 61,658 for NIC2.
Below are the stats for the storage NICs. Both NICs are also used for iSCSI, but currently there is only one VM running on this host and only this one host is attached to the Tintri. But the outbound discards continue even with no VMs running on the Tintri.
Question about the NFS settings. I left them as default until I came across a best practices on Tintri's site, although it doesn't mention using these settings for the T8xx series.
When using high-bandwidth 10GigE networks, the following settings are recommended:
Net.TcpipHeapSize = 32
Net.TcpipHeapMax = 128
NFS.MaxVolumes = 64
I changed the settings on one ESXi host for now to test. Haven't seen any change.
Question about the NFS settings. I left them as default until I came across a best practices on Tintri's site, although it doesn't mention using these settings for the T8xx series.
When using high-bandwidth 10GigE networks, the following settings are recommended:
Net.TcpipHeapSize = 32
Net.TcpipHeapMax = 128
NFS.MaxVolumes = 64
I changed the settings on one ESXi host for now to test. Haven't seen any change.
Code:
/> cd /net/pNics/vmnic5
/net/pNics/vmnic5/> cat stats
device {
-- General Statistics:
Rx Packets:3078708
Tx Packets:1520882
Rx Bytes:25539468527
Tx Bytes:149940582
Rx Errors:0
Tx Errors:0
Rx Dropped:0
Tx Dropped:0
Multicast:1500
Collisions:0
Rx Length Errors:0
Rx Over Errors:0
Rx CRC Errors:0
Rx Frame Errors:0
Rx Fifo Errors:0
Rx Missed Errors:0
Tx Aborted Errors:0
Tx Carrier Errors:0
Tx Fifo Errors:0
Tx Heartbeat Errors:0
Tx Window Errors:0
Module Interface Rx packets:0
Module Interface Tx packets:0
Module Interface Rx dropped:0
Module Interface Tx dropped:0
-- Driver Specific Statistics:
rx_packets : 3078714
tx_packets : 1520887
rx_bytes : 25539504882
tx_bytes : 149941862
rx_errors : 0
tx_errors : 0
rx_dropped : 0
tx_dropped : 0
multicast : 1501
collisions : 0
rx_over_errors : 0
rx_crc_errors : 0
rx_frame_errors : 0
rx_fifo_errors : 0
rx_missed_errors : 0
tx_aborted_errors : 0
tx_carrier_errors : 0
tx_fifo_errors : 0
tx_heartbeat_errors : 0
rx_pkts_nic : 3078722
tx_pkts_nic : 1520695
rx_bytes_nic : 25564131733
tx_bytes_nic : 162089854
lsc_int : 3
tx_busy : 0
non_eop_descs : 0
broadcast : 1984
rx_no_buffer_count : 0
tx_timeout_count : 0
tx_restart_queue : 0
rx_long_length_errors : 0
rx_short_length_errors : 0
tx_flow_control_xon : 0
rx_flow_control_xon : 0
tx_flow_control_xoff : 0
rx_flow_control_xoff : 0
rx_csum_offload_errors : 0
rx_header_split : 0
alloc_rx_page_failed : 0
alloc_rx_buff_failed : 0
rx_no_dma_resources : 0
hw_rsc_aggregated : 0
hw_rsc_flushed : 0
fdir_match : 0
fdir_miss : 0
fdir_overflow : 0
fcoe_bad_fccrc : 0
fcoe_last_errors : 0
rx_fcoe_dropped : 0
rx_fcoe_packets : 0
rx_fcoe_dwords : 0
fcoe_noddp : 0
fcoe_noddp_ext_buff : 0
tx_fcoe_packets : 0
tx_fcoe_dwords : 0
os2bmc_rx_by_bmc : 0
os2bmc_tx_by_bmc : 0
os2bmc_tx_by_host : 0
os2bmc_rx_by_host : 0
tx_queue_0_packets : 1520887
tx_queue_0_bytes : 149941862
tx_queue_1_packets : 0
tx_queue_1_bytes : 0
tx_queue_2_packets : 0
tx_queue_2_bytes : 0
tx_queue_3_packets : 0
tx_queue_3_bytes : 0
tx_queue_4_packets : 0
tx_queue_4_bytes : 0
rx_queue_0_packets : 202826
rx_queue_0_bytes : 1519595198
rx_queue_1_packets : 2875888
rx_queue_1_bytes : 24019909684
rx_queue_2_packets : 0
rx_queue_2_bytes : 0
rx_queue_3_packets : 0
rx_queue_3_bytes : 0
rx_queue_4_packets : 0
rx_queue_4_bytes : 0
tx_pb_0_pxon : 0
tx_pb_0_pxoff : 0
tx_pb_1_pxon : 0
tx_pb_1_pxoff : 0
tx_pb_2_pxon : 0
tx_pb_2_pxoff : 0
tx_pb_3_pxon : 0
tx_pb_3_pxoff : 0
tx_pb_4_pxon : 0
tx_pb_4_pxoff : 0
tx_pb_5_pxon : 0
tx_pb_5_pxoff : 0
tx_pb_6_pxon : 0
tx_pb_6_pxoff : 0
tx_pb_7_pxon : 0
tx_pb_7_pxoff : 0
rx_pb_0_pxon : 0
rx_pb_0_pxoff : 0
rx_pb_1_pxon : 0
rx_pb_1_pxoff : 0
rx_pb_2_pxon : 0
rx_pb_2_pxoff : 0
rx_pb_3_pxon : 0
rx_pb_3_pxoff : 0
rx_pb_4_pxon : 0
rx_pb_4_pxoff : 0
rx_pb_5_pxon : 0
rx_pb_5_pxoff : 0
rx_pb_6_pxon : 0
rx_pb_6_pxoff : 0
rx_pb_7_pxon : 0
rx_pb_7_pxoff : 0
Code:
/net/pNics/vmnic5/> cd /net/pNics/vmnic7
/net/pNics/vmnic7/> cat stats
device {
-- General Statistics:
Rx Packets:6221163
Tx Packets:8717772
Rx Bytes:28650214901
Tx Bytes:50595013796
Rx Errors:0
Tx Errors:0
Rx Dropped:0
Tx Dropped:0
Multicast:1551
Collisions:0
Rx Length Errors:0
Rx Over Errors:0
Rx CRC Errors:0
Rx Frame Errors:0
Rx Fifo Errors:0
Rx Missed Errors:0
Tx Aborted Errors:0
Tx Carrier Errors:0
Tx Fifo Errors:0
Tx Heartbeat Errors:0
Tx Window Errors:0
Module Interface Rx packets:0
Module Interface Tx packets:0
Module Interface Rx dropped:0
Module Interface Tx dropped:0
-- Driver Specific Statistics:
rx_packets : 6221179
tx_packets : 8717787
rx_bytes : 28650224720
tx_bytes : 50595023506
rx_errors : 0
tx_errors : 0
rx_dropped : 0
tx_dropped : 0
multicast : 1552
collisions : 0
rx_over_errors : 0
rx_crc_errors : 0
rx_frame_errors : 0
rx_fifo_errors : 0
rx_missed_errors : 0
tx_aborted_errors : 0
tx_carrier_errors : 0
tx_fifo_errors : 0
tx_heartbeat_errors : 0
rx_pkts_nic : 6221187
tx_pkts_nic : 8026311
rx_bytes_nic : 28699991139
tx_bytes_nic : 50613591674
lsc_int : 3
tx_busy : 0
non_eop_descs : 0
broadcast : 2086
rx_no_buffer_count : 0
tx_timeout_count : 0
tx_restart_queue : 0
rx_long_length_errors : 0
rx_short_length_errors : 0
tx_flow_control_xon : 0
rx_flow_control_xon : 0
tx_flow_control_xoff : 0
rx_flow_control_xoff : 0
rx_csum_offload_errors : 0
rx_header_split : 0
alloc_rx_page_failed : 0
alloc_rx_buff_failed : 0
rx_no_dma_resources : 0
hw_rsc_aggregated : 0
hw_rsc_flushed : 0
fdir_match : 0
fdir_miss : 0
fdir_overflow : 0
fcoe_bad_fccrc : 0
fcoe_last_errors : 0
rx_fcoe_dropped : 0
rx_fcoe_packets : 0
rx_fcoe_dwords : 0
fcoe_noddp : 0
fcoe_noddp_ext_buff : 0
tx_fcoe_packets : 0
tx_fcoe_dwords : 0
os2bmc_rx_by_bmc : 0
os2bmc_tx_by_bmc : 0
os2bmc_tx_by_host : 0
os2bmc_rx_by_host : 0
tx_queue_0_packets : 8717787
tx_queue_0_bytes : 50595023506
tx_queue_1_packets : 0
tx_queue_1_bytes : 0
tx_queue_2_packets : 0
tx_queue_2_bytes : 0
tx_queue_3_packets : 0
tx_queue_3_bytes : 0
tx_queue_4_packets : 0
tx_queue_4_bytes : 0
rx_queue_0_packets : 245875
rx_queue_0_bytes : 1737630756
rx_queue_1_packets : 5975304
rx_queue_1_bytes : 26912593964
rx_queue_2_packets : 0
rx_queue_2_bytes : 0
rx_queue_3_packets : 0
rx_queue_3_bytes : 0
rx_queue_4_packets : 0
rx_queue_4_bytes : 0
tx_pb_0_pxon : 0
tx_pb_0_pxoff : 0
tx_pb_1_pxon : 0
tx_pb_1_pxoff : 0
tx_pb_2_pxon : 0
tx_pb_2_pxoff : 0
tx_pb_3_pxon : 0
tx_pb_3_pxoff : 0
tx_pb_4_pxon : 0
tx_pb_4_pxoff : 0
tx_pb_5_pxon : 0
tx_pb_5_pxoff : 0
tx_pb_6_pxon : 0
tx_pb_6_pxoff : 0
tx_pb_7_pxon : 0
tx_pb_7_pxoff : 0
rx_pb_0_pxon : 0
rx_pb_0_pxoff : 0
rx_pb_1_pxon : 0
rx_pb_1_pxoff : 0
rx_pb_2_pxon : 0
rx_pb_2_pxoff : 0
rx_pb_3_pxon : 0
rx_pb_3_pxoff : 0
rx_pb_4_pxon : 0
rx_pb_4_pxoff : 0
rx_pb_5_pxon : 0
rx_pb_5_pxoff : 0
rx_pb_6_pxon : 0
rx_pb_6_pxoff : 0
rx_pb_7_pxon : 0
rx_pb_7_pxoff : 0In a "best" world, sure - but it shouldn't matter, and shouldn't cause discards. The hosts aren't seeing it, so it seems to be tintri -> switch more than anything else. Do me a favor - fail the array over to the second controller (it's in the hardware page under diagnose) - lets see if the other controller is also doing it. If so, my next bet is going to be switch firmware. If not, then it may be a funky nic (it can happen).
I suck!
Network was totally my fault. I was looking to add replication over the data network the other day and screwed up the command to add VLAN 50 to the port-channel on switch B. Instead of adding VLAN 50, I added ALL VLANs except for 50. Still worked since the data VLAN was one of the 4094 VLANs, but yeah it caused the outbound discards.
Looks much better now.
Network was totally my fault. I was looking to add replication over the data network the other day and screwed up the command to add VLAN 50 to the port-channel on switch B. Instead of adding VLAN 50, I added ALL VLANs except for 50. Still worked since the data VLAN was one of the 4094 VLANs, but yeah it caused the outbound discards.
Looks much better now.
No, I was going to email them the config and when I pasted it into an email, I was like "I am an idiot". Then I fixed it.
So I haven't really had a chance to run many tests, but I did test to see the different in reboot time on a Windows 2012 VM. I rebooted the VM while running on our NetApp and it took 32 seconds. I then migrated it to Tintri and rebooted it and it only took 15 seconds. Hell, I was watching it on console and VMware Tools was running before I even saw the login screen. It was the opposite on the NetApp.
Other disk speed tests always show sub-ms access time, while the NetApp hits around 6ms for read. Starting sometime next week I will be moving some actually production VMs on to the Tintri.
Other disk speed tests always show sub-ms access time, while the NetApp hits around 6ms for read. Starting sometime next week I will be moving some actually production VMs on to the Tintri.
No, I was going to email them the config and when I pasted it into an email, I was like "I am an idiot". Then I fixed it.
Been there, done that, got the T-shirt
defuseme2k
[H]ard|Gawd
- Joined
- Oct 7, 2004
- Messages
- 1,074
getting our demo t880 up and online. Got a couple of tintri dudes coming to town next week... its a lab... Cisco gave us two FIs and we're using those with ports in application mode to run the 10gb interfaces for the Tintri. Problem is, upstream switch is a POS 1gb netgear thing. We're going to have to create a new vnic and force it out "path a" so that we don't have a chance of crossing that upstream switch to reach the controller or it'll drop to 1gb :\. Should be fun !
We're just getting stuff up and off the ground, so a folding table = rack. I'll have to post a picture or two a bit later. Sad state of affairs in that room so far.
We're just getting stuff up and off the ground, so a folding table = rack. I'll have to post a picture or two a bit later. Sad state of affairs in that room so far
.Good. So I still have no good reason to upgrade to vSphere 6 yet.
What I would like to know is what exactly is "Host" latency. And not the definition I found in Tintri's support portal.
"Host: The latency attributed to the ESX host. This value is calculated by subtracting the network, storage and disk latency values from the end-to-end latency measurement Provided by vCenter."
That is a very general statement which obviously would require more research on the host side to see where the latency is coming from. Not that I am having any latency issues. The only time latency is high is when performing a bunch of Storage vMotions and it seems to show mostly "Host" latency. In that type of situation, what from the ESXi host would attribute to the latency?
What I would like to know is what exactly is "Host" latency. And not the definition I found in Tintri's support portal.
"Host: The latency attributed to the ESX host. This value is calculated by subtracting the network, storage and disk latency values from the end-to-end latency measurement Provided by vCenter."
That is a very general statement which obviously would require more research on the host side to see where the latency is coming from. Not that I am having any latency issues. The only time latency is high is when performing a bunch of Storage vMotions and it seems to show mostly "Host" latency. In that type of situation, what from the ESXi host would attribute to the latency?
Riccochet
Fully [H]
- Joined
- Apr 11, 2007
- Messages
- 29,905
The only "Host" latency I can think of would be memory/cpu contention.
It's memory/cpu contention, kernel queueing, kernel contention in the IO stack, etc. Anything that can't be pinned to "network" or "the storage device". With NFS, it's a bit more nebulous based on the lack of SCSI end-to-end, which is what makes the Tintri nice as it not only has the host latency stat from ESX, but also can derive one from other places to confirm its reliability
Gotcha. So I guess when it comes to that, the higher model systems will be able to handle more load and not just more capacity?
So far everything has been great with our Tintri. One thing that really impresses me is the SAN to SAN replication. I enabled replication on a 650 GB file server only a few hours ago and it has already been replicated from NY to NJ over the same 1 Gb connection that I was using for Zerto. Now when I tried to use Zerto, it would easily take at least 24 hours to replicate something that large because it is reading the entire VMDK file from the hypervisor level.
Granted, once Zerto finishes the initial replication it is pretty good at keeping the VMs up to date but it still has additional networking overhead on the hypervisor.
Granted, once Zerto finishes the initial replication it is pretty good at keeping the VMs up to date but it still has additional networking overhead on the hypervisor.
Not convinced that it's apples to apples when comparing array based replication to software based replication that runs from a VM. Most (all?) other hardware vendors offer array based replication as well.
I agree, but it beats our NetApp SnapMirror as well, and I didn't believe it myself when it said the replica was up to date and cloned it in the recovery site just to be sure. I still have more to replicate, so I will do a test and pay closer attention to how long it takes.
And although Zerto vs Tintri may not be apples to apples, based on my experience with Zerto, what would have taken a week, took less than a day with Tintri.
Oh and with SnapMirror we have to play around with the TCP window size to try and get the best performance. With Tintri, I just enable it and pick a schedule.
Watch the logical replication graph - that'll tell you how much it's taking out of the actual data stream. Triply more so if it understands the base-tree that the VM was created from
defuseme2k
[H]ard|Gawd
- Joined
- Oct 7, 2004
- Messages
- 1,074
wtf on a 1gb link? Holy moly, that is insane.
This is what happens when you hire the guys who wrote NetApp, Data Domain, and RAID to start your company.
On a secondary note, interviewing with the guy who wrote RAID? Freaky. As. Hell. You're just sitting there, thinking: "My entire career is based on your work..."
On a secondary note, interviewing with the guy who wrote RAID? Freaky. As. Hell. You're just sitting there, thinking: "My entire career is based on your work..."
Let's think about this. Theoretical throughput of a 1gb link is 125 MB/s, or 450 GB/hr. During initial replication it's a pure copy since no data is present yet at the replicated site.
So seeding 270 GB in 45 minutes gives us a throughput of 360 GB/hr. Not convinced that insane is the word I'd use to describe copying data at a rate of 360 GB/hr, or at 80% line speed.
Now, if that VM wasn't the first to go over the wire and there is already data at the replication site then de-dupe comes into play which is likely rate-limited by CPU more so than network or disk. It would be interesting to know how much data was actually transferred to the replication site. Though it's safe to say that compressed and de-duped replication is slower than 360 GB/hr, meaning that if less than 270 GB were pushed then the throughput was lower. (It may still be faster than for other vendors.)
Last edited: