UBNT Edgemax site-to-site performance?

B

bds1904

Gawd
Joined
Aug 10, 2011
Messages
1,007
Does anyone have any real world vpn performance numbers for the ERlite, ER8 and/or the ER8-Pro?

Ipsec seems pretty easy to find info on, but it's all sha1/aes128. Any numbers on sha1/aes256?

Openvpn is a whole other story, I can't find much of anything that isn't from right after the release of the hardware. Again, I'm looking for any numbers that are sha1/aes256 udp and possibly with and without compression.

Anyone got testing results?
 
EdgeRouter Lite

OS: FreeBSD 10 (Linux will perform similar)

OpenVPN (OpenSSL)

proto udp
cipher AES-192-CBC
comp-lzo

Performance
Maxes out at ~830kbyte/s (single threaded, maxes out one core), due to OpenVPN's design hardware acceleration doesn't make any difference.

You're going to see pretty much similar performance in all versions.

//Danne
 
I'm using the built in IPSEC site-to-site tunnel, with a VTI binding as seen in the KB article: http://community.ubnt.com/t5/EdgeMA...dgeMAX-VTI-Example-on-EdgeRouter/ta-p/1106117

I have a 50mbps/5mbps cable connection on one end, with an ER-Lite, and a 60mbps/60mbps fiber connection with an ER-8 on the other end. Doing Iperf between two machines on the LAN on each end, I saw 5mbps in one direction and 40mbps in the other.

The connection was being used fairly heavily at the time, so I don't know if 40mbps is the upper limit of the tunnel, or if I ran out of bandwidth on my cable connection.

Edit: Didn't see the aes256 requirement.
 
diizzy said:
Performance
Maxes out at ~830kbyte/s (single threaded, maxes out one core), due to OpenVPN's design hardware acceleration doesn't make any difference.
Click to expand...

That seems odd to me because pfsense can use cpu with aes-ni or a amd geode crypto accelerator (aes-128) to improve openvpn performance.
 
Last edited:
Well, I think I'm going to order a lot of 5 hp t5740 thin clients, 5x intel 1000/pro pt dual nic's and 5x pcie expansion kits for the thin client.

It's just for a personal project, and at $100 per unit with 40mbit+ openvpn throughput the price can't be beat. If all the endpoint had static ip's I would go with the erl and aes-128 ipsec but I need the (easy) ability to handle the dynamic ip's.

Good thing I'm familiar with pfsense.
 
Getting 40+ out of an Atom N280 CPU seems really optimistic to me but oh well, what are you going to use the NICs for? It's a 1000mbit NIC (Broadcom from what I can tell which is decent) in that box which you'll hardly saturate, you can use VLANs if you need multiple interfaces. In this case I doubt you'll see any noticeable difference between Intel och Broadcom. In terms of performance you're probably better off running vanilla FreeBSD and using the 4BSD scheduler than ULE due to the single core CPU.
//Danne
 
Last edited:
It's just easier to have more than 1 physcial interface when you are dealing with dhcp on wan's. In the past I have run into ISP's equipment that doesn't play well with router-on-a-stick topologies.

For instance, one of comcast's modems had to be powered on and in sync before the interface was plugged into the switch otherwise it wouldn't accept a dhcp request from the ROOS. Even if the modem dropped sync then came back it wouldn't accept a dhcp request again. It was a very odd problem. Ever since then I have avoided roos if the wan is dhcp.

Roos with static ip wan's always works well.

I typed that wrong btw, 20mbit+.
 
@ bds1904

If you're just going for 20mbit (or so) the TP-Link WDR4900 (v1) might be a very good option in that regard running OpenVPN.

@ /usr/home

Since both platforms (UBNT's firmware and OpenWRT) are based on Linux it's relevant. BSD also performs similar using OpenVPN so not much of a diference there...
You can use hardware encryption using the ERL but OpenVPN doesn't benefit from it at least not on MIPS devices as I mentioned above, interestingly OpenWRT supports it (I haven't tested but there are commits about it) but not the vendor provided firmware.
//Danne
 
Last edited:
IPsec SH1A you can see 200Mbps on the ERL
128 AES tops out at 130Mbps on the ERL
256 AES tops out around 50Mbps on the ERL


The ER Pro model almost doubles the performance of the numbers above.

OpenVPN performance is terrible and will remain that way until it gets acceleration in the kernel....which is not going to happen anytime soon.


If you want beastly fast IPsec VPN hardware devices, go purchase a pair of Zyxel USG 110s 300-400Mbps with failover.
 
diizzy said:
@ Mackintire
No, OpenVPN will still not improve (at least noticable) as I've said before...
https://doc.pfsense.org/index.php/File:Alix2d3_vpn_throughput.png
//Danne
Click to expand...

There is some issue with the openvpn package that is causing the issue.

Either way i don't see the issue being solved anytime soon, much like Mikrotik's lack of udp openvpn support. It is either a licensing issue or a mips cpu issue.

I did find some hp gt7720 thin clients that are of interest for running pfsense. Dual core 2.3ghz and pcie expansion should make a decent little router.
 
Sigh, the only thing that's "wrong" is how OpenVPN is designed it has nothing to do with hardware or platform and sure as isn't MIPS as I run it on my OpenWRT boxes (Atheros SoC) just fine (UDP and all) and my ERLs (Cavium SoC) running FreeBSD.
//Danne
 
Just to update I did pick up a HP T5740 (Atom N280 1.66GHz 1GB memory) w/ expansion chassis for $50 and threw in a dual port pcie intel nic. I did some performance testing on it and was quite surprised with the performance. Tests were T5740 to VM on dual L5420 server.

Code: 
OpenVPN UDP AES-128 No compression: 58Mb/sec
OpenVPN UDP AES-128 Adaptive Compression: 90Mb/sec

OpenVPN UDP AES-256 No compression: 50Mb/sec
OpenVPN UDP AES-256 Adaptive Compression: 86Mb/sec

Routing capability: ~450Mb/sec

All tests were performed with iperf on 2 PC's behind the routers, default window size 5 concurrent connections.

The real impressive part was AES-256 no compression single stream iperf test, 49.6Mb/sec!

I think I'm going to spring for a couple more of the T5740's for 2 lower bandwidth sites and 3 GT7725's (2 for the main connection running carp and 1 high(er) bandwidth site). If my math is right the GT7725 should do 85Mb/sec+ AES-256 no compression.
 
Last edited:
I can do about 14-15mbit/s on a TP-Link WDR3600 (OpenWRT) using AES-192 so slightly faster than the EdgeRouter Lite.
//Danne
 
So, updating the firmware on the WDR3600 I managed to do about 18-19mbit/s instead. All speeds are real world over the Internet speeds not on the same network.

Changed a few libs too,
(e)glibc --> musl
openssl --> polarssl (mbedtls)
http://projects.pyret.net/dump/openwrt/r45385-collectd/
WDR3500, WDR3600, WDR4300, Archer C5 and C7v2, WD MyNet N600 and N750.

If anyone wants to give a go, most useful stuff is already included such as Samba, NFS etc
Transmission is a trunk snapshot as release doesn't support PolarSSL yet.

Full package list:
Code: 
root@OpenWrt:~# opkg list-installed
base-files - 157-r45385
block-mount - 2015-04-10-cc526b9dd7471270dae586433ec610760df54cc0
busybox - 1.23.2-1
bwm-ng - 0.6-1
chat - 2.4.7-5
collectd - 5.4.2-1
collectd-mod-conntrack - 5.4.2-1
collectd-mod-cpu - 5.4.2-1
collectd-mod-df - 5.4.2-1
collectd-mod-disk - 5.4.2-1
collectd-mod-dns - 5.4.2-1
collectd-mod-interface - 5.4.2-1
collectd-mod-iptables - 5.4.2-1
collectd-mod-load - 5.4.2-1
collectd-mod-logfile - 5.4.2-1
collectd-mod-memory - 5.4.2-1
collectd-mod-network - 5.4.2-1
collectd-mod-ping - 5.4.2-1
collectd-mod-processes - 5.4.2-1
collectd-mod-protocols - 5.4.2-1
collectd-mod-tcpconns - 5.4.2-1
collectd-mod-uptime - 5.4.2-1
collectd-mod-vmem - 5.4.2-1
comgt - 0.32-25
comgt-directip - 0.32-25
comgt-ncm - 0.32-25
curl - 7.40.0-3
ddns-scripts - 2.4.0-1
dnsmasq - 2.73rc4-1
dropbear - 2014.65-2
e2fsprogs - 1.42.12-1
firewall - 2015-02-26
fstools - 2015-04-10-cc526b9dd7471270dae586433ec610760df54cc0
hostapd-common - 2015-03-25-1
ip6tables - 1.4.21-1
iptables - 1.4.21-1
iptables-mod-conntrack-extra - 1.4.21-1
iptables-mod-filter - 1.4.21-1
iptables-mod-ipopt - 1.4.21-1
iw - 3.17-1
iwinfo - 2015-03-23-40f2844fadc05f4a4de7699dbc12fee295b7057b
jshn - 2015-03-22-b8d9b382e39823850331edc2a92379173daf1be3
jsonfilter - 2014-06-19-cdc760c58077f44fc40adbbe41e1556a67c1b9a9
kernel - 3.18.11-1-0b660be3f37e5f2d018f190b69bd2a4b
kmod-ath - 3.18.11+2015-03-09-3
kmod-ath9k - 3.18.11+2015-03-09-3
kmod-ath9k-common - 3.18.11+2015-03-09-3
kmod-bridge - 3.18.11-1
kmod-cfg80211 - 3.18.11+2015-03-09-3
kmod-crypto-aes - 3.18.11-1
kmod-crypto-arc4 - 3.18.11-1
kmod-crypto-core - 3.18.11-1
kmod-crypto-hash - 3.18.11-1
kmod-dnsresolver - 3.18.11-1
kmod-fs-exportfs - 3.18.11-1
kmod-fs-ext4 - 3.18.11-1
kmod-fs-nfs - 3.18.11-1
kmod-fs-nfs-common - 3.18.11-1
kmod-fs-nfsd - 3.18.11-1
kmod-fs-vfat - 3.18.11-1
kmod-gpio-button-hotplug - 3.18.11-1
kmod-ifb - 3.18.11-1
kmod-ip6tables - 3.18.11-1
kmod-ipt-conntrack - 3.18.11-1
kmod-ipt-conntrack-extra - 3.18.11-1
kmod-ipt-core - 3.18.11-1
kmod-ipt-filter - 3.18.11-1
kmod-ipt-ipopt - 3.18.11-1
kmod-ipt-nat - 3.18.11-1
kmod-ipv6 - 3.18.11-1
kmod-ledtrig-usbdev - 3.18.11-1
kmod-lib-crc-ccitt - 3.18.11-1
kmod-lib-crc16 - 3.18.11-1
kmod-lib-textsearch - 3.18.11-1
kmod-llc - 3.18.11-1
kmod-mac80211 - 3.18.11+2015-03-09-3
kmod-mii - 3.18.11-1
kmod-nf-conntrack - 3.18.11-1
kmod-nf-conntrack6 - 3.18.11-1
kmod-nf-ipt - 3.18.11-1
kmod-nf-ipt6 - 3.18.11-1
kmod-nf-nat - 3.18.11-1
kmod-nf-nathelper - 3.18.11-1
kmod-nls-base - 3.18.11-1
kmod-nls-cp437 - 3.18.11-1
kmod-nls-cp850 - 3.18.11-1
kmod-nls-iso8859-1 - 3.18.11-1
kmod-nls-utf8 - 3.18.11-1
kmod-ppp - 3.18.11-1
kmod-pppoe - 3.18.11-1
kmod-pppox - 3.18.11-1
kmod-sched-connmark - 3.18.11-1
kmod-sched-core - 3.18.11-1
kmod-scsi-core - 3.18.11-1
kmod-slhc - 3.18.11-1
kmod-stp - 3.18.11-1
kmod-tun - 3.18.11-1
kmod-usb-core - 3.18.11-1
kmod-usb-net - 3.18.11-1
kmod-usb-net-cdc-eem - 3.18.11-1
kmod-usb-net-cdc-ether - 3.18.11-1
kmod-usb-net-cdc-mbim - 3.18.11-1
kmod-usb-net-cdc-ncm - 3.18.11-1
kmod-usb-net-cdc-subset - 3.18.11-1
kmod-usb-net-huawei-cdc-ncm - 3.18.11-1
kmod-usb-net-qmi-wwan - 3.18.11-1
kmod-usb-net-rndis - 3.18.11-1
kmod-usb-net-sierrawireless - 3.18.11-1
kmod-usb-ohci - 3.18.11-1
kmod-usb-printer - 3.18.11-1
kmod-usb-serial - 3.18.11-1
kmod-usb-serial-sierrawireless - 3.18.11-1
kmod-usb-serial-wwan - 3.18.11-1
kmod-usb-storage - 3.18.11-1
kmod-usb-wdm - 3.18.11-1
kmod-usb2 - 3.18.11-1
libblkid - 2.25.2-4
libblobmsg-json - 2015-03-22-b8d9b382e39823850331edc2a92379173daf1be3
libc - 1.1.7-1
libcurl - 7.40.0-3
libevent2 - 2.0.22-1
libext2fs - 1.42.12-1
libgcc - 4.8-linaro-1
libip4tc - 1.4.21-1
libip6tc - 1.4.21-1
libiptc - 1.4.21-1
libiwinfo - 2015-03-23-40f2844fadc05f4a4de7699dbc12fee295b7057b
libiwinfo-lua - 2015-03-23-40f2844fadc05f4a4de7699dbc12fee295b7057b
libjson-c - 0.12-1
libjson-script - 2015-03-22-b8d9b382e39823850331edc2a92379173daf1be3
libltdl - 2.4-1
liblua - 5.1.5-1
liblzo - 2.08-1
libncurses - 5.9-1
libnfnetlink - 1.0.1-1
libnl-tiny - 0.1-4
liboping - 1.6.2-1
libpcap - 1.5.3-1
libpolarssl - 1.3.10-1
libpopt - 1.16-1
libpthread - 1.1.7-1
librpc - 2015-04-10-308e9964bfb623773dc0dcc99ef9d18d1551d6ae
librt - 1.1.7-1
libubox - 2015-03-22-b8d9b382e39823850331edc2a92379173daf1be3
libubus - 2015-01-22-2d660c519d2fcff95248da9f4fd9b37d61f9eb09
libubus-lua - 2015-01-22-2d660c519d2fcff95248da9f4fd9b37d61f9eb09
libuci - 2015-04-09.1-1
libuci-lua - 2015-04-09.1-1
libusb-1.0 - 1.0.19-1
libuuid - 2.25.2-4
libwrap - 7.6-1
libxtables - 1.4.21-1
lua - 5.1.5-1
luci - git-15.100.60956-5ef2946-1
luci-app-ddns - 2.2.2-1
luci-app-firewall - git-15.100.60956-5ef2946-1
luci-app-p910nd - git-15.100.60956-5ef2946-1
luci-app-qos - git-15.100.60956-5ef2946-1
luci-app-samba - git-15.100.60956-5ef2946-1
luci-app-transmission - git-15.100.60956-5ef2946-1
luci-app-upnp - git-15.100.60956-5ef2946-1
luci-base - git-15.100.60956-5ef2946-1
luci-lib-ip - git-15.100.60956-5ef2946-1
luci-lib-nixio - git-15.100.60956-5ef2946-1
luci-mod-admin-full - git-15.100.60956-5ef2946-1
luci-proto-3g - git-15.100.60956-5ef2946-1
luci-proto-ppp - git-15.100.60956-5ef2946-1
luci-theme-bootstrap - git-15.100.60956-5ef2946-1
miniupnpd - 1.9.20150307-1
mtd - 20
nano - 2.4.0-1
netifd - 2015-04-09-92d2aea918f650f309f753349457028032d53280
nfs-kernel-server - 1.3.2-2
nfs-kernel-server-utils - 1.3.2-2
odhcp6c - 2015-04-10-c3bbeced0f204b6b9571148ae84227105baaf179
odhcpd - 2015-04-10-e23972527f93a3d2a5412400384519ff550708b7
openvpn-polarssl - 2.3.6-4
opkg - 9c97d5ecd795709c8584e972bfdf3aee3a5b846d-7
p910nd - 0.97-4
portmap - 6.0-4
ppp - 2.4.7-5
ppp-mod-pppoe - 2.4.7-5
procd - 2015-04-10-27159f21f76b973a9fa3ec92b8fee2e390d43a43
qos-scripts - 1.2.1-7
rpcd - 2015-03-30-311c85e7d9a8f7fee17e65afc371f4fd0c8cd588
rsync - 3.1.1-2
rsyncd - 3.1.1-2
samba36-server - 3.6.25-3
swap-utils - 2.25.2-4
swconfig - 10
tc - 3.19.0-1
tcpdump - 4.5.1-4
terminfo - 5.9-1
transmission-daemon - 2.84-1
tune2fs - 1.42.12-1
uboot-envtools - 2014.10-2
ubox - 2015-04-02-6fbafd7d5b2835d00eb9d1684e7c6ccf907177b8
ubus - 2015-01-22-2d660c519d2fcff95248da9f4fd9b37d61f9eb09
ubusd - 2015-01-22-2d660c519d2fcff95248da9f4fd9b37d61f9eb09
uci - 2015-04-09.1-1
uhttpd - 2015-03-30-b9178b9357798ae23a5724333cc6572d14f23958
uhttpd-mod-ubus - 2015-03-30-b9178b9357798ae23a5724333cc6572d14f23958
usb-modeswitch - 2014-08-26-993a9a542791953c4804f7ddbb3a07756738e37a
usbreset - 4
wpad-mini - 2015-03-25-1
wwan - 2014-07-17-1
zlib - 1.2.8-1

Use at your own risk etc but it works fine for me (tm) :)

As a side note, you can overclock these boxes apparently so I'd guess you'll end up at around 25mbit/s by doing so.

//Danne
 
Last edited:
You must log in or register to reply here.
Back
Top