Permalänk
teklager
Skrivet av box4mm:

Great thank you, im interested in a router that can do wireguard 1000/1000 both directions simultaneously, wireless (dont need high speeds but need wireless)

edit: im guessing APU1-4 cant? if you can specify OS also for tests

Hey @box4mm,

finally, I found some time to run the wireguard throughput test for you.

I took two TLsense i7 routers, installed Ubuntu on both of them and configured a Wireguard connection between them. Here's how it looks like

And here's how the topology looks like. As you see it's very simple - two routers connected with a cable.
This should be roughly equivalent to a typical client-server connection between your router at home and server somewhere on the internet (Mullvad or similar).

I opened 3 terminals for each router (6 in total).
On wireguard-server I executed
1. `iperf3 -s -p 5000` (first iperf server for download)
2. `iperf3 -s -p 5001` (second iperf server for upload test)
3. `htop` (to see memory/CPU utilization)

On wireguard-client I executed
1. `iperf3 -c 10.10.0.1 -t 100 -p 5000` (download test)
2. `iperf3 -c 10.10.0.1 -R -t 100 -p 5001` (reverse / upload test)
3. `htop` (to see memory/CPU utilization)

To summarize, I'm testing upload and download speed at the same time. Both connections are running through a wireguard tunnel.

The results are excellent. I'm getting a full gigabit in both directions simultaneously, with about 40% CPU utilization.
In practice, the upload/download speed is at 890-900Mbit/s in both directions due to the VPN overhead. That, of course, can't be helped, as this is how VPN works.
Without Wireguard, the throughput is 940Mbit/s in both directions. The "missing" 40-50Mbit/s is the encapsulation overhead from Wireguard.

I made a screenshot of the 6 terminals while the test was running.

The server is on top, and the client is on the bottom.

I planned to run this test again on APU, but it takes quite a lot of time to set it up, so it will have to wait until next weekend.

Permalänk
Medlem
Skrivet av teklager:

Hey @box4mm,

finally, I found some time to run the wireguard throughput test for you.

I took two TLsense i7 routers, installed Ubuntu on both of them and configured a Wireguard connection between them. Here's how it looks like
https://i.imgur.com/zL6C8B7.jpg

And here's how the topology looks like. As you see it's very simple - two routers connected with a cable.
This should be roughly equivalent to a typical client-server connection between your router at home and server somewhere on the internet (Mullvad or similar).

https://i.imgur.com/zlro8XI.png

I opened 3 terminals for each router (6 in total).
On wireguard-server I executed
1. `iperf3 -s -p 5000` (first iperf server for download)
2. `iperf3 -s -p 5001` (second iperf server for upload test)
3. `htop` (to see memory/CPU utilization)

On wireguard-client I executed
1. `iperf3 -c 10.10.0.1 -t 100 -p 5000` (download test)
2. `iperf3 -c 10.10.0.1 -R -t 100 -p 5001` (reverse / upload test)
3. `htop` (to see memory/CPU utilization)

To summarize, I'm testing upload and download speed at the same time. Both connections are running through a wireguard tunnel.

The results are excellent. I'm getting a full gigabit in both directions simultaneously, with about 40% CPU utilization.
In practice, the upload/download speed is at 890-900Mbit/s in both directions due to the VPN overhead. That, of course, can't be helped, as this is how VPN works.
Without Wireguard, the throughput is 940Mbit/s in both directions. The "missing" 40-50Mbit/s is the encapsulation overhead from Wireguard.

I made a screenshot of the 6 terminals while the test was running. https://i.imgur.com/2QgGNev.png
The server is on top, and the client is on the bottom.

I planned to run this test again on APU, but it takes quite a lot of time to set it up, so it will have to wait until next weekend.

This is amazing, thank you for all that work!
Yes, with 40% CPU maybe the i7 is overkill, i understand these tests are for sure time consuming.
With ubuntu its probably the case that its so well implemented that the cpu does way less work then windows for example, or even openwrt/opnsense (just guessing).

My initial thoughts is, i work in IT etc, but im not sure i will be savvy enough to handle ubuntu as a mainrouter is it a modified version in some way for just alittle bit more usefriendly? Or maybe basic configuration is something that would be included somehow?

Otherwise ive been playing around with OPNSense and i must say its a really good GUI, perhaps Ubuntu has some "GUI-package" one can install for use with router? I dont know enough about these things.

If anyone else knows more about this, please feel free to help

Thanks

PS: Because it takes such a long time to setup these tests, maybe you can try TLSense i5 first, and if its possible to try with both Ubuntu and OPNSense, when theres time.
Im definately interested

EDIT: I see TLSense i5 out of stock, so ye then maybe "APU4C4" is the best next test :), or either APU maybe if they perform the same, then we would get a good idea of how the range from APU -> i5 -> i7 is in terms of performance

Permalänk
teklager
Skrivet av box4mm:

Yes, with 40% CPU maybe the i7 is overkill, i understand these tests are for sure time consuming.
With Ubuntu its probably the case that its so well implemented that the cpu does way less work then windows for example, or even openwrt/opnsense (just guessing).

I used Ubuntu because I had two boxes with already installed OS. I believe OpenWRT will have the same performance. In fact, I have one APU2 with OpenWRT so I can run another test quickly... more about this below.

Skrivet av box4mm:

My initial thoughts is, i work in IT etc, but im not sure i will be savvy enough to handle Ubuntu as a mainrouter is it a modified version in some way for just alittle bit more usefriendly? Or maybe basic configuration is something that would be included somehow?

I would not recommend Ubuntu on a router to anyone I used it only for benchmarking purposes. I suggest using OpenWRT for wireguard. It has a relatively nice web UI for configuring everything. There's no need to mess with the terminal. I find it pleasant to use.

Skrivet av box4mm:

EDIT: I see TLSense i5 out of stock, so ye then maybe "APU4C4" is the best next test :), or either APU maybe if they perform the same, then we would get a good idea of how the range from APU -> i5 -> i7 is in terms of performance

Yeah, the i5 are out, but they should be back in about 2 weeks. See this performance comparison for different models: https://teklager.se/en/knowledge-base/apu-vs-tlsense-cpu-perf... I believe i5 will run at gigbit both ways as well.

I executed a quick test with OpenWRT as a wireguard server. The download throughput was 750Mbit/s - more than I expected. The upload was at ~520Mbit/s. Interestingly, it's not identical. I suppose encryption is more performant than decryption. These tests weren't executed simultaneously. If I run upload and download at the same time, the numbers drop by half. I think this is a great result for a router that consumes 6W of power

Permalänk
Medlem
Skrivet av Kieeps:

Paketet som används i denna tråden till WG är mer en gui för att göra interfacet med GW, dom som jobbar på pfsense ogillar dock att man gör på detta sätt och rekommenderar att man ska vänta tills dom fått in WG i kernel (https://redmine.pfsense.org/issues/8786)
personligen funderar jag på att gå över till Opnsense eftersom dom känns mycket snabbare att implentera nya funktioner och uppdaterar oftare... Men vad vet jag... Pfsense är säkert bättre på nått sätt som känns orelevant för mej

Kör OpnSense här. Tokstabil brandvägg, aldrig strulat för mig. Har labbat lite med Wireguard på brandväggen och tycker det funkat bra hittills!
Då i form av att jag kopplar upp mig till mitt hemnätverk när jag är utanför hemmet. Har som sagt funkat kanon.

Permalänk
Medlem
Skrivet av teklager:

I used Ubuntu because I had two boxes with already installed OS. I believe OpenWRT will have the same performance. In fact, I have one APU2 with OpenWRT so I can run another test quickly... more about this below.

I would not recommend Ubuntu on a router to anyone I used it only for benchmarking purposes. I suggest using OpenWRT for wireguard. It has a relatively nice web UI for configuring everything. There's no need to mess with the terminal. I find it pleasant to use.

Yeah, the i5 are out, but they should be back in about 2 weeks. See this performance comparison for different models: https://teklager.se/en/knowledge-base/apu-vs-tlsense-cpu-perf... I believe i5 will run at gigbit both ways as well.

I executed a quick test with OpenWRT as a wireguard server. The download throughput was 750Mbit/s - more than I expected. The upload was at ~520Mbit/s. Interestingly, it's not identical. I suppose encryption is more performant than decryption. These tests weren't executed simultaneously. If I run upload and download at the same time, the numbers drop by half. I think this is a great result for a router that consumes 6W of power
https://i.imgur.com/kIlExH8.png

Awesome job your doing, makes me want to support you guys just for that
Yeah the i5 probably also does gigabit both ways, i guess i will have to wait however many weeks until its in stock

In the meantime ill be checking out opnsense, openwrt and other router OS you have listed.
OPNSense has a really nice UI indeed but it seems abit to much for simple homenetworking.

I do like however, that u can see the firewall log in realtime, are there similar feature with openwrt?

Thanks again for testing, I am correct in assuming i will have to wait for i5 correct?

EDIT: My plan is to use my current Asus 86u in AP-mode for wifi, i read openwrt is best for those kind of things?

Permalänk
Medlem
Skrivet av -=fredrik=-:

Kör OpnSense här. Tokstabil brandvägg, aldrig strulat för mig. Har labbat lite med Wireguard på brandväggen och tycker det funkat bra hittills!
Då i form av att jag kopplar upp mig till mitt hemnätverk när jag är utanför hemmet. Har som sagt funkat kanon.

opnsense är absolut ett alternativ, kommer försöka kika på flera alternativ i vm-miljö

Permalänk
Medlem
Skrivet av teklager:

I used Ubuntu because I had two boxes with already installed OS. I believe OpenWRT will have the same performance. In fact, I have one APU2 with OpenWRT so I can run another test quickly... more about this below.

I would not recommend Ubuntu on a router to anyone I used it only for benchmarking purposes. I suggest using OpenWRT for wireguard. It has a relatively nice web UI for configuring everything. There's no need to mess with the terminal. I find it pleasant to use.

Yeah, the i5 are out, but they should be back in about 2 weeks. See this performance comparison for different models: https://teklager.se/en/knowledge-base/apu-vs-tlsense-cpu-perf... I believe i5 will run at gigbit both ways as well.

I executed a quick test with OpenWRT as a wireguard server. The download throughput was 750Mbit/s - more than I expected. The upload was at ~520Mbit/s. Interestingly, it's not identical. I suppose encryption is more performant than decryption. These tests weren't executed simultaneously. If I run upload and download at the same time, the numbers drop by half. I think this is a great result for a router that consumes 6W of power
https://i.imgur.com/kIlExH8.png

For some reason i cant edit my previous post any more, thats why im quoting again.
So is it correct that there is no possibility for more testing with the TLSense i5 until roughly 2weeks?

Permalänk
teklager
Skrivet av box4mm:

Awesome job your doing, makes me want to support you guys just for that
Yeah the i5 probably also does gigabit both ways, i guess i will have to wait however many weeks until its in stock

Thanks for your kind words! Yeah, i5 should be back in stock in about 2-3 weeks.

Skrivet av box4mm:

In the meantime ill be checking out opnsense, openwrt and other router OS you have listed.
OPNSense has a really nice UI indeed but it seems abit to much for simple homenetworking.

Not sure if OPNsense is "too much". To me, it seems quite intuitive to use. Maybe other users can also comment on this?

Skrivet av box4mm:

I do like however, that u can see the firewall log in realtime, are there similar feature with openwrt?

I think this is where OpenWRT is lacking. Firewall logs aren't enabled by default, and it's not super straightforward to get a log of dropped packets. It's possible to do, but the user experience is far from OPNSense in this area.

Skrivet av box4mm:

Thanks again for testing, I am correct in assuming i will have to wait for i5 correct?

Yes, some patience is required here

Skrivet av box4mm:

EDIT: My plan is to use my current Asus 86u in AP-mode for wifi, i read openwrt is best for those kind of things?

If you plan to use your Asus as an access point, then you can use any router OS.
If you would want to have WiFi inside of the main router than I would recommend OpenWRT because it has much better WiFi support.

Permalänk
Medlem
Skrivet av teklager:

Thanks for your kind words! Yeah, i5 should be back in stock in about 2-3 weeks.

Not sure if OPNsense is "too much". To me, it seems quite intuitive to use. Maybe other users can also comment on this?

I think this is where OpenWRT is lacking. Firewall logs aren't enabled by default, and it's not super straightforward to get a log of dropped packets. It's possible to do, but the user experience is far from OPNSense in this area.

Yes, some patience is required here

If you plan to use your Asus as an access point, then you can use any router OS.
If you would want to have WiFi inside of the main router than I would recommend OpenWRT because it has much better WiFi support.

Would this mini-pcie card work with TLSense i5 or i7? I guess so
http://www.commell.com.tw/Product/Peripheral/PCI%20Express%20...

Edit: Do you still have the testsetup running?
Would be interesting to try bidirectional iperf with wireguard on opnsense / openwrt, the reason is that they are both userspace and not in kernel like ubuntu, so would be nice to know if its possible in those, since in windows for example its not, doesnt matter what my hardware is

Permalänk
teklager
Skrivet av box4mm:

Would this mini-pcie card work with TLSense i5 or i7? I guess so
http://www.commell.com.tw/Product/Peripheral/PCI%20Express%20...

In principle, it should work. It runs on a PCI interface that is present in the TLSense i5/i7 box.
I have not tested this specific card so I can't guarantee that it's compatible, but as far as I can tell, it should be fine.

Skrivet av box4mm:

Edit: Do you still have the testsetup running?
Would be interesting to try bidirectional iperf with wireguard on opnsense / openwrt, the reason is that they are both userspace and not in kernel like ubuntu, so would be nice to know if its possible in those, since in windows for example its not, doesnt matter what my hardware is

I can run some more tests this weekend

Permalänk
Medlem

Jag körde lite tester nu när jag fått min i7 från Teklager och jag kör mot mullvad. Har något år till på nordvpn som jag kan testa med om jag vill, men jag tror inte att jag ids om ingen verkligen vill ha det.

Jag har alltså en PFSense på hårdvara. Det är Teklagers snabbaste tysta burk med i7. Den andra "burken" är i realiteten en virtuell applance på VMware.

Jag får bättre prestanda via Wireguard än med OpenVPN när jag har bättre hårdvara.

Speedtestet får jag genom att ladda hem en "fil" från Tele2.
wget -O /dev/null http://speedtest.tele2.net/10GB.zip --bind-address x.x.x.x

Beroende på vilken IP adress jag binder det till så kommer jag skickas via olika vägar.

Hur som helst. PF = PFSense på hårdvara core i7. VM = VMware.
PF OVPN 110s
PF WG 100s
PF Inet 94s

VM OVPN 109s
VM WG 115s
VM Inet 96s

Inet är naturligtvis när det går rakt ut på internet.

För att få bättre precision tänkte jag köra hem en 100GB stor "fil". Återkommer när jag då vet mer.

Helt klart är i alla fall att via Mullvad får man riktigt bra prestanda och kan i stort fylla en gigabitlänk. Det är såklart viss overhead med VPNm så att det inte blir exakt lika bra är självklart.
Dessutom bor jag på söder, så min latency är riktigt låg mot Mullvads servrar som enligt platstjänsten ska ligga någonstans i Årsta. Dvs, det är geografiskt väldigt nära och ping indikerar oerhört låg latency.

Permalänk
Medlem

Jag pravade med en 100GB randomiserad fil.
PF OVPN 19m 33s
PF WG 16m 58s
PF Inet 16m 7s

VM OVPN 20m 21s
VM WG ---
VM Inet 16m 43s

WG på min VMware är helt enkelt lite för instabil. Filen kommer ned, men inte utan att det blir ett gäng avbrott då den återansluter.

Wireguard har lägre latency och bättre prestanda mot Mullvad. Givetvis kan begränsningar ligga på mullvad-sidan, men jag tror att det helt enkelt är wireguard som är snabbare än OpenVPN.

Dock kan instabiliteten på svagare CPU (eller är det minne) göra att man väljer att avvakta med att använda Wireguard i PFSense förrän den läggs in i kernel.

Permalänk
Medlem
Skrivet av Talisker00:

Jag provade med en 100GB randomiserad fil.
PF OVPN 19m 33s
PF WG 16m 58s
PF Inet 16m 7s

VM OVPN 20m 21s
VM WG ---
VM Inet 16m 43s

WG på min VMware är helt enkelt lite för instabil. Filen kommer ned, men inte utan att det blir ett gäng avbrott då den återansluter.

Wireguard har lägre latency och bättre prestanda mot Mullvad. Givetvis kan begränsningar ligga på mullvad-sidan, men jag tror att det helt enkelt är wireguard som är snabbare än OpenVPN.

Dock kan instabiliteten på svagare CPU (eller är det minne) göra att man väljer att avvakta med att använda Wireguard i PFSense förrän den läggs in i kernel.

PF OVPN 19m 33s
PF WG 16m 58s
PF Inet 16m 7s

VM OVPN 20m 21s
VM WG 20m 32s
VM Inet 16m 43s

Nu fick jag igenom en så stor fil på VM WG.
------------------------------------------------------------
Edit:
Komplettering på den snabba PF-brandväggen:
OpenVPN mot Mullvad 1 TB stor fil:
3h 14m

Wireguard mot Mullvad 1 TB stor fil:
2h 47m

Ja, Wireguard är helt klart snabbare än OpenVPN på en kraftig burk, men är det värt det lilla extra besväret? Jag tycker det.

Permalänk
Medlem
Skrivet av Talisker00:

PF OVPN 19m 33s
PF WG 16m 58s
PF Inet 16m 7s

VM OVPN 20m 21s
VM WG 20m 32s
VM Inet 16m 43s

Nu fick jag igenom en så stor fil på VM WG.
------------------------------------------------------------
Edit:
Komplettering på den snabba PF-brandväggen:
OpenVPN mot Mullvad 1 TB stor fil:
3h 14m

Wireguard mot Mullvad 1 TB stor fil:
2h 47m

Ja, Wireguard är helt klart snabbare än OpenVPN på en kraftig burk, men är det värt det lilla extra besväret? Jag tycker det.

intressanta uppgifter Talisker00 , jag kör OpenVPN på en 250/250 lina (pfSense i5 3450 8 GB ram med Intels nic i) även om jag anser WireGuard har chanser att bli framtidens VPN protokoll så känner jag ingen brådska att implementera det av samma åsikt som @gonace skriver här

Permalänk
Medlem

Tackar för infon indeed Talisker00, jag har också fått min hårdvara och jag har samma resultat som dig, galet bra prestanda på wireguard mullvad.
Jag får ut 1000/1000 samtidigt, minus overhead

Permalänk
Medlem

Jag kör Mullvad via min router ifrån Teklager med Wireguard. Dock behöver jag öppna en port i Openwrt för att få bra anslutning till mitt Xbox. Är det någon som kan förklara lite snabbt hur man gör och vilka värden man ska trycka i?

Får en port ifrån mullvad och en port ifrån Xboxet.

Permalänk
Medlem
Skrivet av teklager:

Hey @box4mm,

finally, I found some time to run the wireguard throughput test for you.

I took two TLsense i7 routers, installed Ubuntu on both of them and configured a Wireguard connection between them. Here's how it looks like
https://i.imgur.com/zL6C8B7.jpg

And here's how the topology looks like. As you see it's very simple - two routers connected with a cable.
This should be roughly equivalent to a typical client-server connection between your router at home and server somewhere on the internet (Mullvad or similar).

https://i.imgur.com/zlro8XI.png

I opened 3 terminals for each router (6 in total).
On wireguard-server I executed
1. `iperf3 -s -p 5000` (first iperf server for download)
2. `iperf3 -s -p 5001` (second iperf server for upload test)
3. `htop` (to see memory/CPU utilization)

On wireguard-client I executed
1. `iperf3 -c 10.10.0.1 -t 100 -p 5000` (download test)
2. `iperf3 -c 10.10.0.1 -R -t 100 -p 5001` (reverse / upload test)
3. `htop` (to see memory/CPU utilization)

To summarize, I'm testing upload and download speed at the same time. Both connections are running through a wireguard tunnel.

The results are excellent. I'm getting a full gigabit in both directions simultaneously, with about 40% CPU utilization.
In practice, the upload/download speed is at 890-900Mbit/s in both directions due to the VPN overhead. That, of course, can't be helped, as this is how VPN works.
Without Wireguard, the throughput is 940Mbit/s in both directions. The "missing" 40-50Mbit/s is the encapsulation overhead from Wireguard.

I made a screenshot of the 6 terminals while the test was running. https://i.imgur.com/2QgGNev.png
The server is on top, and the client is on the bottom.

I planned to run this test again on APU, but it takes quite a lot of time to set it up, so it will have to wait until next weekend.

What CPU is that, and did you perform the same test using OpenVPN, if so what speed did you then get?

Permalänk
Medlem

Har inte riktigt haft tid för detta men nu börjar det klia i fingrarna vad är bästa OS för att köra WG på en Custom Router I dagsläget? Open WRT har fungerat riktigt bra men är hyfsat less på att interface och annat känns så begränsat och rörigt. Antar att det fortfarande är rätt dött lopp mellan pfsense och opnsense? Initialt var det väl samma kodbas men vet inte hur mkt dom skiljer sig åt idag annat än UI.

Permalänk
teklager
Skrivet av No_Jah:

What CPU is that, and did you perform the same test using OpenVPN, if so what speed did you then get?

That was `i7-4510U` CPU. I remember running a similar test with OpenVPN and NordVPN about 2 years ago. At that time I believe I got 700Mbit/s but suspect the limitation was on the NordVPN side or the ISP side (not verified though).

Skrivet av improwise:

Har inte riktigt haft tid för detta men nu börjar det klia i fingrarna vad är bästa OS för att köra WG på en Custom Router I dagsläget? Open WRT har fungerat riktigt bra men är hyfsat less på att interface och annat känns så begränsat och rörigt. Antar att det fortfarande är rätt dött lopp mellan pfsense och opnsense? Initialt var det väl samma kodbas men vet inte hur mkt dom skiljer sig åt idag annat än UI.

I have some thoughts on this: https://teklager.se/en/pfsense-vs-opnsense/

Permalänk
Medlem
Skrivet av teklager:

That was `i7-4510U` CPU. I remember running a similar test with OpenVPN and NordVPN about 2 years ago. At that time I believe I got 700Mbit/s but suspect the limitation was on the NordVPN side or the ISP side (not verified though).

I have some thoughts on this: https://teklager.se/en/pfsense-vs-opnsense/

Ok, that CPU seems to be very similar to the CPU I use (Xeon E3-1220L v2) performnce wise. I didn't know it could potentially reach that kind of speeds.
https://www.cpubenchmark.net/compare/Intel-i7-4510U-vs-Intel-...

Permalänk
Medlem
Skrivet av teklager:

Any thoughts on the WireGuard implementations specifically? My network setup at home isn't really all that complex these days when everything is in the cloud