DHCP not working - Transmit (Tx) checksum offload problem (#1) · Issues · Greg Olsen / lxc-devuan

Closed

Opened 2016-06-07T10:23Z by Greg Olsen @gregolsen

DHCP not working - Transmit (Tx) checksum offload problem

Synopsis: Container can't get DHCP assigned IP address

The failed sequence looks like this:

Client DHCPDISCOVER(lxcbr0), followed by Server DHCPOFFER(lxcbr0)
Client does not follow with DHCPREQUEST as required per protocol
Consequently neither will DHCPACK be sent from server to acknowledge

Don't blame Devuan for this.

It's an old issue related to Linux transmit (Tx) checksum offload handling for virtual devices such as:

bridges, veth devices, etc.

For a more in-depth explanation see Root Cause below.

Solution

Here's the solution I've used reliably for many years with my Xen VMs, and more recently with LXC:

Install ethtool:
```
apt-get install -y ethtool
```
Turn off Tx checksum offloading on the LXC bridge:

There's more than one way to do this, but the cleanest way I've found is to simply add an "up" command to the bridge interface definition.

Example 'up' command ($IFACE = bridge interface):
```
up /sbin/ethtool -K $IFACE tx off  # <== TURN OFF TX CHECKSUM OFFLOAD
```
I manually define all my bridges.

Example bridge definition in/etc/network/interfaces:

auto lxcbr0 iface lxcbr0 inet static bridge_fd 0 bridge_stp 0 bridge_ports none address 10.0.0.1 netmask 255.255.0.0 up /sbin/ethtool -K $IFACE tx off # <== TURN OFF TX CHECKSUM OFFLOAD ```

<br>
- *CentOS/RHEL/SUSE* example (**untested**):

     Add to interface config `/etc/sysconfig/network/ifcfg-lxcbr0`:

     ```shell

ETHTOOL_OPTIONS='-K iface tx off' ```

Other solutions include, setting an iptables rule on either the POSTROUTING or OUTPUT chains of the mangle table. At this time I don't do this, therefore I don't have an example to provide.

IMHO, the iptables rule solution might be prone to breakage as subsequent rule changes can lose the checksum rule.

However if maximum performance on a prod server is important, I suggest using the OUTPUT chain of the mangle table to calculate checksums for DHCP.

Root Cause

The root cause of the issue is a missing or incorrect checksum for packets transmitted from virtual devices, and a DHCLIENT that rejects packets with bad checksums.

There's a lot of confusion about the cause.

First and foremost, know this is not a problem with Devuan, and nor is it a problem with LXC.

It's an old issue related to Linux transmit (Tx) checksum offload handling for virtual devices:

bridge's, veth's, etc.

The kernel Linux defers calculating checksums until packet egress on physical devices only.

Therefore this is not a kernel bug, but a design decision.

As I understand it, it's because there's no need to calculate checksum's when packets transmit via in-memory copy, as is the case for virtual devices. Makes sense. Why consume CPU performing the calculation when there's no physical network?

There's also the partial checksum offload feature which was introduced as a performance optimization. I don't fully understand it. Apparently it has resulted in incorrect checksums that can cause DHCP to reject packets.

Disabling Tx checksum offload circumvents the issue.

On the surface it may seem like there's a problem with the DHCLIENT , especially if other VMs/Containers work fine on the same bridge. However all this implies is some patching has been done to accept packets with bad checksums. Depending on ones perspective this is either good, or it's bad. IMHO, both perspectives have some merit.

DHCP not working - Transmit (Tx) checksum offload problem

Synopsis: Container can't get DHCP assigned IP address

Solution

Root Cause

Related links