Two NICS + One subnet + Linux = Headaches

Meethune (mostly) and I have been trying to wrap up the final bits of our new Xen toy at the office, and had run into a roadblock on the networking end of things. We finally got it all figured out today and, seeing as he’s beaten me to the punch,here’s his writeup

Edit - More details:

Our goal for the server was twofold; familiarize ourselves with Xen and use Xen to allow us to more effectively test various software, servers, etc. To this end, we did a fresh Gentoo install (sans nptl) and attempted to use a dual-NIC setup to segregate Xen/system net traffic. We discovered very quickly that Linux (2.6.{11-12}) does not allow for two NICs on the same subnet out of the box.

Our first attempt(s) were complete failures resulting in no DHCP address and no pings for eth1 (second NIC) while eth0 functioned perfectly. After a bit of digging using tcpdump, we found that the packets from eth1 simply weren’t getting routed directly since the kernel was trying to push them all thru eth0. Meethune did a bit of digging and re-discovered the iproute2 utility and more specifically the Split Access section of their How-To.

We were able to get eth1 to communicate with any of the machines on our LAN by using:

ip route add default via gateway.ip.address dev eth1 table XEN
ip rule add from eth1.ip.address table XEN

but every time we tried to get outside our net via eth1, tcpdump would show the machine completely bypassing our router and doing ARP requests directly.

I did a bit of digging into the kernel network parameters and found ‘arp_filter’. Basically, the kernel allows any interface on the system to reply to an ARP request for any interface on the system. The logic (as stated in Documentation/networking/ip-sysctl.txt) is that since IP addresses are “owned by the complete host”, this behaviour “increases the chance of successful communication”. I’m not sure I really agree with this, but I can’t say I’m up on all the arguments and it doesn’t matter here anyway.

In short, we added net.ipv4.conf.default.arp_filter = 1 to /etc/sysctl.conf and we now have (hopefully) a fully functioning setup…