Here’s a quick list of helpful commands to use when trying to debug the OpenWrt routing tables. I ran across some issues recently where a cluster running metallb in L2 mode was failing to get traffic, and the first place I looked was the router. Here are a few commands that ended up being very helpful - in my case to prove that the router was completely fine.

Print all of the firewall rules on uci

uci show firewall

Print all of the traffic rules

fw4 print

fw4 is sort of like iptables but less ancient.

Note: fw4 print also does some basic validation of whether the config are valid, since uci can add parameters that are invalid which will cause those config sections to be ignored. BUT there ARE some changes that are valid in UCI but will be considered invalid by fw4! like log_limit

Manually edit the uci firewall rules

vi /etc/config/firewall

This can be very helpful in order to add logging (and log limits) to specific rules to see when they’re triggered.

Reload the firewall config manually

/etc/init.d/firewall reload

Manual reload of the config is only really necessary when editing the firewall config by hand

Run ip neigh to test ARP route

Install the package

opkg install ip-full
opkg install iputils-arping

And then check if the ARP list is correct

ip neigh

The traffic for the ARP on a host can also be watched with

sudo tcpdump -i wlp3s0 -n -e -vvv arp

(replace wlp3s0 with whatever your network interface is called.)

If the ARP list isn’t right, it’s possible to add a permanent entry with ip neigh replace, but it’s worth considering whether this is a workable solution long-term.

In my case, just knowing that the ARP list was wrong led me to find that one of my network cards just wasn’t responding properly to ARP requests. Rather than add a manual entry, I specified a preferred node for managing my metallb load balancer that didn’t have this problem.