AdGuard Home: Taming the DNS Loop

Installing AdGuard Home (AGH) via the iStoreOS app store was straightforward. The initial setup wizard (:3000) guided me through setting ports. I changed AGH’s DNS port to :5353 to avoid conflict with iStoreOS’s built-in dnsmasq (running on :53).

The goal: Clients -> iStoreOS (dnsmasq :53) -> AdGuard Home (:5353) -> Public DNS (e.g., 1.1.1.1).

Configuration involved:

1. dnsmasq: Set it to not forward upstream queries found in /etc/resolv.conf, and explicitly forward all queries to AGH listening on the loopback address: server=127.0.0.1#5353.
2. AdGuard Home: Set its upstream DNS servers to Cloudflare/Google.
3. DHCP: Configure iStoreOS’s DHCP server (Option 6) to advertise the router’s IP (10.0.0.1) as the DNS server for all clients.

Initially, this resulted in DNS resolution failures and context deadline exceeded errors in the AGH logs. Diagnosis: a DNS loop. Dnsmasq was forwarding to AGH, but AGH, possibly due to a misconfiguration or startup order issue, was somehow trying to query dnsmasq back.

Adding to the confusion was a “ghost configuration.” I had initially tried installing AGH via a curl script before using the app store version. This seemed to leave conflicting settings. The breakthrough came almost accidentally: running the AdGuard Home binary directly from the command line with the -s run flag (./AdGuardHome -s run) triggered some kind of self-reset or reinitialization. After this, restarting the service properly allowed the correct DNS chain (dnsmasq -> AGH -> Public) to establish, and DNS resolution started working beautifully across my network.

The Network Speed Crisis: A Virtualization Bottleneck

With DNS filtering active, I ran a speed test. Disaster! My gigabit fiber connection, which previously delivered ~950 Mbps, was now capped at a measly 400 Mbps.

The troubleshooting began:

• Disable AdGuard Home: No change. DNS filtering wasn’t the bottleneck.
• Check Physical Links: Used ethtool eth0 and ethtool eth1 within the iStoreOS VM to confirm the virtual NICs were negotiating at 2500 Mbps with the PVE bridges. Links were fine.
• Enable Software Flow Offloading: Toggled this common OpenWrt optimization. No significant change.

The problem had to lie deeper, likely within the virtualization layer itself. Research pointed towards a common issue with high-speed networking in VMs: the CPU struggling to handle the sheer volume of network packets being processed by a single core for the virtual NIC.

The Solution: Multiqueue

Proxmox VE (and the underlying KVM/QEMU) supports a feature called Multiqueue VirtIO-Net. This allows the network packet processing load for a virtual NIC to be spread across multiple vCPU cores assigned to the VM.

1. Shut down the iStoreOS VM.
2. In the PVE web UI, go to the VM’s “Hardware” tab.
3. Select the VirtIO network device corresponding to the LAN interface (net0 in my case, connected to vmbr1).
4. Click “Edit”.
5. Check the “Multiqueue” box.
6. Set the “Queues” number equal to the number of vCPU cores assigned to the VM (e.g., if you gave it 4 cores, set Queues to 4).
7. Repeat steps 3-6 for the WAN network device (net1, connected to vmbr0).
8. Start the iStoreOS VM.

Ran the speed test again. Success! Speeds jumped right back up to the ~950 Mbps range. The virtualization bottleneck was eliminated.

Tailscale Triumph: Conquering Remote Access

My final major goal was setting up remote access, primarily to manage services like my NAS (planned OMV VM) and potentially the router itself when away from home. I opted for Tailscale due to its ease of use and availability in the iStoreOS app store.

Installation was simple. Following the command-line instructions (tailscale up --advertise-routes=10.0.0.0/24) connected the router to my Tailnet and advertised my home LAN subnet. However, while the router appeared online in the Tailscale admin console, I couldn’t access any devices on my home LAN (like 10.0.0.1) from my phone connected via Tailscale.

Troubleshooting Round 1: Firewall

Pinging 10.0.0.1 from my phone resulted in “Destination Port Unreachable.” This screamed firewall. iStoreOS hadn’t automatically created a firewall zone or rules for the new tailscale0 interface.

Troubleshooting Round 2: Interface & Zone

Following guidance from GitHub issues, I manually edited /etc/config/network to define the tailscale0 interface:

config interface 'tailscale'
        option proto 'none'
        option device 'tailscale0'

Then, in the LuCI Firewall settings (Network -> Firewall), I:

1. Created a new firewall zone named tailscale.
2. Set its Input/Output/Forward policies to ACCEPT.
3. Assigned the tailscale0 network interface to this zone.
4. Crucially, created a Forwarding rule allowing traffic from the tailscale zone to the lan zone.

Saved and applied. Tried connecting again. The “Port Unreachable” error was gone, but now attempts to access 10.0.0.1just… hung. Loading indefinitely.

Troubleshooting Round 3: MTU/MSS

This new symptom – indefinite loading – strongly suggested an MTU (Maximum Transmission Unit) mismatch problem, common with VPNs and tunnels like Tailscale. Packets were likely getting fragmented or dropped because their size exceeded what the tunnel could handle. The standard fix is MSS Clamping (Maximum Segment Size), where the router rewrites TCP packet headers to request smaller segment sizes.

The Final Fix: iptables MSS Clamping

Added the following rules via Network -> Firewall -> Custom Rules:

# Tailscale MSS Clamping for IPv4
iptables -t mangle -A FORWARD -o tailscale0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

# Tailscale MSS Clamping for IPv6 (if using IPv6 with Tailscale)
ip6tables -t mangle -A FORWARD -o tailscale0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Restarted the firewall. Tried accessing 10.0.0.1 from my phone via Tailscale one last time.

Victory! The iStoreOS login page loaded instantly. Remote access was finally conquered.

Conclusion: A System Forged in Debugging

This journey, from a simple IPv6 issue to a fully virtualized N100 soft router with custom services and robust remote access, was far more challenging than I initially anticipated. Every step seemed to uncover a new problem, requiring deep dives into networking concepts, Linux internals, and virtualization quirks.

However, each problem solved solidified my understanding and resulted in a final system that is incredibly powerful, stable, and perfectly tailored to my needs. The N100 platform with Proxmox and iStoreOS has proven to be an outstanding foundation. The debugging war is over, and the era of enjoying a truly capable home lab has begun.