Syed Jahanzaib – سید جہانزیب – Personal Blog to Share Knowledge !

February 14, 2026

Building ISP-Grade DNS Infrastructure Using DNSDIST + VRRP (50K~100K Users Design Model)

Filed under: Linux Related — Tags: , , , , , , , , — Syed Jahanzaib / Pinochio~:) @ 7:11 PM

Building ISP-Grade DNS Infrastructure Using DNSDIST + VRRP (50K~100K Users Design Model)

~under review

  • Author: Syed Jahanzaib ~A Humble Human being! nothing else 😊
  • Platform: aacable.wordpress.com
  • Category: ISP Infrastructure / DNS Engineering
  • Audience: ISP Engineers, NOC Teams, Network Architects

⚠️ Disclaimer & Note on Writing Style

Every network environment is unique. A solution that works effectively in one infrastructure may require modification in another. Readers are strongly encouraged to understand the underlying concepts and adapt the guidance according to their own architecture, operational policies, and risk tolerance.

Blind copy-paste implementation without proper validation, testing, and change management is never recommended — especially in production environments. Always ensure proper backups and risk assessment before applying any configuration.

The content shared here is based on hands-on experience from real-world deployments, ISP environments, lab testing, and continuous learning. While I strive for technical accuracy, no technical implementation is entirely free from the possibility of error. Constructive discussion and alternative approaches are always welcome.

Due to professional commitments, it is not always feasible to publish highly detailed or multi-part write-ups. The technical logic and implementation details are written based on my own practical experience. AI tools such as ChatGPT are used only to refine grammar, structure, and presentation — not to generate the core technical concepts.

This blog is not intended for client acquisition or follower growth. It exists solely to share practical knowledge and real-world experience with the community.

Thank you for your understanding and continued support.

📌 Article Roadmap >What This Guide Covers

In this detailed ISP-grade DNS architecture guide, I have covered the following sections:

  1. Introduction & Design Objectives
    Explains why traditional DNS fails in ISP networks and defines the core engineering objectives for a scalable, highly available DNS architecture.
  2. Scope & Audience
    Clarifies what is included in this guide and who will benefit most from it.
  3. High-Level Architecture Overview
    Presents the recommended DNS infrastructure model using dnsdist + VRRP, including role separation and failure domains.
  4. Capacity Planning & Traffic Expectations
    Discusses realistic QPS and sizing models for 50K–100K subscribers, including cache hit assumptions and peak load calculations.
  5. dnsdist Frontend Configuration
    Covers dnsdist installation, load-balancing policy selection, backend pools, rate limiting and health checks.
  6. Recursive & Authoritative Server Setup
    Provides detailed guidance for configuring recursive and authoritative BIND instances, including isolation and security hardening.
  7. Keepalived + VRRP High Availability Setup
    Walks through VRRP configuration, priority planning, timers, split-brain prevention, and process tracking.
  8. Kernel & OS Level Optimizations
    Covers performance tuning at the OS level (network, limits, buffer sizes) for high-packet-rate DNS workloads.
  9. Monitoring & Observability Architecture
    Prescribes a monitoring stack with metrics, dashboards and alerting targets for production operations.
  10. Scaling Beyond 100K Users
    Explains how to grow the architecture horizontally and introduces future-ready concepts like Anycast and multi-datacenter distribution.
  11. Operational Workflows & Maintenance
    Shares best practices for rolling upgrades, backups, failover testing, and lifecycle management.
  12. FAQ & Edge-Case Scenarios
    Answers common implementation questions and illustrates practical traffic-routing examples.
  13. Appendix / Production-Ready Config Snippets
    Includes tested, copy-ready configuration examples for dnsdist, Keepalived and BIND.

Introduction

In most Pakistani cable-net ISPs, DNS is treated as a secondary service , until it fails. When DNS fails, customers report “Internet not working” even though PPPoE is connected and routing is fine.

DNS is core infrastructure. For ISPs serving 50,000~100,000+ subscribers, DNS must be:

  • Highly available
  • Scalable
  • Secure
  • Monitored
  • Redundant

Design Objectives & Scope

  1. Design Objectives

The objective of this DNS architecture is to build a production-grade, high-availability, scalable DNS infrastructure suitable for medium to large ISPs (50,000–100,000 subscribers), with clear separation of roles, deterministic failover behavior, and measurable performance boundaries.

This design is built around the following core engineering principles:

1.1 Infrastructure-Level Redundancy

Failover must not depend on:

  • Subscriber CPE behavior
  • Operating system DNS retry timers
  • Application-layer retries

Redundancy must be handled at the infrastructure level using:

  • VRRP floating IP
  • Dual dnsdist frontend nodes
  • Backend health checks

Failover target: ≤ 3 seconds convergence.

1.2 Separation of Recursive and Authoritative Roles

Recursive and Authoritative DNS must not coexist on the same server in ISP-scale deployments.

This design enforces:

  • Dedicated authoritative server(s)
  • Dedicated recursive server pool
  • Controlled routing via dnsdist

Benefits:

  • Security isolation
  • Independent performance tuning
  • Contained failure domains
  • Clear operational visibility

1.3 Horizontal Scalability

The architecture must allow:

  • Adding new recursive servers without service interruption
  • Increasing QPS handling capacity without redesign
  • Backend pool expansion without client configuration change

Scaling must be horizontal-first, not vertical-only.

1.4 Deterministic Failover

Failover logic must be:

  • Script-based
  • Process-aware
  • Health-check driven
  • Predictable under load

VRRP must:

  • Immediately relinquish VIP if dnsdist stops
  • Promote standby node within controlled detection interval

1.5 Abuse Resistance & Operational Hardening

The DNS layer must include:

  • Rate limiting
  • ANY query suppression
  • Backend health checks
  • ACL-based query restriction
  • Recursive exposure protection

This prevents:

  • Amplification abuse
  • Internal malware flooding
  • Resource exhaustion attacks
  • Backend overload during update storms

1.6 Performance Measurability

The system must allow:

  • QPS measurement
  • Backend latency tracking
  • Cache hit ratio monitoring
  • Failover verification testing
  • Resource utilization visibility

No production DNS infrastructure should operate without measurable metrics.

  1. Scope of This Deployment Blueprint

This document covers:

  • Full deployment sequence from OS preparation to HA activation
  • dnsdist frontend configuration with aggressive tuning
  • BIND authoritative configuration
  • BIND recursive configuration
  • Keepalived VRRP configuration
  • Kernel-level performance tuning
  • Capacity planning logic
  • Failure testing methodology
  • Production hardening recommendations
  1. Out of Scope (Explicitly)

The following are not covered in this blueprint:

  • Global Anycast BGP-based DNS distribution
  • DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT) frontend implementation
  • Multi-datacenter geo-distributed architecture
  • Commercial DNS hardware appliance comparison benchmarking
  • DNSSEC zone signing strategy

These may be addressed in future parts.

  1. Intended Audience

This document is intended for:

  • ISP Network Architects
  • NOC Engineers
  • Systems Administrators
  • Broadband Infrastructure Operators
  • Technical leads in 50K–100K subscriber environments

This is not a beginner tutorial.
It assumes familiarity with:

  • Linux system administration
  • BIND
  • Networking fundamentals
  • VRRP
  • Basic ISP architecture
  1. Expected Outcome

After implementing this design, the ISP should achieve:

  • Infrastructure-level DNS high availability
  • Predictable failover behavior
  • Controlled recursive exposure
  • Measurable QPS performance
  • Reduced subscriber outage perception
  • Scalable DNS backend architecture

DNS transitions from:

“Just another Linux service”

to

“Core ISP control-plane infrastructure.”

DNSDIST! what is it?

This guide explains how to build a professional DNS architecture using:

  • DNSDIST as frontend DNS load balancer
  • Recursive / Authoritative separation <<<
  • VRRP-based High Availability
  • Packet cache & rate limiting
  • Scalable backend design

🔹Is DNSDISTIndustry-Style? Or Hobby Level?

DNSDIST is absolutely industry-grade.

It is:

  • Developed by PowerDNS
  • Used by:
    • Large hosting providers
    • Cloud providers
    • IX-level DNS infrastructures
    • Serious ISPs
  • Designed specifically for:
    • High QPS
    • DNS DDoS mitigation
    • Load balancing authoritative & recursive farms

This is NOT a lab tool.
It is widely deployed in production worldwide.

Key Architectural Shift

Old Model:
Redundancy at edge (client).

New Model:
Redundancy at core (infrastructure).

That is the fundamental upgrade in DNS architecture philosophy.

🔹Recommended Architecture for 50k+ ISP

Minimum Safe Production Design

  • 2x dnsdist (HA)
  • 3–4x Recursive Servers
  • 2x Authoritative Servers
  • Separate VLANs
  • Monitoring + Rate limiting

🔹 Hardware Guideline (Recursive)

Per node:

  • 8–16 CPU cores
  • 32–64 GB RAM
  • NVMe (for logs)
  • 10G NIC preferred

DNS is mostly CPU + RAM heavy (cache efficiency matters).

🔹 Why DNSDIST Becomes Useful at 50k+ Scale

Without DNSDIST:

  • Clients directly hit recursive
  • No centralized rate limiting
  • No traffic shaping
  • Harder to isolate DDoS
  • Hard to scale cleanly

With DNSDIST:

✔ Central traffic control
✔ Backend pool management
✔ Active health checks
✔ Per-IP QPS limiting
✔ Easy horizontal scaling
✔ Easy separation (auth vs rec)

🔹 What Serious ISPs Actually Do

At this size, typical models are:

  • Model A – DNSDIST+ Unbound/BIND cluster

Very common

  • Model B – Anycast DNS (advanced tier)

Used by larger national ISPs

  • Model C – Appliance-based (Infoblox, F5 DNS, etc.)

Expensive, enterprise heavy

DNSDISTsits between open-source and enterprise appliance , very powerful balance.

🔹 Would I Recommend dnsdist for 50k+ ISP?

Yes > if:

  • You want scalable architecture
  • You want control
  • You want DDoS handling layer
  • You want future growth to 150k–200k users

No > if:

  • Very small budget
  • No in-house Linux expertise
  • No monitoring culture

🔹 Strategic Advice

At 50k+ subscribers:

  • Single DNS server is negligence
  • Single dnsdist is risky
  • Proper HA + scaling is mandatory

DNS outage at this scale = full network outage perception.

🔹 Final Verdict

For 50k+ ISP:

DNSDIST is:
✔ Industry proven
✔ Production ready
✔ Cost effective
✔ Scalable

  • It is not overkill.
  • It is appropriate engineering.

Traditional DNS Models in Pakistani Cable ISPs

Executive Context – The Pakistani Cable ISP Reality

In many Pakistani cable-net environments:

  • MikroTik PPPoE NAS handles subscribers
  • RADIUS authenticates
  • One or two BIND servers provide DNS
  • No frontend load balancer
  • No recursive/authoritative separation
  • No QPS monitoring
  • No health checks

Common symptoms at 30K–80K subscribers:

  • CPU spikes during Android update waves
  • Recursive server freeze
  • Cache poisoning attempts
  • DNS amplification attempts
  • Failover delays when one DNS IP stops responding

Traditional “Primary/Secondary DNS” model relies on client retry timers. That is not infrastructure-grade redundancy. Modern ISP design must shift failover responsibility from client to infrastructure.

Architectural Philosophy

Why Single DNS Server is Wrong

  • Single server = single point of failure.
  • Even if uptime is 99.5%, subscriber perception during outage is 0%.

Why Primary / Secondary is Not Enough

Primary/Secondary:

  • Client decides when to retry.
  • Retry delay depends on OS resolver behavior.

This causes:

  • 5–30 seconds browsing delay
  • Perceived outage
  • Increased support calls

Infrastructure-level redundancy is superior.

Control Plane vs Data Plane

We separate roles:

Control Plane (dnsdist):

  • Load balancing
  • Rate limiting
  • Traffic classification
  • Health monitoring

Data Plane:

  • Recursive resolution
  • Authoritative zone serving

This allows independent scaling.

Recommended Modern ISP DNS Architecture

Client → VRRP VIP (DNSDIST)

┌──────────────────┐
│   dnsdist HA x2  │
└──────────────────┘
               |
┌──────────────┬─────────────────┐
│ Auth Pool    │ Rec Pool        │
│ (BIND) x2    │(BIND/Unbound) x2|
└──────────────┴─────────────────┘

🔹 Operational Best Practices

✔ Monitoring (Prometheus/Grafana)
✔ Log sampling only (avoid full query logging)
✔ Separate management VLAN
✔ Disable recursion on authoritative
✔ Disable public access to backend IPs

🔹 Result

  • No Single Point of Failure
  • Clean separation (Auth vs Rec)
  • Scalable horizontally
  • Controlled DDoS surface

Why This Design Works

✔ Zero backend exposure
✔ Clean separation of duties
✔ Easy scaling (add more recursive nodes in DNS VLAN)
✔ Maintenance without downtime
✔ Audit-friendly (clear segmentation)

DNS ISP infra mein critical service hai 🙂

  • RADIUS down ho to login issue hota hai,
  • DNS down ho to poora internet “down” lagta hai.

🔎 1️⃣ Is The Architecture Correct For 100k Users?

Your design:

  • Clients
↓
VRRP VIP
↓
2x dnsdist (HA)
↓
Auth Pool + Rec Pool
↓
2x Recursive + 1x Auth

This is industry-standard L7 DNS load-balancer model.

Used by:

  • Mid-size ISPs
  • Hosting providers
  • MSPs
  • Regional broadband operators

So yes > conceptually correct.

🔎 2️⃣ 100k Users → What Load Does That Mean?

Typical ISP DNS usage:

  • 3–10 QPS per subscriber during peak
  • 100k subs × avg 2–3 active at same moment
  • Realistic peak: 15k–40k QPS

During Netflix / Android updates / cache expiry bursts:

  • 50k+ QPS spikes possible

Our LAB config (10k cache entries, 50 QPS limit) is too small for that.

  • Architecture is fine.
  • Sizing must change.

🔵 3️⃣ What Would Be Required For 100k Subscribers?

✅ dnsdist Layer

Minimum recommended per node:

  • 8–16 vCPU
  • 16–32 GB RAM
  • Packet cache 500k–1M entries
  • NIC tuned for high PPS
  • IRQ affinity tuned
  • RPS/RFS enabled

Example production packet cache:

pc = newPacketCache(500000, {maxTTL=300})

✅ Recursive Layer

For 100k subs:

Two recursive servers are borderline.

Better:

  • 3–4 recursive nodes
  • Each 8–16 cores
  • 32 GB RAM
  • Proper ulimit tuning
  • Large resolver cache

In BIND:

  • max-cache-size 8g
  • recursive-clients 50000;

✅ Authoritative Layer

  • Authoritative load is typically very low.
  • 1 primary + 1 secondary recommended.

✅ Network Layer

Must ensure:

  • Multicast allowed (VRRP)
  • NIC offloading tuned
  • Firewall not bottlenecking
  • MTU correct
  • No stateful inspection on DNS traffic

🔎 4️⃣ Is dnsdist Used In Serious ISP Deployments?

Yes.

dnsdist (by PowerDNS) is widely used in:

  • ISPs
  • CDN providers
  • Hosting companies
  • Enterprise resolvers
  • Cloud operators

It is not hobby software.

It supports:

  • 1M+ QPS on proper hardware
  • Advanced rate limiting
  • Geo routing
  • DNS filtering
  • DoT/DoH frontend

🔎 5️⃣Is OUR Current Lab Enough For 100k?

In current lab sizing:

❌ No (hardware too small)
❌ Cache too small
❌ Recursive count too small

But:

✔ Architecture pattern is correct
✔ Failover model correct
✔ Separation correct
✔ Routing logic correct

So design is scalable.

🔵 6️⃣ Real-World Upgrade Path For 100k ISP

I would recommend:

  • 2x dnsdist (active/active possible)
  • 3x recursive nodes
  • 2x authoritative nodes
  • Anycast (optional future)
  • Monitoring (Prometheus + Grafana)

🔎 7️⃣ Real Question: Single VIP or Dual IP?

For 100k users:
Better to provide clients:

  • Primary DNS: VIP
  • Secondary DNS: VIP (same)

Redundancy handled at server layer.

Or:

Active/Active with ECMP or Anycast if advanced.

🔵 8️⃣ Where Would This Design Break?

It would break if:

  • Recursive servers undersized
  • Cache too small
  • CPU too low
  • Too aggressive rate limiting
  • No kernel tuning

Not because of architecture.

🎯 Final Professional Answer

Yes > this architecture is absolutely suitable for 100k subscribers.

But:

  • It must be deployed on proper hardware
  • properly tuned
  • and monitored.

Your lab has proven:

  • Design works
  • HA works
  • Routing works
  • Backend failover works

That is exactly what matters before production.

Deployment Blueprint – Exact Sequence

We use the following topology:

✅ Finalized Lab IP Plan

Hostname Role IP
DD-VRRP-IP Floating VIP 10.10.2.160
LAB-DD1 dnsdist-1 10.10.2.161
LAB-DD2 dnsdist-2 10.10.2.162
LAB-AUTH1 Authoritative BIND 10.10.2.163
LAB-REC1 Recursive BIND 10.10.2.164
LAB-REC2 Recursive BIND 10.10.2.165
LAB-CLIENT1 Test Windows 10.10.2.166

Very clean numbering 👍

🔎 Important Design Note (Very Important)

Right now everything is in:

  • 10.10.2.0/24

For lab this is OK.

But remember in production:

  • dnsdist public interface
  • backend DNS VLAN
  • management VLAN

should ideally be separated.

For lab → single subnet is fine.

How Many VMs Required?

Minimum lab set:

Role Qty
dnsdist 2
BIND Authoritative 1
BIND Recursive 2
Windows Client 1
(Optional Monitoring) 1

✅ Total Minimum: 6 VMs

(7 if you add monitoring like Zabbix/Prometheus later)

Minimum Hardware Sizing (LAB Only)

Since this is not production load:

🔹 dnsdist VM (each)

  • 2 vCPU
  • 2 GB RAM
  • 20 GB disk
  • 2 NICs (Recommended)
    • NIC1 → VLAN-2 (Public simulation)
    • NIC2 → DNS VLAN (Backend network)

🔹 BIND Authoritative

  • 2 vCPU
  • 2 GB RAM
  • 20 GB disk
  • 1 NIC (DNS VLAN)

🔹 BIND Recursive (each)

  • 2 vCPU
  • 2 GB RAM
  • 20 GB disk
  • 1 NIC (DNS VLAN)

🔹 Windows Client

  • 2 vCPU
  • 4 GB RAM
  • 40 GB disk
  • 1 NIC (VLAN-2)

💡 Lab Total Resource Footprint

Approx:

  • 12–14 vCPU
  • 14–16 GB RAM

Very manageable in VMware test cluster.

Few Queries for above scheme

✅ Query #1

Should internal users get:

  • Only VIP → 10.10.2.160
    OR
  • Two real IPs → 10.10.2.161 and 10.10.2.162 ?

🔹 Correct Answer (With VRRP)

If you are using:

  • 2x dnsdist
  • VRRP Floating IP (10.10.2.160)

👉 Clients should receive ONLY the VIP (10.10.2.160)

Why?

Because:

  • VIP always exists
  • If dnsdist-1 fails → VIP moves to dnsdist-2
  • Clients don’t need to know which node is active
  • Clean failover

This is standard HA design.

🔹 When Would You Give 2 IPs?

You would give:

  • Primary DNS: 10.10.2.160(VIP)
  • Secondary DNS: 10.10.2.162 (optional)

Only if:

  • You are not fully trusting VRRP
  • Or you want additional redundancy layer
  • Or you are not using floating IP

But in proper HA design:

One VIP is enough.

🔹 Best Practice for 50k+ ISP

Subscribers receive:

  • Primary DNS: 10.10.2.160
  • Secondary DNS: 10.10.2.160

Yes > same IP twice is fine when using VRRP HA.

You may find it strange, but The redundancy is at server layer, not IP layer.

✅ Query #2

Authoritative used for internal + external > how will it function?

This is about traffic separation.

Remember your architecture:

Internet → dnsdist (VIP)
|
--------------------------------
|                  |
Authoritative Pool Recursive Pool

dnsdist decides where query goes.

🔹 Case A > External Client Query

Example:

External user queries:

ns1.yourisp.com

Flow:

Internet → VIP → dnsdist → Authoritative (10.10.10.10)

Recursive pool is NOT involved.

🔹 Case B > Internal Subscriber Query

Subscriber asks:

google.com

Flow:

Subscriber → VIP → dnsdist → Recursive pool

Authoritative not involved.

🔹 Case C > Internal Query for ISP Domain

Subscriber asks:

portal.yourisp.com

Flow:

Subscriber → VIP → dnsdist → Authoritative

Works same as external.

🔹 How Does dnsdist Know Where to Send?

Usually:

Option 1 > Domain-based routing (Recommended)

  • addAction(RegexRule(“yourisp.com”), PoolAction(“auth”))
  • addAction(AllRule(), PoolAction(“rec”))

Everything else → recursive

Your own domains → authoritative

🔹 Important Best Practice

On Authoritative server:

❌ Disable recursion

In BIND:

  • recursion no;

So even if misrouted traffic comes, it won’t resolve internet domains.

🔹 Very Important for ISP

Recursive servers:

  • Should allow only subscriber IP ranges
  • Should not be open resolver to world

Authoritative:

  • Should answer only hosted zones
  • Should not do recursion

dnsdist enforces clean split.

🔹 Final Clean Answers

Query #1:

Give clients ONLY the VIP (10.10.2.160).

Query #2:

dnsdist routes queries to:

  • Authoritative pool for your domains
  • Recursive pool for everything else

Both internal and external clients can use same VIP > routing logic handles separation.

Architecture Overview (Layer-by-Layer Flow)

This DNS architecture is not simply a dual-server deployment.
It is a layered control-plane model designed to:

  • Contain failures
  • Classify traffic
  • Absorb load bursts
  • Maintain deterministic failover
  • Enable horizontal scaling

The system is divided into five logical layers.

Layer 1 – Subscriber Access Layer

This is the ingress layer.

Traffic Origin:

  • PPPoE subscribers
  • CGNAT subscribers
  • Internal LAN clients
  • Management clients (if allowed)

Subscribers are configured to use:

DNS = 10.10.2.160 (DD-VRRP-IP)

Key property:
Subscribers never see backend servers.
They only see the VIP.

This ensures:

  • No backend IP exposure
  • No client-side failover logic
  • Simplified DHCP configuration
  • Clean abstraction layer

Failure containment:
Even if one dnsdist node fails, the VIP floats. Clients are unaware.

Layer 2 – Frontend Control Plane (dnsdist HA Pair)

Nodes:
LAB-DD1 (10.10.2.161)
LAB-DD2 (10.10.2.162)

Floating IP:
10.10.2.160

Role:
DNS traffic controller and policy engine.

Responsibilities:

  1. Accept UDP/TCP 53 traffic
  2. Apply ACL rules
  3. Apply rate limiting
  4. Drop abusive queries
  5. Classify domain type
  6. Route to correct backend pool
  7. Cache responses
  8. Monitor backend health

This is the most critical layer in the system.

It does NOT perform recursive resolution.
It performs traffic governance.

2.1 VRRP Behavior

VRRP ensures:

  • Only one frontend holds 10.10.2.160 at a time.
  • If MASTER fails, BACKUP becomes MASTER.
  • If dnsdist process fails, VIP relinquished.

Failover flow:

dnsdist crash → keepalived detects → priority lost → VIP moves → service restored in 2–3 seconds.

This removes dependency on:

  • Client retry timers
  • Secondary DNS IP logic
  • Application resolver behavior

Failover is deterministic.

Layer 3 – Traffic Classification Engine (Inside dnsdist)

Once a DNS packet arrives at VIP:

dnsdist evaluates rules in order.

Example logic:

If domain suffix = zaibdns.lab
→ send to “auth” pool

Else
→ send to “rec” pool

Additionally:

  • ANY query dropped
  • Excess QPS per IP dropped
  • Non-allowed subnet rejected

This classification stage is critical.

Without classification:

  • Recursive and authoritative mix
  • Backend tuning conflicts
  • Security boundaries blur

dnsdist enforces traffic discipline.

Layer 4 – Backend Pools

There are two independent backend pools:

AUTH Pool:
LAB-AUTH1 (10.10.2.163)

REC Pool:
LAB-REC1 (10.10.2.164)
LAB-REC2 (10.10.2.165)

These pools are isolated.

dnsdist maintains health status per server.

4.1 Authoritative Pool

Purpose:
Serve local zones only.

Properties:

  • recursion disabled
  • publicly queryable (if required)
  • low QPS compared to recursive

Failure impact:
Only local zone resolution affected.

Does NOT affect internet browsing.

4.2 Recursive Pool

Purpose:
Resolve internet domains.

Properties:

  • recursion enabled
  • restricted to subscriber subnet
  • large cache memory
  • high concurrency settings

Failure behavior:

If REC1 fails:
dnsdist stops sending traffic to it.
REC2 continues serving.

If both fail:
Service disruption occurs.

This is why horizontal scaling is recommended for 100K users.

Layer 5 – Internet Resolution Layer

Recursive servers query:

  • Root servers
  • TLD servers
  • Authoritative internet servers

This layer is outside ISP control.

However:

Packet cache in dnsdist reduces external dependency frequency.

High cache hit ratio = lower external latency.

End-to-End Query Flow Example

Scenario 1: Subscriber queries www.google.com

Step 1:
Client sends query to 10.10.2.160

Step 2:
dnsdist receives packet

Step 3:
Suffix does NOT match local zone

Step 4:
dnsdist forwards to REC pool

Step 5:
Recursive server checks cache
If cache miss → resolves via internet
If cache hit → replies immediately

Step 6:
dnsdist optionally caches packet

Step 7:
Response sent to subscriber

Scenario 2: Subscriber queries www.zaibdns.lab

Step 1:
Packet arrives at VIP

Step 2:
Suffix matches local zone

Step 3:
dnsdist forwards to AUTH pool

Step 4:
Authoritative server responds

Step 5:
dnsdist relays response

Recursive servers are never involved.

Failure Domain Isolation

Let’s analyze impact per failure.

Failure: LAB-REC1 crash
Impact: 50% recursive capacity lost
Mitigation: REC2 continues

Failure: LAB-AUTH1 crash
Impact: Local zone fails
Internet browsing unaffected

Failure: LAB-DD1 crash
Impact: VIP moves to LAB-DD2
Subscriber impact: ~2–3 seconds max

Failure: dnsdist process crash on MASTER
Impact: VIP released immediately
Failover triggered

Failure: Kernel UDP overload on one frontend
Impact: Traffic handled by second frontend if VRRP triggered

This layered model ensures limited blast radius.

Logical Separation of Concerns

Layer Responsibility Failure Impact
Subscriber Query origin None
dnsdist Traffic governance Frontend failover
AUTH pool Local zones Local zone only
REC pool Internet resolution Internet browsing
Internet External resolution External dependency

Clear separation improves troubleshooting.

Why This Layered Model Matters

Without layering:

  • Recursive and authoritative mixed
  • No policy enforcement
  • No health-driven routing
  • No horizontal scaling path

With layering:

  • Each component has defined responsibility
  • Each failure has defined boundary
  • Scaling can be targeted
  • Security can be enforced per layer

This is the difference between:

“Two DNS servers”

and

“A DNS infrastructure.”

Now tell me where you want to go next:

  1. Detailed Logical Traffic Flow with timing metrics
  2. VRRP State Machine Deep Explanation
  3. Backend Health Check Mechanics Deep Dive
  4. Performance Modeling & Cache Efficiency Analysis
  5. Deployment Sequence – Step 1 (Aggressive OS Hardening)

We continue building this as a full engineering whitepaper.

OS Preparation (All Servers)

Ubuntu 22.04 recommended.

Disable systemd-resolved

Reason:

  • Ubuntu binds 127.0.0.53:53 by default.
  • dnsdist requires port 53.

Commands:

sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved
sudo rm /etc/resolv.conf
echo "nameserver 8.8.8.8" > /etc/resolv.conf

🔹 Production Notes for Ubuntu 22

✔ Ubuntu 22 is stable for ISP DNS use
✔ Works fine with Keepalived
✔ Supports high kernel tuning
✔ Good for 10k–50k+ QPS per node (proper hardware required)

🔹 Important Tuning (Must Do in Production)

In /etc/sysctl.conf:

net.core.rmem_max=25000000
net.core.wmem_max=25000000
net.core.netdev_max_backlog=50000

Then:

sudo sysctl -p

Without kernel tuning, high QPS performance suffer karega.

🎯 Lab Build Order (Important)

Always follow this order:

1️⃣ Backend first (BIND servers working standalone)
2️⃣ Then dnsdist (single node)
3️⃣ Then HA (Keepalived)

Never start with HA first.

🔵 Final Zone Design

Zone name:
zaibdns.lab
Primary NS:
ns1.zaibdns.lab
Test records:
www.zaibdns.lab
portal.zaibdns.lab

Authoritative DNS Configuration (LAB-AUTH1)

🔷 Now Configure on LAB-AUTH1 (10.10.2.163)

🔵 STEP 1 > Install BIND

sudo apt update
sudo apt install bind9 bind9-utils bind9-dnsutils -y

Verify service:

sudo systemctl status bind9

It should show:

  • Active: active (running)

If not running:

sudo systemctl start bind9

🔵 STEP 2 > Configure BIND as Authoritative Only

Edit options file:

sudo nano /etc/bind/named.conf.options

Replace entire content with:

options {
directory "/var/cache/bind";
recursion no;
allow-query { any; };
listen-on { 10.10.2.163; };
listen-on-v6 { none; };
};

Save and exit.

🔵 STEP 3 > Define Zone

Edit:

sudo nano /etc/bind/named.conf.local

Add this at bottom:

zone "zaibdns.lab" {
type master;
file "/etc/bind/db.zaibdns.lab";
};

Save.

🔵 STEP 4 > Create Zone File

sudo nano /etc/bind/db.zaibdns.lab

Paste this:

$TTL 86400
@ IN SOA ns1.zaibdns.lab. admin.zaibdns.lab. (
2026021401
3600
1800
604800
86400 )
IN NS ns1.zaibdns.lab.
ns1 IN A 10.10.2.163
www IN A 10.10.2.163
portal IN A 10.10.2.163

Save.

🔵 STEP 5 > Check Configuration (Very Important)

Run:

sudo named-checkconf

No output = good.

Then:

sudo named-checkzone zaibdns.lab /etc/bind/db.zaibdns.lab

It must say:

OK

If error appears, stop and fix.

🔵 STEP 6 > Restart BIND

sudo systemctl restart bind9
sudo systemctl status bind9

Ensure it is running.

🔵 STEP 7 > Test Authoritative Function

From another VM (LAB-DD1 or LAB-REC1):

dig @10.10.2.163 www.zaibdns.lab

You should see:

ANSWER SECTION:
http://www.zaibdns.lab. 86400 IN A 10.10.2.163

🔵 STEP 8 > Confirm Recursion Is Disabled

Test:

dig @10.10.2.163 google.com

It should FAIL (no answer section).

If it resolves google.com → recursion not disabled properly.

🎯 Expected Result

Authoritative server should:

✔ Resolve zaibdns.lab records
✔ NOT resolve internet domains
✔ Respond on 10.10.2.163 only

🎯 When This Works

Once AUTH server working for zaibdns.lab

🚀 Next Phase

Now we move to:

🔵 PHASE 2 > Recursive DNS Setup

On:

  • LAB-REC1 (10.10.2.164)
  • LAB-REC2 (10.10.2.165)

We will configure them as:

  • Recursive-only resolvers
  • Allow queries only from 10.10.2.0/24
  • Disable zone hosting
  • Enable caching
  • Ready for dnsdist pool

🔵 STEP 1 > Install BIND (On BOTH REC1 & REC2)

Run on each:

sudo apt install bind9 bind9-utils bind9-dnsutils -y

Verify:

sudo systemctl status bind9

Must show active (running).

🔵 STEP 2 > Configure Recursive Resolver

Edit:

sudo nano /etc/bind/named.conf.options

Replace entire content with this (adjust listen IP per server):

🔹 On LAB-REC1 (10.10.2.164)

options {
directory "/var/cache/bind";
recursion yes;
allow-recursion { 10.10.2.0/24; };
allow-query { 10.10.2.0/24; };
listen-on { 10.10.2.164; };
listen-on-v6 { none; };
dnssec-validation auto;
};

🔹 On LAB-REC2 (10.10.2.165)

Same config, just change:

listen-on { 10.10.2.165; };

🔵 STEP 3 > Remove Default Zones (Optional but Clean)

on both REC servers, Open:

sudo nano /etc/bind/named.conf.local

Make sure it is empty or has no zones.

Recursive servers should not host zones.

🔵 STEP 4 > Validate Config

Run on both:

sudo named-checkconf

No output = good.

🔵 STEP 5 > Restart BIND (on both rec bind servers)

sudo systemctl restart bind9
sudo systemctl status bind9

Must be running.

🔵 STEP 6 > Test Recursive Function

From LAB-DD1 or any other VM node:

Test REC1:

dig @10.10.2.164 google.com

Test REC2:

dig @10.10.2.165 google.com

You should see:

  • ANSWER SECTION populated
  • NOERROR
  • No AA flag

🔵 STEP 7 > Test ACL Restriction

From LAB-AUTH1 (allowed subnet), it should work.

Later when Windows client configured outside allowed range, recursion should be blocked (we will test that later).

🎯 Expected Behavior

Recursive servers should:

✔ Resolve google.com
✔ Cache responses
✔ NOT host zaibdns.lab
✔ Only allow 10.10.2.0/24
✔ Listen only on their IP

🔎 Quick Verification

Also test:

dig @10.10.2.164 www.zaibdns.lab

It should NOT resolve (because recursion won’t find internal zone).

That confirms clean separation.

Once both REC1 & REC2 successfully resolve google.com,

Move forward …

Kernel Aggressive Tuning (All DNS Servers)

Add to /etc/sysctl.conf:

net.core.rmem_max=67108864
net.core.wmem_max=67108864
net.core.netdev_max_backlog=500000
net.ipv4.udp_mem=262144 524288 1048576
net.ipv4.udp_rmem_min=16384
net.ipv4.udp_wmem_min=16384
net.ipv4.ip_local_port_range=1024 65000
fs.file-max=1000000

Apply:

sudo sysctl -p

Increase file descriptors:

ulimit -n 1000000

Reason:

High QPS requires high UDP buffer capacity and file descriptor availability.

dnsdist Configuration (LAB-DD1 & LAB-DD2)

 1# LAB-DD1 (LAB-DD1 = 10.10.2.161)

Install dnsdist from official PowerDNS repository.

(Full Production Version)

🔹 Recommended Method (Official Repository)

  • Do NOT rely on very old distro packages.
  • Use PowerDNS official repo for production.

Step 1 > Add PowerDNS Repo

sudo apt install -y curl gnupg2
curl -fsSL https://repo.powerdns.com/FD380FBB-pub.asc | sudo gpg --dearmor -o /usr/share/keyrings/pdns.gpg

Add repo file:

echo "deb [signed-by=/usr/share/keyrings/pdns.gpg]
http://repo.powerdns.com/ubuntu jammy-dnsdist-17 main" | sudo tee /etc/apt/sources.list.d/pdns.list

Step 2 > Install

sudo apt update
sudo apt install dnsdist

Verify:

sudo systemctl status dnsdist

Step 3 > Enable & Start

sudo systemctl enable dnsdist
sudo systemctl start dnsdist

Check status:

sudo systemctl status dnsdist

🔹 Default Config Location

/etc/dnsdist/dnsdist.conf

Configure dnsdist

 /etc/dnsdist/dnsdist.conf

Delete everything and paste:

setLocal("0.0.0.0:53")
addACL("10.10.2.0/24")
-- Packet Cache
pc = newPacketCache(500000, {maxTTL=300})
getPool("rec"):setCache(pc)
-- Why:
-- 500k entries supports high subscriber base.
-- TTL limited to prevent stale responses.
-- Abuse Protection
addAction(QTypeRule(DNSQType.ANY), DropAction())
addAction(MaxQPSIPRule(200), DropAction())
-- Why:
-- ANY queries are amplification risk.
-- 200 QPS per IP is safe baseline.
-- Backend Health Checks
newServer({address="10.10.2.163:53", pool="auth", checkType="A", checkName="zaibdns.lab.", checkInterval=5})
newServer({address="10.10.2.164:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
newServer({address="10.10.2.165:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
-- Why:
-- Backend marked DOWN if health check fails.
-- Routing
local suffixes = newSuffixMatchNode()
suffixes:add(newDNSName("zaibdns.lab."))
addAction(SuffixMatchNodeRule(suffixes), PoolAction("auth"))
addAction(AllRule(), PoolAction("rec"))
-- Monitoring
controlSocket("127.0.0.1:5199")
#Save

Load Balancing Policy Selection (Critical Design Decision)

dnsdist supports multiple server selection policies. Choosing the correct one directly affects latency and failure behavior.

Recommended for ISP Recursive Pool

setServerPolicy(leastOutstanding)

Why:

  • Distributes traffic based on active outstanding queries
  • Prevents overloading a single backend
  • Maintains low latency under burst traffic

Alternative Models

Policy Use Case Notes
firstAvailable Simple failover Not ideal for load distribution
wrandom Weighted random Good when backend hardware differs
chashed Consistent hashing Useful for cache stickiness

Recommendation:
For equal hardware recursive pool → use leastOutstanding.

🧠 Why SuffixMatchNode Is Better

Regex:

  • Easy to break
  • Dot escaping messy
  • Trailing dot issues

SuffixMatchNode:

  • DNS-aware matching
  • Exact domain match
  • Used in serious deployments

After editing Restart service

sudo systemctl restart dnsdist

TEST Routing Logic

From any other VM:

🔸 Test Authoritative Routing

dig @10.10.2.161 http://www.zaibdns.lab

Expected:

  • Correct answer
  • AA flag present

🔸 Test Recursive Routing

dig @10.10.2.161 google.com

Expected:

  • Resolves normally
  • No AA flag

🔎 What Should Happen Internally

For:

http://www.zaibdns.lab

  • dnsdist → AUTH pool → 10.10.2.163

For:

google.com

  • dnsdist → REC pool → 10.10.2.164 / 165

🎯 When This Works

You now have:

✔ Smart DNS routing
✔ Proper separation
✔ Backend load distribution
✔ DNS traffic control layer

After confirming both tests work, we will:

🔵 Add dnsdist on LAB-DD2
🔵 Configure Keepalived
🔵 Implement VRRP VIP 10.10.2.160
🔵 Perform real failover testing

BUT FIRST understands the DIG flags that will help you understand the results correctly.

🔎 Where Do We See DNS Flags?

You already saw them in dig output:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61032
;; flags: qr aa rd

That flags: line is what you inspect.

🔹 Meaning of Important Flags

When you run:

dig @10.10.2.161 www.zaibdns.lab

You’ll see:

flags: qr aa rd

Here is what each means:

Flag Meaning
qr Query Response (this is a reply)
aa Authoritative Answer
rd Recursion Desired (client asked for recursion)
ra Recursion Available (server supports recursion)
ad Authenticated Data (DNSSEC validated)

🔵 What You Should Expect

🔹 For zaibdns.lab (Authoritative Path)

Expected:

status: NOERROR
flags: qr aa rd

Important:

  • aa must be present ✅
  • ra should NOT appear (since auth server doesn’t recurse)

🔹 For google.com (Recursive Path)

Expected:

status: NOERROR
flags: qr rd ra

Important:

  • aa should NOT be present ❌
  • ra must be present ✅

That proves recursion happened.

🔎 Cleaner Output (Easier to Read)

Instead of full dig, use:

dig @10.10.2.161 www.zaibdns.lab +noall +answer +authority +comments

Example output:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR
;; flags: qr aa rd

This makes flags very clear.

🔬 Extra Debug Mode (Very Useful)

For detailed packet view:

dig @10.10.2.161 www.zaibdns.lab +dnssec +multi

Or full raw:

dig @10.10.2.161 www.zaibdns.lab +trace

🧠 How To Validate dnsdist Routing Using ‘DIG’ Flags

When testing through dnsdist:

Authoritative test:

dig @10.10.2.161 http://www.zaibdns.lab

Look for:

✔ aa

Recursive test:

dig @10.10.2.161 google.com

Look for:

✔ ra
❌ no aa

🎯 Why Flags Matter in ISP World

In real ISP troubleshooting:

  • If aa missing → authoritative routing broken
  • If ra missing → recursion disabled
  • If REFUSED → ACL issue
  • If SERVFAIL → backend failure

Flags are your first debugging indicator.

🚀 You Have Now Built

✔ Authoritative backend
✔ Recursive backend
✔ Intelligent dnsdist routing
✔ DNS flag-level validation

 

Right now:

  • LAB-DD1 (10.10.2.161) → Working
  • LAB-DD2 (10.10.2.162) → Not configured yet
  • VIP planned → 10.10.2.160

Goal:

  • Clients will use only 10.10.2.160
  • If DD1 fails → DD2 takes over automatically

🔵 Configure LAB-DD2 (Clone of DD1)

STEP 1 > Install dnsdist on LAB-DD2

On:

  • LAB-DD2 (10.10.2.162)

Install same way:

sudo apt install dnsdist -y

STEP 2 > Copy Same Config

Edit:

sudo nano /etc/dnsdist/dnsdist.conf

Paste SAME config as DD1, but change listen IP:

setLocal("10.10.2.162:53")
addACL("10.10.2.0/24")
newServer({address="10.10.2.163:53", pool="auth"})
newServer({address="10.10.2.164:53", pool="rec"})
newServer({address="10.10.2.165:53", pool="rec"})
local suffixes = newSuffixMatchNode()
suffixes:add(newDNSName("zaibdns.lab."))
addAction(SuffixMatchNodeRule(suffixes), PoolAction("auth"))
addAction(AllRule(), PoolAction("rec"))

Restart:

sudo systemctl restart dnsdist

STEP 3 > Test DD2 Directly

From any VM:

dig @10.10.2.162 www.zaibdns.lab
dig @10.10.2.162 google.com

Both must work exactly like DD1.

Once DD2 works, we move to:

🔵NEXT PHASE – HA !!!!!!!! (or HAHAHAHA 😉

VRRP High Availability (Keepalived)

Now we move to HA layer.

H.A  PHASE > Install Keepalived (VRRP)

We will:

  • Install Keepalived on BOTH DD1 & DD2
  • Configure floating IP = 10.10.2.160
  • Make DD1 MASTER
  • Make DD2 BACKUP

🔹 STEP 1 > Install Keepalived (On BOTH DD1 & DD2)

On LAB-DD1:

sudo apt install keepalived -y

🔹 STEP 2 > Configure LAB-DD1 (MASTER)

On LAB-DD1:

sudo nano /etc/keepalived/keepalived.conf

Paste:

global_defs { router_id LAB_DD1 } vrrp_script chk_dnsdist { script “systemctl is-active –quiet dnsdist” interval 2 fall 1 rise 1 } vrrp_instance VI_DNS { state MASTER interface ens160 virtual_router_id 51 priority 150 advert_int 1 authentication { auth_type PASS auth_pass lab123 } virtual_ipaddress { 10.10.2.160 } track_script { chk_dnsdist } } Save.

🔹 STEP 3 > Configure LAB-DD2 (BACKUP)

On LAB-DD2:

sudo nano /etc/keepalived/keepalived.conf

Paste:

global_defs {
router_id LAB_DD2
}
vrrp_script chk_dnsdist {
script "systemctl is-active --quiet dnsdist"
interval 2
fall 1
rise 1
}
vrrp_instance VI_DNS {
state BACKUP
interface ens160
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass lab123
}
virtual_ipaddress {
10.10.2.160
}
track_script {
chk_dnsdist
}
}

Save.

VRRP Engineering Considerations

  1. Advertise Interval

advert_int 1

  • 1 second provides fast failover
  • Avoid sub-second unless absolutely required
  1. Priority Design

Primary:

priority 150

Secondary:

priority 100

Avoid equal priorities to prevent master flapping.

  1. Split-Brain Prevention

Ensure:

  • VRRP runs on isolated VLAN
  • No L2 loops
  • Proper STP configuration
  • Monitoring for dual-master condition
  1. Health-Based VRRP Tracking (Recommended)

Track dnsdist process:

track_script {
chk_dnsdist
}

Failover should occur if:

  • dnsdist crashes
  • Backend unreachable
  • System load critical

This avoids “IP is up but service is down” scenario.

🔹 STEP 4 > Start Keepalived

On BOTH:

sudo systemctl enable keepalived
sudo systemctl start keepalived

🔹 STEP 5 > Verify VIP

On DD1:

  • ip a

You should see:

  • 10.10.2.160

On DD2:

You should NOT see VIP (since it’s backup).

🔹 STEP 5 > Test VIP

From any VM:

dig @10.10.2.160 www.zaibdns.lab
dig @10.10.2.160 google.com

Both must work.

🔥 STEP 6 > Failover Test (Important)

🔎 What This Does

  • Every 2 seconds it checks:

systemctl is-active dnsdist

  • If dnsdist stops → health check fails
  • MASTER immediately drops state
  • BACKUP becomes MASTER
  • VIP moves

No weight calculations.
No partial priority logic.
Clean failover.

🔁 Apply Configuration

On BOTH nodes:

sudo systemctl restart keepalived

Confirm VIP is on DD1:

ip a | grep 10.10.2.160

🔥 Test Failover

On DD1:

sudo systemctl stop dnsdist

Within ~2 seconds:

  • VIP disappears from DD1
  • VIP appears on DD2

Test:

dig @10.10.2.160 google.com

Should continue working.

🔁 Test Recovery

Start dnsdist again on DD1:

sudo systemctl start dnsdist

VIP should move back to DD1 (because higher priority 150).

🎯 Now You Have

✔ Service-aware failover
✔ Proper HA behavior
✔ Clean VIP movement
✔ Production-style design

More Failover Testing…

now we test what really matters: failover.

You already have:

  • DD1 (MASTER)
  • DD2 (BACKUP)
  • VIP = 10.10.2.160
  • dnsdist listening on 0.0.0.0:53
  • Keepalived running

Now we validate HA properly.

🔵 STEP 1 > Confirm Who Owns VIP

On DD1:

  • ip a | grep 10.10.2.160

On DD2:

  • ip a | grep 10.10.2.160

Expected:

  • VIP visible on DD1 only
  • Not visible on DD2

🔵 STEP 2 > Baseline DNS Test

From any other VM:

dig @10.10.2.160 google.com
dig @10.10.2.160 www.zaibdns.lab

Confirm both resolve.

🔵 STEP 3 > Test Soft Failover (Service Failure)

Now simulate dnsdist crash on MASTER.

On DD1:

sudo systemctl stop dnsdist

Wait few seconds.

🔎 Check VIP Movement

On DD2:

ip a | grep 10.10.2.160

VIP should now appear on DD2.

On DD1:

ip a | grep 10.10.2.160

VIP should be gone.

🔎 Test DNS During Failover

From another VM:

dig @10.10.2.160 google.com

It should still resolve.

That means:

✔ Keepalived detected dnsdist failure
✔ VIP moved
✔ Clients unaffected

🔵 STEP 4 > Restore MASTER

On DD1:

sudo systemctl start dnsdist

Wait few seconds.

Check:

ip a | grep 10.10.2.160

Depending on config:

  • VIP may return to DD1 (if preemption active)
  • Or stay on DD2

If you want VIP to always return to DD1, ensure:

  • priority higher on DD1

(which you already set 110 vs 100)

🔵 STEP 5 > Hard Failover (Real Test)

Now simulate full server failure.

On DD1:

sudo poweroff

Check from another VM:

dig @10.10.2.160 google.com

It should still work.

On DD2:

ip a | grep 10.10.2.160

VIP must be present.

🔵 STEP 6 > Continuous Resolution Test (While Failing Over)

From any non-DD node:

while true; do dig @10.10.2.160 google.com +short; sleep 1; done

Now stop dnsdist on DD1.

Expected:

  • 0–1 failed query max
  • Resolution resumes automatically
  • No manual intervention

🔵 What We Are Actually Testing

Keepalived config:

track_script {
chk_dnsdist
}

This means:

If pidof dnsdist fails → priority drops → VIP moves.

This is correct HA design.

🔥 Common Problems During Failover

If VIP does not move:

  • Wrong interface name in keepalived config
  • Multicast blocked in VLAN
  • Firewall blocking VRRP (protocol 112)
  • Wrong virtual_router_id mismatch

🎯 Final Expected Result

You should achieve:

✔ Service failover without manual action
✔ Clients always use single IP
✔ Zero backend changes
✔ DNS resolution continues

🎯 Final Lab Architecture Now

Client → 10.10.2.160 (VIP)
|
dnsdist HA
|
Auth + Rec backend

You now have a properly built DNS stack:

  • Authoritative backend (LAB-AUTH1)
  • Recursive backend (LAB-REC1 / LAB-REC2)
  • dnsdist routing layer (DD1 / DD2)
  • VRRP floating VIP (10.10.2.160)
  • Service-aware failover using keepalived

This is structurally identical to how many mid-size ISPs deploy DNS control layers.

Backend Failure Test (Very Important)

Stop recursive on REC1:

sudo systemctl stop bind9

Now query via VIP:

dig @10.10.2.160 google.com

It should still resolve via REC2.

That validates backend redundancy.

4️⃣ Authoritative Isolation Test

Stop AUTH:

sudo systemctl stop bind9

Now:

dig @10.10.2.160 www.zaibdns.lab

Should fail.

But:

dig @10.10.2.160 google.com

Should still work.

That confirms clean pool separation.

🎯 What You Have Achieved

You have built:

  • Layered DNS architecture
  • Pool-based routing
  • High availability with VRRP
  • Service-aware failover
  • Controlled recursion
  • Authoritative isolation
  • Backend redundancy

This is not “lab toy” level anymore.
This is real network engineering.

 Why Separate Recursive and Authoritative?

  • Security isolation
  • Prevents recursive abuse
  • Better performance tuning
  • DDoS containment
  • Operational clarity

Where to Use Public vs Private IP

  • dnsdist → Public IP or subscriber-facing IP
  • Recursive servers → Private IP only
  • Authoritative → Private or Public (if serving public zones)

Golden Rules

  1. Separate recursive and authoritative.
  2. Never expose recursive publicly.
  3. Always monitor QPS.
  4. Always use packet cache.
  5. Always implement health checks.
  6. Test failover periodically.
  7. Plan scaling before congestion.
  8. Do not rely on client-side failover.

Scaling for 50K–100K Subscribers

Estimated Peak QPS

  • 50K users → 15K–25K QPS
  • 100K users → 30K–50K QPS

Capacity Planning Model (QPS Engineering Approach)

For ISP-grade DNS design, sizing must be derived from realistic subscriber behavior instead of theoretical hardware limits.

Baseline Estimation Formula

Peak QPS = Active Subscribers × Avg Queries per Second per Subscriber

Where:

  • Typical residential subscriber generates: 5–15 QPS
  • Business subscriber may generate: 20–50 QPS
  • During peak (evening streaming + mobile apps), bursts can reach 2× baseline.

Example (100,000 Subscribers)

If:

  • 60% concurrently active
  • Avg 8 QPS per active user

100,000 × 0.6 × 8 = 480,000 QPS peak

With:

  • 70–85% cache hit rate (well-tuned resolver)
  • Backend recursion load reduces significantly

Engineering Rule

Always size recursive backend for:

  • 1.5× projected peak QPS
  • Ability to survive single-node failure (N+1 model)

This ensures performance stability during:

  • Cache cold start
  • DDoS bursts
  • Backend node outage

Recommended Hardware

  • dnsdist: 8–16 cores, 16–32GB RAM
  • Recursive: 8–16 cores, 32GB RAM
  • Authoritative: Moderate load

Failure Scenarios Tested

  • Stop dnsdist → VIP moves
  • Stop keepalived → Backup takes over
  • Power off DD1 → DD2 becomes MASTER
  • Stop REC1 → Traffic moves to REC2
  • Stop AUTH → Only authoritative queries fail

Common Deployment Issues

  • systemd-resolved conflict on port 53
  • dnsdist not listening on VIP
  • Incorrect interface name in keepalived
  • VRRP blocked by VMware security settings
  • Regex routing errors

🔹 Security Controls (ISP Best Practice)

  • Firewall:
    • Allow UDP/TCP 53 → dnsdist only
    • Block direct access to backend IPs
  • Recursive ACL:
    • Allow only subscriber IP ranges
  • Rate limiting enabled on dnsdist
  • Disable recursion on authoritative

Best Practices for Pakistani Cable ISPs

  • Never run single DNS server
  • Always separate recursive & authoritative
  • Always use health checks
  • Monitor QPS continuously
  • Use packet cache
  • Use VRRP for frontend HA
  • Never expose recursive servers publicly

ISP GRADE TUNING – SJZ

Now we move from functional lab to ISP-grade tuning.

All changes below go into:

/etc/dnsdist/dnsdist.conf

(on BOTH DD1 and DD2)

Restart dnsdist after modifications.

🔵 1️⃣ Enable Packet Cache (Very Important)

This dramatically reduces load on recursive servers.

Add near top:

-- Packet cache (10k entries, 60s max TTL)
pc = newPacketCache(10000, {maxTTL=60, minTTL=0, temporaryFailureTTL=10})
getPool("rec"):setCache(pc)

What this does:

  • Caches recursive responses
  • Offloads REC1 / REC2
  • Improves latency
  • Handles burst traffic

For real ISP scale → 100k+ entries.

🔵 2️⃣ Enable Rate Limiting (Basic DDoS Protection)

Add:

-- Basic rate limiting (per IP)
addAction(MaxQPSIPRule(50), DropAction())

Meaning:

  • If a single IP sends >50 queries/sec → drop
  • Protects against abuse

For ISP production:

  • Adjust threshold based on subscriber profile

🔵 3️⃣ Basic Abuse Protection

Add:

-- Drop ANY queries (reflection attack prevention)
addAction(QTypeRule(DNSQType.ANY), DropAction())
-- Drop CHAOS queries (version.bind)
addAction(AndRule({QClassRule(DNSClass.CH), QTypeRule(DNSQType.TXT)}), DropAction())

Prevents:

  • Amplification attacks
  • Version probing

🔵 4️⃣ Backend Health Checks (Very Important)

Replace your newServer() lines with health checks:

newServer({
address="10.10.2.163:53",
pool="auth",
checkType="A",
checkName="zaibdns.lab.",
checkInterval=5
})
newServer({
address="10.10.2.164:53",
pool="rec",
checkType="A",
checkName="google.com.",
checkInterval=5
})
newServer({
address="10.10.2.165:53",
pool="rec",
checkType="A",
checkName="google.com.",
checkInterval=5
})

Now dnsdist:

  • Automatically marks backend DOWN if it fails
  • Stops sending traffic to dead backend

🔵 5️⃣ Enable Logging (Lightweight)

Add:

setVerboseHealthChecks(true)

To log health check failures.

For query logging (not recommended in production):

addAction(AllRule(), LogAction("/var/log/dnsdist-queries.log"))

⚠ Only use in lab > high overhead.

🔵 6️⃣ Enable TCP Support Tuning

Add:

setMaxTCPClientThreads(10)
setMaxTCPConnectionsPerClient(20)

Prevents TCP abuse.

Also increase UDP socket buffers (system-level):

sudo sysctl -w net.core.rmem_max=26214400
sudo sysctl -w net.core.wmem_max=26214400

🔵 7️⃣ Enable Metrics Export (Very Powerful)

Add:

controlSocket("127.0.0.1:5199")

Restart dnsdist.

Then:

dnsdist -c

Inside console:

showServers()
showPools()
showCacheHitResponseCounts()

You’ll see:

  • Query counts
  • Latency
  • Backend state
  • Cache hits

🔵 8️⃣ Optional: Prometheus Exporter (ISP Grade)

Add:

webserver("0.0.0.0:8083")
setWebserverConfig({password="admin123", apiKey="secret"})

Then access:

You get live stats.

⚠ Secure properly in production.

🔵 Example Clean Production Block (Recommended Final Version)

Here is consolidated core tuning block:

setLocal("0.0.0.0:53")
addACL("10.10.2.0/24")
-- Packet cache
pc = newPacketCache(10000, {maxTTL=60})
getPool("rec"):setCache(pc)
-- Abuse protection
addAction(QTypeRule(DNSQType.ANY), DropAction())
addAction(MaxQPSIPRule(50), DropAction())
-- Health checks
newServer({address="10.10.2.163:53", pool="auth", checkType="A", checkName="zaibdns.lab.", checkInterval=5})
newServer({address="10.10.2.164:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
newServer({address="10.10.2.165:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
local suffixes = newSuffixMatchNode()
suffixes:add(newDNSName("zaibdns.lab."))
addAction(SuffixMatchNodeRule(suffixes), PoolAction("auth"))
addAction(AllRule(), PoolAction("rec"))
controlSocket("127.0.0.1:5199")

🎯 What You Have Now

If you enable all above:

✔ Caching layer
✔ Backend health detection
✔ Rate limiting
✔ Basic abuse protection
✔ Failover HA
✔ Metrics visibility
✔ TCP control

This is now serious ISP-grade DNS architecture.

Few thoughts about architecture … SJz

🔎 1️⃣ Is The Architecture Correct For 100k Users?

Your design:

Clients
↓
VRRP VIP
↓
2x dnsdist (HA)
↓
Auth Pool + Rec Pool
↓
2x Recursive + 1x Auth

This is industry-standard L7 DNS load-balancer model.

Used by:

  • Mid-size ISPs
  • Hosting providers
  • MSPs
  • Regional broadband operators

So yes > conceptually correct.

🔎 2️⃣ 100k Users → What Load Does That Mean?

Typical ISP DNS usage:

  • 3–10 QPS per subscriber during peak
  • 100k subs × avg 2–3 active at same moment
  • Realistic peak: 15k–40k QPS

During Netflix / Android updates / cache expiry bursts:

  • 50k+ QPS spikes possible

Your LAB config (10k cache entries, 50 QPS limit) is too small for that.

Architecture is fine.
Sizing must change.

🔵 3️⃣ What Would Be Required For 100k Subscribers?

✅ dnsdist Layer

Minimum recommended per node:

  • 8–16 vCPU
  • 16–32 GB RAM
  • Packet cache 500k–1M entries
  • NIC tuned for high PPS
  • IRQ affinity tuned
  • RPS/RFS enabled

Example production packet cache:

  • pc = newPacketCache(500000, {maxTTL=300})

✅ Recursive Layer

For 100k subs:

Two recursive servers are borderline.

Better:

  • 3–4 recursive nodes
  • Each 8–16 cores
  • 32 GB RAM
  • Proper ulimit tuning
  • Large resolver cache

In BIND:

  • max-cache-size 8g;
  • recursive-clients 50000;

✅ Authoritative Layer

Auth load is typically very low.

1 primary + 1 secondary recommended.

✅ Network Layer

Must ensure:

  • Multicast allowed (VRRP)
  • NIC offloading tuned
  • Firewall not bottlenecking
  • MTU correct
  • No stateful inspection on DNS traffic

🔎 4️⃣ Is dnsdist Used In Serious ISP Deployments?

Yes.

dnsdist (by PowerDNS) is widely used in:

  • ISPs
  • CDN providers
  • Hosting companies
  • Enterprise resolvers
  • Cloud operators

It is not hobby software.

It supports:

  • 1M+ QPS on proper hardware
  • Advanced rate limiting
  • Geo routing
  • DNS filtering
  • DoT/DoH frontend

🔎 5️⃣ Is OUR Current Lab Enough For 100k?

In current lab sizing:

❌ No (hardware too small)
❌ Cache too small
❌ Recursive count too small

But:

✔ Architecture pattern is correct
✔ Failover model correct
✔ Separation correct
✔ Routing logic correct

So design is scalable.

🔵 6️⃣ Real-World Upgrade Path For 100k ISP

I would recommend:

2x dnsdist (active/active possible)
3x recursive nodes
2x authoritative nodes
Anycast (optional future)
Monitoring (Prometheus + Grafana)

🔎 7️⃣ Real Question: Single VIP or Dual IP?

For 100k users:

Better to provide clients:

  • Primary DNS: VIP
  • Secondary DNS: VIP (same)

Redundancy handled at server layer.

Or:

Active/Active with ECMP or Anycast if advanced.

🔵 8️⃣ Where Would This Design Break?

It would break if:

  • Recursive servers undersized
  • Cache too small
  • CPU too low
  • Too aggressive rate limiting
  • No kernel tuning

Not because of architecture.

One more topic to go with (zaib)

How load balancing will be done via DNSDIST?

Since users are asking since publishing of this post, here is the reply. (zaib)

You defined:

newServer({address="10.10.2.164:53", pool="rec"})
newServer({address="10.10.2.165:53", pool="rec"})

And:

addAction(AllRule(), PoolAction("rec"))

So when a client queries:

google.com

Flow is:

Client → VIP → dnsdist → REC pool → REC1 or REC2

2️⃣ How Does dnsdist Distribute Traffic?

By default, dnsdist uses least outstanding queries (latency-aware load balancing).

That means:

  • It does NOT strictly do round-robin
  • It sends traffic to the server with fewer active queries
  • It prefers lower-latency backends

So it is intelligent load balancing, not naive rotation.

3️⃣ Will Load Be Even?

Not exactly 50/50.

Distribution depends on:

  • Backend response time
  • Current query backlog
  • Health status
  • TCP/UDP mix

If both servers are equal hardware and same latency:

→ Load will be very close to balanced.

If one server is slightly faster:

→ It may receive slightly more traffic.

This is good behavior.

4️⃣ What About Cache?

Important detail:

You enabled packet cache on dnsdist:

pc = newPacketCache(...)
getPool("rec"):setCache(pc)

That means:

  • First query hits recursive
  • Subsequent identical queries may be answered directly from dnsdist
  • Backend load reduces
  • Cache hits handled at frontend

So backend distribution applies only to cache misses.

5️⃣ What Happens If One Recursive Fails?

If REC1 fails:

  • Health check fails
  • dnsdist marks it DOWN
  • All traffic goes to REC2 automatically
  • No manual action required

That’s real production-grade behavior.

6️⃣ If You Want Strict Round Robin

You can force it:

setServerPolicy(roundrobin)

But this is NOT recommended in ISP production.

Latency-aware balancing is better.

7️⃣ How To Verify Load Balancing Live

On dnsdist console:

dnsdist -c
showServers()

You will see:

  • queries handled per backend
  • latency
  • state (UP/DOWN)

Run repeated:

dig @10.10.2.160 google.com

And watch counters increment.

8️⃣ Important ISP Insight

For 100K subscribers:

2 recursive servers is minimum.

Better production design:

  • 3 recursive nodes
  • Or 2 strong nodes + 1 backup

dnsdist will distribute automatically.

Final Answer for Load Balancing via DNSDIST

Yes > dnsdist will load balance between both recursive servers automatically.

It uses intelligent latency-aware distribution, not basic round-robin.

It also automatically removes failed backends from rotation.

🎯 Final Professional Answer

Yes > this architecture is absolutely suitable for 100k subscribers.

But:

  • It must be deployed on proper hardware,
  • properly tuned,
  • and monitored.

OUR lab has proven:

  • Design works
  • HA works
  • Routing works
  • Backend failover works

That is exactly what matters before production.

Final Conclusion

dnsdist + VRRP + backend separation is a production-grade DNS architecture suitable for 50K–100K subscriber ISPs.

This design provides:

  • High availability
  • Intelligent routing
  • Backend redundancy
  • Security controls
  • Cost efficiency

Important:
dnsdist is not a DDoS appliance.
Edge filtering still required.

For Pakistani cable-net ISPs, this model delivers enterprise-level stability without expensive hardware appliances.

DNS is core infrastructure. Design it accordingly.

 Syed Jahanzaib