Building ISP-Grade DNS Infrastructure Using DNSDIST + VRRP (50K~100K Users Design Model)
~under review
- Author: Syed Jahanzaib ~A Humble Human being! nothing else
- Platform: aacable.wordpress.com
- Category: ISP Infrastructure / DNS Engineering
- Audience: ISP Engineers, NOC Teams, Network Architects
⚠️ Disclaimer & Note on Writing Style
Every network environment is unique. A solution that works effectively in one infrastructure may require modification in another. Readers are strongly encouraged to understand the underlying concepts and adapt the guidance according to their own architecture, operational policies, and risk tolerance.
Blind copy-paste implementation without proper validation, testing, and change management is never recommended — especially in production environments. Always ensure proper backups and risk assessment before applying any configuration.
The content shared here is based on hands-on experience from real-world deployments, ISP environments, lab testing, and continuous learning. While I strive for technical accuracy, no technical implementation is entirely free from the possibility of error. Constructive discussion and alternative approaches are always welcome.
Due to professional commitments, it is not always feasible to publish highly detailed or multi-part write-ups. The technical logic and implementation details are written based on my own practical experience. AI tools such as ChatGPT are used only to refine grammar, structure, and presentation — not to generate the core technical concepts.
This blog is not intended for client acquisition or follower growth. It exists solely to share practical knowledge and real-world experience with the community.
Thank you for your understanding and continued support.
📌 Article Roadmap >What This Guide Covers
In this detailed ISP-grade DNS architecture guide, I have covered the following sections:
- Introduction & Design Objectives
Explains why traditional DNS fails in ISP networks and defines the core engineering objectives for a scalable, highly available DNS architecture. - Scope & Audience
Clarifies what is included in this guide and who will benefit most from it. - High-Level Architecture Overview
Presents the recommended DNS infrastructure model using dnsdist + VRRP, including role separation and failure domains. - Capacity Planning & Traffic Expectations
Discusses realistic QPS and sizing models for 50K–100K subscribers, including cache hit assumptions and peak load calculations. - dnsdist Frontend Configuration
Covers dnsdist installation, load-balancing policy selection, backend pools, rate limiting and health checks. - Recursive & Authoritative Server Setup
Provides detailed guidance for configuring recursive and authoritative BIND instances, including isolation and security hardening. - Keepalived + VRRP High Availability Setup
Walks through VRRP configuration, priority planning, timers, split-brain prevention, and process tracking. - Kernel & OS Level Optimizations
Covers performance tuning at the OS level (network, limits, buffer sizes) for high-packet-rate DNS workloads. - Monitoring & Observability Architecture
Prescribes a monitoring stack with metrics, dashboards and alerting targets for production operations. - Scaling Beyond 100K Users
Explains how to grow the architecture horizontally and introduces future-ready concepts like Anycast and multi-datacenter distribution. - Operational Workflows & Maintenance
Shares best practices for rolling upgrades, backups, failover testing, and lifecycle management. - FAQ & Edge-Case Scenarios
Answers common implementation questions and illustrates practical traffic-routing examples. - Appendix / Production-Ready Config Snippets
Includes tested, copy-ready configuration examples for dnsdist, Keepalived and BIND.
Introduction
In most Pakistani cable-net ISPs, DNS is treated as a secondary service , until it fails. When DNS fails, customers report “Internet not working” even though PPPoE is connected and routing is fine.
DNS is core infrastructure. For ISPs serving 50,000~100,000+ subscribers, DNS must be:
- Highly available
- Scalable
- Secure
- Monitored
- Redundant
Design Objectives & Scope
- Design Objectives
The objective of this DNS architecture is to build a production-grade, high-availability, scalable DNS infrastructure suitable for medium to large ISPs (50,000–100,000 subscribers), with clear separation of roles, deterministic failover behavior, and measurable performance boundaries.
This design is built around the following core engineering principles:
1.1 Infrastructure-Level Redundancy
Failover must not depend on:
- Subscriber CPE behavior
- Operating system DNS retry timers
- Application-layer retries
Redundancy must be handled at the infrastructure level using:
- VRRP floating IP
- Dual dnsdist frontend nodes
- Backend health checks
Failover target: ≤ 3 seconds convergence.
1.2 Separation of Recursive and Authoritative Roles
Recursive and Authoritative DNS must not coexist on the same server in ISP-scale deployments.
This design enforces:
- Dedicated authoritative server(s)
- Dedicated recursive server pool
- Controlled routing via dnsdist
Benefits:
- Security isolation
- Independent performance tuning
- Contained failure domains
- Clear operational visibility
1.3 Horizontal Scalability
The architecture must allow:
- Adding new recursive servers without service interruption
- Increasing QPS handling capacity without redesign
- Backend pool expansion without client configuration change
Scaling must be horizontal-first, not vertical-only.
1.4 Deterministic Failover
Failover logic must be:
- Script-based
- Process-aware
- Health-check driven
- Predictable under load
VRRP must:
- Immediately relinquish VIP if dnsdist stops
- Promote standby node within controlled detection interval
1.5 Abuse Resistance & Operational Hardening
The DNS layer must include:
- Rate limiting
- ANY query suppression
- Backend health checks
- ACL-based query restriction
- Recursive exposure protection
This prevents:
- Amplification abuse
- Internal malware flooding
- Resource exhaustion attacks
- Backend overload during update storms
1.6 Performance Measurability
The system must allow:
- QPS measurement
- Backend latency tracking
- Cache hit ratio monitoring
- Failover verification testing
- Resource utilization visibility
No production DNS infrastructure should operate without measurable metrics.
- Scope of This Deployment Blueprint
This document covers:
- Full deployment sequence from OS preparation to HA activation
- dnsdist frontend configuration with aggressive tuning
- BIND authoritative configuration
- BIND recursive configuration
- Keepalived VRRP configuration
- Kernel-level performance tuning
- Capacity planning logic
- Failure testing methodology
- Production hardening recommendations
- Out of Scope (Explicitly)
The following are not covered in this blueprint:
- Global Anycast BGP-based DNS distribution
- DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT) frontend implementation
- Multi-datacenter geo-distributed architecture
- Commercial DNS hardware appliance comparison benchmarking
- DNSSEC zone signing strategy
These may be addressed in future parts.
- Intended Audience
This document is intended for:
- ISP Network Architects
- NOC Engineers
- Systems Administrators
- Broadband Infrastructure Operators
- Technical leads in 50K–100K subscriber environments
This is not a beginner tutorial.
It assumes familiarity with:
- Linux system administration
- BIND
- Networking fundamentals
- VRRP
- Basic ISP architecture
- Expected Outcome
After implementing this design, the ISP should achieve:
- Infrastructure-level DNS high availability
- Predictable failover behavior
- Controlled recursive exposure
- Measurable QPS performance
- Reduced subscriber outage perception
- Scalable DNS backend architecture
DNS transitions from:
“Just another Linux service”
to
“Core ISP control-plane infrastructure.”
DNSDIST! what is it?
This guide explains how to build a professional DNS architecture using:
- DNSDIST as frontend DNS load balancer
- Recursive / Authoritative separation <<<
- VRRP-based High Availability
- Packet cache & rate limiting
- Scalable backend design
🔹Is DNSDISTIndustry-Style? Or Hobby Level?
DNSDIST is absolutely industry-grade.
It is:
- Developed by PowerDNS
- Used by:
- Large hosting providers
- Cloud providers
- IX-level DNS infrastructures
- Serious ISPs
- Designed specifically for:
- High QPS
- DNS DDoS mitigation
- Load balancing authoritative & recursive farms
This is NOT a lab tool.
It is widely deployed in production worldwide.
Key Architectural Shift
Old Model:
Redundancy at edge (client).
New Model:
Redundancy at core (infrastructure).
That is the fundamental upgrade in DNS architecture philosophy.
🔹Recommended Architecture for 50k+ ISP
Minimum Safe Production Design
- 2x dnsdist (HA)
- 3–4x Recursive Servers
- 2x Authoritative Servers
- Separate VLANs
- Monitoring + Rate limiting
🔹 Hardware Guideline (Recursive)
Per node:
- 8–16 CPU cores
- 32–64 GB RAM
- NVMe (for logs)
- 10G NIC preferred
DNS is mostly CPU + RAM heavy (cache efficiency matters).
🔹 Why DNSDIST Becomes Useful at 50k+ Scale
Without DNSDIST:
- Clients directly hit recursive
- No centralized rate limiting
- No traffic shaping
- Harder to isolate DDoS
- Hard to scale cleanly
With DNSDIST:
✔ Central traffic control
✔ Backend pool management
✔ Active health checks
✔ Per-IP QPS limiting
✔ Easy horizontal scaling
✔ Easy separation (auth vs rec)
🔹 What Serious ISPs Actually Do
At this size, typical models are:
- Model A – DNSDIST+ Unbound/BIND cluster
Very common
- Model B – Anycast DNS (advanced tier)
Used by larger national ISPs
- Model C – Appliance-based (Infoblox, F5 DNS, etc.)
Expensive, enterprise heavy
DNSDISTsits between open-source and enterprise appliance , very powerful balance.
🔹 Would I Recommend dnsdist for 50k+ ISP?
Yes > if:
- You want scalable architecture
- You want control
- You want DDoS handling layer
- You want future growth to 150k–200k users
No > if:
- Very small budget
- No in-house Linux expertise
- No monitoring culture
🔹 Strategic Advice
At 50k+ subscribers:
- Single DNS server is negligence
- Single dnsdist is risky
- Proper HA + scaling is mandatory
DNS outage at this scale = full network outage perception.
🔹 Final Verdict
For 50k+ ISP:
DNSDIST is:
✔ Industry proven
✔ Production ready
✔ Cost effective
✔ Scalable
- It is not overkill.
- It is appropriate engineering.
Traditional DNS Models in Pakistani Cable ISPs
Executive Context – The Pakistani Cable ISP Reality
In many Pakistani cable-net environments:
- MikroTik PPPoE NAS handles subscribers
- RADIUS authenticates
- One or two BIND servers provide DNS
- No frontend load balancer
- No recursive/authoritative separation
- No QPS monitoring
- No health checks
Common symptoms at 30K–80K subscribers:
- CPU spikes during Android update waves
- Recursive server freeze
- Cache poisoning attempts
- DNS amplification attempts
- Failover delays when one DNS IP stops responding
Traditional “Primary/Secondary DNS” model relies on client retry timers. That is not infrastructure-grade redundancy. Modern ISP design must shift failover responsibility from client to infrastructure.
Architectural Philosophy
Why Single DNS Server is Wrong
- Single server = single point of failure.
- Even if uptime is 99.5%, subscriber perception during outage is 0%.
Why Primary / Secondary is Not Enough
Primary/Secondary:
- Client decides when to retry.
- Retry delay depends on OS resolver behavior.
This causes:
- 5–30 seconds browsing delay
- Perceived outage
- Increased support calls
Infrastructure-level redundancy is superior.
Control Plane vs Data Plane
We separate roles:
Control Plane (dnsdist):
- Load balancing
- Rate limiting
- Traffic classification
- Health monitoring
Data Plane:
- Recursive resolution
- Authoritative zone serving
This allows independent scaling.
Recommended Modern ISP DNS Architecture
Client → VRRP VIP (DNSDIST)
┌──────────────────┐
│ dnsdist HA x2 │
└──────────────────┘
|
┌──────────────┬─────────────────┐
│ Auth Pool │ Rec Pool │
│ (BIND) x2 │(BIND/Unbound) x2|
└──────────────┴─────────────────┘
🔹 Operational Best Practices
✔ Monitoring (Prometheus/Grafana)
✔ Log sampling only (avoid full query logging)
✔ Separate management VLAN
✔ Disable recursion on authoritative
✔ Disable public access to backend IPs
🔹 Result
- No Single Point of Failure
- Clean separation (Auth vs Rec)
- Scalable horizontally
- Controlled DDoS surface
Why This Design Works
✔ Zero backend exposure
✔ Clean separation of duties
✔ Easy scaling (add more recursive nodes in DNS VLAN)
✔ Maintenance without downtime
✔ Audit-friendly (clear segmentation)
DNS ISP infra mein critical service hai 🙂
- RADIUS down ho to login issue hota hai,
- DNS down ho to poora internet “down” lagta hai.
🔎 1️⃣ Is The Architecture Correct For 100k Users?
Your design:
-
Clients ↓ VRRP VIP ↓ 2x dnsdist (HA) ↓ Auth Pool + Rec Pool ↓ 2x Recursive + 1x Auth
This is industry-standard L7 DNS load-balancer model.
Used by:
- Mid-size ISPs
- Hosting providers
- MSPs
- Regional broadband operators
So yes > conceptually correct.
🔎 2️⃣ 100k Users → What Load Does That Mean?
Typical ISP DNS usage:
- 3–10 QPS per subscriber during peak
- 100k subs × avg 2–3 active at same moment
- Realistic peak: 15k–40k QPS
During Netflix / Android updates / cache expiry bursts:
- 50k+ QPS spikes possible
Our LAB config (10k cache entries, 50 QPS limit) is too small for that.
- Architecture is fine.
- Sizing must change.
🔵 3️⃣ What Would Be Required For 100k Subscribers?
✅ dnsdist Layer
Minimum recommended per node:
- 8–16 vCPU
- 16–32 GB RAM
- Packet cache 500k–1M entries
- NIC tuned for high PPS
- IRQ affinity tuned
- RPS/RFS enabled
Example production packet cache:
pc = newPacketCache(500000, {maxTTL=300})
✅ Recursive Layer
For 100k subs:
Two recursive servers are borderline.
Better:
- 3–4 recursive nodes
- Each 8–16 cores
- 32 GB RAM
- Proper ulimit tuning
- Large resolver cache
In BIND:
- max-cache-size 8g
- recursive-clients 50000;
✅ Authoritative Layer
- Authoritative load is typically very low.
- 1 primary + 1 secondary recommended.
✅ Network Layer
Must ensure:
- Multicast allowed (VRRP)
- NIC offloading tuned
- Firewall not bottlenecking
- MTU correct
- No stateful inspection on DNS traffic
🔎 4️⃣ Is dnsdist Used In Serious ISP Deployments?
Yes.
dnsdist (by PowerDNS) is widely used in:
- ISPs
- CDN providers
- Hosting companies
- Enterprise resolvers
- Cloud operators
It is not hobby software.
It supports:
- 1M+ QPS on proper hardware
- Advanced rate limiting
- Geo routing
- DNS filtering
- DoT/DoH frontend
🔎 5️⃣Is OUR Current Lab Enough For 100k?
In current lab sizing:
❌ No (hardware too small)
❌ Cache too small
❌ Recursive count too small
But:
✔ Architecture pattern is correct
✔ Failover model correct
✔ Separation correct
✔ Routing logic correct
So design is scalable.
🔵 6️⃣ Real-World Upgrade Path For 100k ISP
I would recommend:
- 2x dnsdist (active/active possible)
- 3x recursive nodes
- 2x authoritative nodes
- Anycast (optional future)
- Monitoring (Prometheus + Grafana)
🔎 7️⃣ Real Question: Single VIP or Dual IP?
For 100k users:
Better to provide clients:
- Primary DNS: VIP
- Secondary DNS: VIP (same)
Redundancy handled at server layer.
Or:
Active/Active with ECMP or Anycast if advanced.
🔵 8️⃣ Where Would This Design Break?
It would break if:
- Recursive servers undersized
- Cache too small
- CPU too low
- Too aggressive rate limiting
- No kernel tuning
Not because of architecture.
🎯 Final Professional Answer
Yes > this architecture is absolutely suitable for 100k subscribers.
But:
- It must be deployed on proper hardware
- properly tuned
- and monitored.
Your lab has proven:
- Design works
- HA works
- Routing works
- Backend failover works
That is exactly what matters before production.
Deployment Blueprint – Exact Sequence
We use the following topology:
✅ Finalized Lab IP Plan
| Hostname | Role | IP |
| DD-VRRP-IP | Floating VIP | 10.10.2.160 |
| LAB-DD1 | dnsdist-1 | 10.10.2.161 |
| LAB-DD2 | dnsdist-2 | 10.10.2.162 |
| LAB-AUTH1 | Authoritative BIND | 10.10.2.163 |
| LAB-REC1 | Recursive BIND | 10.10.2.164 |
| LAB-REC2 | Recursive BIND | 10.10.2.165 |
| LAB-CLIENT1 | Test Windows | 10.10.2.166 |
Very clean numbering 👍
🔎 Important Design Note (Very Important)
Right now everything is in:
- 10.10.2.0/24
For lab this is OK.
But remember in production:
- dnsdist public interface
- backend DNS VLAN
- management VLAN
should ideally be separated.
For lab → single subnet is fine.
How Many VMs Required?
Minimum lab set:
| Role | Qty |
| dnsdist | 2 |
| BIND Authoritative | 1 |
| BIND Recursive | 2 |
| Windows Client | 1 |
| (Optional Monitoring) | 1 |
✅ Total Minimum: 6 VMs
(7 if you add monitoring like Zabbix/Prometheus later)
Minimum Hardware Sizing (LAB Only)
Since this is not production load:
🔹 dnsdist VM (each)
- 2 vCPU
- 2 GB RAM
- 20 GB disk
- 2 NICs (Recommended)
- NIC1 → VLAN-2 (Public simulation)
- NIC2 → DNS VLAN (Backend network)
🔹 BIND Authoritative
- 2 vCPU
- 2 GB RAM
- 20 GB disk
- 1 NIC (DNS VLAN)
🔹 BIND Recursive (each)
- 2 vCPU
- 2 GB RAM
- 20 GB disk
- 1 NIC (DNS VLAN)
🔹 Windows Client
- 2 vCPU
- 4 GB RAM
- 40 GB disk
- 1 NIC (VLAN-2)
💡 Lab Total Resource Footprint
Approx:
- 12–14 vCPU
- 14–16 GB RAM
Very manageable in VMware test cluster.
Few Queries for above scheme
✅ Query #1
Should internal users get:
- Only VIP → 10.10.2.160
OR - Two real IPs → 10.10.2.161 and 10.10.2.162 ?
🔹 Correct Answer (With VRRP)
If you are using:
- 2x dnsdist
- VRRP Floating IP (10.10.2.160)
👉 Clients should receive ONLY the VIP (10.10.2.160)
Why?
Because:
- VIP always exists
- If dnsdist-1 fails → VIP moves to dnsdist-2
- Clients don’t need to know which node is active
- Clean failover
This is standard HA design.
🔹 When Would You Give 2 IPs?
You would give:
- Primary DNS: 10.10.2.160(VIP)
- Secondary DNS: 10.10.2.162 (optional)
Only if:
- You are not fully trusting VRRP
- Or you want additional redundancy layer
- Or you are not using floating IP
But in proper HA design:
One VIP is enough.
🔹 Best Practice for 50k+ ISP
Subscribers receive:
- Primary DNS: 10.10.2.160
- Secondary DNS: 10.10.2.160
Yes > same IP twice is fine when using VRRP HA.
You may find it strange, but The redundancy is at server layer, not IP layer.
✅ Query #2
Authoritative used for internal + external > how will it function?
This is about traffic separation.
Remember your architecture:
Internet → dnsdist (VIP) | -------------------------------- | | Authoritative Pool Recursive Pool
dnsdist decides where query goes.
🔹 Case A > External Client Query
Example:
External user queries:
ns1.yourisp.com
Flow:
Internet → VIP → dnsdist → Authoritative (10.10.10.10)
Recursive pool is NOT involved.
🔹 Case B > Internal Subscriber Query
Subscriber asks:
google.com
Flow:
Subscriber → VIP → dnsdist → Recursive pool
Authoritative not involved.
🔹 Case C > Internal Query for ISP Domain
Subscriber asks:
portal.yourisp.com
Flow:
Subscriber → VIP → dnsdist → Authoritative
Works same as external.
🔹 How Does dnsdist Know Where to Send?
Usually:
Option 1 > Domain-based routing (Recommended)
- addAction(RegexRule(“yourisp.com”), PoolAction(“auth”))
- addAction(AllRule(), PoolAction(“rec”))
Everything else → recursive
Your own domains → authoritative
🔹 Important Best Practice
On Authoritative server:
❌ Disable recursion
In BIND:
- recursion no;
So even if misrouted traffic comes, it won’t resolve internet domains.
🔹 Very Important for ISP
Recursive servers:
- Should allow only subscriber IP ranges
- Should not be open resolver to world
Authoritative:
- Should answer only hosted zones
- Should not do recursion
dnsdist enforces clean split.
🔹 Final Clean Answers
Query #1:
Give clients ONLY the VIP (10.10.2.160).
Query #2:
dnsdist routes queries to:
- Authoritative pool for your domains
- Recursive pool for everything else
Both internal and external clients can use same VIP > routing logic handles separation.
Architecture Overview (Layer-by-Layer Flow)
This DNS architecture is not simply a dual-server deployment.
It is a layered control-plane model designed to:
- Contain failures
- Classify traffic
- Absorb load bursts
- Maintain deterministic failover
- Enable horizontal scaling
The system is divided into five logical layers.
Layer 1 – Subscriber Access Layer
This is the ingress layer.
Traffic Origin:
- PPPoE subscribers
- CGNAT subscribers
- Internal LAN clients
- Management clients (if allowed)
Subscribers are configured to use:
DNS = 10.10.2.160 (DD-VRRP-IP)
Key property:
Subscribers never see backend servers.
They only see the VIP.
This ensures:
- No backend IP exposure
- No client-side failover logic
- Simplified DHCP configuration
- Clean abstraction layer
Failure containment:
Even if one dnsdist node fails, the VIP floats. Clients are unaware.
Layer 2 – Frontend Control Plane (dnsdist HA Pair)
Nodes:
LAB-DD1 (10.10.2.161)
LAB-DD2 (10.10.2.162)
Floating IP:
10.10.2.160
Role:
DNS traffic controller and policy engine.
Responsibilities:
- Accept UDP/TCP 53 traffic
- Apply ACL rules
- Apply rate limiting
- Drop abusive queries
- Classify domain type
- Route to correct backend pool
- Cache responses
- Monitor backend health
This is the most critical layer in the system.
It does NOT perform recursive resolution.
It performs traffic governance.
2.1 VRRP Behavior
VRRP ensures:
- Only one frontend holds 10.10.2.160 at a time.
- If MASTER fails, BACKUP becomes MASTER.
- If dnsdist process fails, VIP relinquished.
Failover flow:
dnsdist crash → keepalived detects → priority lost → VIP moves → service restored in 2–3 seconds.
This removes dependency on:
- Client retry timers
- Secondary DNS IP logic
- Application resolver behavior
Failover is deterministic.
Layer 3 – Traffic Classification Engine (Inside dnsdist)
Once a DNS packet arrives at VIP:
dnsdist evaluates rules in order.
Example logic:
If domain suffix = zaibdns.lab
→ send to “auth” pool
Else
→ send to “rec” pool
Additionally:
- ANY query dropped
- Excess QPS per IP dropped
- Non-allowed subnet rejected
This classification stage is critical.
Without classification:
- Recursive and authoritative mix
- Backend tuning conflicts
- Security boundaries blur
dnsdist enforces traffic discipline.
Layer 4 – Backend Pools
There are two independent backend pools:
AUTH Pool:
LAB-AUTH1 (10.10.2.163)
REC Pool:
LAB-REC1 (10.10.2.164)
LAB-REC2 (10.10.2.165)
These pools are isolated.
dnsdist maintains health status per server.
4.1 Authoritative Pool
Purpose:
Serve local zones only.
Properties:
- recursion disabled
- publicly queryable (if required)
- low QPS compared to recursive
Failure impact:
Only local zone resolution affected.
Does NOT affect internet browsing.
4.2 Recursive Pool
Purpose:
Resolve internet domains.
Properties:
- recursion enabled
- restricted to subscriber subnet
- large cache memory
- high concurrency settings
Failure behavior:
If REC1 fails:
dnsdist stops sending traffic to it.
REC2 continues serving.
If both fail:
Service disruption occurs.
This is why horizontal scaling is recommended for 100K users.
Layer 5 – Internet Resolution Layer
Recursive servers query:
- Root servers
- TLD servers
- Authoritative internet servers
This layer is outside ISP control.
However:
Packet cache in dnsdist reduces external dependency frequency.
High cache hit ratio = lower external latency.
End-to-End Query Flow Example
Scenario 1: Subscriber queries www.google.com
Step 1:
Client sends query to 10.10.2.160
Step 2:
dnsdist receives packet
Step 3:
Suffix does NOT match local zone
Step 4:
dnsdist forwards to REC pool
Step 5:
Recursive server checks cache
If cache miss → resolves via internet
If cache hit → replies immediately
Step 6:
dnsdist optionally caches packet
Step 7:
Response sent to subscriber
Scenario 2: Subscriber queries www.zaibdns.lab
Step 1:
Packet arrives at VIP
Step 2:
Suffix matches local zone
Step 3:
dnsdist forwards to AUTH pool
Step 4:
Authoritative server responds
Step 5:
dnsdist relays response
Recursive servers are never involved.
Failure Domain Isolation
Let’s analyze impact per failure.
Failure: LAB-REC1 crash
Impact: 50% recursive capacity lost
Mitigation: REC2 continues
Failure: LAB-AUTH1 crash
Impact: Local zone fails
Internet browsing unaffected
Failure: LAB-DD1 crash
Impact: VIP moves to LAB-DD2
Subscriber impact: ~2–3 seconds max
Failure: dnsdist process crash on MASTER
Impact: VIP released immediately
Failover triggered
Failure: Kernel UDP overload on one frontend
Impact: Traffic handled by second frontend if VRRP triggered
This layered model ensures limited blast radius.
Logical Separation of Concerns
| Layer | Responsibility | Failure Impact |
| Subscriber | Query origin | None |
| dnsdist | Traffic governance | Frontend failover |
| AUTH pool | Local zones | Local zone only |
| REC pool | Internet resolution | Internet browsing |
| Internet | External resolution | External dependency |
Clear separation improves troubleshooting.
Why This Layered Model Matters
Without layering:
- Recursive and authoritative mixed
- No policy enforcement
- No health-driven routing
- No horizontal scaling path
With layering:
- Each component has defined responsibility
- Each failure has defined boundary
- Scaling can be targeted
- Security can be enforced per layer
This is the difference between:
“Two DNS servers”
and
“A DNS infrastructure.”
Now tell me where you want to go next:
- Detailed Logical Traffic Flow with timing metrics
- VRRP State Machine Deep Explanation
- Backend Health Check Mechanics Deep Dive
- Performance Modeling & Cache Efficiency Analysis
- Deployment Sequence – Step 1 (Aggressive OS Hardening)
We continue building this as a full engineering whitepaper.
OS Preparation (All Servers)
Ubuntu 22.04 recommended.
Disable systemd-resolved
Reason:
- Ubuntu binds 127.0.0.53:53 by default.
- dnsdist requires port 53.
Commands:
sudo systemctl stop systemd-resolved sudo systemctl disable systemd-resolved sudo rm /etc/resolv.conf echo "nameserver 8.8.8.8" > /etc/resolv.conf
🔹 Production Notes for Ubuntu 22
✔ Ubuntu 22 is stable for ISP DNS use
✔ Works fine with Keepalived
✔ Supports high kernel tuning
✔ Good for 10k–50k+ QPS per node (proper hardware required)
🔹 Important Tuning (Must Do in Production)
In /etc/sysctl.conf:
net.core.rmem_max=25000000 net.core.wmem_max=25000000 net.core.netdev_max_backlog=50000
Then:
sudo sysctl -p
Without kernel tuning, high QPS performance suffer karega.
🎯 Lab Build Order (Important)
Always follow this order:
1️⃣ Backend first (BIND servers working standalone)
2️⃣ Then dnsdist (single node)
3️⃣ Then HA (Keepalived)
Never start with HA first.
🔵 Final Zone Design
Zone name: zaibdns.lab Primary NS: ns1.zaibdns.lab Test records: www.zaibdns.lab portal.zaibdns.lab
Authoritative DNS Configuration (LAB-AUTH1)
🔷 Now Configure on LAB-AUTH1 (10.10.2.163)
🔵 STEP 1 > Install BIND
sudo apt update sudo apt install bind9 bind9-utils bind9-dnsutils -y
Verify service:
sudo systemctl status bind9
It should show:
- Active: active (running)
If not running:
sudo systemctl start bind9
🔵 STEP 2 > Configure BIND as Authoritative Only
Edit options file:
sudo nano /etc/bind/named.conf.options
Replace entire content with:
options {
directory "/var/cache/bind";
recursion no;
allow-query { any; };
listen-on { 10.10.2.163; };
listen-on-v6 { none; };
};
Save and exit.
🔵 STEP 3 > Define Zone
Edit:
sudo nano /etc/bind/named.conf.local
Add this at bottom:
zone "zaibdns.lab" {
type master;
file "/etc/bind/db.zaibdns.lab";
};
Save.
🔵 STEP 4 > Create Zone File
sudo nano /etc/bind/db.zaibdns.lab
Paste this:
$TTL 86400 @ IN SOA ns1.zaibdns.lab. admin.zaibdns.lab. ( 2026021401 3600 1800 604800 86400 ) IN NS ns1.zaibdns.lab. ns1 IN A 10.10.2.163 www IN A 10.10.2.163 portal IN A 10.10.2.163
Save.
🔵 STEP 5 > Check Configuration (Very Important)
Run:
sudo named-checkconf
No output = good.
Then:
sudo named-checkzone zaibdns.lab /etc/bind/db.zaibdns.lab
It must say:
OK
If error appears, stop and fix.
🔵 STEP 6 > Restart BIND
sudo systemctl restart bind9 sudo systemctl status bind9
Ensure it is running.
🔵 STEP 7 > Test Authoritative Function
From another VM (LAB-DD1 or LAB-REC1):
dig @10.10.2.163 www.zaibdns.lab
You should see:
ANSWER SECTION:
http://www.zaibdns.lab. 86400 IN A 10.10.2.163
🔵 STEP 8 > Confirm Recursion Is Disabled
Test:
dig @10.10.2.163 google.com
It should FAIL (no answer section).
If it resolves google.com → recursion not disabled properly.
🎯 Expected Result
Authoritative server should:
✔ Resolve zaibdns.lab records
✔ NOT resolve internet domains
✔ Respond on 10.10.2.163 only
🎯 When This Works
Once AUTH server working for zaibdns.lab
🚀 Next Phase
Now we move to:
🔵 PHASE 2 > Recursive DNS Setup
On:
- LAB-REC1 (10.10.2.164)
- LAB-REC2 (10.10.2.165)
We will configure them as:
- Recursive-only resolvers
- Allow queries only from 10.10.2.0/24
- Disable zone hosting
- Enable caching
- Ready for dnsdist pool
🔵 STEP 1 > Install BIND (On BOTH REC1 & REC2)
Run on each:
sudo apt install bind9 bind9-utils bind9-dnsutils -y
Verify:
sudo systemctl status bind9
Must show active (running).
🔵 STEP 2 > Configure Recursive Resolver
Edit:
sudo nano /etc/bind/named.conf.options
Replace entire content with this (adjust listen IP per server):
🔹 On LAB-REC1 (10.10.2.164)
options {
directory "/var/cache/bind";
recursion yes;
allow-recursion { 10.10.2.0/24; };
allow-query { 10.10.2.0/24; };
listen-on { 10.10.2.164; };
listen-on-v6 { none; };
dnssec-validation auto;
};
🔹 On LAB-REC2 (10.10.2.165)
Same config, just change:
listen-on { 10.10.2.165; };
🔵 STEP 3 > Remove Default Zones (Optional but Clean)
on both REC servers, Open:
sudo nano /etc/bind/named.conf.local
Make sure it is empty or has no zones.
Recursive servers should not host zones.
🔵 STEP 4 > Validate Config
Run on both:
sudo named-checkconf
No output = good.
🔵 STEP 5 > Restart BIND (on both rec bind servers)
sudo systemctl restart bind9 sudo systemctl status bind9
Must be running.
🔵 STEP 6 > Test Recursive Function
From LAB-DD1 or any other VM node:
Test REC1:
dig @10.10.2.164 google.com
Test REC2:
dig @10.10.2.165 google.com
You should see:
- ANSWER SECTION populated
- NOERROR
- No AA flag
🔵 STEP 7 > Test ACL Restriction
From LAB-AUTH1 (allowed subnet), it should work.
Later when Windows client configured outside allowed range, recursion should be blocked (we will test that later).
🎯 Expected Behavior
Recursive servers should:
✔ Resolve google.com
✔ Cache responses
✔ NOT host zaibdns.lab
✔ Only allow 10.10.2.0/24
✔ Listen only on their IP
🔎 Quick Verification
Also test:
dig @10.10.2.164 www.zaibdns.lab
It should NOT resolve (because recursion won’t find internal zone).
That confirms clean separation.
Once both REC1 & REC2 successfully resolve google.com,
Move forward …
Kernel Aggressive Tuning (All DNS Servers)
Add to /etc/sysctl.conf:
net.core.rmem_max=67108864 net.core.wmem_max=67108864 net.core.netdev_max_backlog=500000 net.ipv4.udp_mem=262144 524288 1048576 net.ipv4.udp_rmem_min=16384 net.ipv4.udp_wmem_min=16384 net.ipv4.ip_local_port_range=1024 65000 fs.file-max=1000000
Apply:
sudo sysctl -p
Increase file descriptors:
ulimit -n 1000000
Reason:
High QPS requires high UDP buffer capacity and file descriptor availability.
dnsdist Configuration (LAB-DD1 & LAB-DD2)
1# LAB-DD1 (LAB-DD1 = 10.10.2.161)
Install dnsdist from official PowerDNS repository.
(Full Production Version)
🔹 Recommended Method (Official Repository)
- Do NOT rely on very old distro packages.
- Use PowerDNS official repo for production.
Step 1 > Add PowerDNS Repo
sudo apt install -y curl gnupg2 curl -fsSL https://repo.powerdns.com/FD380FBB-pub.asc | sudo gpg --dearmor -o /usr/share/keyrings/pdns.gpg
Add repo file:
echo "deb [signed-by=/usr/share/keyrings/pdns.gpg] http://repo.powerdns.com/ubuntu jammy-dnsdist-17 main" | sudo tee /etc/apt/sources.list.d/pdns.list
Step 2 > Install
sudo apt update sudo apt install dnsdist
Verify:
sudo systemctl status dnsdist
Step 3 > Enable & Start
sudo systemctl enable dnsdist sudo systemctl start dnsdist
Check status:
sudo systemctl status dnsdist
🔹 Default Config Location
/etc/dnsdist/dnsdist.conf
Configure dnsdist
/etc/dnsdist/dnsdist.conf
Delete everything and paste:
setLocal("0.0.0.0:53")
addACL("10.10.2.0/24")
-- Packet Cache
pc = newPacketCache(500000, {maxTTL=300})
getPool("rec"):setCache(pc)
-- Why:
-- 500k entries supports high subscriber base.
-- TTL limited to prevent stale responses.
-- Abuse Protection
addAction(QTypeRule(DNSQType.ANY), DropAction())
addAction(MaxQPSIPRule(200), DropAction())
-- Why:
-- ANY queries are amplification risk.
-- 200 QPS per IP is safe baseline.
-- Backend Health Checks
newServer({address="10.10.2.163:53", pool="auth", checkType="A", checkName="zaibdns.lab.", checkInterval=5})
newServer({address="10.10.2.164:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
newServer({address="10.10.2.165:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
-- Why:
-- Backend marked DOWN if health check fails.
-- Routing
local suffixes = newSuffixMatchNode()
suffixes:add(newDNSName("zaibdns.lab."))
addAction(SuffixMatchNodeRule(suffixes), PoolAction("auth"))
addAction(AllRule(), PoolAction("rec"))
-- Monitoring
controlSocket("127.0.0.1:5199")
#Save
Load Balancing Policy Selection (Critical Design Decision)
dnsdist supports multiple server selection policies. Choosing the correct one directly affects latency and failure behavior.
Recommended for ISP Recursive Pool
setServerPolicy(leastOutstanding)
Why:
- Distributes traffic based on active outstanding queries
- Prevents overloading a single backend
- Maintains low latency under burst traffic
Alternative Models
| Policy | Use Case | Notes |
| firstAvailable | Simple failover | Not ideal for load distribution |
| wrandom | Weighted random | Good when backend hardware differs |
| chashed | Consistent hashing | Useful for cache stickiness |
Recommendation:
For equal hardware recursive pool → use leastOutstanding.
🧠 Why SuffixMatchNode Is Better
Regex:
- Easy to break
- Dot escaping messy
- Trailing dot issues
SuffixMatchNode:
- DNS-aware matching
- Exact domain match
- Used in serious deployments
After editing Restart service
sudo systemctl restart dnsdist
TEST Routing Logic
From any other VM:
🔸 Test Authoritative Routing
dig @10.10.2.161 http://www.zaibdns.lab
Expected:
- Correct answer
- AA flag present
🔸 Test Recursive Routing
dig @10.10.2.161 google.com
Expected:
- Resolves normally
- No AA flag
🔎 What Should Happen Internally
For:
- dnsdist → AUTH pool → 10.10.2.163
For:
google.com
- dnsdist → REC pool → 10.10.2.164 / 165
🎯 When This Works
You now have:
✔ Smart DNS routing
✔ Proper separation
✔ Backend load distribution
✔ DNS traffic control layer
After confirming both tests work, we will:
🔵 Add dnsdist on LAB-DD2
🔵 Configure Keepalived
🔵 Implement VRRP VIP 10.10.2.160
🔵 Perform real failover testing
BUT FIRST understands the DIG flags that will help you understand the results correctly.
🔎 Where Do We See DNS Flags?
You already saw them in dig output:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61032 ;; flags: qr aa rd
That flags: line is what you inspect.
🔹 Meaning of Important Flags
When you run:
dig @10.10.2.161 www.zaibdns.lab
You’ll see:
flags: qr aa rd
Here is what each means:
| Flag | Meaning |
| qr | Query Response (this is a reply) |
| aa | Authoritative Answer |
| rd | Recursion Desired (client asked for recursion) |
| ra | Recursion Available (server supports recursion) |
| ad | Authenticated Data (DNSSEC validated) |
🔵 What You Should Expect
🔹 For zaibdns.lab (Authoritative Path)
Expected:
status: NOERROR flags: qr aa rd
Important:
- aa must be present ✅
- ra should NOT appear (since auth server doesn’t recurse)
🔹 For google.com (Recursive Path)
Expected:
status: NOERROR flags: qr rd ra
Important:
- aa should NOT be present ❌
- ra must be present ✅
That proves recursion happened.
🔎 Cleaner Output (Easier to Read)
Instead of full dig, use:
dig @10.10.2.161 www.zaibdns.lab +noall +answer +authority +comments
Example output:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR ;; flags: qr aa rd
This makes flags very clear.
🔬 Extra Debug Mode (Very Useful)
For detailed packet view:
dig @10.10.2.161 www.zaibdns.lab +dnssec +multi
Or full raw:
dig @10.10.2.161 www.zaibdns.lab +trace
🧠 How To Validate dnsdist Routing Using ‘DIG’ Flags
When testing through dnsdist:
Authoritative test:
dig @10.10.2.161 http://www.zaibdns.lab
Look for:
✔ aa
Recursive test:
dig @10.10.2.161 google.com
Look for:
✔ ra
❌ no aa
🎯 Why Flags Matter in ISP World
In real ISP troubleshooting:
- If aa missing → authoritative routing broken
- If ra missing → recursion disabled
- If REFUSED → ACL issue
- If SERVFAIL → backend failure
Flags are your first debugging indicator.
🚀 You Have Now Built
✔ Authoritative backend
✔ Recursive backend
✔ Intelligent dnsdist routing
✔ DNS flag-level validation
Right now:
- LAB-DD1 (10.10.2.161) → Working
- LAB-DD2 (10.10.2.162) → Not configured yet
- VIP planned → 10.10.2.160
Goal:
- Clients will use only 10.10.2.160
- If DD1 fails → DD2 takes over automatically
🔵 Configure LAB-DD2 (Clone of DD1)
STEP 1 > Install dnsdist on LAB-DD2
On:
- LAB-DD2 (10.10.2.162)
Install same way:
sudo apt install dnsdist -y
STEP 2 > Copy Same Config
Edit:
sudo nano /etc/dnsdist/dnsdist.conf
Paste SAME config as DD1, but change listen IP:
setLocal("10.10.2.162:53")
addACL("10.10.2.0/24")
newServer({address="10.10.2.163:53", pool="auth"})
newServer({address="10.10.2.164:53", pool="rec"})
newServer({address="10.10.2.165:53", pool="rec"})
local suffixes = newSuffixMatchNode()
suffixes:add(newDNSName("zaibdns.lab."))
addAction(SuffixMatchNodeRule(suffixes), PoolAction("auth"))
addAction(AllRule(), PoolAction("rec"))
Restart:
sudo systemctl restart dnsdist
STEP 3 > Test DD2 Directly
From any VM:
dig @10.10.2.162 www.zaibdns.lab dig @10.10.2.162 google.com
Both must work exactly like DD1.
Once DD2 works, we move to:
🔵NEXT PHASE – HA !!!!!!!! (or HAHAHAHA 😉
VRRP High Availability (Keepalived)
Now we move to HA layer.
H.A PHASE > Install Keepalived (VRRP)
We will:
- Install Keepalived on BOTH DD1 & DD2
- Configure floating IP = 10.10.2.160
- Make DD1 MASTER
- Make DD2 BACKUP
🔹 STEP 1 > Install Keepalived (On BOTH DD1 & DD2)
On LAB-DD1:
sudo apt install keepalived -y
🔹 STEP 2 > Configure LAB-DD1 (MASTER)
On LAB-DD1:
sudo nano /etc/keepalived/keepalived.conf
Paste:
global_defs { router_id LAB_DD1 } vrrp_script chk_dnsdist { script “systemctl is-active –quiet dnsdist” interval 2 fall 1 rise 1 } vrrp_instance VI_DNS { state MASTER interface ens160 virtual_router_id 51 priority 150 advert_int 1 authentication { auth_type PASS auth_pass lab123 } virtual_ipaddress { 10.10.2.160 } track_script { chk_dnsdist } } Save.
🔹 STEP 3 > Configure LAB-DD2 (BACKUP)
On LAB-DD2:
sudo nano /etc/keepalived/keepalived.conf
Paste:
global_defs {
router_id LAB_DD2
}
vrrp_script chk_dnsdist {
script "systemctl is-active --quiet dnsdist"
interval 2
fall 1
rise 1
}
vrrp_instance VI_DNS {
state BACKUP
interface ens160
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass lab123
}
virtual_ipaddress {
10.10.2.160
}
track_script {
chk_dnsdist
}
}
Save.
VRRP Engineering Considerations
- Advertise Interval
advert_int 1
- 1 second provides fast failover
- Avoid sub-second unless absolutely required
- Priority Design
Primary:
priority 150
Secondary:
priority 100
Avoid equal priorities to prevent master flapping.
- Split-Brain Prevention
Ensure:
- VRRP runs on isolated VLAN
- No L2 loops
- Proper STP configuration
- Monitoring for dual-master condition
- Health-Based VRRP Tracking (Recommended)
Track dnsdist process:
track_script {
chk_dnsdist
}
Failover should occur if:
- dnsdist crashes
- Backend unreachable
- System load critical
This avoids “IP is up but service is down” scenario.
🔹 STEP 4 > Start Keepalived
On BOTH:
sudo systemctl enable keepalived sudo systemctl start keepalived
🔹 STEP 5 > Verify VIP
On DD1:
- ip a
You should see:
- 10.10.2.160
On DD2:
You should NOT see VIP (since it’s backup).
🔹 STEP 5 > Test VIP
From any VM:
dig @10.10.2.160 www.zaibdns.lab dig @10.10.2.160 google.com
Both must work.
🔥 STEP 6 > Failover Test (Important)
🔎 What This Does
- Every 2 seconds it checks:
systemctl is-active dnsdist
- If dnsdist stops → health check fails
- MASTER immediately drops state
- BACKUP becomes MASTER
- VIP moves
No weight calculations.
No partial priority logic.
Clean failover.
🔁 Apply Configuration
On BOTH nodes:
sudo systemctl restart keepalived
Confirm VIP is on DD1:
ip a | grep 10.10.2.160
🔥 Test Failover
On DD1:
sudo systemctl stop dnsdist
Within ~2 seconds:
- VIP disappears from DD1
- VIP appears on DD2
Test:
dig @10.10.2.160 google.com
Should continue working.
🔁 Test Recovery
Start dnsdist again on DD1:
sudo systemctl start dnsdist
VIP should move back to DD1 (because higher priority 150).
🎯 Now You Have
✔ Service-aware failover
✔ Proper HA behavior
✔ Clean VIP movement
✔ Production-style design
More Failover Testing…
now we test what really matters: failover.
You already have:
- DD1 (MASTER)
- DD2 (BACKUP)
- VIP = 10.10.2.160
- dnsdist listening on 0.0.0.0:53
- Keepalived running
Now we validate HA properly.
🔵 STEP 1 > Confirm Who Owns VIP
On DD1:
- ip a | grep 10.10.2.160
On DD2:
- ip a | grep 10.10.2.160
Expected:
- VIP visible on DD1 only
- Not visible on DD2
🔵 STEP 2 > Baseline DNS Test
From any other VM:
dig @10.10.2.160 google.com dig @10.10.2.160 www.zaibdns.lab
Confirm both resolve.
🔵 STEP 3 > Test Soft Failover (Service Failure)
Now simulate dnsdist crash on MASTER.
On DD1:
sudo systemctl stop dnsdist
Wait few seconds.
🔎 Check VIP Movement
On DD2:
ip a | grep 10.10.2.160
VIP should now appear on DD2.
On DD1:
ip a | grep 10.10.2.160
VIP should be gone.
🔎 Test DNS During Failover
From another VM:
dig @10.10.2.160 google.com
It should still resolve.
That means:
✔ Keepalived detected dnsdist failure
✔ VIP moved
✔ Clients unaffected
🔵 STEP 4 > Restore MASTER
On DD1:
sudo systemctl start dnsdist
Wait few seconds.
Check:
ip a | grep 10.10.2.160
Depending on config:
- VIP may return to DD1 (if preemption active)
- Or stay on DD2
If you want VIP to always return to DD1, ensure:
- priority higher on DD1
(which you already set 110 vs 100)
🔵 STEP 5 > Hard Failover (Real Test)
Now simulate full server failure.
On DD1:
sudo poweroff
Check from another VM:
dig @10.10.2.160 google.com
It should still work.
On DD2:
ip a | grep 10.10.2.160
VIP must be present.
🔵 STEP 6 > Continuous Resolution Test (While Failing Over)
From any non-DD node:
while true; do dig @10.10.2.160 google.com +short; sleep 1; done
Now stop dnsdist on DD1.
Expected:
- 0–1 failed query max
- Resolution resumes automatically
- No manual intervention
🔵 What We Are Actually Testing
Keepalived config:
track_script {
chk_dnsdist
}
This means:
If pidof dnsdist fails → priority drops → VIP moves.
This is correct HA design.
🔥 Common Problems During Failover
If VIP does not move:
- Wrong interface name in keepalived config
- Multicast blocked in VLAN
- Firewall blocking VRRP (protocol 112)
- Wrong virtual_router_id mismatch
🎯 Final Expected Result
You should achieve:
✔ Service failover without manual action
✔ Clients always use single IP
✔ Zero backend changes
✔ DNS resolution continues
🎯 Final Lab Architecture Now
Client → 10.10.2.160 (VIP) | dnsdist HA | Auth + Rec backend
You now have a properly built DNS stack:
- Authoritative backend (LAB-AUTH1)
- Recursive backend (LAB-REC1 / LAB-REC2)
- dnsdist routing layer (DD1 / DD2)
- VRRP floating VIP (10.10.2.160)
- Service-aware failover using keepalived
This is structurally identical to how many mid-size ISPs deploy DNS control layers.
Backend Failure Test (Very Important)
Stop recursive on REC1:
sudo systemctl stop bind9
Now query via VIP:
dig @10.10.2.160 google.com
It should still resolve via REC2.
That validates backend redundancy.
4️⃣ Authoritative Isolation Test
Stop AUTH:
sudo systemctl stop bind9
Now:
dig @10.10.2.160 www.zaibdns.lab
Should fail.
But:
dig @10.10.2.160 google.com
Should still work.
That confirms clean pool separation.
🎯 What You Have Achieved
You have built:
- Layered DNS architecture
- Pool-based routing
- High availability with VRRP
- Service-aware failover
- Controlled recursion
- Authoritative isolation
- Backend redundancy
This is not “lab toy” level anymore.
This is real network engineering.
Why Separate Recursive and Authoritative?
- Security isolation
- Prevents recursive abuse
- Better performance tuning
- DDoS containment
- Operational clarity
Where to Use Public vs Private IP
- dnsdist → Public IP or subscriber-facing IP
- Recursive servers → Private IP only
- Authoritative → Private or Public (if serving public zones)
Golden Rules
- Separate recursive and authoritative.
- Never expose recursive publicly.
- Always monitor QPS.
- Always use packet cache.
- Always implement health checks.
- Test failover periodically.
- Plan scaling before congestion.
- Do not rely on client-side failover.
Scaling for 50K–100K Subscribers
Estimated Peak QPS
- 50K users → 15K–25K QPS
- 100K users → 30K–50K QPS
Capacity Planning Model (QPS Engineering Approach)
For ISP-grade DNS design, sizing must be derived from realistic subscriber behavior instead of theoretical hardware limits.
Baseline Estimation Formula
Peak QPS = Active Subscribers × Avg Queries per Second per Subscriber
Where:
- Typical residential subscriber generates: 5–15 QPS
- Business subscriber may generate: 20–50 QPS
- During peak (evening streaming + mobile apps), bursts can reach 2× baseline.
Example (100,000 Subscribers)
If:
- 60% concurrently active
- Avg 8 QPS per active user
100,000 × 0.6 × 8 = 480,000 QPS peak
With:
- 70–85% cache hit rate (well-tuned resolver)
- Backend recursion load reduces significantly
Engineering Rule
Always size recursive backend for:
- 1.5× projected peak QPS
- Ability to survive single-node failure (N+1 model)
This ensures performance stability during:
- Cache cold start
- DDoS bursts
- Backend node outage
Recommended Hardware
- dnsdist: 8–16 cores, 16–32GB RAM
- Recursive: 8–16 cores, 32GB RAM
- Authoritative: Moderate load
Failure Scenarios Tested
- Stop dnsdist → VIP moves
- Stop keepalived → Backup takes over
- Power off DD1 → DD2 becomes MASTER
- Stop REC1 → Traffic moves to REC2
- Stop AUTH → Only authoritative queries fail
Common Deployment Issues
- systemd-resolved conflict on port 53
- dnsdist not listening on VIP
- Incorrect interface name in keepalived
- VRRP blocked by VMware security settings
- Regex routing errors
🔹 Security Controls (ISP Best Practice)
- Firewall:
- Allow UDP/TCP 53 → dnsdist only
- Block direct access to backend IPs
- Recursive ACL:
- Allow only subscriber IP ranges
- Rate limiting enabled on dnsdist
- Disable recursion on authoritative
Best Practices for Pakistani Cable ISPs
- Never run single DNS server
- Always separate recursive & authoritative
- Always use health checks
- Monitor QPS continuously
- Use packet cache
- Use VRRP for frontend HA
- Never expose recursive servers publicly
ISP GRADE TUNING – SJZ
Now we move from functional lab to ISP-grade tuning.
All changes below go into:
/etc/dnsdist/dnsdist.conf
(on BOTH DD1 and DD2)
Restart dnsdist after modifications.
🔵 1️⃣ Enable Packet Cache (Very Important)
This dramatically reduces load on recursive servers.
Add near top:
-- Packet cache (10k entries, 60s max TTL)
pc = newPacketCache(10000, {maxTTL=60, minTTL=0, temporaryFailureTTL=10})
getPool("rec"):setCache(pc)
What this does:
- Caches recursive responses
- Offloads REC1 / REC2
- Improves latency
- Handles burst traffic
For real ISP scale → 100k+ entries.
🔵 2️⃣ Enable Rate Limiting (Basic DDoS Protection)
Add:
-- Basic rate limiting (per IP) addAction(MaxQPSIPRule(50), DropAction())
Meaning:
- If a single IP sends >50 queries/sec → drop
- Protects against abuse
For ISP production:
- Adjust threshold based on subscriber profile
🔵 3️⃣ Basic Abuse Protection
Add:
-- Drop ANY queries (reflection attack prevention)
addAction(QTypeRule(DNSQType.ANY), DropAction())
-- Drop CHAOS queries (version.bind)
addAction(AndRule({QClassRule(DNSClass.CH), QTypeRule(DNSQType.TXT)}), DropAction())
Prevents:
- Amplification attacks
- Version probing
🔵 4️⃣ Backend Health Checks (Very Important)
Replace your newServer() lines with health checks:
newServer({
address="10.10.2.163:53",
pool="auth",
checkType="A",
checkName="zaibdns.lab.",
checkInterval=5
})
newServer({
address="10.10.2.164:53",
pool="rec",
checkType="A",
checkName="google.com.",
checkInterval=5
})
newServer({
address="10.10.2.165:53",
pool="rec",
checkType="A",
checkName="google.com.",
checkInterval=5
})
Now dnsdist:
- Automatically marks backend DOWN if it fails
- Stops sending traffic to dead backend
🔵 5️⃣ Enable Logging (Lightweight)
Add:
setVerboseHealthChecks(true)
To log health check failures.
For query logging (not recommended in production):
addAction(AllRule(), LogAction("/var/log/dnsdist-queries.log"))
⚠ Only use in lab > high overhead.
🔵 6️⃣ Enable TCP Support Tuning
Add:
setMaxTCPClientThreads(10) setMaxTCPConnectionsPerClient(20)
Prevents TCP abuse.
Also increase UDP socket buffers (system-level):
sudo sysctl -w net.core.rmem_max=26214400 sudo sysctl -w net.core.wmem_max=26214400
🔵 7️⃣ Enable Metrics Export (Very Powerful)
Add:
controlSocket("127.0.0.1:5199")
Restart dnsdist.
Then:
dnsdist -c
Inside console:
showServers() showPools() showCacheHitResponseCounts()
You’ll see:
- Query counts
- Latency
- Backend state
- Cache hits
🔵 8️⃣ Optional: Prometheus Exporter (ISP Grade)
Add:
webserver("0.0.0.0:8083")
setWebserverConfig({password="admin123", apiKey="secret"})
Then access:
You get live stats.
⚠ Secure properly in production.
🔵 Example Clean Production Block (Recommended Final Version)
Here is consolidated core tuning block:
setLocal("0.0.0.0:53")
addACL("10.10.2.0/24")
-- Packet cache
pc = newPacketCache(10000, {maxTTL=60})
getPool("rec"):setCache(pc)
-- Abuse protection
addAction(QTypeRule(DNSQType.ANY), DropAction())
addAction(MaxQPSIPRule(50), DropAction())
-- Health checks
newServer({address="10.10.2.163:53", pool="auth", checkType="A", checkName="zaibdns.lab.", checkInterval=5})
newServer({address="10.10.2.164:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
newServer({address="10.10.2.165:53", pool="rec", checkType="A", checkName="google.com.", checkInterval=5})
local suffixes = newSuffixMatchNode()
suffixes:add(newDNSName("zaibdns.lab."))
addAction(SuffixMatchNodeRule(suffixes), PoolAction("auth"))
addAction(AllRule(), PoolAction("rec"))
controlSocket("127.0.0.1:5199")
🎯 What You Have Now
If you enable all above:
✔ Caching layer
✔ Backend health detection
✔ Rate limiting
✔ Basic abuse protection
✔ Failover HA
✔ Metrics visibility
✔ TCP control
This is now serious ISP-grade DNS architecture.
Few thoughts about architecture … SJz
🔎 1️⃣ Is The Architecture Correct For 100k Users?
Your design:
Clients ↓ VRRP VIP ↓ 2x dnsdist (HA) ↓ Auth Pool + Rec Pool ↓ 2x Recursive + 1x Auth
This is industry-standard L7 DNS load-balancer model.
Used by:
- Mid-size ISPs
- Hosting providers
- MSPs
- Regional broadband operators
So yes > conceptually correct.
🔎 2️⃣ 100k Users → What Load Does That Mean?
Typical ISP DNS usage:
- 3–10 QPS per subscriber during peak
- 100k subs × avg 2–3 active at same moment
- Realistic peak: 15k–40k QPS
During Netflix / Android updates / cache expiry bursts:
- 50k+ QPS spikes possible
Your LAB config (10k cache entries, 50 QPS limit) is too small for that.
Architecture is fine.
Sizing must change.
🔵 3️⃣ What Would Be Required For 100k Subscribers?
✅ dnsdist Layer
Minimum recommended per node:
- 8–16 vCPU
- 16–32 GB RAM
- Packet cache 500k–1M entries
- NIC tuned for high PPS
- IRQ affinity tuned
- RPS/RFS enabled
Example production packet cache:
- pc = newPacketCache(500000, {maxTTL=300})
✅ Recursive Layer
For 100k subs:
Two recursive servers are borderline.
Better:
- 3–4 recursive nodes
- Each 8–16 cores
- 32 GB RAM
- Proper ulimit tuning
- Large resolver cache
In BIND:
- max-cache-size 8g;
- recursive-clients 50000;
✅ Authoritative Layer
Auth load is typically very low.
1 primary + 1 secondary recommended.
✅ Network Layer
Must ensure:
- Multicast allowed (VRRP)
- NIC offloading tuned
- Firewall not bottlenecking
- MTU correct
- No stateful inspection on DNS traffic
🔎 4️⃣ Is dnsdist Used In Serious ISP Deployments?
Yes.
dnsdist (by PowerDNS) is widely used in:
- ISPs
- CDN providers
- Hosting companies
- Enterprise resolvers
- Cloud operators
It is not hobby software.
It supports:
- 1M+ QPS on proper hardware
- Advanced rate limiting
- Geo routing
- DNS filtering
- DoT/DoH frontend
🔎 5️⃣ Is OUR Current Lab Enough For 100k?
In current lab sizing:
❌ No (hardware too small)
❌ Cache too small
❌ Recursive count too small
But:
✔ Architecture pattern is correct
✔ Failover model correct
✔ Separation correct
✔ Routing logic correct
So design is scalable.
🔵 6️⃣ Real-World Upgrade Path For 100k ISP
I would recommend:
2x dnsdist (active/active possible)
3x recursive nodes
2x authoritative nodes
Anycast (optional future)
Monitoring (Prometheus + Grafana)
🔎 7️⃣ Real Question: Single VIP or Dual IP?
For 100k users:
Better to provide clients:
- Primary DNS: VIP
- Secondary DNS: VIP (same)
Redundancy handled at server layer.
Or:
Active/Active with ECMP or Anycast if advanced.
🔵 8️⃣ Where Would This Design Break?
It would break if:
- Recursive servers undersized
- Cache too small
- CPU too low
- Too aggressive rate limiting
- No kernel tuning
Not because of architecture.
One more topic to go with (zaib)
How load balancing will be done via DNSDIST?
Since users are asking since publishing of this post, here is the reply. (zaib)
You defined:
newServer({address="10.10.2.164:53", pool="rec"})
newServer({address="10.10.2.165:53", pool="rec"})
And:
addAction(AllRule(), PoolAction("rec"))
So when a client queries:
google.com
Flow is:
Client → VIP → dnsdist → REC pool → REC1 or REC2
2️⃣ How Does dnsdist Distribute Traffic?
By default, dnsdist uses least outstanding queries (latency-aware load balancing).
That means:
- It does NOT strictly do round-robin
- It sends traffic to the server with fewer active queries
- It prefers lower-latency backends
So it is intelligent load balancing, not naive rotation.
3️⃣ Will Load Be Even?
Not exactly 50/50.
Distribution depends on:
- Backend response time
- Current query backlog
- Health status
- TCP/UDP mix
If both servers are equal hardware and same latency:
→ Load will be very close to balanced.
If one server is slightly faster:
→ It may receive slightly more traffic.
This is good behavior.
4️⃣ What About Cache?
Important detail:
You enabled packet cache on dnsdist:
pc = newPacketCache(...)
getPool("rec"):setCache(pc)
That means:
- First query hits recursive
- Subsequent identical queries may be answered directly from dnsdist
- Backend load reduces
- Cache hits handled at frontend
So backend distribution applies only to cache misses.
5️⃣ What Happens If One Recursive Fails?
If REC1 fails:
- Health check fails
- dnsdist marks it DOWN
- All traffic goes to REC2 automatically
- No manual action required
That’s real production-grade behavior.
6️⃣ If You Want Strict Round Robin
You can force it:
setServerPolicy(roundrobin)
But this is NOT recommended in ISP production.
Latency-aware balancing is better.
7️⃣ How To Verify Load Balancing Live
On dnsdist console:
dnsdist -c showServers()
You will see:
- queries handled per backend
- latency
- state (UP/DOWN)
Run repeated:
dig @10.10.2.160 google.com
And watch counters increment.
8️⃣ Important ISP Insight
For 100K subscribers:
2 recursive servers is minimum.
Better production design:
- 3 recursive nodes
- Or 2 strong nodes + 1 backup
dnsdist will distribute automatically.
Final Answer for Load Balancing via DNSDIST
Yes > dnsdist will load balance between both recursive servers automatically.
It uses intelligent latency-aware distribution, not basic round-robin.
It also automatically removes failed backends from rotation.
🎯 Final Professional Answer
Yes > this architecture is absolutely suitable for 100k subscribers.
But:
- It must be deployed on proper hardware,
- properly tuned,
- and monitored.
OUR lab has proven:
- Design works
- HA works
- Routing works
- Backend failover works
That is exactly what matters before production.
Final Conclusion
dnsdist + VRRP + backend separation is a production-grade DNS architecture suitable for 50K–100K subscriber ISPs.
This design provides:
- High availability
- Intelligent routing
- Backend redundancy
- Security controls
- Cost efficiency
Important:
dnsdist is not a DDoS appliance.
Edge filtering still required.
For Pakistani cable-net ISPs, this model delivers enterprise-level stability without expensive hardware appliances.
DNS is core infrastructure. Design it accordingly.
Syed Jahanzaib
Syed Jahanzaib - سید جہانزیب - Personal Blog to Share Knowledge ! 