Private Link is often described as “give a platform service a private IP.” That description is correct, but incomplete. The private endpoint is only one part of the path. DNS must return that private address to every authorized client, the network must carry the traffic, and the service must authenticate the caller. A mistake in any layer can look like the same vague connection timeout.

In this workshop we build the complete path for Azure SQL Database, then break it three different ways on purpose. Terraform and AzAPI create a hub, an application spoke, a simulated branch network, Azure DNS Private Resolver, a private DNS zone, two small Linux VMs, and a Basic SQL database. Public SQL access stays disabled throughout. The application VM connects with its managed identity, so the final proof uses neither a SQL password nor an inbound SSH session.

Reading and lab time: 35–40 minutesLevel: Intermediate to advancedTools: PowerShell 7, Terraform 1.15.8, Azure/azapi 2.11.0, Azure CLI, Azure Portal

Cost and cleanup warning: Azure DNS Private Resolver endpoints, two B1s VMs, two Standard public IPs, Azure SQL Basic, and a Private Endpoint incur a small charge while deployed. Complete the cleanup section when you finish. The public IPs provide explicit outbound access for package installation only; no inbound SSH or RDP rule is created.

What you will prove

By the end of the lab you will have evidence for five distinct states:

1. The normal SQL hostname resolves publicly, but SQL rejects the connection because public access is disabled.

2. An approved private endpoint alone changes nothing for clients that cannot reach the private DNS zone.

3. Linking an empty private zone produces deterministic NXDOMAIN because the private namespace overrides public recursion.

4. A zone group creates the correct A record and the normal SQL hostname resolves to 10.60.1.x.

5. The VM acquires a SQL token from IMDS and queries the database through Private Link without a stored credential.

You will also validate both hybrid DNS directions: Azure-to-branch resolution for corp.example, and branch-to-Azure resolution for the SQL hostname.

Architecture

Component	Address or setting	Purpose
Hub VNet	10.60.0.0/16	Central DNS and Private Endpoint services
Resolver inbound subnet	10.60.0.0/28	Dedicated, delegated subnet; static endpoint 10.60.0.4
Resolver outbound subnet	10.60.0.16/28	Dedicated, delegated subnet for forwarding rules
Private Endpoint subnet	10.60.1.0/24	SQL Private Endpoint NIC, dynamically assigned 10.60.1.x
Application spoke	10.70.0.0/16	Ubuntu application VM with system-assigned identity
Simulated branch	10.80.0.0/16	Ubuntu DNS VM at 10.80.1.4 running dnsmasq
Private DNS zone	privatelink.database.windows.net	Hosts the SQL private A record
Azure SQL	Basic, TLS 1.2, Entra-only	Public networking disabled; identity is temporary lab admin

The application VNet uses the resolver inbound endpoint as its custom DNS server. The branch DNS server conditionally forwards Azure SQL private namespaces to that same inbound endpoint. In the opposite direction, a resolver forwarding rule sends corp.example. queries to 10.80.1.4, while a ruleset link makes that rule available to the application spoke.

There are two important boundaries here:

• DNS is not routing. A correct answer does not create a network path.

• VNet peering is not transitive. App-to-hub and hub-to-branch peerings do not automatically make app-to-branch traffic possible through the hub. This lab creates direct hub relationships for the DNS endpoints it uses; production hybrid routing normally adds VPN Gateway, ExpressRoute, Azure Firewall, or an NVA deliberately.

Why the normal SQL hostname matters

Applications should connect to:

sql-pl-<unique>.database.windows.net

Azure public DNS returns a CNAME to:

sql-pl-<unique>.privatelink.database.windows.net

The private DNS zone answers that second name with the endpoint's private address. Keep the original database.windows.net hostname in the connection string. Connecting directly to 10.60.1.x or to the privatelink hostname breaks Azure SQL's expected hostname and certificate validation behavior.

Prerequisites

You need:

• an Azure subscription where you can create the listed resources;

• permissions to create an Azure SQL Microsoft Entra administrator;

• Azure CLI authenticated to the exact subscription you intend to use;

• PowerShell 7;

• Terraform 1.15.8;

• an SSH public key for VM provisioning.

The source uses the active Azure CLI subscription's immutable ID instead of its display name. This matters when multiple subscriptions share a name.

az login

$subscriptionId = az account show --query id -o tsv
if (-not $subscriptionId) { throw 'No active Azure CLI subscription.' }

# Inspect the name without copying the ID into screenshots or source files.
az account show --query '{name:name,state:state}' -o table

Generate an ephemeral lab password and SSH key outside the tracked source. Azure requires a SQL administrator password at server creation even though Entra-only authentication disables its use afterward.

New-Item -ItemType Directory -Force .local | Out-Null
ssh-keygen -t ed25519 -f .local/workshop_vm -N '""'

$bytes = New-Object byte[] 36
[Security.Cryptography.RandomNumberGenerator]::Fill($bytes)
$sqlPassword = 'A9!' + [Convert]::ToBase64String($bytes)

$env:TF_VAR_subscription_id = $subscriptionId
$env:TF_VAR_sql_admin_password = $sqlPassword
$env:TF_VAR_vm_admin_ssh_public_key = Get-Content .local/workshop_vm.pub -Raw

The distributable bundle excludes .local, Terraform state, saved plans, keys, passwords, tokens, subscription and tenant IDs, and identity GUIDs.

Validate address space and regional availability

Before deployment, compare the lab CIDRs with existing VNets in the active subscription.

$labCidrs = '10.60.0.0/16','10.70.0.0/16','10.80.0.0/16'
$existing = az network vnet list --query '[].addressSpace.addressPrefixes[]' -o tsv
$conflicts = $labCidrs | Where-Object { $existing -contains $_ }
if ($conflicts) { throw "CIDR safety gate failed: $($conflicts -join ', ')" }
'CIDR safety gate: PASS'

This verified run placed the hub, resolver, endpoint, and VMs in West Europe. The active lab subscription temporarily reported Azure SQL provisioning restrictions in West and North Europe, so the SQL server used Sweden Central. Private Endpoints support cross-region connections, and the network behavior in the lab is unchanged. In your own subscription, set sql_location = "westeurope" when SQL is available there.

Terraform structure

The lab uses a single Terraform root and four staged .tfvars files:

terraform/
├── cloud-init/
│   ├── app.yaml.tftpl
│   └── branch.yaml.tftpl
├── stages/
│   ├── 01-bootstrap.tfvars
│   ├── 02-broken-link.tfvars
│   ├── 03-nxdomain.tfvars
│   └── 04-operate.tfvars
├── .terraform.lock.hcl
├── versions.tf
├── variables.tf
├── locals.tf
├── main.tf
└── outputs.tf

The provider is locked and preflight validation is enabled:

terraform {
  required_version = "= 1.15.8"

  required_providers {
    azapi = {
      source  = "Azure/azapi"
      version = "= 2.11.0"
    }
  }
}

provider "azapi" {
  subscription_id           = var.subscription_id
  enable_preflight          = true
  skip_provider_registration = false
}

All resources use explicit API versions: network and resolver resources use 2025-05-01, private DNS uses 2024-06-01, SQL uses 2025-01-01, VMs use 2025-04-01, and the resource group uses 2025-04-01.

The Boolean stage switches are deliberately small:

variable "enable_private_endpoint"   { type = bool }
variable "enable_private_dns_link"  { type = bool }
variable "enable_dns_zone_group"    { type = bool }
variable "enable_outbound_forwarding" { type = bool }

That lets us change only one DNS dependency at a time and observe the exact symptom.

Initialize and create the bootstrap plan

Set-Location terraform
terraform version
terraform fmt -check -recursive
terraform init
terraform validate

terraform plan `
  -var-file=stages/01-bootstrap.tfvars `
  -out=../.local/01-bootstrap.tfplan

terraform show ../.local/01-bootstrap.tfplan
terraform apply ../.local/01-bootstrap.tfplan

Using a saved plan matters: the reviewed plan is the exact artifact applied. Re-run plan if variables or credentials change rather than applying a stale file.

Explore the network and resolver

The hub has three purpose-built subnets. Azure DNS Private Resolver requires dedicated delegated subnets for inbound and outbound endpoints. A /28 is the minimum size used here; the endpoint addresses must not share a subnet with private endpoints or workloads.

The hub is peered bidirectionally with both spokes:

The resolver has two different jobs:

• Inbound endpoint (10.60.0.4) lets DNS servers or clients outside the linked Azure DNS context query Azure-hosted private zones.

• Outbound endpoint gives Azure DNS a route to custom DNS servers through forwarding rulesets.

The corp.example. rule targets the branch DNS server on UDP and TCP 53. A DNS forwarding ruleset link makes this rule available to the application VNet.

The branch NSG permits TCP and UDP 53 only from 10.60.0.16/28, the resolver outbound subnet, then denies other virtual-network inbound traffic. Neither VM has an SSH or RDP allow rule. Azure VM Run Command is our management channel.

SQL security before Private Link

The SQL server is created with:

properties = {
  publicNetworkAccess       = "Disabl
ed"
  minimalTlsVersion         = "1.2"
  administratorLogin       = "sqlbootstrap"
  administratorLoginPassword = var.sql_admin_password
}

The application VM receives a system-assigned managed identity. Terraform configures that identity as the temporary Microsoft Entra administrator and enables Entra-only authentication. No SQL firewall rule or “Allow Azure services” exception is created.

This separates three controls:

• Network reachability: Private Endpoint and routing.

• Name resolution: CNAME, private zone link, and A record.

• Authentication: managed identity token accepted by Azure SQL.

Passing one does not imply the others pass.

Stage 1 — public resolution, private service

01-bootstrap.tfvars keeps all Private Endpoint switches off:

enable_private_endpoint  = false
enable_private_dns_link = false
enable_dns_zone_group   = false
enable_outbound_forwarding = true

Run the test from the application VM with Azure VM Run Command:

./scripts/Test-PrivateLinkStage.ps1 -Stage bootstrap

The public DNS chain returns a public IP. That is expected: disabling public network access changes the service's acceptance policy; it does not remove its public DNS name. The TCP connection fails because Azure SQL public access is off.

This is a useful diagnostic state. If a supposedly private client sees a public answer, investigate DNS before changing SQL firewall rules.

Stage 2 — approved endpoint, missing zone link

Apply the second stage:

terraform plan `
  -var-file=stages/02-broken-link.tfvars `
  -out=../.local/02-broken-link.tfplan
terraform apply ../.local/02-broken-link.tfplan

The stage enables the SQL Private Endpoint and zone group but intentionally does not link the private zone to the hub. Azure automatically approves a Private Endpoint created in the same tenant with sufficient permissions. The endpoint NIC receives 10.60.1.4 in this run.

The zone group already created a private A record, yet the client cannot see the zone because the hub VNet link is missing. Its resolver therefore follows public DNS and still returns a public address.

This is the classic “the endpoint is healthy, so why is traffic still public?” failure. Endpoint approval and DNS visibility are independent states.

Stage 3 — linked empty zone and NXDOMAIN

The third stage enables the hub zone link but removes the zone group:

terraform plan `
  -var-file=stages/03-nxdomain.tfvars `
  -out=../.local/03-nxdomain.tfplan
terraform apply ../.local/03-nxdomain.tfplan

Now the resolver can see privatelink.database.windows.net, but the SQL record is absent. Because the linked private zone is authoritative for that namespace, Azure DNS does not fall through to the public answer. It returns NXDOMAIN.

./scripts/Test-PrivateLinkStage.ps1 -Stage nxdomain

This symptom is more precise than a timeout: the private namespace is visible, but the expected record is missing. Check whether the zone group exists, whether it references the correct zone resource ID, and whether another automation deleted the A record.

Stage 4 — restore the complete private DNS chain

The operating stage enables all three dependencies:

enable_private_endpoint  = true
enable_private_dns_link = true
enable_dns_zone_group   = true
enable_outbound_forwarding = true

terraform plan `
  -var-file=stages/04-operate.tfvars `
  -out=../.local/04-operate.tfplan
terraform apply ../.local/04-operate.tfplan

The Private Endpoint zone group connects endpoint lifecycle to DNS record lifecycle. The zone link makes that zone resolvable from the hub, and the application VNet sends DNS queries to the hub inbound endpoint.

Flush caches between stages. On Linux with systemd-resolved:

sudo resolvectl flush-caches

Then query the normal SQL hostname:

dig +noall +answer sql-pl-<unique>.database.windows.net

Expected chain:

sql-pl-<unique>.database.windows.net.
  CNAME sql-pl-<unique>.privatelink.database.windows.net.
sql-pl-<unique>.privatelink.database.windows.net.
  A 10.60.1.4

Passwordless SQL over Private Link

The application VM never receives a client secret. Its script requests a token from the Azure Instance Metadata Service at 169.254.169.254:

token="$(curl -fsS -H Metadata:true \
  'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fdatabase.windows.net%2F' \
  | jq -r .access_token)"

The token is never echoed. sqlcmd expects the access-token file in UTF-16LE. The script creates it with mode 600 and installs a trap before use:

token_file="$(mktemp)"
chmod 600 "$token_file"
trap 'rm -f "$token_file"' EXIT HUP INT TERM
printf '%s' "$token" | iconv -f UTF-8 -t UTF-16LE > "$token_file"
unset token

sqlcmd -S "tcp:${sql_fqdn},1433" -d "$database_name" \
  -G -P "$token_file" -N -C -b -Q "$query"

For the lab, the managed identity is the SQL Entra administrator so it can create and query a non-sensitive proof table. Production systems should assign an Entra group as administrator, create a contained database user for the workload identity, and grant only the necessary database roles.

IF OBJECT_ID('dbo.PrivateLinkProof', 'U') IS NULL
BEGIN
    CREATE TABLE dbo.PrivateLinkProof
    (
        ProofId int NOT NULL PRIMARY KEY,
        ProofMessage nvarchar(100) NOT NULL
    );
END;

MERGE dbo.PrivateLinkProof AS target
USING (SELECT 1, N'Passwordless SQL over Azure Private Link') AS source(ProofId, ProofMessage)
ON target.ProofId = source.ProofId
WHEN MATCHED THEN UPDATE SET ProofMessage = source.ProofMessage
WHEN NOT MATCHED THEN INSERT VALUES (source.ProofId, source.ProofMessage);

SELECT DB_NAME() AS DatabaseName, ProofMessage
FROM dbo.PrivateLinkProof WHERE ProofId = 1;

The verified result is a private DNS answer, a successful encrypted SQL session, and the message Passwordless SQL over Azure Private Link. A token never appears in logs, state, or screenshots.

Bidirectional hybrid DNS

Azure to branch

The application VM sends legacy.corp.example to the resolver inbound endpoint. The linked forwarding ruleset matches corp.example., uses the outbound endpoint, and sends the query to 10.80.1.4. dnsmasq answers authoritatively:

legacy.corp.example.  60  IN  A  10.80.1.4

Branch to Azure

The branch server has conditional forwards for both SQL namespaces:

server=/database.windows.net/10.60.0.4
server=/privatelink.database.windows.net/10.60.0.4

A query to its local dnsmasq instance receives the SQL Private Endpoint address, and a TCP check to 10.60.1.4:1433 succeeds.

In production, replace the simulated branch VNet with on-premises DNS servers reachable over VPN Gateway or ExpressRoute. Permit DNS only between the resolver subnets and approved DNS servers, and preserve both UDP and TCP 53.

Final drift check

Re-run a refreshed plan using the operating inputs:

terraform plan -refresh=true `
  -var-file=stages/04-operate.tfvars

The verified result is:

No changes. Your infrastructure matches the configuration.

That final refresh is important. A successful application test does not prove that the deployed control-plane configuration still matches the source.

Troubleshooting matrix

Symptom	Likely layer	What to check
SQL hostname returns a public IP	DNS visibility	VNet custom DNS, resolver inbound IP, private zone link
Private Endpoint approved but client remains public	DNS visibility	Zone link, client resolver path, cache
NXDOMAIN after adding the zone link	Private record	Zone group, A record, private zone name
Private IP resolves but TCP 1433 fails	Routing/NSG	Peering, UDRs, NSGs, endpoint status
TLS or hostname error	Connection string	Use the normal *.database.windows.net FQDN
Login failed for user '<token-identified principal>'	Identity/SQL authorization	Entra admin, contained user, token resource
Azure cannot resolve corp.example	Outbound resolver	Ruleset link, longest-suffix rule, target DNS reachability, TCP/UDP 53
Branch cannot resolve Azure SQL privately	Inbound resolver	Conditional forwarder target, private zone link, A record
Intermittent old answers	Cache	TTL, resolvectl flush-caches, DNS server caches

Resolver forwarding rules are suffix-based. More-specific matches win, so document rules centrally and avoid overlapping ownership. A root . forwarding rule is powerful but can accidentally redirect all unresolved queries; use it only with a deliberate hybrid DNS design.

Production hardening

This disposable lab chooses visibility and low setup friction. A production design should go further:

• Remove VM public IPs and provide controlled outbound access with NAT Gateway, Azure Firewall, or private package mirrors.

• Use VPN Gateway or ExpressRoute for real on-premises connectivity; the branch VNet here is only a simulation.

• Place shared Private Endpoints in a governed connectivity subscription when central ownership is required, but confirm application teams can operate and troubleshoot the dependency.

• Use Azure Policy to deny public network access, require approved Private DNS zone groups, and audit orphaned endpoints.

• Assign an Entra group as SQL administrator. Create a contained user for each managed identity and grant the minimum database permissions.

• Protect DNS with narrow NSGs and routing, monitor query/health signals, and maintain redundant on-premises forwarders.

• Treat VNet peering, UDRs, DNS links, and Private Endpoint approval as separate change-controlled objects.

• If using Azure Firewall DNS Proxy, align spoke custom DNS and resolver forwarding to avoid loops.

• Design for split-horizon DNS before migration. Test normal public service FQDNs from every client network.

• Manage Terraform state remotely with encryption, locking, RBAC, and workload identity federation/OIDC in CI/CD.

Cleanup

Review a saved destroy plan before applying it:

# Azure SQL will not delete its Entra administrator while Entra-only
# authentication is enabled. The supplied wrapper performs this ordered step.
./scripts/Invoke-PrivateLinkLab.ps1 -Stage destroy
terraform show .local/destroy.tfplan

# Apply through the wrapper so the bounded Azure consistency retries are active.
./scripts/Invoke-PrivateLinkLab.ps1 -Stage destroy -Apply

If you run the commands manually, disable Entra-only authentication before the destroy plan so Azure can remove the temporary lab Entra administrator:

$sqlServer = (terraform output -raw sql_fqdn).Split('.')[0]
az sql server ad-only-auth disable `
  --resource-group rg-private-link-dns-we `
  --name $sqlServer `
  --output none

Then verify the resource group no longer exists:

az group exists --name rg-private-link-dns-we

The source ZIP contains the Markdown workshop, Terraform and AzAPI configuration, provider lockfile, stage files, scripts, cover, sanitized screenshots, publishing brief, and validation transcript. It intentionally excludes caches, state, plans, credentials, IDs, and tokens.

DNS Resolver child deletion can be eventually consistent. The supplied wrapper makes up to three bounded destroy attempts, waiting 30 seconds and saving a fresh plan for only the remaining objects when Azure briefly returns CannotDeleteResource.

Key takeaways

• A Private Endpoint supplies the private service interface; it does not fix client DNS by itself.

• A zone group manages endpoint records, while a VNet link controls who can resolve the zone.

• A linked empty private zone correctly returns NXDOMAIN; that is a useful diagnostic signal.

• The normal Azure SQL FQDN must remain in the connection string.

• Resolver inbound and outbound endpoints solve opposite hybrid DNS directions.

• VNet peering is nontransitive, and successful DNS does not prove routing.

• Managed identity removes stored database credentials, but database authorization still needs least-privilege design.

• Deliberate failure stages make Private Link troubleshooting reproducible instead of mysterious.

References

• Azure DNS Private Resolver architecture

• Azure DNS Private Resolver endpoints and rulesets

• Azure SQL Private Link overview

• Private DNS virtual network links

• Private Link in a hub-and-spoke network

• Hub-spoke network topology

• Microsoft Entra-only authentication for Azure SQL

• Managed identity token acquisition through IMDS

• sqlcmd authentication

azure-private-link-dns-hub-spoke-deep-dive-final.zip

Download ZIP • 5.16MB