- 2 days ago
- 12 min read
High availability is not a diagram. It is an observed behavior.
In this workshop, I built a small but genuinely redundant Azure web tier, injected seven real failure conditions, and measured the application continuously instead of trusting the status shown by the fault runner. The result is a practical Azure Chaos Studio lab that combines the new Workspace and Scenario experience with a classic agent-based experiment.
The most useful finding was not that the core route stayed online. It was that the first network-latency test did not inject the delay I expected. The failed result is preserved below, along with the diagnosis, one repair, and one successful rerun.
> Test date: 28 July 2026
> Workload region: West Europe
> Chaos Studio Workspace region: North Europe
> Fault duration: Five minutes per fault
> Probe interval: Two seconds with a new HTTP connection
Figure 1 — Original workshop artwork created for this article.
WORKSHOP OVERVIEW
The workload has two Ubuntu 22.04 virtual machines in availability zones 1 and 2. Neither VM has a public IP address, and there is no inbound SSH rule. A Standard Load Balancer is the only public entry point. Its explicit outbound rule gives the private VMs the egress needed for package installation, the Chaos Agent, and the application’s external dependency check.
Nginx fronts a dependency-free Python service with four routes:
• / returns the instance name, zone, application version, and UTC timestamp.
• /healthz is the local Load Balancer health probe.
• /dependency makes a fresh HTTPS request to the Microsoft Entra OpenID metadata endpoint.
• /telemetry returns privacy-safe CPU/load and memory readings.
The chaos control plane is deliberately hybrid:
• A Chaos Studio Workspace runs Compute Zone Down, DNS Outage, and Microsoft Entra ID Outage Scenarios.
• A classic agent-based experiment runs CPU pressure, physical memory pressure, and outbound network latency inside the zone-1 VM.
Microsoft describes Workspaces as the top-level resource that discovers scoped Azure resources and recommends outage-pattern Scenarios. Workspaces and Scenarios are currently in public preview, while classic experiments remain available for custom fault compositions. The source is listed by title in the references section without an outbound link.
OBJECTIVES
By the end of the workshop, you will have:
1. A zone-redundant Linux web tier with private VM NICs.
2. A Standard Load Balancer with an explicit outbound rule.
3. A Chaos Studio Workspace with least-privilege execution identity.
4. A classic Chaos Agent experiment for in-guest pressure and latency.
5. A probe CSV containing every observed request.
6. A Markdown and JSON report with success rate, p50, p95, p99, longest outage, and recovery time.
7. A repeatable cleanup phase that removes the isolated resource group.
PREREQUISITES
Use PowerShell 7 and a current Azure CLI. The account running the bootstrap needs authority to create role assignments, such as Owner or User Access Administrator plus the required resource permissions.
The script checks:
• Azure CLI authentication and the active subscription.
• Registration of Microsoft.Chaos.
• Installation or update of the preview chaos CLI extension.
• Role-assignment authority.
• West Europe availability zones and Standard_B2s availability.
• Regional VM quota.
• Local PowerShell dependencies.
• Resource-name and resource-group collisions.
Microsoft’s current CLI workflow covers Workspace creation, discovery, Scenario configuration, validation, permission repair, execution, and cleanup. The relevant Microsoft Learn CLI management guide is listed by title in the references section.
COST AND SAFETY
This lab creates billable resources: two Standard_B2s VMs and disks, a Standard Load Balancer, a public Load Balancer frontend, network resources, Chaos Studio resources, and Azure Monitor metric data. Pricing depends on subscription and region, so check the Azure pricing calculator before deployment.
Use a dedicated non-production subscription or tightly isolated resource group. The script refuses to deploy if rg-bcwc-chaos-deepdive-weu already exists. It does not include cleanup in the All phase; deletion requires the explicit Cleanup phase and -ConfirmCleanup.
The permissions are scoped to the lab:
• The Workspace system-assigned identity receives Reader for discovery, Virtual Machine Contributor for VM actions, and Network Contributor for temporary NSG actions.
• The classic experiment identity receives Reader only on the targeted zone-1 VM.
• A user-assigned identity is attached to the VMs for the Chaos Agent.
Do not copy these role assignments to a broader production scope without reviewing the blast radius.
ARCHITECTURE
]]
Figure 2 — Two private zonal VMs behind one Standard Load Balancer. Workspace Scenarios operate through Azure resource actions; the classic experiment injects faults through the in-guest agent.
The Workspace can live in North Europe while targeting resources in West Europe. Microsoft documents Workspaces as logical resources that can operate on Azure resources in other regions. Regional support can still vary by fault and target type, so verify the current Chaos Studio regional availability documentation before choosing a location.
DOWNLOAD THE AUTOMATION
Download:
• azure-chaos-studio-deep-dive-workshop.zip — contains the guarded PowerShell .ps1 automation for deployment, fault, analysis, export, and cleanup phases.
• chaos-results.csv — the privacy-safe request-level evidence from this run.
• chaos-results-summary.md — the human-readable measured results.
The script exposes these phases:
Authenticate
Preflight
Deploy
Workspace
AgentExperiment
Baseline
RunWorkspaceScenarios
RunAgentExperiment
Analyze
Export
Cleanup
All
Every command in this article avoids printing tenant IDs, subscription IDs, object IDs, tokens, private keys, or complete VM resource IDs.
STEP 1 — AUTHENTICATE AND RUN THE NON-CHARGEABLE PREFLIGHT
Open PowerShell 7 in the folder containing the script:
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase Authenticate
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase Preflight
The preflight registers Microsoft.Chaos when needed, updates the preview extension, checks quota and zonal availability, and exits before creating billable resources.
The resource group name is a safety boundary. If it already exists, stop and choose a different isolated name rather than adopting an unknown environment.
STEP 2 — DEPLOY THE REDUNDANT WEB TIER
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase Deploy
The deployment creates:
• One virtual network and web subnet.
• One NSG with inbound HTTP only.
• Two private VM NICs.
• Two Ubuntu 22.04 Standard_B2s VMs in zones 1 and 2.
• One Standard public IP for the Load Balancer frontend.
• One Standard Load Balancer, backend pool, HTTP rule, health probe, and outbound rule.
• One user-assigned identity for the Chaos Agent.
Cloud-init installs Nginx and the local Python service. The VMs do not expose SSH.
Figure 3 — An actual PowerShell 7 window rendering the sanitized resource inventory and the three completed Workspace Scenario runs.portal-resource-invento
Figure 4 — The isolated Azure resource group. Account data, subscription identifiers, and generated disk suffixes are redacted.
STEP 3 — CREATE THE WORKSPACE AND DISCOVER SCENARIOS
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase Workspace
The Workspace is created in North Europe with the West Europe resource group as its discovery scope. The script grants the managed identity only the roles required by the selected actions, refreshes recommendations, creates three five-minute configurations, and validates them before a run:
• ComputeZoneDown-1.1 / zone1-five-minutes
• DnsOutage-1.0 / dns-five-minutes
• EntraOutage-1.0 / entra-five-minutes
The Scenario approach is useful because it starts with an outage pattern rather than a single resource action. The Workspace discovers applicable resources and composes the required action sequence.
az chaos scenario list
--workspace-name csw-bcwc-chaos-neu
--resource-group rg-bcwc-chaos-deepdive-weu
--output tablepowershell-workload-ev
Figure 5 — The Entra Outage Scenario configuration after a successful five-minute run. DNS Outage is visible with the same successful latest-run state.
STEP 4 — ENABLE THE AGENT AND CREATE THE CLASSIC EXPERIMENT
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase AgentExperiment
The script:
1. Attaches the user-assigned identity.
2. Enables the Chaos Studio agent target and capabilities on the zone-1 VM.
3. Installs and validates the Linux Chaos Agent extension.
4. Creates a system-assigned identity for the experiment.
5. Grants that experiment identity Reader on the targeted VM only.
6. Validates the three sequential five-minute steps.
The original experiment definition is:
Step 1 CPU Pressure 85% PT5M
Step 2 Physical Memory Pressure 75% PT5M
Step 3 Outbound Network Latency 750ms PT5M
destination: Microsoft Entra OpenID metadata host on TCP 443
CPU Pressure, Physical Memory Pressure, and outbound Network Latency are documented agent-based faults for Linux in the Chaos Studio fault and action library.
Figure 6 — The restored classic experiment definition, showing the five-minute CPU and physical-memory steps.
Figure 7 — The original hostname-filtered, 750 ms outbound latency step. The temporary repair used for the rerun was not left in the experiment definition.
STEP 5 — RECORD THE BASELINE
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase Baseline
For 60 seconds, the probe requests / and /dependency every two seconds with a fresh connection. The acceptance check also confirms that responses were served by both availability zones.
Observed baseline:
| Route | Samples | Success | p50 | p95 | Longest outage |
|---|---:|---:|---:|---:|---:|
| Core / | 30 | 100% | 53 ms | 75 ms | 0 s |
| /dependency | 29 | 100% | 67 ms | 108 ms | 0 s |
The baseline matters because a latency fault is only proven when it produces a material increase relative to the normal dependency path.
STEP 6 — RUN THE WORKSPACE SCENARIOS
Run all three Workspace Scenarios:
.\azure-chaos-studio-deep-dive-workshop.ps1
-Phase RunWorkspaceScenarios
-WorkspaceScenario All
Or select one:
.\azure-chaos-studio-deep-dive-workshop.ps1
-Phase RunWorkspaceScenarios
-WorkspaceScenario Compute
For each run, the probe begins before the disruption, continues through the five-minute action, and stays active for two minutes after reversal.
Figure 8 — Compute Zone Down, DNS Outage, and Microsoft Entra ID Outage all completed successfully in the Workspace.
COMPUTE ZONE DOWN
The Workspace shut down resources in zone 1 for five minutes. The Load Balancer continued sending traffic to zone 2.
• Core requests: 182/182 succeeded
• Core p95: 68 ms
• Longest core outage: 0 seconds
• Zone-1 return after fault reversal: 5.6 seconds
This passed the availability gate: at least 99% core success, no consecutive outage over 15 seconds, zone 2 serving during the shutdown, and zone 1 returning within five minutes.
DNS OUTAGE
The Workspace applied a temporary outbound port-53 NSG deny. That broke the fresh dependency lookup without taking down the local web route.
• Core requests: 218/218 succeeded
• Dependency success: 72.50%
• Dependency p95: 6,002 ms
• Longest dependency outage: 144 seconds
• Dependency recovery after reversal: 1 second
This is the intended resilience boundary: the service still answered its core route while the external dependency route visibly failed.
MICROSOFT ENTRA ID OUTAGE
The Entra Scenario applied the outage pattern through NSG actions.
• Core requests: 216/216 succeeded
• Dependency success: 64.76%
• Dependency p95: 6,003 ms
• Longest dependency outage: 296.1 seconds
• First five successful dependency probes after reversal: immediate
The /dependency route proved the outage, while / remained available.
STEP 7 — RUN THE AGENT-BASED EXPERIMENT
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase RunAgentExperiment
CPU PRESSURE
The action requested 85% pressure on the zone-1 VM for five minutes.
• Core success: 100%
• Core p95: 78 ms
• Core p99: 854 ms
• Azure Monitor peak CPU: 75.92%
The requested pressure level is the experiment input; the Azure Monitor observation is the evidence of what the VM actually reached. They are deliberately reported as separate values.
PHYSICAL MEMORY PRESSURE
The action requested 75% physical memory pressure for five minutes.
• Core success: 100%
• Core p95: 73 ms
• Core p99: 130 ms
The core route stayed well under the two-second p95 gate.
OUTBOUND NETWORK LATENCY — FIRST ATTEMPT
The first latency action failed at the experiment level. More importantly, the measurement did not show the intended delay:
• Core success: 100%
• Dependency success: 100%
• Dependency p95: 89 ms
• Required increase over baseline: at least 500 ms
• Result: FAILportal-agent-experimen
Figure 9 — CPU and memory completed, but the initial hostname-filtered network-latency action failed. This result is retained as evidence.
The diagnosis found that the Linux traffic-control rule had been created for the IP addresses resolved when the fault started. During the run, the Microsoft Entra OpenID metadata host returned a different address set. Traffic followed the new addresses and bypassed the original filters.
That behavior is consistent with the documented hostname-filter behavior: address information is retrieved when the fault starts. A hostname that rotates across front-door addresses is therefore a weak target for a deterministic five-minute latency assertion.
The failed action also left a tc netem root queue discipline on the VM. I removed only that exact rule and verified that no test queue discipline remained.
ONE-TIME REPAIR AND RERUN
The acceptance plan allowed one repair for an obvious lab defect. For the rerun, the script temporarily replaced the hostname filter with the stable Microsoft Entra IPv4 ranges used by the service and executed only the latency step. After the run, it restored the original three-step experiment definition.
.\azure-chaos-studio-deep-dive-workshop.ps1
-Phase RunAgentExperiment
-RetryLatencyWithStableRanges
Observed rerun:
• Core requests: 165/165 succeeded
• Dependency requests: 98/98 succeeded
• Dependency p95: 3,108 ms
• Increase over the 108 ms baseline p95: 3,000 ms
• Result: PASS
The injected delay was 750 ms per outbound packet path, but one HTTPS dependency request consists of several network exchanges. End-to-end request latency was therefore much higher than 750 ms. That is exactly why the external probe is the acceptance signal—not the fault parameter alone.
portal-agent-experimen
Figure 10 — The repaired latency-only execution completed for five minutes and 45 seconds.portal-agent-experimen
Figure 11 — Both executions remain visible: the failed first run and the successful one-time retry. Execution identifiers are redacted.
STEP 8 — ANALYZE AND EXPORT
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase Analyze
.\azure-chaos-studio-deep-dive-workshop.ps1 -Phase Export
The CSV schema is:
Timestamp,Test,Route,StatusCode,LatencyMs,Instance,Zone,DependencyStatus,ErrorClass
The analysis aligns request timestamps with the actual action windows returned by Chaos Studio. Transition and post-fault samples remain in the raw CSV, while the gate calculations use the action window for each named fault.
Figure 12 — The privacy-safe measured result table rendered in an actual PowerShell 7 window.
MEASURED FINDINGS
| Fault | Core result | Dependency or recovery result | Gate |
|---|---|---|---|
| Baseline | 100%, p95 75 ms | Dependency p95 108 ms | PASS |
| Compute Zone Down | 100%, p95 68 ms | Zone 1 returned in 5.6 s | PASS |
| CPU Pressure | 100%, p95 78 ms | Azure Monitor peak 75.92% | PASS |
| Physical Memory Pressure | 100%, p95 73 ms | No core outage | PASS |
| Network Latency — initial | 100%, p95 66 ms | Dependency p95 89 ms | FAIL |
| Network Latency — retry | 100%, p95 65 ms | Dependency p95 3,108 ms | PASS |
| DNS Outage | 100%, p95 73 ms | 144 s dependency outage; recovery 1 s | FAULT PROVED |
| Entra ID Outage | 100%, p95 69 ms | 296.1 s dependency outage; immediate recovery | FAULT PROVED |
The core HTTP route achieved 100% success during every executed fault. That is a strong result for this narrow service boundary, but it does not mean the application was fully functional: DNS and Entra failures deliberately made the dependency route unavailable.
That distinction is the central lesson. A green Load Balancer health probe proves that the local process is serving. It does not prove that login, token discovery, or another remote dependency works.
WHAT I WOULD CHANGE FOR PRODUCTION
This two-VM lab is intentionally small. For a production service, I would add:
• An application-level readiness probe that distinguishes local liveness from dependency readiness.
• An explicit user journey or synthetic transaction, not only route probes.
• Azure Monitor alerts and workbook visualizations aligned to each fault window.
• Retry, timeout, circuit-breaker, and cached-metadata behavior for identity dependencies.
• Capacity testing that proves the surviving zone can carry the full production load.
• A documented rollback owner and a change window.
• Policy-controlled Chaos Studio permissions and an approval process.
• Repeated runs at different times to expose DNS, front-door, and regional variance.
TROUBLESHOOTING
NGINX RETURNS 404 AFTER CLOUD-INIT
The first deployment installed the site link after Nginx was already running. Restart Nginx after enabling the site, then recheck /healthz. This was an obvious lab configuration defect, repaired before the measured baseline.
THE PROBE JOB STARTS BUT WRITES NO VALID SAMPLES
Use Start-ThreadJob in PowerShell 7. In an interpolated URL, delimit variables next to the path:
$scheme = 'http'
$uri = "${scheme}://${Ip}${Path}?probe=$([Guid]::NewGuid())"
Without the braces, PowerShell can interpret the adjacent characters as part of the variable name.
THE LATENCY ACTION COMPLETES BUT LATENCY DOES NOT INCREASE
Check:
1. The destination and port filter.
2. The current DNS answers from the VM.
3. The tc filters and root queue discipline.
4. Whether the hostname maps to rotating service addresses.
5. End-to-end latency on the dependency route, not only experiment status.
Do not turn a failed gate green by changing the report. Preserve the run, repair one clear lab defect, and rerun once.
WORKSPACE VALIDATION REPORTS MISSING PERMISSIONS
Role assignments can take several minutes to propagate. Confirm that the Workspace identity has only the roles required by the Scenario, then rerun validation. The CLI guide documents config validate and config fix-permissions; use --what-if before an automatic permission repair when reviewing a broader scope.
A SCENARIO APPEARS UNDER THE WRONG RESOURCE TYPE
Preview catalogs can contain similarly named Scenario variants. Verify the exact Scenario ID before creating a configuration. In this lab, an early unused zone1-five-minutes configuration was attached to the SQL Managed Instance variant. It was never run, was removed during evidence review, and the intended ComputeZoneDown-1.1 configuration remained intact.
CLEANUP
Delete only the isolated lab resource group:
.\azure-chaos-studio-deep-dive-workshop.ps1
-Phase Cleanup
-ConfirmCleanup
The script starts the resource-group deletion and polls Azure until the group is confirmed absent. Local CSV, JSON, Markdown, and screenshot evidence remains in the outputs` folder.
FINAL TAKEAWAY
This workload survived the local compute failures I injected: zone loss, CPU pressure, and memory pressure did not interrupt the core route. It also degraded in the way the service boundary predicted: DNS and Entra outages broke the dependency route while the local service stayed online.
The failed first latency attempt was just as valuable as the passes. It exposed a test-design assumption about rotating hostname addresses and a cleanup edge case in the agent action. Chaos engineering is not a green-checkmark exercise. The experiment itself must be observable, falsifiable, and honest.
OFFICIAL MICROSOFT REFERENCES
• Workspaces in Azure Chaos Studio
• Manage Chaos Studio Workspaces and Scenarios with the Azure CLI
• Azure Chaos Studio fault and action library
• Regional availability of Azure Chaos Studio
Comments