Lab 16 — Resilient IaaS Operations and Recovery

azure administrator series

Azure Workshopz — Lab 16

Two green VM status icons do not prove that an application can serve its data. A load balancer needs the correct health signal, an NSG must permit the actual client path, and file recovery needs a usable recovery point and verified original bytes.

This IaaS capstone combines Azure CLI, PowerShell 7, Bicep, Python, Linux administration, KQL, and Azure Portal inspection. Build two application instances, investigate three separate incidents, and recover one missing file without rebuilding either original VM.

Use an independent disposable environment in the existing Visual Studio Enterprise subscription, West Europe. These exercises connect the compute, networking, monitoring, and recovery topics in the AZ-104 study guide.

Download Lab 16 scripts, guest application, KQL queries, and learning material — ZIP

Follow the walkthrough, then use the three challenges, progressive hints, separate solutions, and eight knowledge checks for independent practice.

Architecture and operating boundaries

Conceptual Azure-and-coding illustration.

Use two nonzonal Ubuntu 24.04 LTS Standard_B2s VMs in one aligned availability set configured for two fault domains and five update domains. Each VM has a 32 GiB Standard SSD OS disk and a separate 4 GiB Standard SSD data disk at LUN 0. Record actual assigned domains rather than inferring them from configured limits.

The VNet is 10.116.0.0/24 with workload subnet 10.116.0.0/26. A public Standard Load Balancer forwards TCP 80 to backend TCP 8080. Its HTTP probe uses the data-aware /health endpoint. Permit the verified workstation IPv4 /32 for application traffic and the AzureLoadBalancer source for probes separately; deny other inbound traffic, including SSH.

Neither VM has a public IP or inbound NAT mapping. Administer through Run Command. Disable default subnet outbound access and implicit rule SNAT; use an explicit outbound rule allocating 1,024 ports per instance. No Bastion, NAT Gateway, or VM scale set is required. See outbound-rule guidance.

Port 80 serves only synthetic training content over HTTP. Send no credentials or sensitive data through it. Azure Monitor Agent supplies Heartbeat and limited local0 Syslog from both VMs. A stateful probe-health metric alert uses an enabled receiver-free action group. Only VM A is protected by the dedicated LRS Recovery Services vault; VM B remains an independent control.

Keep a refreshed estimate within EUR12 active consumption plus EUR5 cleanup reserve inside the EUR20 ceiling. Include both VMs/disks, load balancer/IP, backup/snapshots, restored disks, staging, logs, and alert evaluation. Stop new exercises at five hours after chargeable creation and target cleanup before six. Initial backup waits are bounded by 90 minutes, restores by 45 minutes, and telemetry/alert transitions by 30 minutes, shortened by the overall deadline.

1 Confirm the intended context and stable client source

Use Windows, PowerShell 7.5 or newer, installed Azure CLI/Bicep, Python, PSScriptAnalyzer, and the package's compatible installed Az capabilities. Sign in separately to CLI and PowerShell and match tenant, subscription, and operator.

$state = 'C:\LabPrivate\Lab16\state.json'
.\Start-Lab16.ps1 -StatePath $state -Stage Preflight
.\Start-Lab16.ps1 -StatePath $state -Stage Preflight -Execute

Preview mode performs no state IO, deployments, guest execution, formatting, backups, restores, HTTP traffic, changes, or deletion. Executed Preflight is read-only in Azure. Review permissions, providers, policy, locks, image/SKU and availability-set support, quotas, backup/restore capabilities, addresses, names, prices, and current costs. It does not upgrade tools, register providers, grant operator permissions, exempt policy, or substitute a SKU.

Six fresh proxy-disabled IPv4 checks across two endpoints must agree before deployment and public tests. If the source changes, stop rather than widen access or silently change the design. Keep the ACL-protected private manifest outside the download and honor its exclusive lock. Cleanup remains available through management access.

2 Interpret Bicep, review what-if, and deploy

Actual package source — load-balancer configuration.

.\Start-Lab16.ps1 -StatePath $state -Stage Foundation -Execute
# Repeat only after reviewing the corresponding private results.
.\Start-Lab16.ps1 -StatePath $state -Stage Foundation -ReviewedGovernance -ReviewedWhatIf -Execute

Read the foundation and restore templates and their compiled ARM representations. Keep deployments Incremental and inspect every what-if change. Exact IDs, tags, returned parent/child relationships, and pending-operation records identify ownership. A familiar name or prefix is not permission to adopt or delete a resource.

The first review may initialize empty owned groups for Azure validation. Resume their recorded manifest rather than replacing it. Reject unexpected contents or changes. Once applications/protection are established, do not replay an empty baseline template to recover a fault; use the targeted repair.

Inspect the availability set's limits and each VM's actual fault/update domains. This is placement inspection, not a host, zone, or datacenter-outage experiment.

3 Verify networking and effective access

Inspect exact NIC memberships in the application and outbound pools, the frontend 80→backend 8080 rule, its probe association, floating IP disabled, implicit SNAT disabled, and explicit outbound rule. Verify actual agent/outbound connectivity independently of inbound traffic.

• AllowWorkstation, priority 100: recorded workstation /32 to TCP 8080.

• AllowProbe, priority 110: AzureLoadBalancer to TCP 8080/8081.

• DenyOtherInbound, priority 200: remaining inbound traffic denied.

Inspect effective rules and the rule selected for the real client flow. Confirm no instance public IP, SSH NAT mapping, or other administration exposure; check blocked external TCP 22 at the frontend. Probe traffic has its own source path and does not prove client access. Established flow state may survive rule changes, so use new connections. See NSG behavior.

4 Prepare persistent data and data-aware application health

.\Start-Lab16.ps1 -StatePath $state -Stage Applications -Execute

Before formatting, verify each exact VM, recorded disk, LUN, blank signatures, and separation from root/OS/temporary volumes. Never choose a guessed device letter. Create ext4 and mount the original filesystem persistently by UUID at /srv/az104-lab16. Generate at most 5 MiB of synthetic files and record names, sizes, SHA-256, UID/GID, and mode under a dedicated unprivileged owner. Reboot each original VM and verify the changed boot identity, persistent mount, and unchanged hashes.

Actual package source — application health includes data integrity.

The systemd-managed Python standard-library HTTP service listens on 8080. /health validates the mounted original dataset, not just the open port. /data performs fresh file hashing and returns a bounded synthetic instance label/result. /diagnostics retains actual client-address details privately. The public HTML view presents only synthetic instance/readiness/hash information.

Requests with explicit UUID markers emit allowlisted local0 information events. Unmarked recurring probes remain quiet to bound ingestion. There is no HTTP write endpoint. Keep raw addresses, IDs, credentials, and private guest state out of shared captures.

5 Protect VM A and select a usable pre-loss recovery point

.\Start-Lab16.ps1 -StatePath $state -Stage Protection -Execute
.\Start-Lab16.ps1 -StatePath $state -Stage Backup -Execute

Protect A's original OS and data disks in the dedicated LRS Recovery Services vault. Use the Standard daily policy with seven-day retention and two-day Instant Restore retention. Keep existing vaults/workloads outside the exercise. Record exact protected-item/policy/vault identities, required narrowly scoped restore assignments, and any service-created group through actual backup relationships.

Stop writes/service activity on A as required by the guarded implementation, synchronize and unmount the data filesystem, then trigger one on-demand backup. Wait for the entire returned job to reach successful completion; configured protection or a completed snapshot subtask is not enough.

Record the exact job ID, recovery-point ID, UTC time, consistency, tier, and source OS/data relationships privately. Remount/restart A and verify original data afterward. The point must predate file loss and meet the intended on-demand retention. If a response is uncertain, reconcile the recorded job instead of submitting a duplicate. Inspect the full job and point in Portal.

6 Establish monitoring and the complete baseline

.\Start-Lab16.ps1 -StatePath $state -Stage Monitoring -Execute
.\Start-Lab16.ps1 -StatePath $state -Stage Baseline -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Baseline -Execute

Use official Azure Monitor Agent and the required Network Watcher Agent, narrow local0 collection, unique synthetic markers, and independent Heartbeat. The workspace uses 31-day retention and a 1 GiB daily cap; bounded collection/traffic and cleanup are the primary cost controls.

Confirm both local data-aware endpoints, original file metadata/hashes, fresh public requests reaching both original labels, approved client source, healthy probes, actual placement, effective NSG controls, explicit outbound connectivity, and A's completed recovery point together. Default hash distribution is not round-robin or a promise of equal responses.

Inspect actual DipAvailability values and dimensions before enabling the alert. Calibrate its healthy scale from returned data rather than confusing a fraction and percentage. The stateful rule evaluates a one-minute average below the recorded healthy scale every minute. Record actual alert state separately from the metric. Its enabled action group has no external receivers.

7 Query fresh independent telemetry

Use the downloadable queries with exact recorded VM IDs, UTC start, and new explicit request markers. Replace placeholders privately. Heartbeat itself has no HTTP marker; any observation marker attached to a heartbeat summary is an investigator's correlation label, not agent payload.

let StartUtc = todatetime('<RECORDED_OBSERVATION_START_UTC>');
let EndUtc = now();
let VmA = '<EXACT_RECORDED_VM_A_RESOURCE_ID>';
let VmB = '<EXACT_RECORDED_VM_B_RESOURCE_ID>';
let MarkerA = '<UNIQUE_FRESH_VM_A_REQUEST_MARKER>';
let MarkerB = '<UNIQUE_FRESH_VM_B_REQUEST_MARKER>';
Syslog
| where TimeGenerated between (StartUtc .. EndUtc)
| where _ResourceId =~ VmA or _ResourceId =~ VmB
| where Facility == 'local0'
| extend Event = parse_json(SyslogMessage)
| where toint(Event.lab) == 16
| extend Instance=iff(_ResourceId =~ VmA, 'A', 'B'), Marker=tostring(Event.marker)
| where (Instance == 'A' and Marker == MarkerA) or (Instance == 'B' and Marker == MarkerB)
| project TimeGenerated, EventTime, Instance, Marker,
          Healthy=tobool(Event.healthy), HttpStatus=toint(Event.httpStatus)

Actual query source — exact VM identities and unique request markers.

Inspect a separate exact-VM Heartbeat summary using max(TimeGenerated) and sample count. Distinguish event time from ingestion delay. A heartbeat does not prove file/application health, and a request blocked before reaching a VM may emit no Syslog row. Correlate the actual traffic/control evidence rather than using absence alone as a diagnosis. See the Heartbeat and Syslog schemas.

8 Incident one — incorrect probe port

.\Invoke-Lab16Fault.ps1 -StatePath $state -FaultKind ProbePort -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Fault -FaultKind ProbePort -Execute

Actual package source — controlled fault injection.

First verify no listener exists on 8081. Change only the recorded probe port from 8080 to 8081, preserving other properties. The probe NSG rule still permits that port, but neither application listens there. Gather actual unhealthy metrics and failed new connections while local application/data health, guest agents, memberships, hashes, and NSG configuration stay intact.

Observe the alert's actual Fired state within the bounded window. Established TCP flows can survive Standard Load Balancer probe failure, so an old keep-alive response is not the new-connection test. See health-probe behavior.

.\Restore-Lab16.ps1 -StatePath $state -FaultKind ProbePort -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Recovery -FaultKind ProbePort -Execute
.\Restore-Lab16.ps1 -StatePath $state -FaultKind ProbePort -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Baseline -Execute

Restore the exact captured probe through PowerShell using the current owned resource. Preserve unrelated frontends, pools, rules, and outbound settings. Verify healthy metrics/new requests reaching both original labels, unchanged data, actual alert resolution, and no write on repetition. Start the next incident only after a new healthy baseline.

9 Incident two — client traffic denied while probes stay healthy

.\Invoke-Lab16Fault.ps1 -StatePath $state -FaultKind NsgAccess -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Fault -FaultKind NsgAccess -Execute

Change only AllowWorkstation's access from Allow to Deny. Preserve its /32, priority 100, application port, and other fields; leave the probe rule unchanged. Inspect the effective/matched rule for the actual source/destination and gather failed new client requests alongside healthy probes/local applications/original data.

.\Restore-Lab16.ps1 -StatePath $state -FaultKind NsgAccess -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Recovery -FaultKind NsgAccess -Execute
.\Restore-Lab16.ps1 -StatePath $state -FaultKind NsgAccess -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Baseline -Execute

Restore the captured access value after checking the complete rule and surrounding state. Unexpected drift stops the repair. Do not add a higher-priority broad allow, widen source addresses, open SSH, or change the probe. Confirm fresh traffic, both labels/hashes, original source rule, and no write on repetition. If the workstation IP changes, stop public tests and recover the recorded mutation through management access.

10 Incident three — file loss with application-health failover

.\Invoke-Lab16Fault.ps1 -StatePath $state -FaultKind FileLoss -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Fault -FaultKind FileLoss -Execute

Delete only the recorded /srv/az104-lab16/marker.txt on VM A. Observe the actual missing-file error and data-aware health failure. A's original VM, root filesystem, data attachment/mount, guest agent, service process, and remaining file hashes must remain healthy.

Verify B's unchanged data and local health. After probe propagation, new public requests should use B's recorded label while A is excluded from new flows. This is application-health exclusion—not a host/datacenter-outage claim. Confirm no probe, NSG, pool, or original disk setting was changed and that A's exact completed pre-loss point remains the selected source.

11 Restore to new disks and copy only the missing file

.\Restore-Lab16.ps1 -StatePath $state -FaultKind FileLoss -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Recovery -FaultKind FileLoss -Execute
.\Restore-Lab16.ps1 -StatePath $state -FaultKind FileLoss -Execute
.\Test-Lab16.ps1 -StatePath $state -Phase Baseline -Execute

Actual package source — captured configuration and targeted recovery.

Restore the exact pre-deletion point into new disks through PowerShell. Never select Replace existing, recreate A, swap its original disks, or boot the restored OS disk. Keep that OS disk for inventory only. Verify the completed job's source, point, target, and returned disk relationships; identify the data disk using actual job metadata and original source/LUN relationship, not name or size alone.

Attach only the identified restored data disk temporarily to A at LUN 1. Its filesystem UUID can duplicate the original. Resolve the exact physical attachment rather than UUID alone and add no fstab entry. Mount at /srv/az104-lab16-restored read-only with noload to avoid ext4 journal replay. Reject root/OS/temporary or ambiguous devices.

Verify all recovered original hashes before copying. Copy only the absent file to the original mount, preserving recorded UID/GID/mode, and refuse an unexpected existing target rather than overwriting it. Recheck every original hash, VM/disk/mount identity, and B's unchanged control. Unmount/detach the recovered copy.

Verify A's data-aware health, both healthy probes, and new requests reaching both original labels. Repetition should reuse recorded successful results with no new job, disk, attachment, or copy. An uncertain operation requires exact reconciliation before another submission. See VM disk-restore options.

12 Export evidence and clean up protection first

Create the sanitized output directory inside your already ACL-protected private run directory before exporting.

.\Export-Lab16Evidence.ps1 -StatePath $state -OutputDirectory 'C:\LabPrivate\Lab16\sanitized' -Execute
.\Remove-Lab16.ps1 -StatePath $state -Execute
.\Remove-Lab16.ps1 -StatePath $state -Execute

Export allowlisted results with stable sanitized labels. Keep private state, raw IDs/addresses, credentials, restore artifacts, and full raw captures private. Source images illustrate implementation; for your run, capture the actual relevant Portal configuration, metrics, responses, job relationships, and cleanup outcomes.

Disable alerts, stop protection, and request normal backup-data deletion before removing permissions/dependencies. Wait for confirmed terminal states. Keep soft delete enabled and use supported modern lifecycle operations without downgrading APIs or forcing permanent purge.

Remove only recorded disposable operational copies/restore-point collections, restored disks, staging dependencies, workload/network/monitoring resources, assignments, and deployment records, then verified-empty owned groups. Preserve shared Network Watcher and unrelated resources. Verify active-resource/assignment absence twice. Report retained vault/workspace records and expiry separately; deleted operational snapshots no longer supply disk recovery merely because metadata remains. See Backup soft-delete guidance.

Three symptom-only challenges

Case 1 New requests fail but local applications work

Both VMs and local application/data checks remain healthy, yet new public load-balanced connections fail after a controlled change. Identify the smallest difference using fresh traffic, local checks, backend-health metrics, configuration, and alert evidence. Recover without VM redeployment, healthy-service restarts, data changes, broader NSG access, or a rebuilt load balancer.

Case 2 Healthy backends but the client cannot connect

After a new baseline, fresh client requests fail again while backend probes and local data remain healthy. Investigate the actual client path and compare this with Case 1. Recover without a broad allow, source-address change, SSH exposure, port change, or rebuilding resources. Verify successful fresh connections and no configuration write on repetition.

Case 3 A required file is gone on A

A and its agent stay healthy but one original file cannot be read; other files are intact and B continues serving new requests. Select a verified pre-loss point and recover only the missing original file, including owner/permissions. Preserve A's original VM/disks and B. Identify any temporary restored disk unambiguously, mount read-only without journal replay, and remove the attachment afterward.

Submit symptoms, hypotheses, UTC-bounded observations, the exact changed control or point relationship, smallest repair, new recovery traffic, original hashes/metadata, and safe-repeat evidence. A restore status alone does not prove recovered bytes. Keep raw state/IDs private and stop on ambiguity or unfinished service operations.

Progressive hints

Case 1

1. Separate VM/process health from backend eligibility for new flows.

2. Compare probe protocol/port/path/association with the actual listener and saved baseline.

3. Check 8081 versus the 8080 listener; the separate probe rule permits both ports.

4. Use Restore-Lab16.ps1 -FaultKind ProbePort, preserve unrelated settings, then verify fresh traffic, metrics, actual alert resolution, and a no-op repeat.

Case 2

1. Probe and workstation requests have different sources.

2. Inspect effective/matched rules for the real source and backend application port.

3. Compare AllowWorkstation's access field while confirming AllowProbe is unchanged.

4. Use Restore-Lab16.ps1 -FaultKind NsgAccess to restore the exact field, not a broader path; use new connections.

Case 3

1. Compare A's data-aware local health, remaining files, independent guest health, and B's new responses.

2. Select the exact completed point from before deletion; a later point or early snapshot subtask is insufficient.

3. Match restored disk/source/LUN job metadata; UUID can be shared with the original copy.

4. Use Restore-Lab16.ps1 -FaultKind FileLoss, verified LUN 1, read-only noload, and a single metadata-preserving file copy. Unmount/detach and verify no extra job on repeat.

Separate solutions

Case 1 The probe targets an unused port

The captured difference is 8080→8081 on the selected HTTP probe. The application still works on 8080 and NSG probe access is unchanged. Combine actual unhealthy metrics, no wrong-port listener, new connection failures, preserved local data, and actual alert state. Restore the captured probe using current owned configuration; verify original traffic/metrics/hashes and safe repetition.

Case 2 The workstation rule denies new flows

Only the client rule's Allow→Deny access differs. Probe traffic remains separately permitted, so healthy backend metrics do not establish client reachability. Inspect the actual matching rule, restore the captured field without widening scope, and verify new successful connections from the original source.

Case 3 Recover the original file from the correct point

Use A's exact successful pre-loss backup. Restore new disks, never boot the recovered OS, and identify the data copy by job/source/LUN metadata. Verify exact physical LUN 1 despite duplicate UUID, mount read-only without journal replay, confirm original hashes, and copy only the absent file with recorded UID/GID/mode. Detach after validation. Repetition must add no job/disk/attachment/file change.

For any interrupted operation, use Reconcile-Lab16.ps1 -StatePath $state -Execute and the exact recorded provenance before resumption. A timeout is not permission to guess completion or resubmit a restore.

Eight knowledge checks

1. What does fault/update-domain inspection establish?

   Answer: Configured availability-set limits and actual assigned placement. It does not simulate a host/datacenter outage, availability-zone placement, or every continuity scenario.

2. Why inspect metrics and new flows when local apps work?

   Answer: An incorrect probe can make healthy listeners ineligible. Established TCP flows may continue, so new connections expose the intended symptom.

3. How can backends be healthy while the client is blocked?

   Answer: Probe and client sources match different NSG rules. Check the actual client flow's effective/matched rule.

4. Why verify both explicit outbound connectivity and disabled implicit SNAT?

   Answer: Disabling a default does not create an alternative. Pool membership, explicit rule, frontend/port allocation, and actual outbound dependencies must work together.

5. Why use data-aware HTTP health for file loss?

   Answer: A listener can stay open without the required data. Exclude A from new flows while B serves its intact dataset, without calling it a VM/datacenter outage.

6. Why not simply choose the newest point?

   Answer: A point after deletion can preserve the missing-file state. Select the exact completed pre-loss point and verify source relationships.

7. How should a restored disk with duplicate UUID be mounted?

   Answer: Identify the exact job/source/LUN and physical attachment; reject dangerous/ambiguous targets. Use read-only noload and no restored-copy fstab entry.

8. What establishes recovery and what does a Fired alert not establish?

   Answer: Original bytes/owner/mode, all other hashes, preserved identities/mounts/B, fresh data-aware health/traffic, and no extra job/disk/attachment/copy on repetition. Alert state with empty receivers does not establish external delivery.

Verification checklist

• Review syntax/static/application checks, all Bicep builds, Azure validation, incremental what-if, source stability, current pricing, and ownership.

• Verify actual availability placement, both persistent mounts/reboots, original file hashes/metadata, effective rules, explicit outbound connectivity, blocked SSH, and fresh traffic reaching both instances.

• Verify A's entire completed backup, exact pre-loss point, consistency/tier, and original disk relationships.

• For probe and NSG incidents, establish distinct actual symptoms and preserved independent controls; repair captured configuration and prove no-op repetition.

• For file loss, establish missing original data with healthy VM/B controls, exact new-disk restore provenance, safe read-only attachment, original-file recovery, and no extra work on repetition.

• Inspect fresh Heartbeat/Syslog and actual alert Fired/Resolved states separately from receiver configuration.

• Reconcile uncertain operations, retain soft deletion, preserve shared resources, and confirm active-resource/assignment absence twice.

• Extract the package and verify archive and individual file SHA-256 values before use.

Local package checks

The package's recorded local checks include 24 Python application/guest-guard tests, 18 PowerShell files parsed with zero parser errors, three Bicep builds, and 120 offline safety cases including 20 preview variants. PSScriptAnalyzer recorded zero Error-level findings and 38 retained, reviewed warnings. Python tests use local temporary files, loopback HTTP, and mocked guest/disk/IMDS/service operations; safety cases cover context/provenance guards, pending operations, locking, receiver restrictions, preview-before-IO behavior, and adversarial ZIP metadata.

Further resilience topics

Availability-set placement, backup recovery, and regional disaster recovery address different scopes. For Site Recovery, review prerequisites, replication, target network/storage, RPO/RTO, recovery points, isolated test failover, test cleanup, and failover/failback. Use Lab 14 and the Azure-to-Azure recovery guide when adapting the design.

VM scale sets add orchestration/model/repair/replacement behavior; replacement is not targeted file recovery. Compare Lab 10. Route selection and monitor configuration are also distinct from NSG and authorization; revisit Lab 13 for network diagnostics. Production designs need reviewed TLS/authentication, dependency-aware health, complete protection, and approved alert delivery.

Download the complete Lab 16 package

Previous: Lab 15 — Secure PaaS Administration Capstone All workshops: Azure Workshopz