Lab 15 — Secure PaaS Administration Capstone

azure administrator series

Azure Workshopz — Lab 15

Your application is running—but is it authorized to read the right container? Does its storage hostname still resolve through Private Link? Is staging refusing requests because the code cannot start, or because a release setting says it is not ready?

In this capstone, combine Azure CLI, PowerShell 7, Bicep, Node.js, KQL, and Azure Portal inspection. Build a small App Service application with a private managed-identity data path, investigate three separate incidents, then practise release promotion, rollback, and manual scaling. Preserve the original synthetic data and unrelated controls throughout each recovery.

Allow 240–300 minutes, including propagation and service waits. Use a disposable training environment. This workshop connects subjects from the Microsoft AZ-104 study guide; it is independent community training.

Download Lab 15 scripts, application source, and learning material — ZIP

Follow the walkthrough, then use the three challenges, progressive hints, separate solutions, and eight knowledge checks for independent practice.

Architecture and boundaries

Conceptual Azure-and-coding illustration.

Create one Linux Standard S1 plan with one worker, production version 1, and a staging slot running version 2. Give the two slots distinct system-assigned identities. Both read the same private dataset container using Storage Blob Data Reader at that exact container scope. An empty decoy container provides a deliberately wrong scope for one incident.

Use the existing Visual Studio Enterprise subscription and West Europe. The VNet is 10.115.0.0/24; integration uses 10.115.0.0/26, delegated to Microsoft.Web/serverFarms, and the Blob private endpoint uses 10.115.0.64/27. Both slots integrate with the same integration subnet. Stop on conflicting address space.

App Service application and SCM hostnames remain publicly addressed but accept only the verified workstation IPv4 /32, with unmatched requests denied. VNet integration supplies outbound connectivity; it does not make application ingress private. Storage public-network access, anonymous Blob access, and Shared Key remain disabled from creation. Use the normal storage hostname, not a hard-coded private IP or hosts-file entry. See App Service access restrictions and Storage private endpoints.

The application uses Node.js 24 LTS and standard-library HTTP, DNS, filesystem, and hashing functions. No external application dependency, storage key, SAS, VM, or database is required. HTTPS-only, minimum TLS 1.2, Always On, and disabled FTP/SCM basic publishing credentials belong in the initial configuration.

Use narrow App Service console/HTTP and Blob read/write diagnostics with one dedicated Log Analytics workspace: 31-day retention and a 1 GiB daily cap. Bounded traffic and collection are the primary cost controls; a daily cap is not a guaranteed spending limit. One stateful readiness log alert uses an enabled action group without external receivers.

1 Prepare the private run and review preflight

Use Windows, PowerShell 7.5 or newer, installed Azure CLI/Bicep, Node.js, PSScriptAnalyzer, and the compatible installed Az capabilities required by the package. Sign in to CLI and PowerShell separately and match the intended tenant, subscription, and operator.

$state = 'C:\LabPrivate\Lab15\state.json'
.\Start-Lab15.ps1 -StatePath $state -Stage Preflight
.\Start-Lab15.ps1 -StatePath $state -Stage Preflight -Execute

Without -Execute, the operational interfaces describe the action only: no deployment, uploads, guest execution, application traffic, swaps, scaling, or deletion. Executed Preflight inspects Azure without creating resources. Review dependencies, permissions, providers, policy, locks, quotas, names, runtime/SKU availability, addresses, pricing, and existing costs. No automatic upgrade, registration, operator grant, policy exemption, or substitution is authorized.

Require six fresh proxy-disabled IPv4 checks across two independent endpoints to agree. Repeat before deployment and public testing. If the address changes, stop public exercises without broadening the allow rule. Keep private state outside the download, protect it with Windows ACLs, and use the exclusive manifest lock.

In Azure Portal, confirm the intended subscription and West Europe. Keep account information, tenant/subscription identifiers, addresses, tokens, identity headers, and full raw captures private.

2 Interpret Bicep and deploy the protected foundation

Read templates/foundation.bicep and its compiled ARM representation. Identify parameters, variables, resource dependencies, outputs, the plan/slot relationship, subnet delegation, endpoint/DNS relationships, and creation-time access restrictions.

Actual package source — Bicep declarations.

.\Start-Lab15.ps1 -StatePath $state -Stage Foundation -Execute
# Acknowledge reviews only after inspecting their private results.
.\Start-Lab15.ps1 -StatePath $state -Stage Foundation -ReviewedGovernance -ReviewedWhatIf -Execute

Keep deployments Incremental. Inspect validation and every what-if change before acknowledging the corresponding gate. Review switches record inspection; they do not disable policy. Exact IDs, ownership tags, returned relationships, and pending-operation records establish ownership. A matching prefix or familiar name does not.

Inspect S1/one worker, Linux/Node 24 LTS, HTTPS/TLS, Always On, disabled publishing credentials, and application plus SCM restrictions for each slot. Inspect both VNet integrations, subnet delegation, the approved Blob private endpoint, DNS zone group, private zone, and exact VNet link with registration disabled. Confirm storage Standard_LRS, HTTPS/TLS 1.2, and the three disabled access mechanisms. Endpoint approval does not establish authorization by itself.

3 Deploy two releases and initialize private data

.\Start-Lab15.ps1 -StatePath $state -Stage Applications -Execute

Deploy extracted ZIP packages through Azure CLI using Microsoft Entra authentication, with remote build and run-from-package disabled. Production uses version 1; staging uses version 2. Record package and independently inspected deployed-file SHA-256 hashes. A deployment success message alone is not content verification. See ZIP deployment guidance.

Actual package source — Node.js application behavior.

App Service supplies a local identity endpoint/header for requesting a Storage OAuth token. Use them internally and keep the token in memory. Never return or log credentials. Each slot has its own identity; see App Service managed identities.

Temporarily give production Storage Blob Data Contributor at the dataset container only. Startup initialization runs when LAB_INITIALIZE_DATA=true and creates bounded synthetic files conditionally. On a retry, compare an existing blob's exact bytes and hash; never overwrite unexpected data. Disable initialization and remove that recorded data-role assignment before accepting the baseline. Retain only the two dataset Reader relationships and keep the decoy container empty.

Mark dataset configuration, expected hashes, the slot label, and LAB_RELEASE_READY as slot-specific. Code-derived release metadata follows the deployed files; the slot label stays with its slot.

• /health reports local readiness, release, and slot; success alone does not test storage.

• /data performs fresh OAuth Blob reads and reports file names/sizes/SHA-256 comparisons, dataset hash, and a unique request marker.

• When readiness is false, the root page, /health, and /data return the synthetic 503 before reading storage.

• /diagnostics bypasses readiness and performs a fresh OAuth read. Retain its DNS/socket/identity metadata privately; use sanitized results for sharing.

4 Establish monitoring and the complete baseline

.\Start-Lab15.ps1 -StatePath $state -Stage Monitoring -Execute
.\Start-Lab15.ps1 -StatePath $state -Stage Baseline -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Baseline -Execute

Configure diagnostics before final settings snapshots because logging enablement can affect related app configuration. Monitoring first establishes collection, generates bounded fresh requests, and waits for its tables; then it configures the alert with query validation enabled. Follow recorded resume/reconciliation instructions if an operation is uncertain rather than submitting it again.

Confirm both releases read the original data through the recorded private endpoint using their own identities. Check trusted TLS, exact deployed hashes, all data hashes, sticky settings, ingress/SCM controls, integrations, endpoint approval, and Reader scopes together. Use new connections, unique markers, and no proxy/cache.

Check denial from an independent source outside the workstation rule. Inspect storage public access disabled and the actual public-path rejection without exposing the in-memory credential. An unauthenticated rejection alone does not identify a network control. In Portal, inspect the actual relevant settings, responses, and logs rather than treating a browser permission prompt as an application incident.

5 Correlate fresh requests with KQL

Filter private queries to the exact lab resource and recorded UTC observation window. Replace placeholders with fresh recorded values; the code below is a query, not a result table.

let StartUtc = todatetime('<RECORDED_OBSERVATION_START_UTC>');
let EndUtc = now();
let SlotId = '<EXACT_RECORDED_STAGING_RESOURCE_ID>';
let Marker = '<UNIQUE_FRESH_REQUEST_MARKER>';
AppServiceConsoleLogs
| where TimeGenerated between (StartUtc .. EndUtc)
| where _ResourceId =~ SlotId
| extend Event = parse_json(ResultDescription)
| where Event.lab15 == true and tostring(Event.marker) == Marker
| project TimeGenerated, Slot=tostring(Event.slot),
          Marker=tostring(Event.marker), HttpStatus=toint(Event.httpStatus),
          ReadinessFalse=tobool(Event.readinessFalse)

Actual query source — correlate your fresh observations.

Find the storage request using its recorded ClientRequestId, which comes from x-ms-client-request-id. Check OAuth and the expected requester privately. Project operation/status/duration and the synthetic correlation needed for the question; exclude raw addresses, URIs, token-related fields, and identities. Matching request IDs provide stronger correlation than similar timestamps. See the AppServiceConsoleLogs and StorageBlobLogs schemas.

Use summarize and bin(TimeGenerated, 1m) for a short filtered trend. Distinguish event time from ingestion delay. Bound propagation/telemetry transitions to 30 minutes and the remaining operational deadline; an empty query result is not automatically proof that no request occurred.

6 Incident one — diagnose the authorization scope

.\Invoke-Lab15Fault.ps1 -StatePath $state -FaultKind RbacScope -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Fault -FaultKind RbacScope -Execute

Actual package source — controlled CLI configuration changes.

The exercise removes staging's recorded dataset Reader assignment and creates Reader at the empty decoy container. Compare the exact principal, role definition, and full scope. Gather the actual failed staging storage response, unchanged private resolution/endpoint, new production success, and unchanged code/data. A generic HTTP 403 alone is not a diagnosis.

Managed-identity authorization can be cached. Restarting the app or requesting another token does not guarantee immediate revocation or refreshed permission behavior. Use bounded observations; if the expected symptom does not emerge in time, restore the recorded mutation and keep the observation distinct from a verified reproduction.

.\Restore-Lab15.ps1 -StatePath $state -FaultKind RbacScope -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Recovery -FaultKind RbacScope -Execute
.\Restore-Lab15.ps1 -StatePath $state -FaultKind RbacScope -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Baseline -Execute

Restore the captured dataset Reader relationship and remove only the recorded decoy assignment. Confirm fresh successful reads/original hashes in both slots. Repetition should verify state without another assignment. Do not grant storage-account-wide access or remove unexpected permissions to force a pass. Reconcile a recorded partial assignment transition before resuming.

7 Incident two — restore the shared private DNS relationship

.\Invoke-Lab15Fault.ps1 -StatePath $state -FaultKind PrivateDns -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Fault -FaultKind PrivateDns -Execute

Delete only the recorded private DNS VNet link. Observe fresh changed resolution and actual failed storage reads after bounded propagation. The link is shared, so both slots' data paths can fail. Keep process health separate from data access. Check endpoint approval, both identities/Reader scopes, public access disabled, original data, and unchanged packages.

.\Restore-Lab15.ps1 -StatePath $state -FaultKind PrivateDns -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Recovery -FaultKind PrivateDns -Execute
.\Restore-Lab15.ps1 -StatePath $state -FaultKind PrivateDns -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Baseline -Execute

Restore the exact captured link name, zone, VNet relationship, and disabled registration. Confirm private resolution and original bytes in both slots, with one unchanged link on repetition. Do not use hosts-file overrides, a hard-coded IP, another identity, public storage, or a broader role to bypass the dependency.

8 Incident three — recover staging readiness and inspect the alert

.\Invoke-Lab15Fault.ps1 -StatePath $state -FaultKind SlotSetting -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Fault -FaultKind SlotSetting -Execute

Change only staging's sticky LAB_RELEASE_READY from true to false. Check the synthetic HTTP 503, new production HTTP 200, unchanged code hashes, and an independently successful staging /diagnostics storage read. The ordinary /data route intentionally refuses before reading storage while readiness is false. This is an application-setting incident, not an Azure infrastructure outage.

The stateful readiness log alert uses a five-minute lookback and one-minute evaluation with count greater than zero. Its structured event filter targets the exact run, reserved marker, staging slot, and readiness-failure fields. Observe actual Fired state separately from matching log rows. The enabled action group contains no external receiver.

.\Restore-Lab15.ps1 -StatePath $state -FaultKind SlotSetting -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Recovery -FaultKind SlotSetting -Execute
.\Restore-Lab15.ps1 -StatePath $state -FaultKind SlotSetting -Execute
.\Test-Lab15.ps1 -StatePath $state -Phase Baseline -Execute

Actual package source — restore captured settings while preserving unrelated keys.

Compare the full configuration against the captured snapshot and the single permitted difference; restore the exact saved value. Unexpected drift stops repair. Confirm HTTP 200, original hashes, production health, and no write on repetition. Stop fault requests and observe actual alert resolution after its window permits it. Application recovery and alert resolution use different timelines; alert state is not notification delivery.

9 Promote with preview, roll back, and scale workers

.\Start-Lab15.ps1 -StatePath $state -Stage Release -Execute
.\Start-Lab15.ps1 -StatePath $state -Stage Scaling -Execute

Require all incidents recovered and a fresh healthy baseline. Warm-up uses /health, accepting HTTP 200. Inspect staging under target configuration during manual swap preview, including its private data path and hashes, before completing promotion. Verify version 2 in production/version 1 in staging, then swap back and verify original placement.

Identities, VNet integrations, and sticky settings stay attached to their slots. Both identities therefore need dataset Reader independently. Recheck TLS and restrictions. A pending preview cancellation differs from rolling back completed promotion. If interrupted, reconcile the exact operation and current versions; blindly repeating a swap can undo the intended release. See deployment-slot behavior.

Demonstrate manual capacity 1 → 2 → 1 while retaining S1. Inspect actual worker inventory as well as returned capacity and verify private reads/original hashes at each transition. Do not promise equal response distribution. Keep raw worker identities private and use stable sanitized labels.

10 Export your evidence and remove the disposable run

Create the sanitized output directory inside your already ACL-protected private run directory before exporting.

.\Export-Lab15Evidence.ps1 -StatePath $state -OutputDirectory 'C:\LabPrivate\Lab15\sanitized' -Execute
.\Remove-Lab15.ps1 -StatePath $state -Execute
.\Remove-Lab15.ps1 -StatePath $state -Execute

Require a refreshed six-hour peak estimate within EUR15 active consumption, retaining EUR5 for cleanup. Stop exercises at five hours and target cleanup before six. Stopping an app does not stop plan billing. Include two S1 workers at peak, Private Link, storage, ingestion, and alert evaluation.

Disable alerts, remove exact diagnostics/assignments, disconnect integrations, and delete only recorded resources and deployment histories, then the verified-empty owned group. Unexpected contents or ambiguous pending operations stop deletion. Verify absence twice. Use normal workspace soft deletion and report its 14-day recovery window separately from active-resource absence. See workspace deletion guidance.

Keep tokens, identity headers, raw identifiers, addresses, private state, and unredacted diagnostic captures out of shared evidence. Use genuine Portal observations for your run and deterministic privacy masks/crops without changing the results.

Three symptom-only challenges

Case 1 Staging cannot read the dataset

Staging runs the expected release but fresh data requests fail; production still reads original files. Determine where staging's request stops. Gather competing hypotheses, actual request evidence, and a comparison establishing the smallest changed control. Recover without replacing identity, redeploying code, broadening permissions, generating a SAS, enabling Shared Key, or changing storage networking. Verify original hashes and safe repetition.

Case 2 Both slots lose the data path

Both slots still run their code but fresh dataset reads fail. Their packages and identities remain unchanged. Investigate shared dependencies and distinguish data access from application-process health. Recover without an IP/hosts-file workaround, alternate hostname, broader role, public storage, or endpoint recreation. Prove both slots use the intended private path.

Case 3 Staging returns HTTP 503

Staging produces a synthetic readiness response; production still returns HTTP 200 and both code hashes remain original. Decide whether the process, data dependency, release, or configuration explains it. Correlate new log evidence and actual alert state. Recover without redeployment, plan replacement, storage-access changes, or production changes.

For each case, submit the symptom, competing hypotheses, UTC-bounded observations, exact configuration difference, smallest repair, new successful requests, original hashes, and no-op repeat evidence. Keep raw object IDs/private state private. Unexpected drift is a stop condition.

Progressive hints

Case 1

1. Compare release, DNS destination, identity, authorization, and requested object separately.

2. Inspect principal, role definition, and full scope—not only a role display name.

3. Compare staging's dataset and decoy container assignments with the saved baseline.

4. Restore the recorded dataset Reader relationship with Restore-Lab15.ps1 -FaultKind RbacScope, remove only the recorded decoy assignment, then verify new reads and repetition. Account for authorization caching.

Case 2

1. Start with the dependencies both slots share.

2. Compare fresh normal-hostname resolution with the recorded endpoint.

3. Inspect the private zone's exact VNet link; records can exist without a usable network relationship.

4. Use Restore-Lab15.ps1 -FaultKind PrivateDns and verify both private data paths without bypasses.

Case 3

1. Read the response body; 503 alone is not a cause.

2. Use the private /diagnostics fresh OAuth read, not readiness-gated /data, to test the dependency.

3. Compare staging's sticky readiness setting with the snapshot and production.

4. Use Restore-Lab15.ps1 -FaultKind SlotSetting, preserve unrelated keys, verify no-op repetition, stop fault traffic, and observe actual alert resolution separately.

Separate solutions

Case 1 Reader is assigned at the wrong container

The decoy assignment does not cover the dataset request. Establish the actual authorization response plus changed principal/role/scope, unchanged private resolution, production's new success, and unchanged bytes. Restore the captured dataset Reader assignment and delete only the recorded decoy assignment. Use Recovery validation, repeat the repair, then run a new Baseline. Reconcile a partial assignment transition by its exact pending record instead of adopting or removing unrelated assignments.

Case 2 The private DNS VNet link is missing

Deleting the shared link removes the intended resolution relationship, not the endpoint or data. Establish changed fresh answers and failed reads while endpoint approval, identities, roles, public-access denial, and bytes remain intact. Restore the exact link name/zone/VNet with registration disabled. Verify both slots resolve privately and return the original hashes; repetition should create no second link.

Case 3 Staging readiness is false

The synthetic readiness setting declines the request before storage is read. Confirm independent diagnostics success, unchanged code, production HTTP 200, and the single sticky setting difference. Restore the captured value without redeploying. Verify HTTP 200 and original hashes; repeat without a write. Wait for the alert's actual Resolved state after the failure rows age out. Empty action receivers establish configuration, not delivery.

Eight knowledge checks

1. Why still restrict application and SCM ingress when App Service has VNet integration?

   Answer: Integration supplies outbound connectivity, not private inbound access. Protect both public surfaces independently with the verified source and unmatched denial.

2. Which three parts of a data role relationship must match the request?

   Answer: Exact principal, role definition, and resource scope. Reader at the decoy container does not cover the dataset.

3. Why can an approved endpoint be unusable after a DNS-link change?

   Answer: Approval does not supply the resolver path. Both slots use the same VNet/DNS relationship, so both data paths can be affected.

4. Does HTTP 403 prove RBAC caused the problem?

   Answer: No. Compare the actual diagnostic, exact changed scope, private resolution, endpoint, identity, healthy control request, and unchanged content.

5. Can restarting the application guarantee immediate managed-identity permission refresh?

   Answer: No. Token/authorization caching can delay behavior, and there is no guaranteed forced token refresh before expiry. Use bounded observations.

6. How do you distinguish the readiness incident from a platform outage?

   Answer: Compare the synthetic response, setting difference, private diagnostics read, production health, and code hashes. Restore the captured setting only.

7. What stays with production after version 2 is promoted?

   Answer: Its slot-specific label/settings, identity, and VNet integration. Each identity needs Reader independently because code moves while those relationships stay.

8. Why can the alert stay Fired after HTTP 200 returns?

   Answer: Historical rows remain in the lookback and alert evaluation follows its own schedule. An action group without receivers does not establish external delivery.

Verification checklist

• Review syntax/static checks, application tests, Bicep compilation, Azure validation, and incremental what-if.

• Confirm both releases, independently inspected deployed bytes, original dataset hashes, correct slot identities/scopes, approved private resolution, trusted TLS, and application/SCM restrictions.

• Record independent-source denial and storage public-access behavior with the relevant configuration.

• For each incident, establish the actual symptom and exact cause, preserve unrelated controls, recover with new requests, and verify repetition changes nothing.

• Correlate fresh application/storage events and actual alert Fired/Resolved states.

• Verify preview, promotion, rollback, sticky controls, actual worker counts, and hashes after scaling.

• Reconcile uncertain operations, export only sanitized evidence, and verify resource/assignment absence twice after cleanup.

• Extract the download and compare archive and individual file SHA-256 values before use.

Local package checks

The package's recorded local checks include 17 Node application tests, 16 PowerShell files parsed with zero parser errors, two Bicep builds, and 102 offline safety cases including 21 preview interfaces. PSScriptAnalyzer recorded zero Error-level findings and 29 retained, reviewed warnings. The application tests use synthetic/mock storage, DNS, and identity fixtures with actual local HTTP responses; safety cases exercise local locking, context/identity guards, pending states, and adversarial ZIP fixtures.

Further administration topics

For custom domains, distinguish DNS ownership validation, routing, hostname binding, certificate issuance/binding, trust, and renewal. Use the custom-domain and certificate guides when adapting the Azure-provided hostname design. Keep existing registrar/DNS/certificate material outside this exercise.

Manual scale-out, Azure Monitor metric/scheduled autoscale, and Premium automatic scaling are different mechanisms. Choose a mechanism intentionally and review its prerequisites, bounds, cooldown, and downstream behavior. A slot rollback changes release placement; use Lab 12's App Service backup/restore material for the separate recovery topic. Production identity/bootstrap and external alert routing also need their own reviewed lifecycle and delivery design.

Download the complete Lab 15 package

Previous: Lab 14 — Azure Backup, Restore, and Site Recovery Next: Lab 16 — Resilient IaaS Operations and Recovery All workshops: Azure Workshopz