ZIP SHA-256: 62095515ab0c9aa0db7e1655c63865f091c77b44edb1987804ef24762b8d44a9. Verify the ZIP checksum. Use the ZIP to run the lab: it preserves the complete folder structure, pinned dependencies and executable files. Keep the extracted source separate from your private execution directory.
AI-generated conceptual illustration. Genuine Azure Portal, CLI and GitHub evidence is separately labeled below.
Audience: Intermediate Windows/PowerShell learners. Duration: 3–4 hours.
Individual Markdown, PowerShell, KQL and checksum downloads may use a .txt viewing extension. Use the complete ZIP to run: it retains original names and the full folder structure. Source controller (text)
A green pipeline is useful. Knowing which identity ran it, which plan it applied and which state it changed is better. In this workshop you build a small private delivery path, establish a healthy baseline and then break three different parts of that path: federation, state locking and infrastructure consistency.
You diagnose each fault using a healthy control and evidence from Azure, Terraform and GitHub Actions. You repair only the owned fault, repeat the repair safely and remove the complete disposable environment.
Observed workshop status: PASSED. Azure/GitHub rehearsal passed on 10 November 2026: all 23 required checks, nine independent evidence observations and four real approval refusals. Final workload and Foundation plans had zero drift. Both owned groups were verified absent twice; all 27 unrelated baseline resources were preserved. Local validation passed 116 Pester, 40 Python and six Terraform tests, with zero analyzer errors and 19 disclosed warnings. Archive and publication verification are recorded separately. Expected results below describe acceptance conditions. A result becomes observed evidence only when it has a genuine capture and a matching sanitized report. The cover is conceptual artwork; it is separate from the twenty environment screenshots.
Workshop blocks
Block 1 — Scenario and architecture
Objective. Follow one change from a Windows operator to a private runner, an Azure identity and a private state blob. Distinguish the operator's bootstrap permissions from the pipeline's deliberately smaller permissions.
The Windows controller creates two disposable resource groups. The Foundation group contains a private Ubuntu runner, outbound NAT, a private state account, monitoring and two user-assigned managed identities. The Workload group contains a small VNet, a subnet and two NSGs. The exercise NSG has a synthetic marker tag; the canary NSG gives you an independent object that must remain unchanged.
Windows operator ── authenticated dispatch ──> private GitHub repository
│ workflow_dispatch, main
▼
private Ubuntu runner
│ │
GitHub OIDC token │ private DNS / HTTPS
▼ ▼
deployment UAMI Blob state backend
│ tfstate/workload.tfstate
▼
Workload RG
exercise NSG + canary
Independent control: separate UAMI, Reader on the Workload RG only
Operator Foundation/Federation state: private local directory, not the blob above
The runner has no managed identity, VM public IP or inbound administration route. Bootstrap uses Azure managed Run Command. NAT provides outbound connectivity; it does not make the VM an inbound endpoint. The state account disables public network access and Shared Key authorization. Its Blob private endpoint and private DNS link give the runner an internal route to the backend.
Boundary | Workshop default | Evidence to inspect |
Foundation network | 10.176.0.0/22; runner 10.176.0.0/24; private endpoint 10.176.1.0/24 | Private runner NIC, DNS resolution and state networking |
Workload network | 10.177.0.0/24; subnet 10.177.0.0/26 | Reviewed plan and final Azure inventory |
Deployment identity | Contributor on the new Workload RG; Storage Blob Data Contributor on the exact new state container | Two explicit role scopes, no subscription role |
Control identity | Reader on the new Workload RG | Fresh read-only login succeeds during the deployment-identity fault |
Workflow scope | Private repository, main, manual dispatch only | Actual claims, workflow ref and run attempt |
State ownership | Operator Foundation state and runner Workload state remain separate | Exact state paths, backend key and ownership ledger |
Expected evidence. A baseline validates a fresh control login, private state access, a no-change workload plan and the correct run marker. It does not infer health from an old successful workflow.
Recovery. If an address range overlaps an existing VNet, stop before deployment and adjust the reviewed design consistently. Never attach this workshop to a production state account, runner or resource group.
Block 2 — Preflight and operating limits
Objective. Establish the exact Azure/GitHub context, usable screenshot access and a conservative cost forecast before creating paid resources.
Use Windows with PowerShell 7.6.9, Azure CLI 2.88.0 and Terraform 1.15.8. AzureRM is pinned to 5.8.0 with its dependency lockfile. Git, GitHub CLI and OpenSSH client tools are also required. The Ubuntu image and GitHub runner release are resolved and recorded during preflight, including their immutable version or checksum. Do not replace a pinned dependency with latest while reproducing a report.
The bootstrap operator needs permission to create the two groups, their resources, scoped role assignments and federated credentials. GitHub authentication needs permission to create and manage the disposable private repository and its workflows/runners. OIDC removes an Azure client secret from pipeline login; bootstrap GitHub authentication and short-lived runner registration tokens still require private handling.
# Run from the extracted workshop folder in PowerShell 7.
az login
az account set --subscription '<your-selected-subscription-id>'
gh auth status
$subscriptionId = (az account show --query id -o tsv).Trim()
# Keep state, keys, raw logs and captures outside this source tree and Git.
$private = Join-Path $env:LOCALAPPDATA (
'AzureWorkshopz\TerraformDelivery\' + [guid]::NewGuid().ToString('N'))
$state = Join-Path $private 'run.json'
$lab = Join-Path (Get-Location) 'Invoke-TerraformDeliveryWorkshop.ps1'
& $lab -Stage Preflight -StatePath $state `
-SubscriptionId $subscriptionId -CaptureAccessVerified
Set -CaptureAccessVerified only after opening the authenticated Azure Portal and GitHub repository/workflow surfaces and proving that a readable CLI capture can be saved. A later display of retained command output is acceptable when its label preserves the original command, execution UTC and source checksum. It must not be described as fresh execution.
Preflight checks the selected tenant/subscription, existing resources, policies, required providers, operator permissions, regional B2s availability/quota, image availability, address overlap and current West Europe rates. Existing resource identifiers and raw results stay in the private execution directory. The initial review must resolve any restrictive policy before deploying; an assigned policy is not proof that its effects are harmless.
Cost boundary. The total operational limit is €20, including a €5 cleanup reserve. Deployment requires a conservative six-hour active estimate of at most €15 using current regional prices. Begin cleanup by 4½ hours and target verified absence before six hours. This is an operator/controller limit, not an Azure-enforced spending cap. Azure billing can arrive later, and a forecast is not an invoice. If any stage cannot finish within the remaining window, proceed to the exact-owned-resource cleanup path.
Portal checkpoint. Confirm that the selected subscription is correct and all three capture surfaces are usable. Read the printed forecast before approving the Foundation plan.
Capture 01 · CLI. Retained real Preflight and reviewed Foundation plan: 27 creates, no updates or deletes, with current EUR5.07 six-hour estimate and EUR5 cleanup reserve. Display time differs from original execution UTC.
Expected evidence. A timestamped passing preflight and a reviewable plan limited to the new disposable scopes. No paid resources exist at the end of Preflight.
Recovery. Fix authentication, quota, policy or capture access before continuing. Preserve the same private ledger for this run; use a new private directory for a new run. If an operation is interrupted later, reconcile its recorded Azure operation or GitHub run before using -Resume on that operation. Never start a competing controller against the same state.
Block 3 — Deploy Foundation, observe claims and add federation
Objective. Build the private execution path, then create trust from claims actually issued to this repository.
& $lab -Stage Plan -Phase Foundation -StatePath $state
# Review the complete printed Foundation plan and its saved checksum.
& $lab -Stage Deploy -Phase Foundation -StatePath $state `
-Execute -ReviewedPlan -AcknowledgeCost
& $lab -Stage Bootstrap -StatePath $state -Execute
Bootstrap installs verified tool/runner downloads and creates the private execution repository. Every dispatched job receives a fresh ephemeral runner registration on the same disposable VM. Registration disappears after one job; the VM is not reimaged between jobs. This lab runs only its reviewed main workflows, with no untrusted pull-request job. Ephemeral registration alone does not provide full host isolation. See GitHub's self-hosted runner security guidance.
Capture 02 · Portal. The native VM network view shows private IP10.176.0.4, no NIC public IP and a priority4096 deny-all inbound rule. NAT supplies separate outbound access.
Capture 03 · Portal. Native storage properties show Shared Key access disabled, TLS1.2, public network access disabled and one private endpoint connection.
Capture 04 · Portal. Native workload-group IAM filtered to the two workshop identities: deployment Contributor and independent control Reader, both scoped to this resource.
Capture 05 · GitHub. The new execution repository is private, on main, and contains the workflow, infrastructure and guarded delivery source. Existing repositories were preserved.
The claims workflow requests an OIDC token and retains only an allowlisted claim report, never the token. Check its issuer, audience, repository identity and main ref. The exact sub is an input to the next phase. GitHub subject formats can differ for newer repositories, so copying an old repo:owner/name:ref:... example is not a valid observation. id-token: write permits token issuance; Azure role assignments still determine authorization. See GitHub OIDC claims and Azure federation setup.
Capture 06 · GitHub. Actual claims-only job succeeded: issuer, audience and main-branch subject are shown; immutable private IDs are redacted. The JWT was never printed.
& $lab -Stage Plan -Phase Federation -StatePath $state
# Review the two exact federated credentials, including observed issuer/sub/aud.
& $lab -Stage Deploy -Phase Federation -StatePath $state `
-Execute -ReviewedPlan
The Federation phase adds exact trust to the deployment and control identities. It does not add a wildcard subject or GitHub Environment. Allow a bounded interval for federation and role propagation, then prove fresh logins. A stale successful token does not validate a newly changed trust relationship.
Capture 07 · CLI. Retained actual Federation plan, produced only after safe claims discovery. Private identifiers and exact subject bytes remain in the protected ledger.
Capture 08 · Portal. The native credential details were reopened after repair: issuer, main-branch subject and Azure token-exchange audience match the originally observed tuple. Immutable private IDs are redacted. No Portal edit was submitted.
Portal checkpoint. Inspect the deployment identity's resource-group and container scopes, the control identity's Reader scope and the exact federated credential. Inspect state networking and the runner's lack of public/inbound access.
Expected evidence. Claims match the new private repository and its main workflow; fresh control and deployment logins work; the runner resolves and reaches the private blob endpoint.
Recovery. If claims or scopes differ, stop and correct the new owned configuration through a fresh reviewed plan. Do not broaden the subject or assign subscription Contributor to make a test pass. Reconcile any failed initial apply against actual Terraform state before retrying.
Block 4 — Review a saved plan and apply that exact binary
Objective. Establish a healthy delivery baseline and prove that approval identifies one plan artifact, one run attempt and one hash.
& $lab -Stage Pipeline -Action Plan -StatePath $state -Execute
Open the exact plan.yml run. Review its complete human-readable workload diff and safe metadata. The raw binary and full JSON remain in the private artifact/private execution directory: Terraform plan files and terraform show -json can contain sensitive values even when terminal output redacts them. Treat repository artifact access accordingly. See saved Terraform plans and JSON output sensitivity.
Capture 09 · GitHub. Native successful Plan run: binary SHA-256, semantic hash, private artifact name and five creates with no updates or deletes. The separate metadata enforces the 15-minute review lifetime.
Capture 10 · CLI. Four genuine separate workflow refusals. Each original result records an intended error, Terraform not started and zero Terraform commands. Approval expires after 15 minutes; artifact retention is one day. This later display preserves original execution times and checksums and performs no execution.
Copy the exact run ID, run attempt, artifact ID and reviewed SHA256 from this run. The values below are learner inputs, not example evidence.
$review = @{
PlanRunId = '<exact-plan-run-id>'
PlanRunAttempt = 1 # Replace if the reviewed attempt is different.
ArtifactId = '<exact-private-artifact-id>'
PlanSha256 = '<reviewed-64-character-SHA256>'
}
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan @review
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
# Preserve this exact ORIGINAL successful approval for the refusal tests.
$baselineReview = @{} + $review
The approval lifetime is 15 minutes. The private artifact retention is one day. These are different controls: an artifact can still exist after it is too old to apply. Wrong hashes, wrong run attempts, mismatched provenance, expired plans and consumed approvals must be rejected. Review the article first; generate a new plan when ready to review and apply promptly.
Apply checks the source/ref/commit, artifact metadata, backend and state bindings, then makes a fresh semantic comparison before applying the original reviewed binary. The fresh comparison is a consistency check, not a replacement unreviewed apply plan. These checks do not atomically freeze Azure against other writers. Backend locking protects state operations; an external ARM change can still race with plan/apply. Keep the lab scopes exclusive and validate the resulting state.
There is no protected GitHub Environment approval in this workshop. -ReviewedPlan and the exact run/hash inputs record the operator's explicit approval; they do not impersonate an independent GitHub reviewer.
Capture 11 · GitHub. The genuine successful Apply run confirms applied_original_plan=true for the reviewed SHA-256, with independent canary health and five creates.
Capture 12 · Portal. The live workload group contains the separate canary NSG, exercise NSG and workload VNet. The preceding independent control job verified the canary configuration hash and zero workload drift.
GitHub checkpoint. The successful Apply run names the reviewed plan identity. The subsequent Validate run performs fresh identity, state and no-drift checks. Portal workload objects carry this run's marker; the canary is present.
Prove three distinct approval refusals
Run these immediately after the successful baseline Apply, while the original plan is younger than 15 minutes. Keep the execution repository's main commit unchanged throughout all four guard tests. Each -ExpectedFailure dispatch must produce the specific recorded refusal before Terraform starts. A missing runner or generic failed job does not count.
# 1. Same plan, wrong independently supplied review hash.
$wrongHash = @{} + $baselineReview
$firstHex = if ($baselineReview.PlanSha256[0] -eq '0') { '1' } else { '0' }
$wrongHash.PlanSha256 = $firstHex + $baselineReview.PlanSha256.Substring(1)
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @wrongHash
# Required refusal: REVIEWED_HASH_MISMATCH; terraform_started=false.
# 2. Exact successful plan approval has already been consumed.
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @baselineReview
# Required refusal: PLAN_ALREADY_CONSUMED; terraform_started=false.
# 3. Use the actual evidence artifact from the SAME plan run, not its plan artifact.
$repo = (Get-Content -LiteralPath $state -Raw | ConvertFrom-Json).repository
$artifactList = gh api "repos/$repo/actions/runs/$($baselineReview.PlanRunId)/artifacts" |
ConvertFrom-Json
$evidenceName = 'evidence-{0}-{1}' -f $baselineReview.PlanRunId, $baselineReview.PlanRunAttempt
$evidenceArtifact = @($artifactList.artifacts | Where-Object name -CEQ $evidenceName)
if ($evidenceArtifact.Count -ne 1) { throw 'Exact evidence artifact not found; reconcile this run.' }
$wrongArtifact = @{} + $baselineReview
$wrongArtifact.ArtifactId = [string]$evidenceArtifact[0].id
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @wrongArtifact
# Required refusal: ARTIFACT_PROVENANCE_MISMATCH; terraform_started=false.
The fourth refusal uses this original plan after genuine expiry in Block 8. Do not edit its timestamps or consumption receipt. Preserve the actual failed-run results and fresh healthy validation; the final acceptance audit binds each refusal to its original source.
Expected evidence. Original-binary apply succeeds, fresh control/deployment access is healthy and a fresh workload plan has no changes. A successful historical run or a merely downloadable artifact does not satisfy this baseline.
Recovery. If approval expires or anything changes, discard that approval and create/review a new plan. Do not edit metadata, extend a timestamp or regenerate a binary under an old hash. Diagnose an access error before attempting broader permissions.
Block 5 — Troubleshoot an exact OIDC trust failure
Objective. Distinguish a federation failure from runner networking, state access and Azure authorization failures.
Start with a fresh healthy validation. The fault changes only the deployment identity's exact federated subject to a known wrong value. The independent control identity retains its exact trust and Reader scope.
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
& $lab -Stage Fault -Scenario Oidc -StatePath $state -Execute
Inspect the actual drill-oidc.yml job: deployment token exchange must fail for the intended subject mismatch while a fresh control login/read succeeds. This is an expected fault result, not a generic successful workflow claim. A timeout, missing runner, DNS error or expired bootstrap token is not the intended failure.
Propagation checkpoint. A successful subject-update PUT does not prove that token exchange sees the change. Microsoft documents cached older FIC data and recommends retries. During rehearsal, the first three fresh attempts did not establish the intended failure; the first, 48 seconds after the PUT, returned EXPECTED_OIDC_FAILURE_DID_NOT_OCCUR with control healthy and zero Terraform commands. The fourth fresh attempt produced the intended AADSTS700213 rejection at 19:09:26 UTC, after the original 18:47:55 UTC injection. Fresh repairs then passed at 19:11:43 and 19:14:03 UTC. Cache delay remains a plausible inference rather than a proved cause. Reconcile an attempt before bounded fresh retries of the unchanged configuration; use Fault -Resume only for its recorded pending operation. Elapsed time alone is not proof. Microsoft FIC propagation guidance.
Capture 13 · GitHub. Fresh OIDC drill observed AADSTS700213 for the faulted deployment subject while the independent control stayed healthy. No Terraform command or Azure resource write occurred.
& $lab -Stage Repair -Scenario Oidc -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
# Repeat repair to prove that restoring the intended subject is safe.
& $lab -Stage Repair -Scenario Oidc -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
Capture 14 · CLI. Retained genuine fault and two distinct fresh repair jobs. Original execution times and original result hashes are shown; this later display executes no commands. Earlier propagation attempts remain recorded as failures in the private history.
Checkpoint. Compare the failing exchange with the allowlisted observed claims and the actual deployment FIC. Preserve the error category and UTC, not the JWT. A failed token exchange occurs before a backend request, so a missing storage log row is not proof of missing federation evidence.
Expected evidence. Fresh control succeeds during the fault; deployment exchange fails for the subject mismatch; both recover after exact restoration; repeated repair and validation pass.
Recovery. Restore the recorded owned subject. Never use a wildcard or alter the control identity. If there is an interrupted operation, reconcile that exact run before repeating repair.
Block 6 — Diagnose an owned state-lock conflict
Objective. Observe contention without corrupting state or interfering with an unknown owner's lease.
The AzureRM backend uses Blob-native locking. This exercise acquires a 60-second finite lease on the exact owned state blob, runs Terraform with a short lock timeout, verifies the lock conflict and releases only the recorded lease ID. A fresh control read remains healthy. See AzureRM backend locking and Terraform lock recovery.
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
& $lab -Stage Exercise -Scenario StateLock -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
One bounded drill includes the post-release HEAD availability check used as repeat-safety evidence. It does not need a second lease acquisition for acceptance. Optional: repeat the full Exercise -Scenario StateLock only for additional rehearsal, after a fresh healthy validation and within the cleanup window; that separate drill acquires its own new finite lease.
Capture 15 · CLI. Genuine fresh live drill after the diagnostic-retention fix. Original native argv and stderr bytes were independently retrieved and verified against the workflow diagnostic hashes. Paths and the run prefix are redacted; the stderr is an excerpt. The earlier drill lacked retained native bytes and remains separate history. State serial 3 to 3 and original hashes are shown. This later display executes no commands.
Capture 16 · Portal. Native StorageBlobLogs for the final validated lock rehearsal show the owned acquire (201), properties reads (200), release (200) and successful recovery operations. Exact UTC, OAuth authentication and service correlation IDs are preserved.
Checkpoint. Inspect the drill-lease.yml report and corroborating StorageBlobLogs. The report must show the intended Terraform lock conflict, owned-lease release and successful post-release operation. Compare the state content hash and serial before/after: the drill must not change state. Logs are a corroborating data-plane record, not a substitute for the Terraform diagnostic or hash check.
Read the pinned backend behavior. Terraform 1.15.8 reads blob properties and checks the locked lease status before calling AcquireLease. It can therefore reject a locked state locally without sending a conflicting lease-acquisition request. Do not require HTTP 409 or a LeaseAlreadyPresent response as the only acceptable symptom. Terraform 1.15.8 Azure backend source.
Accepted live evidence. A fresh drill retained native Terraform exit 1 at 19:57:04 UTC, with a five-second lock timeout and these verified diagnostic excerpts:
Error: Error acquiring the state lock
Error message: state blob is already locked
blob metadata "terraformlockid" was empty
The metadata message accompanies the workshop's externally injected lease; it is not a separate fault to repair. The owned lease was released, the recovery plan returned 0, HEAD confirmed availability and state SHA256/serial remained unchanged at 3 → 3. Eleven matching Blob log rows corroborated acquisition (201), held-lease property reads (200) and release (200), with no required HTTP 409. Original journal/stderr bytes were retrieved privately and their hashes matched the workflow's sanitized diagnostic record.
Evidence-retention history. The earlier 19:17:24 UTC workflow reported serial 2 → 2 and recovery, but the pinned runner cleared RUNNER_TEMP before the outer collector retrieved its journals. Those bytes are unrecoverable and were not used as the accepted native diagnostic. The correction moved only private journals and command streams to a protected job directory under RUNNER_WORKSPACE; authentication, Terraform data and plans remained in RUNNER_TEMP. The fresh Validate and StateLock rehearsal then supplied the verified replacement proof above. The four real approval refusals completed before that source change and retain their original per-run artifact/commit provenance; an old approval cannot authorize a new commit.
Expected evidence. The short-timeout operation fails because the owned lease is held; the next operation succeeds after its release; state hash/serial remain unchanged. The post-release HEAD reports availability without releasing an unknown lease. A second full exercise is optional.
Recovery. Let the finite lease expire if the controller loses its response, then reconcile the recorded operation. Do not use -lock=false, break an unidentified lease or force-unlock a lock you do not own. A resource-group cleanup is safer than improvising against another backend.
Block 7 — Detect and repair one out-of-band tag drift
Objective. Prove that drift belongs to one owned object and repair it through a newly reviewed saved plan.
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
& $lab -Stage Fault -Scenario Drift -StatePath $state -Execute
The Fault stage changes only the exercise NSG's synthetic marker tag from baseline-<run_id> to fault-<run_id> through Azure and dispatches the corrective Plan. The canary NSG must remain unchanged. That plan must refresh the remote object and show the intended tag correction without a replacement, deletion or unrelated network change. Its NSG security rule is unchanged; the tag is the exercise marker. A no-change plan against stale state is not adequate diagnosis.
Capture 17 · CLI. Retained genuine corrective plan reviewed by the operator: only the exercise NSG marker tag returns from its injected value to its recorded baseline. Security rules and other tags are identical; the canary is unchanged. This plan view is separate from subsequent repair execution. Original times and hashes are shown; later display only.
# Review the NEW repair plan, then replace every review value.
$repairReview = @{
PlanRunId = '<repair-plan-run-id>'
PlanRunAttempt = 1
ArtifactId = '<repair-plan-artifact-id>'
PlanSha256 = '<reviewed-repair-plan-SHA256>'
}
& $lab -Stage Repair -Scenario Drift -StatePath $state `
-Execute -ReviewedPlan @repairReview
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
# Repeated Repair must confirm the restored marker without introducing drift.
& $lab -Stage Repair -Scenario Drift -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
Capture 18 · GitHub. The genuine drift-repair Apply used the reviewed original binary: one update, four no-ops, no creates or deletes, with the independent canary healthy. Repeated repair is checked by a separate fresh validation.
Checkpoint. Compare the reviewed repair diff, successful saved-binary Apply and fresh no-change validation. The canary's recorded properties must match its baseline throughout.
Observed recovery. The one-NSG marker correction was applied through the reviewed binary. First and repeated repair checks passed, and the independent evidence report records repeated recovery at 19:33:31 UTC with the canary preserved. The final workload and Foundation validations also reported zero drift.
Expected evidence. Only the exercise tag drifts, only that tag is corrected and a new validation reports no drift. Repeated repair is safe. Do not use a direct Azure tag write as the learner's repair: the point is to review and apply the Terraform correction.
Recovery. If the diff includes another change, stop and explain it before approving a plan. A changed source commit, consumed approval or expired plan requires a new plan/review cycle.
Block 8 — Validate, correlate evidence and clean up twice
Objective. Finish with fresh checks, remove both layers in dependency order and prove that unrelated resources survived.
First prove the fourth guard using the original baseline plan after genuine expiry. Blocks 5–7 normally provide enough elapsed time. If necessary, the loop below waits in bounded 30-second steps, stops at the exercise deadline and never changes metadata or the clock.
$metadataPath = Join-Path $private (
'evidence/run-{0}-{1}/plans-{0}-{1}/plan-metadata.json' -f
$baselineReview.PlanRunId, $baselineReview.PlanRunAttempt)
$baselineMetadata = Get-Content -LiteralPath $metadataPath -Raw | ConvertFrom-Json
$expiredAfter = [DateTimeOffset]::Parse($baselineMetadata.created_utc).AddMinutes(15).AddSeconds(5)
$runLedger = Get-Content -LiteralPath $state -Raw | ConvertFrom-Json
$exerciseStop = [DateTimeOffset]::Parse($runLedger.exercise_stop_utc)
if ($expiredAfter -gt [DateTimeOffset]::UtcNow.AddMinutes(16) -or $expiredAfter -ge $exerciseStop) {
throw 'No safe expiry-test window; reconcile time metadata or begin cleanup.'
}
while ([DateTimeOffset]::UtcNow -lt $expiredAfter) {
if ([DateTimeOffset]::UtcNow -ge $exerciseStop) { throw 'Begin cleanup now.' }
Start-Sleep -Seconds 30
}
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @baselineReview
# Required refusal: PLAN_EXPIRED_OR_FUTURE from genuine AGE, not a future timestamp.
# terraform_started=false; original plan metadata remains unchanged.
& $lab -Stage Validate -StatePath $state -Execute -CollectLogs
The final report must reconcile identity checks, every intended fault, first and repeated recoveries, owned-lease invariants, canary preservation and a fresh no-change workload plan. Corroborate lease operations/statuses by the exact owned blob, OAuth principal and execution window; retain the actual Storage correlation IDs. A Storage correlation ID by itself does not prove a shared runner client-request ID. The optional request-ID query is stronger only when an original runner report retains that same identifier. Corroborate the successful drift-repair NSG write by its exact resource, principal and bounded repair window. Do not label every 403 as the OIDC fault or assume that a denied token exchange reaches Storage. Queries in the KQL folder project only documented fields; the final report must state which matching rows were actually observed.
Capture 19 · CLI. Retained genuine final live validation and source-bound evidence evaluation. All nine independent scenario, guard, final-validation and log observations passed. Cleanup was still pending at this checkpoint; this view does not claim workshop completion. Original workflow, collection and evaluation times are separate from this later display.
Observed final validation. All 23 required live checks were satisfied across 39 recorded check events, and all nine independent evidence observations passed. The four approval refusals were genuine failures before Terraform started. Final workload validation was observed at 20:00:50 UTC and the Foundation no-drift check passed at 20:01:59 UTC. Eight sanitized retained CLI reports are included under docs/evidence/retained-cli/; they are original-execution records displayed later, not commands rerun during capture.
Create and review a destroy plan while the private runner, backend and federation still exist. Copy its own four approval values. A normal create/update plan cannot authorize destroy.
& $lab -Stage Pipeline -Action Plan -StatePath $state -Execute -DestroyPlan
$destroyReview = @{
PlanRunId = '<destroy-plan-run-id>'
PlanRunAttempt = 1
ArtifactId = '<destroy-plan-artifact-id>'
PlanSha256 = '<reviewed-destroy-plan-SHA256>'
}
& $lab -Stage Pipeline -Action Destroy -StatePath $state `
-Execute -ReviewedPlan @destroyReview
# Preview exact owned Foundation cleanup, then approve that preview.
& $lab -Stage Cleanup -StatePath $state
& $lab -Stage Cleanup -StatePath $state -Execute -ReviewedPlan
# Repeated cleanup must verify absence and be a safe no-op.
& $lab -Stage Cleanup -StatePath $state -Execute -ReviewedPlan
& $lab -Stage Export -StatePath $state
The controller retains its exact ownership ledger. Cleanup removes the owned workload before deleting the private backend/runner Foundation. It disables workflows and removes ephemeral runner registrations and owned runtime artifacts. The repository and its workflow history remain private; Azure trust and paid Azure resources are removed. Public downloadable source files are separate from that repository. Never delete an existing repository or resource group merely because its name resembles the lab prefix.
Capture 20 · CLI. Genuine successful reviewed destroy and two actual cleanup checks. Both owned resource groups are absent, all 27 unrelated baseline resource IDs are preserved, and GitHub runner/artifact runtime counts are zero. The private repository and workflow history remain retained. The ledger checksum covers the original checkpoint times shown above; this later display executes no commands.
Observed cleanup. The reviewed workload Destroy succeeded. Exact owned-group absence passed at 20:26:29 UTC and repeated cleanup at 20:33:15 UTC; all 27 unrelated baseline resources were preserved. Both checks finished less than 2½hours after paid deployment began at 18:09:29 UTC, inside the original six-hour operating window. The retained private repository has seven disabled workflows and zero runner registrations or runtime artifacts. Its history remains private; Azure trust and paid workshop resources were removed.
Portal/GitHub checkpoint. The exact two resource groups are absent; owned artifacts and runner registrations are removed; repository workflows are disabled. The repository and its workflow history remain private. Independent repeated Azure absence checks agree. Compare the recorded unrelated-resource baseline by identifiers privately and publish only sanitized counts/results.
Expected evidence. Final acceptance passes, both owned scopes are absent twice, no unrelated baseline resource is missing and cleanup finishes inside the operational window. The export excludes raw plans/state, JWTs, registration tokens, keys, raw logs and unredacted images.
Recovery. If pipeline destroy fails or the runner/backend is unavailable, use the controller's reviewed exact-owned-resource fallback to prevent cost overrun. Reconcile actual remaining resources before another attempt. A failed cleanup is outstanding work; a command returning success without independent absence evidence is not completion.
Block 9 — Learning pack
Try the independent challenge before opening the hints or worked solution. The eight knowledge questions include explanations, not just answer keys. The full learning pack is embedded below and also included as editable files in the download.
Use the troubleshooting table to choose the next discriminating check. Diagnose the layer that failed before changing a permission, retrying an apply or releasing a lock.
Symptom | Discriminating evidence | Safe next step |
Deployment login fails; fresh control login succeeds | Actual issuer/sub/aud versus deployment FIC; token-exchange diagnostic | Restore only the exact owned subject |
Deployment exchange still succeeds soon after subject corruption | Confirm the recorded PUT, actual unchanged fault subject and fresh control; inspect propagation timing | Reconcile the attempt and retry within bounds; the intended fault remains unproved |
Both identities fail before an Azure request | Runner registration, workflow ref, outbound HTTPS and actual GitHub run | Reconcile runner/workflow prerequisites |
Login succeeds; backend access fails | Private DNS, Blob reachability, container role and Shared Key/public-network settings | Fix only the owned route or scope |
Plan fails with a lock conflict | Finite owned lease, exact state key, Terraform lock report | Release only your recorded lease or await its expiry |
Apply refuses an available artifact | Run/attempt/artifact/hash, age, consumed marker and source/state bindings | Create and review a new plan |
Plan corrects an unexpected object | Fresh remote diff and canary baseline | Stop approval and investigate the extra change |
Final logs have no matching request | Workflow UTC/observed request ID and log-ingestion delay | Poll within bounds; report missing correlation honestly |
Cleanup reports success but a group exists | Fresh exact-name absence query and owned inventory | Reconcile and repeat precise cleanup |
Source notes and further reading
The public source map links the official references for OIDC, saved plans, private backends, runners and StorageBlobLogs. Capture guidance separates retained real command output from fresh execution. Validation notes separate local tests from observed Azure/GitHub results. The source is editable in article.md; executable entry point: Invoke-TerraformDeliveryWorkshop.ps1.
Independent challenge
Independent challenge — prove an approval is bound to one plan
You inherit a private Terraform delivery pipeline. The workload is healthy, and a teammate sends you a plan-run link saying “approved.” Before you apply it, demonstrate what this approval actually identifies.
Use only this workshop's owned scopes. Begin with fresh healthy validation. Produce a new normal plan, inspect its exact run attempt/artifact/hash and record those safe metadata values privately. Then deliberately submit a wrong reviewed SHA256 by changing one hexadecimal character. The Apply path must refuse it before running Terraform apply. Keep the original valid values intact for comparison.
After proving refusal, produce and review a fresh plan rather than extending an old approval's expiry. Apply the reviewed original binary if the plan has an intended change. If it is a no-change plan, report that honestly and validate the healthy result. Never introduce a resource change solely to make the challenge more dramatic.
Your handover should answer:
Which exact repository, workflow ref/commit, run attempt and artifact does the approval name?
Which hash did you approve, and which wrong hash was rejected?
Was the rejection an approval check, an Azure login failure or a Terraform apply failure?
What happened to the workload, its canary and its state hash/serial during refusal?
Why would a still-downloadable one-day artifact be too old to apply after the 15-minute approval lifetime?
What race remains possible even after a fresh semantic comparison?
Success criteria. Capture the actual refusal reason without exposing raw plan/state/token contents. Prove the canary and workload are healthy after the test, reconcile the exact failed run and finish within the cleanup window. A syntax error, missing runner or expired login is not a successful wrong-hash test.
If the wrong hash is accepted, stop. Preserve private evidence and use precise owned cleanup. Do not continue with an untrustworthy approval path.
Progressive hints
Progressive hints
Hint 1 — identify the boundary
OIDC answers who may exchange a token. RBAC answers what that identity may do. Private networking answers how it reaches a data plane. A saved-plan approval answers which proposed change the operator reviewed. Diagnose these separately.
Hint 2 — choose a healthy control
The independent Reader UAMI needs a new token during the deployment-subject fault. The canary NSG needs a current property comparison during marker drift. An old successful run is neither control.
Hint 3 — inspect the approval tuple
Start with run ID, run attempt, artifact ID and reviewed SHA256. Then compare source/ref/commit, backend and state metadata. A plan cannot become a different plan while keeping an old approval.
Hint 4 — use the failure location
A subject mismatch fails at token exchange. A state lease conflict occurs when Terraform tries to lock its backend. A drift plan can authenticate and read state successfully while still proposing a correction. Look for the first intended failure; do not treat every red job as the same exercise.
Hint 5 — leases and state are different evidence
The finite owned lease can change while state content stays identical. Compare blob content hash and Terraform serial before/after the drill, then prove a new operation can acquire the backend lock. Do not infer unchanged state solely from an unchanged file size.
Hint 6 — make a safe repeat useful
Repeat repair after validating the first recovery. OIDC should restore the same exact subject. Drift's repeated repair should validate the restored tag rather than reapply a consumed plan. The lease drill already includes a post-release HEAD availability check. If you optionally repeat the full lock drill, it must acquire a new owned finite lease and release it again.
Hint 7 — do not outwait cleanup
The exercise stop time is earlier than the cleanup deadline. A failure to get attractive screenshots is not a reason to keep paid resources running. Retain actual output privately, clean up, then label any later display as retained evidence.
Worked solution
Worked solution
This is a diagnostic sequence, not a claim that the current authoring run passed. Use the genuine final reports for observed errors, timestamps and request correlations.
1. Establish the baseline
Complete Preflight, reviewed Foundation deploy, Bootstrap, observed-claim Federation plan/deploy and a reviewed saved-plan Apply. Run fresh pipeline Validate. Record the exercise marker, canary properties, state hash/serial and healthy independent control. Keep private identifiers in the private ledger.
2. Diagnose and restore OIDC
Run Fault -Scenario Oidc -Execute. The controller changes only the deployment FIC subject and dispatches the drill. Inspect a fresh control login/read and the actual deployment token-exchange failure. Compare the allowlisted actual issuer/subject/audience with the wrong subject. Avoid printing the bearer token.
Run Repair -Scenario Oidc -Execute. It restores the recorded exact subject and dispatches fresh success checks. Validate, repeat Repair, then validate again. Do not change the control FIC or broaden Azure roles.
3. Diagnose and release an owned state lease
Run Exercise -Scenario StateLock -Execute from a healthy baseline. The workflow acquires a new finite 60-second lease on the owned backend blob, proves Terraform's short lock timeout fails for that conflict, releases only its recorded lease ID and proves recovery. Verify state content hash and serial are unchanged. The same bounded drill's post-release HEAD confirms availability as repeat-safety evidence; a second full drill is optional, not required for acceptance.
A lease acquire/release event corroborates data-plane activity; it does not independently prove that Terraform diagnosed the intended error. Keep both forms of evidence. The pinned backend can reject a locked status after reading properties before it attempts acquisition, so HTTP 409 is not mandatory. Never break an unknown lease or suppress locking.
4. Diagnose and repair drift through Terraform
Run Fault -Scenario Drift -Execute. It changes only the owned exercise NSG marker and dispatches the corrective Plan. Review the actual plan for exactly that tag change and inspect the unchanged canary. Copy this plan's run ID, attempt, artifact ID and reviewed hash.
& $lab -Stage Repair -Scenario Drift -StatePath $state `
-Execute -ReviewedPlan @repairReview
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
& $lab -Stage Repair -Scenario Drift -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
The first Repair applies the original reviewed correction binary. The repeated Repair confirms the restored state instead of reapplying a consumed approval. Any extra resource change, replacement or changed source requires investigation and a fresh review.
5. Solve the independent wrong-hash challenge
Generate a new Plan. Preserve its valid approval tuple. Copy that tuple to a separate variable and change one hexadecimal character of the copied hash, keeping it a valid 64-character SHA256 string. Submit the copied tuple to Apply with explicit review flags. Inspect the exact rejected run and verify it stopped at the expected approval guard before Terraform apply. Do not confuse a parser error or unavailable runner with a guard refusal.
Re-run fresh validation and compare the current workload/canary/state. Produce and review a new plan for the next authorized action. Do not reuse a consumed approval or modify expiry metadata.
6. Finish and clean up
Run consolidated Validate -Execute -CollectLogs for a fresh healthy pipeline, Foundation no-drift and logs. Review a distinct destroy plan and execute its saved binary while the runner/backend/federation exist. Preview exact Foundation cleanup, execute it, repeat it and independently verify both groups' absence and preservation of the unrelated baseline. Disable repository workflows and remove owned runtime artifacts/runner registrations and Azure trust. Retain the repository and its workflow history privately.
If normal pipeline destruction is unavailable, use only the controller's reviewed exact-owned-resource fallback. Keep private artifacts until reconciliation needs are met, but never publish them. Final export contains allowlisted reports and redacted genuine captures, not raw Terraform plans/state, tokens or raw logs.
Eight knowledge questions and explanations
Eight knowledge questions and explanations
1. Does id-token: write grant the workflow Contributor in Azure?
Answer: no. It lets the GitHub job request an OIDC token. Azure still requires a matching federated credential and RBAC authorization. A token exchange can succeed while a resource operation is denied.
2. Why record the actual sub instead of copying a repository-name example?
Answer: the issued subject must match the trust exactly, and GitHub subject formats can differ for newer or reconfigured repositories. Actual issuer, audience, repository identity and ref make that trust reviewable. Never retain the JWT as public evidence.
3. Why does the runner VM have no managed identity?
Answer: the workshop demonstrates explicit GitHub OIDC identities. A usable VM identity could mask a federation error or expand the runner's ambient Azure access. Removing that alternative makes the intended identity path easier to prove.
4. Is an artifact retained for one day valid to apply for one day?
Answer: no. Storage retention and approval lifetime serve different purposes. This lab expires approval after 15 minutes and binds it to exact run/attempt/artifact/hash/source/state metadata. Availability alone does not authorize apply.
5. Why apply the original reviewed binary after a fresh semantic comparison?
Answer: the original binary is the object that was reviewed. The fresh comparison detects relevant changes before using it. Applying a newly generated unreviewed binary under an old hash would replace the approved proposal. Neither check atomically freezes Azure against other writers.
6. Does an ephemeral self-hosted registration mean a fresh VM for every job?
Answer: no. The registration accepts one job and is removed; the disposable VM persists. Disk/process compromise could persist between jobs. This lab runs reviewed main-only dispatches and is unsuitable for untrusted PR execution.
7. Why compare both state content hash and serial during the lease exercise?
Answer: the intended failure is lock contention, not a state update. Content hash detects byte changes and serial checks Terraform's state version progression. Post-release access then proves the owned lease no longer blocks a new operation.
8. Why destroy the workload before Foundation?
Answer: the private runner, exact federation and private state backend are needed to execute the reviewed workload destroy. Removing them first can strand the remote state and force a fallback cleanup. Final acceptance also needs repeated exact absence checks and proof that unrelated resources survived.
Only observed evidence supports environment claims. Original private runtime material is excluded from the public package. Editable source: article.md.
Comments