A completed backup job is not proof that a workload can be recovered.

Recovery is proven only when you can:

• select a valid recovery point;

• restore the required data;

• start the recovered workload;

• verify the restored files;

• confirm that checksums match the original data;

• measure how long recovery actually took.

In this workshop, you will build a complete Azure VM backup and recovery drill using PowerShell 7 and Azure CLI.

You will deploy a small Linux VM with an operating-system disk and a managed data disk. The data disk will contain several test files and a checksum manifest. After creating an Enhanced Azure Backup policy and triggering an on-demand backup, you will deliberately delete and corrupt the protected data.

You will then prove recovery in two different ways:

1. Restore individual files from a recovery point.

2. Restore the complete VM to an alternate location.

Azure Backup stores VM recovery points in a Recovery Services vault. Those recovery points can be used for complete-VM recovery, managed-disk recovery, or individual file recovery.

Cost warningThis workshop creates a virtual machine, managed disks, public IP addresses, a Recovery Services vault, backup recovery points, a staging storage account, and a restored VM.Remove the complete environment after finishing the recovery drill.

Workshop overview

Item	Configuration
Level	Intermediate
Estimated duration	90–180 minutes
Deployment interface	PowerShell 7 and Azure CLI
Azure region	West Europe
Source VM	Ubuntu Linux
Protected disks	OS disk and one data disk
Backup vault	Recovery Services vault
Backup policy	Enhanced
Backup schedule	Every four hours
Vault retention	30 days
Instant Restore retention	7 days
File validation	SHA-256 checksums
File recovery	Recovery-point mount script
Full recovery	Alternate-location VM restore
Recovery measurement	Observed backup and recovery durations

The recovery objective

The goal is not:

Backup job status: Completed

The goal is:

Recovery point selected
          ↓
Deleted data restored
          ↓
Checksums verified
          ↓
Application returned expected content
          ↓
Observed recovery time recorded

What you will learn

By completing this workshop, you will learn how to:

1. Deploy a small Azure Linux VM.

2. Attach and format a managed data disk.

3. Create test files on the data disk.

4. Generate a SHA-256 checksum manifest.

5. Create a Recovery Services vault.

6. Configure vault storage redundancy.

7. Create an Enhanced VM backup policy.

8. Enable backup protection for an Azure VM.

9. Trigger an on-demand backup.

10. Monitor backup jobs.

11. Identify the protected container and backup item.

12. List and select recovery points.

13. Delete and corrupt protected data.

14. Prove that the workload is broken.

15. Generate a Linux file-recovery script.

16. Mount a recovery point to the source VM.

17. Restore selected files.

18. Verify restored files with SHA-256 checksums.

19. Restore the complete VM to another VNet.

20. Validate the recovered VM and data disk.

21. Measure observed recovery times.

22. Document recovery evidence.

23. Troubleshoot common backup and restore failures.

24. Clean up the environment safely.

Architecture

                         Recovery Services Vault
                       rsv-backup-deepdive-weu
                                  │
                                  │ Enhanced policy
                                  │ On-demand backup
                                  ▼
                     ┌──────────────────────────┐
                     │      Recovery point      │
                     │                          │
                     │ OS disk snapshot         │
                     │ Data disk snapshot       │
                     │ VM configuration         │
                     └────────────┬─────────────┘
                                  │
                   ┌──────────────┴──────────────┐
                   │                             │
                   │                             │
          File-level recovery          Full VM recovery
                   │                             │
                   ▼                             ▼
       Mount recovery point             Alternate-location VM
       to the source VM                 in restore VNet
                   │                             │
                   │                             │
       Restore selected files           Validate complete VM
                   │                             │
                   ▼                             ▼
       Verify SHA-256 hashes             Verify SHA-256 hashes
       Validate web service              Validate web service

Source environment

Resource group: rg-backup-deepdive-weu

VNet: 10.40.0.0/16
└── snet-workload: 10.40.1.0/24
    └── vm-backup-deepdive
        ├── OS disk
        ├── Data disk
        │   └── /srv/backupdata
        │       ├── protected/
        │       └── manifest/
        └── Nginx on TCP 8080

Restore environment

Resource group: rg-backup-restore-weu

VNet: 10.50.0.0/16
└── snet-restore: 10.50.1.0/24
    └── vm-backup-recovered
        ├── Restored OS disk
        ├── Restored data disk
        └── Restored Nginx application

Address plan

Component	Address range
Source VNet	10.40.0.0/16
Source workload subnet	10.40.1.0/24
Source VM	10.40.1.4
Restore VNet	10.50.0.0/16
Restore subnet	10.50.1.0/24

Recovery drill matrix

Drill	Failure introduced	Recovery method	Proof
File recovery	Files deleted and configuration corrupted	Mount recovery point	SHA-256 verification
Application recovery	Web content removed	Restore protected files	HTTP response restored
Full VM recovery	Assume source VM is unusable	Alternate-location VM restore	VM, disk, checksums, and HTTP
Disk recovery	Optional disk-only restore	Restore managed disks	Restored disk inventory

Recovery metrics

The workshop records three observed durations:

Metric	Meaning
Backup duration	Time from backup trigger to completed job
File recovery time	Time from mounting the recovery point to checksum verification
Full VM recovery time	Time from restore trigger to successful application validation

These are observed lab measurements, not guaranteed service-level objectives.

1. Verify the prerequisites

You need:

• PowerShell 7;

• Azure CLI;

• OpenSSH client;

• an Azure subscription;

• permission to create compute, network, storage, and backup resources;

• quota for two small virtual machines.

Verify PowerShell:

pwsh --version

Verify Azure CLI:

az version

Verify SSH and SCP:

ssh -V
scp

Confirm that the required commands exist:

$RequiredCommands = @(
    "pwsh"
    "az"
    "ssh"
    "scp"
)

foreach ($Command in $RequiredCommands) {
    $ResolvedCommand = Get-Command `
        -Name $Command `
        -ErrorAction SilentlyContinue

    if (-not $ResolvedCommand) {
        throw "Required command '$Command' was not found."
    }

    [pscustomobject]@{
        Command = $Command
        Path    = $ResolvedCommand.Source
    }
}

Enhanced VM backup policies support multiple backups per day. Hourly schedules can run at intervals of 4, 6, 8, 12, or 24 hours. Enhanced policies also support longer Instant Restore snapshot retention and newer VM and disk capabilities.

2. Sign in to Azure

Authenticate:

az login

List available subscriptions:

az account list `
    --query "[].{
        Name:name,
        SubscriptionId:id,
        TenantId:tenantId,
        IsDefault:isDefault
    }" `
    --output table

Select the target subscription:

$SubscriptionId = "<your-subscription-id>"

az account set `
    --subscription $SubscriptionId

Confirm the active context:

az account show `
    --query "{
        Subscription:name,
        SubscriptionId:id,
        TenantId:tenantId,
        User:user.name
    }" `
    --output table

3. Register the required resource providers

$ResourceProviders = @(
    "Microsoft.Compute"
    "Microsoft.Network"
    "Microsoft.Storage"
    "Microsoft.RecoveryServices"
)

foreach ($Provider in $ResourceProviders) {
    Write-Host `
        "Registering $Provider..." `
        -ForegroundColor Cyan

    az provider register `
        --namespace $Provider `
        --wait
}

Display their states:

az provider list `
    --query "[?namespace=='Microsoft.Compute' || namespace=='Microsoft.Network' || namespace=='Microsoft.Storage' || namespace=='Microsoft.RecoveryServices'].{
        Provider:namespace,
        State:registrationState
    }" `
    --output table

4. Define the workshop variables

Replace $AdminSourceCidr with the public IP address from which you will connect.

Use /32 for one administrator address.

Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"

$SubscriptionId = (
    az account show `
        --query id `
        --output tsv
).Trim()

$TenantId = (
    az account show `
        --query tenantId `
        --output tsv
).Trim()

if ([string]::IsNullOrWhiteSpace($SubscriptionId)) {
    throw "No active Azure subscription was found."
}

$Location = "westeurope"

# Restrict SSH and HTTP access to your own public address.

$AdminSourceCidr = "<your-public-ip-address>/32"

if ($AdminSourceCidr -match "<") {
    throw "Replace `$AdminSourceCidr with your public IP address in CIDR format."
}

# Resource groups

$SourceResourceGroup  = "rg-backup-deepdive-weu"
$VaultResourceGroup   = "rg-backup-vault-weu"
$RestoreResourceGroup = "rg-backup-restore-weu"

# Local workshop directory

$LabRoot = Join-Path `
    $HOME `
    "azure-backup-recovery-drill"

# Source network

$SourceVnetName     = "vnet-backup-deepdive-weu"
$SourceVnetPrefix   = "10.40.0.0/16"
$SourceSubnetName   = "snet-workload"
$SourceSubnetPrefix = "10.40.1.0/24"
$SourceNsgName      = "nsg-backup-deepdive-weu"

# Source VM

$VmName          = "vm-backup-deepdive"
$VmNicName       = "nic-backup-deepdive"
$VmPublicIpName  = "pip-backup-deepdive"
$VmPrivateIp     = "10.40.1.4"
$VmAdminUsername = "azureuser"
$VmSize          = "Standard_B2s"
$OsDiskName      = "osdisk-backup-deepdive"
$DataDiskName    = "datadisk-backup-proof"
$DataDiskSizeGb  = 16

# Recovery Services vault

$VaultName  = "rsv-backup-deepdive-weu"
$PolicyName = "Enhanced-4Hourly-30DayVault"

# Restore network

$RestoreVnetName     = "vnet-backup-restore-weu"
$RestoreVnetPrefix   = "10.50.0.0/16"
$RestoreSubnetName   = "snet-restore"
$RestoreSubnetPrefix = "10.50.1.0/24"
$RestoreNsgName      = "nsg-backup-restore-weu"

# Recovered VM

$RecoveredVmName       = "vm-backup-recovered"
$RecoveredPublicIpName = "pip-backup-recovered"

# Globally unique staging storage account

$UniqueSuffix = Get-Random `
    -Minimum 100000 `
    -Maximum 999999

$RestoreStorageAccount = (
    "stbkprestore$UniqueSuffix"
).ToLowerInvariant()

# Recovery drill measurements

$BackupStartUtc      = $null
$BackupEndUtc        = $null
$FileRestoreStartUtc = $null
$FileRestoreEndUtc   = $null
$FullRestoreStartUtc = $null
$FullRestoreEndUtc   = $null

az account set `
    --subscription $SubscriptionId

Review the configuration:

[pscustomobject]@{
    SubscriptionId       = $SubscriptionId
    Location             = $Location
    SourceResourceGroup  = $SourceResourceGroup
    VaultResourceGroup   = $VaultResourceGroup
    RestoreResourceGroup = $RestoreResourceGroup
    SourceVm             = $VmName
    Vault                = $VaultName
    Policy               = $PolicyName
    AdminSourceCidr      = $AdminSourceCidr
    RestoreStorage       = $RestoreStorageAccount
}

5. Create the local workshop directory

Remove-Item `
    -Path $LabRoot `
    -Recurse `
    -Force `
    -ErrorAction SilentlyContinue

New-Item `
    -ItemType Directory `
    -Path $LabRoot `
    -Force |
    Out-Null

Set-Location $LabRoot

6. Check for existing Azure resources

$WorkshopResourceGroups = @(
    $SourceResourceGroup
    $VaultResourceGroup
    $RestoreResourceGroup
)

$ExistingGroups = @(
    foreach ($ResourceGroupName in $WorkshopResourceGroups) {
        $Exists = az group exists `
            --subscription $SubscriptionId `
            --name $ResourceGroupName `
            --output tsv

        if ($Exists -eq "true") {
            $ResourceGroupName
        }
    }
)

if ($ExistingGroups.Count -gt 0) {
    $ExistingGroups |
        ForEach-Object {
            Write-Host $_ -ForegroundColor Red
        }

    throw "One or more workshop resource groups already exist."
}

7. Create the source and vault resource groups

Create the source resource group:

az group create `
    --subscription $SubscriptionId `
    --name $SourceResourceGroup `
    --location $Location `
    --tags `
        workshop=azure-backup-deep-dive `
        role=protected-workload `
        managedBy=azure-cli `
    --only-show-errors `
    --output none

Create the vault resource group:

az group create `
    --subscription $SubscriptionId `
    --name $VaultResourceGroup `
    --location $Location `
    --tags `
        workshop=azure-backup-deep-dive `
        role=backup-vault `
        managedBy=azure-cli `
    --only-show-errors `
    --output none

8. Create the source network

Create the VNet and workload subnet:

az network vnet create `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $SourceVnetName `
    --location $Location `
    --address-prefixes $SourceVnetPrefix `
    --subnet-name $SourceSubnetName `
    --subnet-prefixes $SourceSubnetPrefix `
    --tags `
        workshop=azure-backup-deep-dive `
        role=source-network `
    --only-show-errors `
    --output none

9. Create the source NSG

Create the NSG:

az network nsg create `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $SourceNsgName `
    --location $Location `
    --tags `
        workshop=azure-backup-deep-dive `
        role=source-security `
    --only-show-errors `
    --output none

Allow SSH only from the administrator address:

az network nsg rule create `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --nsg-name $SourceNsgName `
    --name Allow-Admin-SSH `
    --priority 100 `
    --direction Inbound `
    --access Allow `
    --protocol Tcp `
    --source-address-prefixes $AdminSourceCidr `
    --source-port-ranges "*" `
    --destination-address-prefixes "*" `
    --destination-port-ranges 22 `
    --description "Allow SSH from the workshop administrator." `
    --only-show-errors `
    --output none

Allow the recovery-proof web service:

az network nsg rule create `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --nsg-name $SourceNsgName `
    --name Allow-Admin-HTTP-8080 `
    --priority 110 `
    --direction Inbound `
    --access Allow `
    --protocol Tcp `
    --source-address-prefixes $AdminSourceCidr `
    --source-port-ranges "*" `
    --destination-address-prefixes "*" `
    --destination-port-ranges 8080 `
    --description "Allow recovery validation from the workshop administrator." `
    --only-show-errors `
    --output none

10. Create the source VM public IP

az network public-ip create `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmPublicIpName `
    --location $Location `
    --sku Standard `
    --tier Regional `
    --allocation-method Static `
    --version IPv4 `
    --tags `
        workshop=azure-backup-deep-dive `
        role=source-validation `
    --only-show-errors `
    --output none

11. Create the source VM NIC

az network nic create `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmNicName `
    --location $Location `
    --vnet-name $SourceVnetName `
    --subnet $SourceSubnetName `
    --private-ip-address $VmPrivateIp `
    --public-ip-address $VmPublicIpName `
    --network-security-group $SourceNsgName `
    --tags `
        workshop=azure-backup-deep-dive `
        role=source-workload `
    --only-show-errors `
    --output none

12. Deploy the source VM

az vm create `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --location $Location `
    --nics $VmNicName `
    --image Ubuntu2204 `
    --size $VmSize `
    --admin-username $VmAdminUsername `
    --authentication-type ssh `
    --generate-ssh-keys `
    --security-type Standard `
    --os-disk-name $OsDiskName `
    --storage-sku Standard_LRS `
    --tags `
        workshop=azure-backup-deep-dive `
        role=protected-workload `
    --only-show-errors `
    --output none

Display its state:

az vm get-instance-view `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --query "{
        Name:name,
        ProvisioningState:instanceView.statuses[?starts_with(code,'ProvisioningState/')].displayStatus | [0],
        PowerState:instanceView.statuses[?starts_with(code,'PowerState/')].displayStatus | [0]
    }" `
    --output table

13. Attach a managed data disk

az vm disk attach `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --vm-name $VmName `
    --name $DataDiskName `
    --new `
    --size-gb $DataDiskSizeGb `
    --sku Standard_LRS `
    --lun 0 `
    --only-show-errors `
    --output none

Display the VM disks:

az vm show `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --query "{
        OsDisk:storageProfile.osDisk.name,
        DataDisks:storageProfile.dataDisks[].{
            Name:name,
            Lun:lun,
            SizeGb:diskSizeGb
        }
    }" `
    --output jsonc

14. Configure the data disk and recovery-proof application

The script:

• formats the data disk;

• mounts it at /srv/backupdata;

• creates protected test files;

• creates a SHA-256 manifest;

• configures Nginx on port 8080;

• serves the recovery-proof file directly from the data disk.

$WorkloadBootstrapScript = @'
#!/usr/bin/env bash
set -euo pipefail

export DEBIAN_FRONTEND=noninteractive

apt-get update
apt-get install -y nginx curl jq

DATA_DISK="/dev/disk/azure/scsi1/lun0"
MOUNT_PATH="/srv/backupdata"
PROTECTED_PATH="$MOUNT_PATH/protected"
MANIFEST_PATH="$MOUNT_PATH/manifest"

for attempt in $(seq 1 60); do
    if [ -e "$DATA_DISK" ]; then
        break
    fi

    sleep 5
done

if [ ! -e "$DATA_DISK" ]; then
    echo "Data disk was not found at $DATA_DISK."
    exit 1
fi

if ! blkid "$DATA_DISK" >/dev/null 2>&1; then
    mkfs.ext4 -F "$DATA_DISK"
fi

DISK_UUID="$(blkid -s UUID -o value "$DATA_DISK")"

mkdir -p "$MOUNT_PATH"

if ! grep -q "$DISK_UUID" /etc/fstab; then
    echo "UUID=$DISK_UUID $MOUNT_PATH ext4 defaults,nofail 0 2" >> /etc/fstab
fi

mount -a

mkdir -p "$PROTECTED_PATH"
mkdir -p "$MANIFEST_PATH"

RECOVERY_ID="$(cat /proc/sys/kernel/random/uuid)"
CREATED_UTC="$(date --utc --iso-8601=seconds)"

cat >"$PROTECTED_PATH/recovery-proof.txt" <<EOF
Azure Backup Deep Dive
Recovery proof ID: $RECOVERY_ID
Created UTC: $CREATED_UTC
Protected VM: vm-backup-deepdive
Protected path: /srv/backupdata/protected
EOF

cat >"$PROTECTED_PATH/customer-records.csv" <<'EOF'
CustomerId,CustomerName,ServiceTier,Region
1001,Contoso,Gold,West Europe
1002,Fabrikam,Silver,North Europe
1003,Adventure Works,Gold,West Europe
1004,Northwind,Bronze,UK South
EOF

cat >"$PROTECTED_PATH/application.conf" <<'EOF'
environment=production
service=backup-recovery-proof
listen_port=8080
data_path=/srv/backupdata/protected
recovery_validation=required
EOF

dd \
    if=/dev/urandom \
    of="$PROTECTED_PATH/payload.bin" \
    bs=1M \
    count=4 \
    status=none

chmod 0755 "$MOUNT_PATH"
chmod 0755 "$PROTECTED_PATH"
chmod 0755 "$MANIFEST_PATH"
chmod 0644 "$PROTECTED_PATH"/*

(
    cd "$PROTECTED_PATH"
    sha256sum * >"$MANIFEST_PATH/checksums.sha256"
)

cat >/etc/nginx/sites-available/default <<'NGINX'
server {
    listen 8080 default_server;
    listen [::]:8080 default_server;

    root /srv/backupdata/protected;
    index recovery-proof.txt;

    location / {
        default_type text/plain;
        try_files $uri $uri/ /recovery-proof.txt =404;
    }
}
NGINX

nginx -t
systemctl enable nginx
systemctl restart nginx

echo
echo "=== Data disk ==="
findmnt "$MOUNT_PATH"

echo
echo "=== Protected files ==="
ls -lh "$PROTECTED_PATH"

echo
echo "=== Checksum manifest ==="
cat "$MANIFEST_PATH/checksums.sha256"

echo
echo "=== Checksum validation ==="
(
    cd "$PROTECTED_PATH"
    sha256sum --check "$MANIFEST_PATH/checksums.sha256"
)

echo
echo "=== Application test ==="
curl -fsS http://127.0.0.1:8080
'@

Run the script:

$BootstrapResult = az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --command-id RunShellScript `
    --scripts $WorkloadBootstrapScript `
    --query "value[0].message" `
    --output tsv

Display the result:

$BootstrapResult

15. Capture the baseline checksum evidence

Retrieve the checksum manifest:

$BaselineManifest = az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --command-id RunShellScript `
    --scripts "sudo cat /srv/backupdata/manifest/checksums.sha256" `
    --query "value[0].message" `
    --output tsv

Display it:

$BaselineManifest

Run checksum verification:

$BaselineVerification = az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --command-id RunShellScript `
    --scripts @'
set -e

cd /srv/backupdata/protected

sha256sum \
    --check \
    ../manifest/checksums.sha256
'@ `
    --query "value[0].message" `
    --output tsv

Display it:

$BaselineVerification

Every file should report:

OK

16. Validate the source application

Get the source VM public IP:

$VmPublicIpAddress = (
    az network public-ip show `
        --subscription $SubscriptionId `
        --resource-group $SourceResourceGroup `
        --name $VmPublicIpName `
        --query ipAddress `
        --output tsv
).Trim()

Create the application URL:

$SourceApplicationUrl = "http://${VmPublicIpAddress}:8080"

$SourceApplicationUrl

Test it:

$SourceApplicationResponse = Invoke-WebRequest `
    -Uri $SourceApplicationUrl `
    -TimeoutSec 30

Display the content:

$SourceApplicationResponse.Content

Validate it:

if (
    $SourceApplicationResponse.Content -notmatch
    "Azure Backup Deep Dive"
) {
    throw "The source application did not return the expected recovery proof."
}

Write-Host `
    "PASS: The source workload is healthy before backup." `
    -ForegroundColor Green

17. Create the Recovery Services vault

The Recovery Services vault and protected VM must use a supported regional combination. For this same-region VM backup lab, the vault is created in the same Azure region as the VM. A backup job creates recovery points inside the vault, and those points become the basis for file, disk, or VM recovery.

az backup vault create `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --name $VaultName `
    --location $Location `
    --job-failure-alerts Enable `
    --public-network-access Enable `
    --tags `
        workshop=azure-backup-deep-dive `
        role=recovery-services-vault `
        managedBy=azure-cli `
    --only-show-errors `
    --output none

18. Configure vault storage redundancy

Use locally redundant backup storage for this short-lived lab:

az backup vault update `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --name $VaultName `
    --backup-storage-redundancy LocallyRedundant `
    --only-show-errors `
    --output none

Production noteLRS is used here to reduce lab cost. Evaluate GRS or ZRS against production recovery, durability, regional-outage, and data-residency requirements.

Display the vault:

az backup vault show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --name $VaultName `
    --query "{
        Name:name,
        Location:location,
        State:properties.provisioningState,
        PublicNetworkAccess:properties.publicNetworkAccess
    }" `
    --output jsonc

Display backup properties:

az backup vault backup-properties show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --name $VaultName `
    --output jsonc

19. Build the Enhanced backup policy

The policy performs backups every four hours, keeps operational snapshots for seven days, and retains daily vault recovery points for 30 days.

The policy body uses the Enhanced V2 schedule schema. Microsoft documents this schema with SimpleSchedulePolicyV2, an hourly interval, a schedule window, long-term retention, and Instant Restore retention.

Define the schedule start time:

$ScheduleStartUtc = (
    [DateTime]::UtcNow.Date
        .AddDays(1)
        .AddHours(2)
).ToString(
    "yyyy-MM-ddTHH:mm:ssZ"
)

$ScheduleStartUtc

Build the policy object:

$EnhancedPolicyObject = [ordered]@{
    properties = [ordered]@{
        backupManagementType = "AzureIaasVM"
        policyType           = "V2"
        instantRPDetails     = @{}

        schedulePolicy = [ordered]@{
            schedulePolicyType = "SimpleSchedulePolicyV2"
            scheduleRunFrequency = "Hourly"

            hourlySchedule = [ordered]@{
                interval                    = 4
                scheduleWindowStartTime     = $ScheduleStartUtc
                scheduleWindowDuration      = 24
            }
        }

        retentionPolicy = [ordered]@{
            retentionPolicyType = "LongTermRetentionPolicy"

            dailySchedule = [ordered]@{
                retentionTimes = @(
                    $ScheduleStartUtc
                )

                retentionDuration = [ordered]@{
                    count        = 30
                    durationType = "Days"
                }
            }

            weeklySchedule = [ordered]@{
                daysOfTheWeek = @(
                    "Sunday"
                )

                retentionTimes = @(
                    $ScheduleStartUtc
                )

                retentionDuration = [ordered]@{
                    count        = 8
                    durationType = "Weeks"
                }
            }

            monthlySchedule = [ordered]@{
                retentionScheduleFormatType = "Weekly"

                retentionScheduleWeekly = [ordered]@{
                    daysOfTheWeek = @(
                        "Sunday"
                    )

                    weeksOfTheMonth = @(
                        "First"
                    )
                }

                retentionTimes = @(
                    $ScheduleStartUtc
                )

                retentionDuration = [ordered]@{
                    count        = 6
                    durationType = "Months"
                }
            }
        }

        tieringPolicy = [ordered]@{
            ArchivedRP = [ordered]@{
                tieringMode = "DoNotTier"
                duration    = 0
                durationType = "Invalid"
            }
        }

        instantRpRetentionRangeInDays = 7
        timeZone                      = "UTC"
        protectedItemsCount           = 0
    }
}

Convert it to JSON:

$EnhancedPolicyJson = $EnhancedPolicyObject |
    ConvertTo-Json `
        -Depth 20 `
        -Compress

Review the formatted policy:

$EnhancedPolicyObject |
    ConvertTo-Json `
        -Depth 20

20. Create the Enhanced policy

az backup policy create `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $PolicyName `
    --backup-management-type AzureIaasVM `
    --workload-type VM `
    --policy $EnhancedPolicyJson `
    --only-show-errors `
    --output none

Verify that Azure classifies it as Enhanced:

az backup policy list `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --backup-management-type AzureIaasVM `
    --workload-type VM `
    --policy-sub-type Enhanced `
    --query "[].{
        Name:name,
        PolicyType:properties.policyType,
        Frequency:properties.schedulePolicy.scheduleRunFrequency,
        Interval:properties.schedulePolicy.hourlySchedule.interval,
        InstantRetentionDays:properties.instantRpRetentionRangeInDays
    }" `
    --output table

Display the complete policy:

az backup policy show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $PolicyName `
    --output jsonc

21. Enable backup protection

Resolve the VM resource ID:

$VmResourceId = (
    az vm show `
        --subscription $SubscriptionId `
        --resource-group $SourceResourceGroup `
        --name $VmName `
        --query id `
        --output tsv
).Trim()

Enable protection:

$ConfigureBackupResult = az backup protection enable-for-vm `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --vm $VmResourceId `
    --policy-name $PolicyName `
    --output json |
    ConvertFrom-Json

Display the result:

$ConfigureBackupResult

Wait for the configure-backup job when a job name is returned:

if ($ConfigureBackupResult.name) {
    az backup job wait `
        --subscription $SubscriptionId `
        --resource-group $VaultResourceGroup `
        --vault-name $VaultName `
        --name $ConfigureBackupResult.name `
        --timeout 7200
}

22. Resolve the protected container and backup item

Wait for the backup container:

$ContainerDeadline = (
    Get-Date
).AddMinutes(30)

do {
    $ContainerName = (
        az backup container list `
            --subscription $SubscriptionId `
            --resource-group $VaultResourceGroup `
            --vault-name $VaultName `
            --backup-management-type AzureIaasVM `
            --query "[?properties.friendlyName=='$VmName'] | [0].name" `
            --output tsv
    ).Trim()

    if ([string]::IsNullOrWhiteSpace($ContainerName)) {
        Write-Host `
            "Waiting for the backup container..."

        Start-Sleep -Seconds 30
    }
}
until (
    -not [string]::IsNullOrWhiteSpace(
        $ContainerName
    ) -or
    (Get-Date) -ge $ContainerDeadline
)

if ([string]::IsNullOrWhiteSpace($ContainerName)) {
    throw "The backup container was not registered."
}

Resolve the item:

$ItemDeadline = (
    Get-Date
).AddMinutes(30)

do {
    $ItemName = (
        az backup item list `
            --subscription $SubscriptionId `
            --resource-group $VaultResourceGroup `
            --vault-name $VaultName `
            --backup-management-type AzureIaasVM `
            --workload-type VM `
            --container-name $ContainerName `
            --query "[?properties.friendlyName=='$VmName'] | [0].name" `
            --output tsv
    ).Trim()

    if ([string]::IsNullOrWhiteSpace($ItemName)) {
        Write-Host `
            "Waiting for the backup item..."

        Start-Sleep -Seconds 30
    }
}
until (
    -not [string]::IsNullOrWhiteSpace(
        $ItemName
    ) -or
    (Get-Date) -ge $ItemDeadline
)

if ([string]::IsNullOrWhiteSpace($ItemName)) {
    throw "The backup item was not created."
}

Display both identifiers:

[pscustomobject]@{
    ContainerName = $ContainerName
    ItemName      = $ItemName
}

These identifiers are used by the backup and restore commands.

23. Inspect the protected backup item

az backup item show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --name $ItemName `
    --backup-management-type AzureIaasVM `
    --workload-type VM `
    --output jsonc

24. Trigger the on-demand backup

The first completed backup creates the initial full recovery point. Later VM backups transfer only changed blocks, making subsequent recovery points incremental.

Set the retention date:

$RetainUntil = (
    [DateTime]::UtcNow
        .AddDays(30)
).ToString(
    "dd-MM-yyyy"
)

$RetainUntil

Record the start time:

$BackupStartUtc = [DateTime]::UtcNow

Trigger the backup:

$BackupJob = az backup protection backup-now `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --backup-management-type AzureIaasVM `
    --retain-until $RetainUntil `
    --output json |
    ConvertFrom-Json

Display the job:

$BackupJob

Wait for completion:

az backup job wait `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $BackupJob.name `
    --timeout 14400

Record the completion time:

$BackupEndUtc = [DateTime]::UtcNow

Calculate the observed duration:

$BackupDuration = New-TimeSpan `
    -Start $BackupStartUtc `
    -End $BackupEndUtc

$BackupDuration

25. Verify the completed backup job

az backup job show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $BackupJob.name `
    --query "{
        Name:name,
        Operation:properties.operation,
        Status:properties.status,
        StartTime:properties.startTime,
        EndTime:properties.endTime,
        Duration:properties.duration,
        Entity:properties.entityFriendlyName
    }" `
    --output jsonc

Required status:

Completed

26. List and select the recovery point

List recovery points:

$RecoveryPoints = @(
    az backup recoverypoint list `
        --subscription $SubscriptionId `
        --resource-group $VaultResourceGroup `
        --vault-name $VaultName `
        --container-name $ContainerName `
        --item-name $ItemName `
        --backup-management-type AzureIaasVM `
        --output json |
    ConvertFrom-Json
)

Display them:

$RecoveryPoints |
    ForEach-Object {
        [pscustomobject]@{
            Name              = $_.name
            RecoveryPointTime = $_.properties.recoveryPointTime
            RecoveryPointType = $_.properties.recoveryPointType
            TierType          = $_.properties.recoveryPointTierDetails.type
        }
    } |
    Sort-Object RecoveryPointTime -Descending |
    Format-Table -AutoSize

Select the latest point:

$LatestRecoveryPoint = $RecoveryPoints |
    Sort-Object {
        [DateTime]$_.properties.recoveryPointTime
    } -Descending |
    Select-Object -First 1

$RecoveryPointName = $LatestRecoveryPoint.name
$RecoveryPointTime = [DateTime]$LatestRecoveryPoint.properties.recoveryPointTime

Display the selection:

[pscustomobject]@{
    RecoveryPointName = $RecoveryPointName
    RecoveryPointTime = $RecoveryPointTime
    RecoveryPointType = $LatestRecoveryPoint.properties.recoveryPointType
}

27. Record the pre-failure recovery evidence

$PreFailureEvidence = [ordered]@{
    VmName                = $VmName
    SourceApplicationUrl  = $SourceApplicationUrl
    RecoveryPointName     = $RecoveryPointName
    RecoveryPointTimeUtc  = $RecoveryPointTime.ToUniversalTime()
    BackupStartedUtc      = $BackupStartUtc
    BackupCompletedUtc    = $BackupEndUtc
    BackupDurationMinutes = [math]::Round(
        $BackupDuration.TotalMinutes,
        2
    )
    BaselineManifest      = $BaselineManifest
    BaselineVerification  = $BaselineVerification
}

Display it:

$PreFailureEvidence |
    ConvertTo-Json `
        -Depth 10

28. Break the workload deliberately

The following command:

• deletes recovery-proof.txt;

• deletes customer-records.csv;

• corrupts application.conf;

• truncates payload.bin;

• leaves the original checksum manifest unchanged.

$BreakWorkloadScript = @'
set -euo pipefail

PROTECTED_PATH="/srv/backupdata/protected"
MANIFEST_PATH="/srv/backupdata/manifest/checksums.sha256"

rm -f "$PROTECTED_PATH/recovery-proof.txt"
rm -f "$PROTECTED_PATH/customer-records.csv"

cat >"$PROTECTED_PATH/application.conf" <<'EOF'
environment=corrupted
service=unavailable
listen_port=0
data_path=/dev/null
recovery_validation=failed
EOF

truncate \
    --size 1024 `
    "$PROTECTED_PATH/payload.bin"

echo
echo "=== Current protected files ==="
ls -lh "$PROTECTED_PATH"

echo
echo "=== Expected checksum failures ==="

set +e

(
    cd "$PROTECTED_PATH"
    sha256sum --check "$MANIFEST_PATH"
)

CHECKSUM_EXIT_CODE=$?

set -e

echo
echo "Checksum exit code: $CHECKSUM_EXIT_CODE"

if [ "$CHECKSUM_EXIT_CODE" -eq 0 ]; then
    echo "BREAK_TEST=FAIL"
    exit 1
fi

echo "BREAK_TEST=PASS"
'@

Correct the PowerShell here-string before running by replacing the accidental PowerShell continuation marker in the Linux command:

$BreakWorkloadScript = $BreakWorkloadScript.Replace(
    "truncate `n    --size 1024 ``",
    "truncate `n    --size 1024"
)

Run the break action:

$BreakResult = az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --command-id RunShellScript `
    --scripts $BreakWorkloadScript `
    --query "value[0].message" `
    --output tsv

Display it:

$BreakResult

29. Prove that the application is broken

Test the URL:

try {
    $BrokenApplicationResponse = Invoke-WebRequest `
        -Uri $SourceApplicationUrl `
        -TimeoutSec 15 `
        -ErrorAction Stop

    $BrokenApplicationAvailable = $true
}
catch {
    $BrokenApplicationAvailable = $false
    $BrokenApplicationError     = $_.Exception.Message
}

Display the result:

[pscustomobject]@{
    ApplicationAvailable = $BrokenApplicationAvailable
    Error                = $BrokenApplicationError
}

The expected result is:

ApplicationAvailable    False

Run checksum validation again:

$BrokenChecksumResult = az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --command-id RunShellScript `
    --scripts @'
set +e

cd /srv/backupdata/protected

sha256sum \
    --check \
    ../manifest/checksums.sha256

echo "CHECKSUM_EXIT_CODE=$?"
'@ `
    --query "value[0].message" `
    --output tsv

Display it:

$BrokenChecksumResult

The failure is now measurable and reproducible.

30. Begin the file-level recovery drill

Azure Backup file recovery downloads a script that connects the selected recovery point to a compatible machine. The mounted volumes can then be browsed, individual files copied, and the recovery point disconnected.

Record the recovery start time:

$FileRestoreStartUtc = [DateTime]::UtcNow

Move to the local lab directory:

Set-Location $LabRoot

Capture the current script inventory:

$ExistingRecoveryScripts = @(
    Get-ChildItem `
        -Path $LabRoot `
        -Filter "*.sh" `
        -File `
        -ErrorAction SilentlyContinue |
    Select-Object -ExpandProperty FullName
)

Generate the recovery script:

$MountRecoveryOutput = az backup restore files mount-rp `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --rp-name $RecoveryPointName `
    2>&1 |
    Tee-Object `
        -Variable MountRecoveryOutputLines

Display the output:

$MountRecoveryOutput

Azure returns output similar to:

File downloaded: vm-backup-deepdive_we_123456789.sh
Use password 0123456789abcdef

31. Identify the generated recovery script

$NewRecoveryScripts = @(
    Get-ChildItem `
        -Path $LabRoot `
        -Filter "*.sh" `
        -File |
    Where-Object {
        $_.FullName -notin $ExistingRecoveryScripts
    } |
    Sort-Object LastWriteTime -Descending
)

$RecoveryScript = $NewRecoveryScripts |
    Select-Object -First 1

Validate it:

if (-not $RecoveryScript) {
    throw @"
The file-recovery script was not found.

Review the output from az backup restore files mount-rp.
"@
}

$RecoveryScript |
    Select-Object `
        Name,
        FullName,
        Length,
        LastWriteTime

32. Capture the temporary recovery password

$MountOutputText = (
    $MountRecoveryOutputLines |
    Out-String
)

$RecoveryPasswordMatch = [regex]::Match(
    $MountOutputText,
    "Use password\s+([^\s]+)",
    [System.Text.RegularExpressions.RegexOptions]::IgnoreCase
)

if ($RecoveryPasswordMatch.Success) {
    $RecoveryPassword = $RecoveryPasswordMatch.Groups[1].Value
}
else {
    $RecoveryPassword = $null
}

Display whether the password was found without printing it:

[pscustomobject]@{
    ScriptName       = $RecoveryScript.Name
    PasswordDetected = -not [string]::IsNullOrWhiteSpace(
        $RecoveryPassword
    )
}

Security noteDo not publish the recovery password in screenshots, source control, blog output, or support tickets.The password is temporary and applies only to the generated recovery script.

33. Copy the recovery script to the source VM

scp `
    -o StrictHostKeyChecking=accept-new `
    $RecoveryScript.FullName `
    "${VmAdminUsername}@${VmPublicIpAddress}:~/"

34. Mount the recovery point interactively

Open an SSH session:

ssh `
    -t `
    -o StrictHostKeyChecking=accept-new `
    "${VmAdminUsername}@${VmPublicIpAddress}"

Inside the SSH session, make the script executable:

chmod +x ~/vm-backup-deepdive*.sh

Run it with elevated privileges:

sudo ~/vm-backup-deepdive*.sh

When prompted, enter the temporary password returned by:

az backup restore files mount-rp

The script connects to the recovery point using iSCSI and displays the mounted recovery volumes.

A typical mount path resembles:

/home/azureuser/vm-backup-deepdive-<timestamp>/Volume1
/home/azureuser/vm-backup-deepdive-<timestamp>/Volume2

35. Locate the protected files in the mounted recovery point

Still inside the SSH session, find the recovery-proof file:

RECOVERED_PROOF_FILE="$(
    sudo find \
        /home/azureuser \
        -type f \
        -path "*/protected/recovery-proof.txt" \
        2>/dev/null |
    head -n 1
)"

echo "$RECOVERED_PROOF_FILE"

Validate that the file was found:

if [ -z "$RECOVERED_PROOF_FILE" ]; then
    echo "The protected recovery file was not found."
    exit 1
fi

Resolve the root of the recovered data volume:

RECOVERED_PROTECTED_PATH="$(
    dirname "$RECOVERED_PROOF_FILE"
)"

RECOVERED_DATA_ROOT="$(
    dirname "$RECOVERED_PROTECTED_PATH"
)"

echo "Recovered data root: $RECOVERED_DATA_ROOT"

List the protected files:

sudo ls -lh "$RECOVERED_DATA_ROOT/protected"

Display the recovery manifest:

sudo cat "$RECOVERED_DATA_ROOT/manifest/checksums.sha256"

36. Restore the individual files

Copy the protected files back to the live data disk:

sudo cp \
    -a \
    "$RECOVERED_DATA_ROOT/protected/." \
    /srv/backupdata/protected/

Copy the recovered checksum manifest to a temporary validation path:

sudo cp \
    "$RECOVERED_DATA_ROOT/manifest/checksums.sha256" \
    /tmp/recovered-checksums.sha256

Set the expected permissions:

sudo chmod 0644 /srv/backupdata/protected/*

37. Verify the restored files

Run checksum verification against the manifest from the recovery point:

cd /srv/backupdata/protected

sudo sha256sum \
    --check \
    /tmp/recovered-checksums.sha256

Every protected file should report:

OK

Display the recovered proof:

sudo cat /srv/backupdata/protected/recovery-proof.txt

Display the restored customer records:

sudo cat /srv/backupdata/protected/customer-records.csv

Display the restored configuration:

sudo cat /srv/backupdata/protected/application.conf

Test Nginx:

curl -fsS http://127.0.0.1:8080

Exit the SSH session:

exit

38. Close access to the recovery point

Back in the local PowerShell terminal:

az backup restore files unmount-rp `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --rp-name $RecoveryPointName `
    --only-show-errors `
    --output none

Delete the downloaded script:

Remove-Item `
    -Path $RecoveryScript.FullName `
    -Force `
    -ErrorAction SilentlyContinue

Clear the temporary password:

$RecoveryPassword = $null

39. Measure the file-recovery time

$FileRestoreEndUtc = [DateTime]::UtcNow

$FileRestoreDuration = New-TimeSpan `
    -Start $FileRestoreStartUtc `
    -End $FileRestoreEndUtc

Display the duration:

[pscustomobject]@{
    StartedUtc      = $FileRestoreStartUtc
    CompletedUtc    = $FileRestoreEndUtc
    DurationMinutes = [math]::Round(
        $FileRestoreDuration.TotalMinutes,
        2
    )
}

40. Validate the restored source workload

Test the public application URL again:

$RestoredSourceResponse = Invoke-WebRequest `
    -Uri $SourceApplicationUrl `
    -TimeoutSec 30

Validate it:

if (
    $RestoredSourceResponse.Content -notmatch
    "Azure Backup Deep Dive"
) {
    throw "The file-level restore did not recover the source application."
}

Write-Host `
    "PASS: File-level recovery restored the application." `
    -ForegroundColor Green

Verify the checksums remotely:

$FileRestoreVerification = az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --name $VmName `
    --command-id RunShellScript `
    --scripts @'
set -e

cd /srv/backupdata/protected

sha256sum \
    --check \
    ../manifest/checksums.sha256
'@ `
    --query "value[0].message" `
    --output tsv

Display the result:

$FileRestoreVerification

41. Begin the complete-VM recovery drill

A complete VM recovery is appropriate when:

• the source VM is unavailable;

• the operating system is damaged;

• several disks need to be recovered together;

• a recovery drill requires an isolated VM copy;

• recovery must be tested without replacing the source.

Azure CLI supports alternate-location VM restore by providing a target resource group, VM name, VNet, subnet, and staging storage account.

Record the start time:

$FullRestoreStartUtc = [DateTime]::UtcNow

42. Create the restore resource group

az group create `
    --subscription $SubscriptionId `
    --name $RestoreResourceGroup `
    --location $Location `
    --tags `
        workshop=azure-backup-deep-dive `
        role=alternate-location-restore `
        managedBy=azure-cli `
    --only-show-errors `
    --output none

43. Create the isolated restore VNet

az network vnet create `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RestoreVnetName `
    --location $Location `
    --address-prefixes $RestoreVnetPrefix `
    --subnet-name $RestoreSubnetName `
    --subnet-prefixes $RestoreSubnetPrefix `
    --tags `
        workshop=azure-backup-deep-dive `
        role=restore-network `
    --only-show-errors `
    --output none

The restore VNet is not peered with the source VNet.

This allows the restored VM to be validated as an isolated recovery copy.

44. Create the restore NSG

az network nsg create `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RestoreNsgName `
    --location $Location `
    --tags `
        workshop=azure-backup-deep-dive `
        role=restore-security `
    --only-show-errors `
    --output none

Allow SSH:

az network nsg rule create `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --nsg-name $RestoreNsgName `
    --name Allow-Admin-SSH `
    --priority 100 `
    --direction Inbound `
    --access Allow `
    --protocol Tcp `
    --source-address-prefixes $AdminSourceCidr `
    --source-port-ranges "*" `
    --destination-address-prefixes "*" `
    --destination-port-ranges 22 `
    --only-show-errors `
    --output none

Allow recovery validation:

az network nsg rule create `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --nsg-name $RestoreNsgName `
    --name Allow-Admin-HTTP-8080 `
    --priority 110 `
    --direction Inbound `
    --access Allow `
    --protocol Tcp `
    --source-address-prefixes $AdminSourceCidr `
    --source-port-ranges "*" `
    --destination-address-prefixes "*" `
    --destination-port-ranges 8080 `
    --only-show-errors `
    --output none

45. Create the restore staging storage account

The staging account must be available in the vault’s region for this same-region restore.

Check the name:

az storage account check-name `
    --name $RestoreStorageAccount `
    --query "{
        NameAvailable:nameAvailable,
        Reason:reason,
        Message:message
    }" `
    --output jsonc

Create it:

az storage account create `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RestoreStorageAccount `
    --location $Location `
    --sku Standard_LRS `
    --kind StorageV2 `
    --allow-blob-public-access false `
    --min-tls-version TLS1_2 `
    --tags `
        workshop=azure-backup-deep-dive `
        role=restore-staging `
    --only-show-errors `
    --output none

Resolve its resource ID:

$RestoreStorageAccountId = (
    az storage account show `
        --subscription $SubscriptionId `
        --resource-group $RestoreResourceGroup `
        --name $RestoreStorageAccount `
        --query id `
        --output tsv
).Trim()

46. Trigger the alternate-location VM restore

Ensure the target VM name is unused:

$ExistingRecoveredVm = az vm show `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName `
    --query id `
    --output tsv `
    2>$null

if ($LASTEXITCODE -eq 0) {
    throw "The target recovered VM already exists."
}

Start the restore:

$FullRestoreJob = az backup restore restore-disks `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --rp-name $RecoveryPointName `
    --storage-account $RestoreStorageAccountId `
    --storage-account-resource-group $RestoreResourceGroup `
    --restore-mode AlternateLocation `
    --target-resource-group $RestoreResourceGroup `
    --target-vm-name $RecoveredVmName `
    --target-vnet-name $RestoreVnetName `
    --target-vnet-resource-group $RestoreResourceGroup `
    --target-subnet-name $RestoreSubnetName `
    --output json |
    ConvertFrom-Json

Display the job:

$FullRestoreJob

Wait for completion:

az backup job wait `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $FullRestoreJob.name `
    --timeout 21600

47. Verify the full restore job

az backup job show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $FullRestoreJob.name `
    --query "{
        Name:name,
        Operation:properties.operation,
        Status:properties.status,
        StartTime:properties.startTime,
        EndTime:properties.endTime,
        Duration:properties.duration,
        Entity:properties.entityFriendlyName
    }" `
    --output jsonc

Required status:

Completed

48. Detect the recovered VM

Wait for the VM resource:

$RecoveredVmDeadline = (
    Get-Date
).AddMinutes(30)

do {
    $RecoveredVmId = (
        az vm show `
            --subscription $SubscriptionId `
            --resource-group $RestoreResourceGroup `
            --name $RecoveredVmName `
            --query id `
            --output tsv `
            2>$null
    ).Trim()

    if ([string]::IsNullOrWhiteSpace($RecoveredVmId)) {
        Write-Host `
            "Waiting for the recovered VM resource..."

        Start-Sleep -Seconds 30
    }
}
until (
    -not [string]::IsNullOrWhiteSpace(
        $RecoveredVmId
    ) -or
    (Get-Date) -ge $RecoveredVmDeadline
)

if ([string]::IsNullOrWhiteSpace($RecoveredVmId)) {
    throw "The recovered VM was not created."
}

Display it:

az vm show `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName `
    --query "{
        Name:name,
        Location:location,
        VmSize:hardwareProfile.vmSize,
        OsDisk:storageProfile.osDisk.name,
        DataDisks:storageProfile.dataDisks[].name,
        Nic:networkProfile.networkInterfaces[0].id
    }" `
    --output jsonc

49. Attach the restore NSG to the recovered NIC

Resolve the NIC ID:

$RecoveredNicId = (
    az vm show `
        --subscription $SubscriptionId `
        --resource-group $RestoreResourceGroup `
        --name $RecoveredVmName `
        --query "networkProfile.networkInterfaces[0].id" `
        --output tsv
).Trim()

Extract the NIC resource group and name:

$RecoveredNicParts = $RecoveredNicId.Trim("/") -split "/"

$RecoveredNicResourceGroup = $RecoveredNicParts[
    [array]::IndexOf(
        $RecoveredNicParts,
        "resourceGroups"
    ) + 1
]

$RecoveredNicName = $RecoveredNicParts[-1]

Attach the NSG:

az network nic update `
    --subscription $SubscriptionId `
    --resource-group $RecoveredNicResourceGroup `
    --name $RecoveredNicName `
    --network-security-group $RestoreNsgName `
    --only-show-errors `
    --output none

50. Create and attach a public IP to the recovered VM

Create the public IP:

az network public-ip create `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredPublicIpName `
    --location $Location `
    --sku Standard `
    --tier Regional `
    --allocation-method Static `
    --version IPv4 `
    --tags `
        workshop=azure-backup-deep-dive `
        role=recovered-vm-validation `
    --only-show-errors `
    --output none

Resolve the NIC IP configuration name:

$RecoveredIpConfigName = (
    az network nic show `
        --subscription $SubscriptionId `
        --resource-group $RecoveredNicResourceGroup `
        --name $RecoveredNicName `
        --query "ipConfigurations[0].name" `
        --output tsv
).Trim()

Attach the public IP:

az network nic ip-config update `
    --subscription $SubscriptionId `
    --resource-group $RecoveredNicResourceGroup `
    --nic-name $RecoveredNicName `
    --name $RecoveredIpConfigName `
    --public-ip-address $RecoveredPublicIpName `
    --only-show-errors `
    --output none

51. Start the recovered VM

az vm start `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName `
    --only-show-errors `
    --output none

Wait for the running state:

$RecoveredVmRunningDeadline = (
    Get-Date
).AddMinutes(20)

do {
    $RecoveredPowerState = (
        az vm get-instance-view `
            --subscription $SubscriptionId `
            --resource-group $RestoreResourceGroup `
            --name $RecoveredVmName `
            --query "instanceView.statuses[?starts_with(code,'PowerState/')].code | [0]" `
            --output tsv
    ).Trim()

    Write-Host `
        "Recovered VM power state: $RecoveredPowerState"

    if ($RecoveredPowerState -ne "PowerState/running") {
        Start-Sleep -Seconds 15
    }
}
until (
    $RecoveredPowerState -eq "PowerState/running" -or
    (Get-Date) -ge $RecoveredVmRunningDeadline
)

if ($RecoveredPowerState -ne "PowerState/running") {
    throw "The recovered VM did not reach the running state."
}

52. Validate the recovered guest

Wait for Azure Run Command:

$GuestAgentDeadline = (
    Get-Date
).AddMinutes(20)

$RecoveredGuestResult = $null

do {
    try {
        $RecoveredGuestResult = az vm run-command invoke `
            --subscription $SubscriptionId `
            --resource-group $RestoreResourceGroup `
            --name $RecoveredVmName `
            --command-id RunShellScript `
            --scripts @'
set -e

echo "=== Hostname ==="
hostname

echo
echo "=== Data disk mount ==="
findmnt /srv/backupdata

echo
echo "=== Restored files ==="
ls -lh /srv/backupdata/protected

echo
echo "=== Checksum validation ==="

cd /srv/backupdata/protected

sha256sum \
    --check \
    ../manifest/checksums.sha256

echo
echo "=== Local application ==="
curl -fsS http://127.0.0.1:8080
'@ `
            --query "value[0].message" `
            --output tsv `
            2>$null
    }
    catch {
        $RecoveredGuestResult = $null
    }

    if (-not $RecoveredGuestResult) {
        Write-Host `
            "Waiting for the recovered VM agent..."

        Start-Sleep -Seconds 30
    }
}
until (
    $RecoveredGuestResult -or
    (Get-Date) -ge $GuestAgentDeadline
)

if (-not $RecoveredGuestResult) {
    throw "The recovered VM guest validation did not complete."
}

Display the result:

$RecoveredGuestResult

Confirm that every checksum reports:

OK

53. Validate the recovered application externally

Resolve the recovered public IP:

$RecoveredPublicIpAddress = (
    az network public-ip show `
        --subscription $SubscriptionId `
        --resource-group $RestoreResourceGroup `
        --name $RecoveredPublicIpName `
        --query ipAddress `
        --output tsv
).Trim()

Create the URL:

$RecoveredApplicationUrl = "http://${RecoveredPublicIpAddress}:8080"

$RecoveredApplicationUrl

Wait for the application:

$RecoveredApplicationDeadline = (
    Get-Date
).AddMinutes(10)

$RecoveredApplicationResponse = $null

do {
    try {
        $RecoveredApplicationResponse = Invoke-WebRequest `
            -Uri $RecoveredApplicationUrl `
            -TimeoutSec 20 `
            -ErrorAction Stop
    }
    catch {
        Write-Host `
            "Waiting for the recovered application..."

        Start-Sleep -Seconds 15
    }
}
until (
    $RecoveredApplicationResponse -or
    (Get-Date) -ge $RecoveredApplicationDeadline
)

if (-not $RecoveredApplicationResponse) {
    throw "The recovered application did not become reachable."
}

Validate the response:

if (
    $RecoveredApplicationResponse.Content -notmatch
    "Azure Backup Deep Dive"
) {
    throw "The recovered VM returned unexpected application content."
}

Write-Host `
    "PASS: The complete VM was restored successfully." `
    -ForegroundColor Green

54. Measure the complete-VM recovery time

$FullRestoreEndUtc = [DateTime]::UtcNow

$FullRestoreDuration = New-TimeSpan `
    -Start $FullRestoreStartUtc `
    -End $FullRestoreEndUtc

Display it:

[pscustomobject]@{
    StartedUtc      = $FullRestoreStartUtc
    CompletedUtc    = $FullRestoreEndUtc
    DurationMinutes = [math]::Round(
        $FullRestoreDuration.TotalMinutes,
        2
    )
}

55. Optional: perform a disk-only restore

A disk-only restore is useful when:

• you want to inspect the recovered disks manually;

• the VM requires custom reconstruction;

• a different VM configuration is needed;

• only selected disks are required.

Start an optional disk restore:

$DiskRestoreStartUtc = [DateTime]::UtcNow

$DiskRestoreJob = az backup restore restore-disks `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --rp-name $RecoveryPointName `
    --storage-account $RestoreStorageAccountId `
    --storage-account-resource-group $RestoreResourceGroup `
    --restore-mode AlternateLocation `
    --target-resource-group $RestoreResourceGroup `
    --output json |
    ConvertFrom-Json

Wait for completion:

az backup job wait `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $DiskRestoreJob.name `
    --timeout 21600

List the restored disks:

az disk list `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --query "[].{
        Name:name,
        SizeGb:diskSizeGb,
        Sku:sku.name,
        State:provisioningState,
        OsType:osType
    }" `
    --output table

56. Build the recovery evidence report

$RecoveryEvidence = [ordered]@{
    Workshop = "Azure Backup Deep Dive"

    Source = [ordered]@{
        VmName         = $VmName
        ResourceGroup  = $SourceResourceGroup
        ApplicationUrl = $SourceApplicationUrl
    }

    Backup = [ordered]@{
        Vault                  = $VaultName
        Policy                 = $PolicyName
        RecoveryPoint          = $RecoveryPointName
        RecoveryPointTimeUtc   = $RecoveryPointTime.ToUniversalTime()
        BackupStartedUtc       = $BackupStartUtc
        BackupCompletedUtc     = $BackupEndUtc
        BackupDurationMinutes  = [math]::Round(
            $BackupDuration.TotalMinutes,
            2
        )
    }

    FileRecovery = [ordered]@{
        StartedUtc             = $FileRestoreStartUtc
        CompletedUtc           = $FileRestoreEndUtc
        DurationMinutes        = [math]::Round(
            $FileRestoreDuration.TotalMinutes,
            2
        )
        ChecksumVerified       = $true
        ApplicationVerified    = $true
    }

    FullVmRecovery = [ordered]@{
        VmName                 = $RecoveredVmName
        ResourceGroup          = $RestoreResourceGroup
        ApplicationUrl         = $RecoveredApplicationUrl
        StartedUtc             = $FullRestoreStartUtc
        CompletedUtc           = $FullRestoreEndUtc
        DurationMinutes        = [math]::Round(
            $FullRestoreDuration.TotalMinutes,
            2
        )
        ChecksumVerified       = $true
        ApplicationVerified    = $true
    }
}

Display it:

$RecoveryEvidence |
    ConvertTo-Json `
        -Depth 20

Save a local copy:

$RecoveryEvidencePath = Join-Path `
    $LabRoot `
    "recovery-evidence.json"

$RecoveryEvidence |
    ConvertTo-Json `
        -Depth 20 |
    Set-Content `
        -Path $RecoveryEvidencePath `
        -Encoding utf8

57. Display the observed recovery metrics

$RecoveryMetrics = @(
    [pscustomobject]@{
        Operation       = "On-demand backup"
        StartedUtc      = $BackupStartUtc
        CompletedUtc    = $BackupEndUtc
        DurationMinutes = [math]::Round(
            $BackupDuration.TotalMinutes,
            2
        )
        Result          = "Recovery point created"
    }

    [pscustomobject]@{
        Operation       = "File-level recovery"
        StartedUtc      = $FileRestoreStartUtc
        CompletedUtc    = $FileRestoreEndUtc
        DurationMinutes = [math]::Round(
            $FileRestoreDuration.TotalMinutes,
            2
        )
        Result          = "Checksums verified"
    }

    [pscustomobject]@{
        Operation       = "Complete VM recovery"
        StartedUtc      = $FullRestoreStartUtc
        CompletedUtc    = $FullRestoreEndUtc
        DurationMinutes = [math]::Round(
            $FullRestoreDuration.TotalMinutes,
            2
        )
        Result          = "VM and application verified"
    }
)

$RecoveryMetrics |
    Format-Table -AutoSize

58. Calculate the observed recovery-point age

$FailureTimeUtc = $FileRestoreStartUtc

$ObservedRecoveryPointAge = New-TimeSpan `
    -Start $RecoveryPointTime `
    -End $FailureTimeUtc

Display it:

[pscustomobject]@{
    RecoveryPointTimeUtc = $RecoveryPointTime
    FailureTimeUtc       = $FailureTimeUtc
    ObservedAgeMinutes   = [math]::Round(
        $ObservedRecoveryPointAge.TotalMinutes,
        2
    )
}

This is a lab observation of the selected recovery point’s age when recovery began.

It is not a contractual RPO.

59. Recovery-proof checklist

The recovery drill is complete only when every item is true:

[ ] Backup policy exists
[ ] VM protection is enabled
[ ] Backup job completed
[ ] Recovery point exists
[ ] Baseline checksums were recorded
[ ] Protected files were deleted or corrupted
[ ] Checksum validation failed after corruption
[ ] File recovery script mounted the recovery point
[ ] Individual files were restored
[ ] File checksums returned OK
[ ] Source application returned expected content
[ ] Complete VM was restored to alternate location
[ ] Restored data disk mounted successfully
[ ] Full-VM checksums returned OK
[ ] Recovered application returned expected content
[ ] Recovery durations were recorded
[ ] Recovery evidence was saved

60. Troubleshooting

The Enhanced policy is rejected

Verify the Azure CLI version:

az version

Inspect the policy JSON:

$EnhancedPolicyObject |
    ConvertTo-Json `
        -Depth 20

Check that it contains:

backupManagementType         AzureIaasVM
policyType                   V2
schedulePolicyType           SimpleSchedulePolicyV2
scheduleRunFrequency         Hourly
interval                     4
timeZone                     UTC
instantRpRetentionRangeInDays 7

Confirm the schedule timestamps include a complete date and time:

$ScheduleStartUtc

Expected format:

2026-08-02T02:00:00Z

Do not pass only a time value.

The VM cannot be protected

Check whether it is already protected:

az backup protection check-vm `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --vm $VmName `
    --output jsonc

Verify the VM and vault regions:

$VmLocation = (
    az vm show `
        --subscription $SubscriptionId `
        --resource-group $SourceResourceGroup `
        --name $VmName `
        --query location `
        --output tsv
).Trim()

$VaultLocation = (
    az backup vault show `
        --subscription $SubscriptionId `
        --resource-group $VaultResourceGroup `
        --name $VaultName `
        --query location `
        --output tsv
).Trim()

[pscustomobject]@{
    VmLocation    = $VmLocation
    VaultLocation = $VaultLocation
}

Check recent jobs:

az backup job list `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --query "[].{
        Name:name,
        Operation:properties.operation,
        Status:properties.status,
        Entity:properties.entityFriendlyName,
        Error:properties.errorDetails
    }" `
    --output jsonc

The backup job completed but no recovery point is listed

Wait for recovery-point registration:

Start-Sleep -Seconds 120

List the points again:

az backup recoverypoint list `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --backup-management-type AzureIaasVM `
    --output table

Inspect the backup job details:

az backup job show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $BackupJob.name `
    --output jsonc

The file-recovery script is not downloaded

Run the command from an empty directory:

Set-Location $LabRoot

Get-ChildItem

Generate it again:

az backup restore files mount-rp `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --rp-name $RecoveryPointName

Check that the recovery point supports file recovery:

az backup recoverypoint show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --name $RecoveryPointName `
    --backup-management-type AzureIaasVM `
    --output jsonc

SCP fails

Verify the VM public IP:

$VmPublicIpAddress

Test TCP port 22:

Test-NetConnection `
    -ComputerName $VmPublicIpAddress `
    -Port 22

Verify the source NSG:

az network nsg rule list `
    --subscription $SubscriptionId `
    --resource-group $SourceResourceGroup `
    --nsg-name $SourceNsgName `
    --query "[].{
        Name:name,
        Priority:priority,
        Access:access,
        Source:sourceAddressPrefix,
        Port:destinationPortRange
    }" `
    --output table

Confirm that $AdminSourceCidr still matches the current administrator public IP.

The recovery script cannot mount the point

Inside the VM, check iSCSI:

sudo systemctl status iscsid --no-pager

Check disk devices:

lsblk -f

Check network connectivity:

curl -I https://management.azure.com

Run the recovery script again:

sudo ~/vm-backup-deepdive*.sh

Common causes include:

• an expired recovery script;

• an incorrect temporary password;

• no outbound HTTPS access;

• iSCSI service problems;

• the script being run on an incompatible operating system;

• another file-recovery mount already being active.

Generate a new script if the existing one has expired.

The recovered file cannot be found

Search all mounted recovery volumes:

sudo find \
    /home/azureuser \
    -type f \
    -name "recovery-proof.txt" \
    2>/dev/null

List mounted filesystems:

findmnt

The data disk might appear as a different volume number from the OS disk.

Do not assume it is always Volume2.

Checksums do not match after file recovery

Verify which manifest is being used:

cat /tmp/recovered-checksums.sha256

List the restored files:

ls -lh /srv/backupdata/protected

Run verification:

cd /srv/backupdata/protected

sha256sum \
    --check \
    /tmp/recovered-checksums.sha256

Typical causes include:

• selecting the wrong recovery point;

• copying only some files;

• restoring from the OS volume instead of the data volume;

• changing a file after restore;

• using the corrupted live manifest instead of the recovered manifest.

Complete VM restore fails

Check the staging account region:

az storage account show `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RestoreStorageAccount `
    --query "{
        Name:name,
        Location:location,
        State:provisioningState,
        Sku:sku.name
    }" `
    --output table

Check the target network:

az network vnet subnet show `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --vnet-name $RestoreVnetName `
    --name $RestoreSubnetName `
    --output jsonc

Check the restore job:

az backup job show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --name $FullRestoreJob.name `
    --output jsonc

Common causes include:

• an invalid staging account;

• insufficient compute quota;

• the target VM name already existing;

• an invalid target subnet;

• Azure Policy blocking resource creation;

• insufficient permissions;

• selecting an unsupported recovery point.

The recovered VM exists but does not start

Display the VM status:

az vm get-instance-view `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName `
    --output jsonc

Display the disks:

az vm show `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName `
    --query "storageProfile" `
    --output jsonc

Check the boot diagnostics:

az vm boot-diagnostics get-boot-log `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName

The recovered VM starts but the data disk is not mounted

Run:

az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName `
    --command-id RunShellScript `
    --scripts @'
lsblk -f

echo
cat /etc/fstab

echo
findmnt /srv/backupdata || true

echo
sudo mount -a

echo
findmnt /srv/backupdata || true
'@ `
    --query "value[0].message" `
    --output tsv

The restored disk preserves its filesystem UUID.

The /etc/fstab entry should therefore remount it automatically.

The recovered application is not reachable

Test it inside the VM:

az vm run-command invoke `
    --subscription $SubscriptionId `
    --resource-group $RestoreResourceGroup `
    --name $RecoveredVmName `
    --command-id RunShellScript `
    --scripts @'
systemctl is-active nginx

ss -lntp | grep ':8080' || true

curl -v http://127.0.0.1:8080
'@ `
    --query "value[0].message" `
    --output tsv

Check the recovered NIC NSG:

az network nic show `
    --subscription $SubscriptionId `
    --resource-group $RecoveredNicResourceGroup `
    --name $RecoveredNicName `
    --query "networkSecurityGroup.id" `
    --output tsv

Check the public IP association:

az network nic ip-config show `
    --subscription $SubscriptionId `
    --resource-group $RecoveredNicResourceGroup `
    --nic-name $RecoveredNicName `
    --name $RecoveredIpConfigName `
    --output jsonc

61. Production considerations

A completed backup is not a recovery test

Monitor backup success, but also perform scheduled recovery drills.

A complete drill should verify:

Recovery point selection
Data restoration
Filesystem integrity
Application startup
Network access
Authentication
Dependencies
Monitoring
Security controls
Observed recovery time
Cleanup

Store recovery evidence

For audit and operational readiness, retain:

• backup job ID;

• recovery-point ID;

• backup start and completion time;

• selected restore point time;

• baseline checksum manifest;

• restored checksum results;

• application validation output;

• observed recovery duration;

• operator name;

• test date;

• exceptions and remediation actions.

Use the correct recovery method

Requirement	Recommended recovery method
One deleted file	File-level recovery
One damaged data disk	Disk restore
Damaged VM configuration	Complete VM restore
Source VM replacement	Original-location restore
Isolated recovery test	Alternate-location restore
Regional disaster	Cross-region restore when configured

Test application consistency

This lab uses a simple Linux workload.

Database and transactional workloads can require:

• application-consistent recovery points;

• pre-freeze scripts;

• post-thaw scripts;

• database recovery;

• transaction-log replay;

• application startup ordering;

• data-consistency validation.

Protect the vault

For production workloads, evaluate:

• vault immutability;

• soft delete;

• multi-user authorization;

• Resource Guard;

• private endpoints;

• restricted public network access;

• diagnostic settings;

• alerting;

• role separation.

Use realistic recovery objectives

Do not define RTO and RPO only from service capabilities.

Include:

• incident detection;

• authorization;

• recovery-point selection;

• restore initiation;

• storage recovery;

• VM startup;

• DNS changes;

• load-balancer changes;

• application validation;

• business-owner acceptance.

62. Cleanup

Azure Backup now enforces soft-delete behavior in supported regions and API versions. Deleted backup items can remain recoverable for the configured retention period, which is 14 days by default. Vault deletion may move the vault into a soft-deleted state rather than removing it permanently immediately.

Close any active file-recovery session

az backup restore files unmount-rp `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --rp-name $RecoveryPointName `
    --only-show-errors `
    --output none `
    2>$null

Delete the restore resource group

az group delete `
    --subscription $SubscriptionId `
    --name $RestoreResourceGroup `
    --yes `
    --no-wait `
    --only-show-errors

Stop protection and request backup-data deletion

az backup protection disable `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --vault-name $VaultName `
    --container-name $ContainerName `
    --item-name $ItemName `
    --backup-management-type AzureIaasVM `
    --delete-backup-data true `
    --yes `
    --only-show-errors

List soft-deleted containers:

az backup vault list-soft-deleted-containers `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --name $VaultName `
    --backup-management-type AzureIaasVM `
    --output table

Attempt to delete the vault

$VaultDeleteOutput = az backup vault delete `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --name $VaultName `
    --force `
    --yes `
    2>&1

$VaultDeleteExitCode = $LASTEXITCODE

Display the result:

$VaultDeleteOutput

Interpret it:

if ($VaultDeleteExitCode -eq 0) {
    Write-Host `
        "Vault deletion or vault soft deletion was accepted." `
        -ForegroundColor Green
}
else {
    Write-Warning @"
The vault could not be deleted immediately.

Secure-by-default soft delete can retain the backup item or vault until the
retention period expires. Review the remaining vault dependencies before
attempting deletion again.
"@
}

Delete the source resource group

az group delete `
    --subscription $SubscriptionId `
    --name $SourceResourceGroup `
    --yes `
    --no-wait `
    --only-show-errors

Delete the vault resource group when possible

Check whether an active vault remains:

$ActiveVaultId = az backup vault show `
    --subscription $SubscriptionId `
    --resource-group $VaultResourceGroup `
    --name $VaultName `
    --query id `
    --output tsv `
    2>$null

If no active vault remains:

if ($LASTEXITCODE -ne 0) {
    az group delete `
        --subscription $SubscriptionId `
        --name $VaultResourceGroup `
        --yes `
        --no-wait `
        --only-show-errors
}
else {
    Write-Warning @"
The vault resource group still contains an active or retained vault.

Do not force-delete unrelated resources. Recheck the vault after the configured
soft-delete retention period.
"@
}

Monitor resource-group deletion

$GroupsToMonitor = @(
    $SourceResourceGroup
    $RestoreResourceGroup
)

$DeletionDeadline = (
    Get-Date
).AddHours(2)

do {
    $RemainingGroups = @(
        foreach ($ResourceGroupName in $GroupsToMonitor) {
            $Exists = az group exists `
                --subscription $SubscriptionId `
                --name $ResourceGroupName `
                --output tsv

            if ($Exists -eq "true") {
                $ResourceGroupName
            }
        }
    )

    Write-Host (
        "{0:u} Remaining resource groups: {1}" -f `
            (Get-Date),
            (
                $RemainingGroups -join ", "
            )
    )

    if ($RemainingGroups.Count -gt 0) {
        Start-Sleep -Seconds 60
    }
}
until (
    $RemainingGroups.Count -eq 0 -or
    (Get-Date) -ge $DeletionDeadline
)

Remove the local workshop directory

Set-Location $HOME

Remove-Item `
    -Path $LabRoot `
    -Recurse `
    -Force `
    -ErrorAction SilentlyContinue

Summary

This workshop created and validated the complete Azure VM recovery lifecycle:

Deploy source VM
          ↓
Attach managed data disk
          ↓
Create protected files
          ↓
Record SHA-256 checksums
          ↓
Create Recovery Services vault
          ↓
Create Enhanced backup policy
          ↓
Enable VM protection
          ↓
Trigger on-demand backup
          ↓
Select recovery point
          ↓
Delete and corrupt data
          ↓
Prove checksum failure
          ↓
Mount recovery point
          ↓
Restore individual files
          ↓
Verify SHA-256 checksums
          ↓
Restore complete VM
          ↓
Validate disks and application
          ↓
Measure observed recovery times
          ↓
Record recovery evidence
          ↓
Delete the workshop

The central principle is:

A successful backup is an input.

A verified restore is the proof.

The most important lessons are:

1. Record checksums before data loss.

2. Trigger and monitor a real backup job.

3. Confirm that a recovery point exists.

4. Break the workload deliberately.

5. Prove that the failure is real.

6. Restore selected files first when that is sufficient.

7. Verify restored data against the original checksum manifest.

8. Test complete-VM recovery separately.

9. Validate the application, not only the VM power state.

10. Measure actual recovery duration.

11. Store recovery evidence.

12. Repeat recovery drills on a schedule.

13. Protect the backup vault from destructive actions.

14. Account for soft-delete behavior during cleanup.

15. Treat recovery as an operational capability rather than a checkbox.