Azure VM Scale Sets Deep Dive — Flexible Orchestration, Autoscaling, Health Monitoring & Self-Healing

Build it. Scale it. Break it. Repair it. Verify it.
Azure Workshopz | Beyond Cloud with Chriz
Level: Intermediate to AdvancedPlatform: Microsoft AzureTools: Azure CLI, Azure Cloud Shell (Bash)Deployment: Azure CLI only — no Terraform, Bicep, or Azure Portal requiredOperating System: Ubuntu Server 22.04 LTSTopics: Compute, Availability Zones, Autoscaling, Monitoring, High Availability, Disaster Recovery Concepts
Introduction
Deploying a virtual machine in Microsoft Azure is straightforward. But what happens when your application experiences an unexpected increase in CPU utilization? How do you automatically provision additional compute capacity? And what happens when an instance becomes unhealthy?
This workshop explores Azure Virtual Machine Scale Sets (VMSS) from an operational perspective.
We will deploy a Flexible Virtual Machine Scale Set across Azure Availability Zones, configure automatic scaling based on CPU utilization, monitor application health, deliberately break an application, and observe Azure automatically replacing the unhealthy instance.
The focus is not just deploying resources. It is understanding what happens when infrastructure needs to scale, recover, and remain operational.
What you will learn
By the end of this workshop, you will be able to:
Deploy an Azure Virtual Machine Scale Set using Flexible orchestration.
Distribute instances across Azure Availability Zones.
Deploy a simple NGINX application automatically using cloud-init.
Configure Azure Monitor autoscaling rules.
Generate CPU load and trigger scale-out operations.
Monitor VM metrics, instance states, and scaling actions.
Configure the Azure Application Health extension.
Enable automatic instance repairs.
Simulate an unhealthy application and verify automatic replacement.
Clean up the complete environment using Azure CLI.
Workshop architecture
We will create the following environment:
Azure Subscription
|
Resource Group
|
Virtual Network
10.42.0.0/16
|
Web Subnet
10.42.1.0/24
|
Azure VM Scale Set
Flexible Orchestration
|
+-----------+-----------+
| | |
Zone 1 Zone 2 Zone 3
| | |
VM-01 VM-02 VM-03
Ubuntu Ubuntu Ubuntu
NGINX NGINX NGINX
| | |
/health /health /health
| | |
+-----------+-----------+
|
Application Health
Extension
|
Azure VMSS Repairs
Unhealthy Instance
|
Replace VM Instance
Azure Monitor
Autoscale
|
Percentage CPU > 65%
|
Scale Out
+1 Instance
|
Percentage CPU < 30%
|
Scale In
-1 InstanceArchitecture note: This lab uses an individual public IP for each VM. There is no shared public application endpoint or load balancer. This makes individual-instance troubleshooting easier and provides explicit outbound connectivity for package installation.
In a production web architecture, use Azure Load Balancer or Application Gateway, controlled inbound access, and a properly designed outbound connectivity solution.
Lab 1 — Prepare the Azure Environment
Step 1 — Open Azure Cloud Shell
Open Azure Cloud Shell at:
Select Bash.
Verify your Azure CLI version and authentication:
az versionaz account show \
--query "{Subscription:name, SubscriptionId:id, TenantId:tenantId}" \
--output tableIf required, select the subscription where you want to deploy the workshop:
az account list --output tableaz account set \
--subscription "<YOUR-SUBSCRIPTION-ID>"All remaining commands should be executed in the same Bash session.
Prerequisites
You need:
An active Azure subscription with permission to create compute, networking, and monitoring resources.
VM quota for up to five Standard_D2s_v3 instances (10 vCPUs).
A region supporting Availability Zones 1, 2, and 3.
Azure CLI with the VMSS and Azure Monitor command groups.
Budget to pay for the deployed resources.
This is a billable lab. Running instances, disks, public IPs, and networking resources can generate charges. Delete the resource group when finished.
Step 2 — Configure workshop variables
We will use environment variables to avoid entering resource names repeatedly.
The random suffix allows several participants to run the workshop independently.
# Workshop configuration
export LOCATION="eastus2"
export SUFFIX=$(openssl rand -hex 3)
# Resource names
export RG="rg-vmss-deepdive-$SUFFIX"
export VMSS="vmss-flex-$SUFFIX"
export VNET="vnet-vmss-$SUFFIX"
export SUBNET="snet-web"
export NSG="nsg-vmss-web-$SUFFIX"
export AUTOSCALE="autoscale-vmss-$SUFFIX"
# Compute configuration
export VM_SIZE="Standard_D2s_v3"
export ADMIN_USER="azureuser"
export INITIAL_COUNT=3
export MIN_COUNT=3
export MAX_COUNT=5
# Display configuration
echo "Resource Group: $RG"
echo "VM Scale Set: $VMSS"
echo "Location: $LOCATION"
echo "VM Size: $VM_SIZE"Expected result: The terminal displays the resource group, VMSS name, region, and VM size that will be used throughout the lab.
Step 3 — Verify VM SKU availability
Before deploying, inspect the VM size and supported zones:
az vm list-skus \
--location "$LOCATION" \
--resource-type virtualMachines \
--size "$VM_SIZE" \
--all \
--query "[?name=='${VM_SIZE}'].{SKU:name,Zones:locationInfo[0].zones,Restrictions:restrictions}" \
--output jsonConfirm that zones 1, 2, and 3 are listed and that the selected SKU is available to your subscription.
If the SKU is restricted, choose a suitable alternative and update VM_SIZE. A VM size must be supported in the requested zones for the deployment to succeed.
Step 4 — Create the resource group
az group create \
--name "$RG" \
--location "$LOCATION" \
--tags Environment=Workshop \
Project=Azure-VMSS-DeepDiveVerify deployment:
az group show \
--name "$RG" \
--query "{Name:name,Location:location,State:properties.provisioningState}" \
--output tableExpected result: The provisioning state is Succeeded.
Lab 2 — Configure Azure Networking
We need a virtual network, a subnet, and a Network Security Group before creating the scale set.
Step 1 — Create the virtual network
az network vnet create \
--resource-group "$RG" \
--name "$VNET" \
--location "$LOCATION" \
--address-prefixes 10.42.0.0/16 \
--subnet-name "$SUBNET" \
--subnet-prefixes 10.42.1.0/24Inspect the network:
az network vnet subnet list \
--resource-group "$RG" \
--vnet-name "$VNET" \
--query "[].{Subnet:name,AddressPrefix:addressPrefix}" \
--output tableStep 2 — Create a Network Security Group
az network nsg create \
--resource-group "$RG" \
--name "$NSG" \
--location "$LOCATION"Allow HTTP traffic to the test application:
az network nsg rule create \
--resource-group "$RG" \
--nsg-name "$NSG" \
--name "Allow-HTTP" \
--priority 100 \
--direction Inbound \
--access Allow \
--protocol Tcp \
--source-address-prefixes Internet \
--destination-address-prefixes "*" \
--destination-port-ranges 80Inspect the configured rule:
az network nsg rule list \
--resource-group "$RG" \
--nsg-name "$NSG" \
--query "[].{Name:name,Priority:priority,Access:access,Port:destinationPortRange}" \
--output tableSecurity note: Port 80 is exposed for demonstration purposes only. No credentials or sensitive data should be hosted on these VMs. SSH port 22 is not opened; we will administer the Linux instances using Azure Run Command.
Lab 3 — Prepare the Application with Cloud-Init
Every VM instance needs to run the same application.
Instead of installing software manually, we will use cloud-init to configure NGINX during the first boot.
This ensures that newly created instances automatically receive the application.
Step 1 — Create the cloud-init configuration
Execute the following command in Azure Cloud Shell:
cat > "${VMSS}-cloud-init.yaml" <<'EOF'
#cloud-config
package_update: true
packages:
- nginx
runcmd:
- [bash, -lc, "hostname > /var/www/html/index.html"]
- [bash, -lc, "echo ok > /var/www/html/health"]
- [systemctl, enable, --now, nginx]
EOFInspect the file:
cat "${VMSS}-cloud-init.yaml"The script performs three operations:
Installs NGINX.
Creates a simple website displaying the VM hostname.
Creates /health, an endpoint returning HTTP 200 when NGINX is running.
The health endpoint is especially important because Azure needs an application-level signal to identify unhealthy instances.
Important: VM provisioning success does not necessarily mean cloud-init has finished installing the application. We will verify that separately.
Lab 4 — Deploy a Flexible Virtual Machine Scale Set
Step 1 — Create the scale set
We will deploy three Ubuntu instances across three Availability Zones.
az vmss create \
--resource-group "$RG" \
--name "$VMSS" \
--location "$LOCATION" \
--orchestration-mode Flexible \
--image Ubuntu2204 \
--vm-sku "$VM_SIZE" \
--instance-count "$INITIAL_COUNT" \
--zones 1 2 3 \
--platform-fault-domain-count 1 \
--admin-username "$ADMIN_USER" \
--generate-ssh-keys \
--vnet-name "$VNET" \
--subnet "$SUBNET" \
--nsg "$NSG" \
--public-ip-per-vm \
--load-balancer "" \
--public-ip-address "" \
--custom-data "${VMSS}-cloud-init.yaml" \
--upgrade-policy-mode ManualThis creates:
One Flexible Virtual Machine Scale Set.
Three Ubuntu virtual machines.
VM network interfaces.
Individual VM public IP addresses.
Managed operating system disks.
VM instances distributed across the selected availability zones on a best-effort basis.
The --platform-fault-domain-count 1 parameter is appropriate for this zone-spanning Flexible deployment.
Step 2 — Inspect the scale set
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "{
Name:name,
OrchestrationMode:orchestrationMode,
Capacity:sku.capacity,
VMSize:sku.name,
Zones:zones,
ProvisioningState:provisioningState
}" \
--output jsonExpected result: The orchestration mode is Flexible, capacity is 3, and provisioning state is Succeeded.
Step 3 — Retrieve the scale set resource ID
We will use this ID to identify the associated VM instances.
export VMSS_ID=$(az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query id \
--output tsv)
echo "$VMSS_ID"Step 4 — List VM instances and availability zones
Flexible orchestration exposes instances as regular Azure virtual machines.
az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
Name:name,
Zone:zones[0],
VMSize:hardwareProfile.vmSize,
State:provisioningState
}" \
--output tableThe output should show three VMs and their respective zones.
Azure attempts to distribute the VMs across the configured zones, but exact placement should be verified rather than assumed. Capacity availability can affect placement.
Step 5 — List public IP addresses
az network public-ip list \
--resource-group "$RG" \
--query "[].{Name:name,IPAddress:ipAddress}" \
--output tableRecord the IP addresses.
When NGINX initialization is complete, you can open an address in your browser:
http://<VM-PUBLIC-IP>Each VM should display its own hostname.
Lab 5 — Validate the Application
Before configuring health monitoring, we must confirm that every VM has a working application.
Step 1 — Retrieve the VM names
mapfile -t VMS < <(
az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].name" \
--output tsv
)
printf '%s\n' "${VMS[@]}"Step 2 — Verify NGINX on every instance
Azure Run Command executes shell commands inside Azure VMs without requiring SSH access.
for VM in "${VMS[@]}"; do
echo "Checking VM: $VM"
az vm run-command invoke \
--resource-group "$RG" \
--name "$VM" \
--command-id RunShellScript \
--scripts \
"cloud-init status --wait" \
"systemctl is-active nginx" \
"curl -fsS http://127.0.0.1/health" \
--query "value[0].message" \
--output tsv
doneExpected result:
active
okThese checks confirm that:
Cloud-init has completed.
NGINX is running.
The local application health endpoint responds successfully.
If NGINX is not active, investigate initialization before proceeding.
Lab 6 — Configure Application Health Monitoring
A VM can report a successful Azure provisioning state while the application inside it is failing.
This is why infrastructure health and application health must be treated separately.
We will install the Azure Application Health extension, which probes the local /health endpoint.
For this workshop, we use version 1.0 with binary health states:
Healthy: The HTTP endpoint responds successfully.
Unhealthy: The endpoint fails or cannot be reached.
Step 1 — Create the health extension configuration
cat > "${VMSS}-health.json" <<'EOF'
{
"protocol": "http",
"port": 80,
"requestPath": "/health"
}
EOFStep 2 — Configure the scale set extension model
The scale set model must contain the extension configuration so new instances created from that model can receive it.
az vmss extension set \
--resource-group "$RG" \
--vmss-name "$VMSS" \
--name ApplicationHealthLinux \
--publisher Microsoft.ManagedServices \
--version 1.0 \
--settings "${VMSS}-health.json"Step 3 — Configure existing Flexible VM instances
With Flexible orchestration, individual instances are managed through the regular Azure VM APIs.
We explicitly configure the application health extension on the existing VMs.
for VM in "${VMS[@]}"; do
echo "Configuring health extension: $VM"
az vm extension set \
--resource-group "$RG" \
--vm-name "$VM" \
--name ApplicationHealthLinux \
--publisher Microsoft.ManagedServices \
--version 1.0 \
--settings "${VMSS}-health.json"
doneStep 4 — Verify extension provisioning
for VM in "${VMS[@]}"; do
echo "VM: $VM"
az vm extension show \
--resource-group "$RG" \
--vm-name "$VM" \
--name ApplicationHealthLinux \
--query "{
Extension:name,
Publisher:publisher,
ProvisioningState:provisioningState
}" \
--output table
doneExpected result: The extension provisioning state is Succeeded for each instance.
Remember that extension installation success is not the same as the application being healthy. We have already verified that the endpoint responds, and later we will deliberately cause it to fail.
Lab 7 — Configure Azure Monitor Autoscaling
Now we will configure the scale set to increase and decrease its instance count automatically.
Our scaling rules will be:
Condition | Action |
Average CPU exceeds 65% for 5 minutes | Add one VM |
Average CPU drops below 30% for 5 minutes | Remove one VM |
Minimum capacity | 3 instances |
Default capacity | 3 instances |
Maximum capacity | 5 instances |
The scaling limits prevent the lab from continuously creating more VMs.
Step 1 — Create the autoscale setting
az monitor autoscale create \
--resource-group "$RG" \
--name "$AUTOSCALE" \
--resource "$VMSS_ID" \
--min-count "$MIN_COUNT" \
--max-count "$MAX_COUNT" \
--count "$INITIAL_COUNT"Step 2 — Configure scale-out
When average CPU utilization exceeds 65% for five minutes, Azure Monitor should add one instance.
az monitor autoscale rule create \
--resource-group "$RG" \
--autoscale-name "$AUTOSCALE" \
--condition "Percentage CPU > 65 avg 5m" \
--scale out 1Step 3 — Configure scale-in
When CPU utilization drops below 30% for five minutes, Azure Monitor should remove one instance.
az monitor autoscale rule create \
--resource-group "$RG" \
--autoscale-name "$AUTOSCALE" \
--condition "Percentage CPU < 30 avg 5m" \
--scale in 1Step 4 — Review the configuration
az monitor autoscale show \
--resource-group "$RG" \
--name "$AUTOSCALE" \
--query "{
Name:name,
Enabled:enabled,
Profiles:profiles
}" \
--output jsonVerify that both rules exist and the configured capacity limits match our workshop.
How autoscaling works: Azure Monitor evaluates metrics periodically. A threshold crossing does not immediately create or delete a VM. Metric aggregation, evaluation intervals, cooldown behavior, instance provisioning, and available capacity influence how quickly changes occur.
Lab 8 — Simulate High CPU Utilization
This is where the workshop becomes more interesting.
Instead of manually changing the number of instances, we will generate actual CPU load and let Azure Monitor react.
Step 1 — Generate CPU load
We will use the Linux yes command to create sustained CPU activity.
The workload runs for a maximum of 15 minutes and automatically terminates.
for VM in "${VMS[@]}"; do
echo "Starting CPU load on: $VM"
az vm run-command invoke \
--resource-group "$RG" \
--name "$VM" \
--command-id RunShellScript \
--scripts \
"for i in 1 2 3 4; do nohup timeout 900 yes > /dev/null 2>&1 < /dev/null & done; echo 'CPU load started'" \
--query "value[0].message" \
--output tsv
doneThis starts four CPU-intensive processes on each initial VM.
The background processes produce continuous CPU load, making it more likely that the average CPU metric will cross the configured threshold.
Step 2 — Monitor the CPU metric
First, identify a VM:
export FIRST_VM="${VMS[0]}"
export FIRST_VM_ID=$(az vm show \
--resource-group "$RG" \
--name "$FIRST_VM" \
--query id \
--output tsv)Inspect its CPU metrics:
az monitor metrics list \
--resource "$FIRST_VM_ID" \
--metric "Percentage CPU" \
--interval PT1M \
--aggregation Average \
--query "value[0].timeseries[0].data[-10:].{
Timestamp:timeStamp,
AverageCPU:average
}" \
--output tableExpected behavior: The CPU metric rises while the workload is running. Metrics may take a few minutes to become available.
Step 3 — Inspect the scale set capacity
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "sku.capacity" \
--output tsvInitially, the result should be:
3If the scale-out rule triggers successfully, the capacity should increase to 4 and potentially 5, depending on sustained CPU usage.
Step 4 — Monitor instance creation
az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
Name:name,
Zone:zones[0],
State:provisioningState
}" \
--output tableRun this command periodically to observe newly created VMs.
You can also compare the scale set capacity before and after the load test.
Step 5 — Verify new instances
New instances should receive the NGINX application from cloud-init and the health extension configuration from the scale set model.
Refresh your instance list:
mapfile -t ALL_VMS < <(
az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].name" \
--output tsv
)
printf '%s\n' "${ALL_VMS[@]}"Check the application health extension on every current instance:
for VM in "${ALL_VMS[@]}"; do
echo "Checking extension on: $VM"
az vm extension show \
--resource-group "$RG" \
--vm-name "$VM" \
--name ApplicationHealthLinux \
--query "{Name:name,State:provisioningState}" \
--output table
doneIf a new instance is missing the extension, investigate its provisioning and scale set model rather than assuming automatic repairs can monitor it.
What did we learn?
Autoscaling is not just increasing an instance count.
Successful scale-out requires the new VM to:
Provision successfully.
Configure its operating system.
Install and start the application.
Become healthy.
Be available to handle workload.
A VM that exists but cannot serve traffic does not provide useful additional capacity.
Lab 9 — Test Scale-In
Now we will remove the artificial CPU workload.
Step 1 — Stop CPU load
for VM in "${VMS[@]}"; do
echo "Stopping CPU load on: $VM"
az vm run-command invoke \
--resource-group "$RG" \
--name "$VM" \
--command-id RunShellScript \
--scripts "pkill -x yes || true" \
--query "value[0].message" \
--output tsv
doneThe load-generation processes have been terminated. Any remaining processes from the original test will also stop when their 15-minute timeout expires.
Step 2 — Observe reduced CPU usage
az monitor metrics list \
--resource "$FIRST_VM_ID" \
--metric "Percentage CPU" \
--interval PT1M \
--aggregation Average \
--query "value[0].timeseries[0].data[-10:].{
Timestamp:timeStamp,
AverageCPU:average
}" \
--output tableAfter enough low-CPU measurements, Azure Monitor should trigger scale-in.
Step 3 — Monitor scale-in
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "{Name:name,Capacity:sku.capacity}" \
--output tableThe scale set should eventually return to the minimum capacity of three instances, provided the scale-in conditions are satisfied.
Scale-in can take longer than the five-minute metric window because of cooldown periods and Azure Monitor's safeguards against unstable scaling behavior.
Lab 10 — Enable Automatic Instance Repairs
We have demonstrated that the platform can react to demand.
Now we will test whether it can react to an application failure.
Azure Virtual Machine Scale Sets can monitor unhealthy instances and automatically trigger repair actions.
The supported repair actions include:
Repair action | Behavior |
Replace | Deletes the unhealthy VM and creates a replacement from the scale set model |
Reimage | Recreates the OS environment from the image |
Restart | Restarts the unhealthy VM |
For this workshop, we will use Replace.
This is important because Replace demonstrates an infrastructure recovery operation rather than simply restarting an application process.
Step 1 — Disable autoscaling for the repair exercise
First, prevent autoscale operations from interfering with the failure simulation.
az monitor autoscale update \
--resource-group "$RG" \
--name "$AUTOSCALE" \
--enabled falseReturn the scale set to three instances if necessary:
az vmss scale \
--resource-group "$RG" \
--name "$VMSS" \
--new-capacity 3Verify the instance count:
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "sku.capacity" \
--output tsvStep 2 — Enable automatic repairs
az vmss update \
--resource-group "$RG" \
--name "$VMSS" \
--enable-automatic-repairs true \
--automatic-repairs-grace-period 30 \
--automatic-repairs-action ReplaceImportant: The 30-minute grace period prevents Azure from prematurely repairing an instance immediately after certain VM or scale set state changes. Automatic repair is not an instantaneous failover mechanism.
Step 3 — Verify the repair policy
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "automaticRepairsPolicy" \
--output jsonThe configuration should indicate that repairs are enabled, with the configured grace period and Replace action.
Step 4 — Verify the repair orchestration service
az vmss get-instance-view \
--resource-group "$RG" \
--name "$VMSS" \
--query "orchestrationServices" \
--output jsonLook for the AutomaticRepairs service.
A running service indicates that automatic repair orchestration is active. If the service is suspended, investigate its state before proceeding.
Lab 11 — Simulate an Unhealthy VM
We will deliberately stop NGINX on one scale set instance.
The VM itself will continue running, but its application will become unavailable.
This demonstrates why application-level health monitoring is essential.
Step 1 — Select a victim instance
export VICTIM=$(az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'] | [0].name" \
--output tsv)
echo "Selected victim: $VICTIM"Step 2 — Record its VM identity
Before causing the failure, record the current VM resource identity.
export VICTIM_ID=$(az vm show \
--resource-group "$RG" \
--name "$VICTIM" \
--query vmId \
--output tsv)
echo "Original VM ID: $VICTIM_ID"This allows us to determine whether Azure subsequently replaced the instance.
Step 3 — Stop NGINX
az vm run-command invoke \
--resource-group "$RG" \
--name "$VICTIM" \
--command-id RunShellScript \
--scripts \
"systemctl stop nginx" \
"systemctl is-active nginx || true" \
"curl -sS http://127.0.0.1/health || true" \
--query "value[0].message" \
--output tsvExpected behavior:
NGINX is no longer active.
The /health endpoint is unavailable.
The VM operating system continues running.
The Application Health extension detects the failing endpoint.
This is an application failure rather than a simulated Azure platform outage.
Step 4 — Check that the VM still exists
az vm get-instance-view \
--resource-group "$RG" \
--name "$VICTIM" \
--query "instanceView.statuses" \
--output jsonThe VM can remain in a running state even though its application is unhealthy.
That is the distinction we want to demonstrate.
Lab 12 — Observe Automatic Recovery
Once the Application Health extension reports the instance as unhealthy and the repair grace period has been satisfied, Azure can initiate the configured Replace operation.
Depending on the state of the environment, this process can take a significant amount of time.
Step 1 — Inspect the repair orchestration service
az vmss get-instance-view \
--resource-group "$RG" \
--name "$VMSS" \
--query "orchestrationServices" \
--output jsonIf automatic repairs are active, the service should report an appropriate running state.
Step 2 — Inspect the Azure Activity Log
az monitor activity-log list \
--resource-group "$RG" \
--offset 2h \
--query "[].{
Timestamp:eventTimestamp,
Operation:operationName.localizedValue,
Status:status.localizedValue
}" \
--output tableLook for operations involving VM deletion, creation, or changes to the scale set.
Activity Log entries may be associated with the individual VMs rather than exclusively with the parent scale set.
Step 3 — List current VM instances
az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
Name:name,
VMId:vmId,
Zone:zones[0],
State:provisioningState
}" \
--output tableA successful Replace operation should remove the unhealthy instance and create another VM to maintain the desired scale set capacity.
Step 4 — Compare the original VM identity
CURRENT_IDS=$(az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].vmId" \
--output tsv)
if printf '%s\n' "$CURRENT_IDS" | grep -qFx "$VICTIM_ID"; then
echo "Original VM still exists."
else
echo "Original VM ID no longer exists in the scale set."
fiInterpretation:
If the original VM ID is no longer present, Azure has removed that instance. Together with the Activity Log and the appearance of a new VM, this provides evidence of a replacement.
Do not interpret a missing VM ID alone as proof of automatic repair. Confirm that the replacement was initiated by the repair mechanism rather than by manual scaling or another operation.
Step 5 — Verify the final capacity
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "{
Name:name,
Capacity:sku.capacity,
RepairPolicy:automaticRepairsPolicy
}" \
--output jsonThe desired capacity should remain three.
Step 6 — Validate recovered application health
Refresh the current VM names:
mapfile -t ALL_VMS < <(
az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].name" \
--output tsv
)Verify the NGINX application on all current instances:
for VM in "${ALL_VMS[@]}"; do
echo "Validating: $VM"
az vm run-command invoke \
--resource-group "$RG" \
--name "$VM" \
--command-id RunShellScript \
--scripts \
"cloud-init status --wait" \
"systemctl is-active nginx" \
"curl -fsS http://127.0.0.1/health" \
--query "value[0].message" \
--output tsv
doneExpected result: Every running instance returns a healthy application response.
This is the central objective of the workshop: Azure should recover the required VM capacity, and the replacement VM should deploy the application correctly without manual software installation.
Lab 13 — Explore Availability Zones
Azure Availability Zones provide physically separate locations within an Azure region, each with independent power, cooling, and networking infrastructure.
A zone-spanning VMSS distributes instances across the configured zones to reduce the impact of failures affecting a single zone.
However, Availability Zones are not the same as multi-region disaster recovery.
Step 1 — Review the zone configuration
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "{
Name:name,
Zones:zones,
Capacity:sku.capacity
}" \
--output jsonStep 2 — Check actual zone placement
az vm list \
--resource-group "$RG" \
--query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
VM:name,
AvailabilityZone:zones[0]
}" \
--output tableCompare the placement before and after scale-out and automatic repair.
Operational questions
Were all three Availability Zones represented?
Did the replacement VM use the same zone as the original?
Could a scale-out operation place another VM in an already populated zone?
What happens if capacity is unavailable in one zone?
Does distributing instances across zones guarantee application availability?
Key lesson: Availability Zone distribution helps improve resilience, but reliable application availability also depends on traffic routing, health-based load balancing, application state, and other dependencies.
Lab 14 — Troubleshooting Scenarios
Now that the workshop is complete, let's examine common operational issues.
Scenario 1 — Autoscaling does not start
Symptoms: CPU is high, but the VM count remains unchanged.
Check the autoscale configuration:
az monitor autoscale show \
--resource-group "$RG" \
--name "$AUTOSCALE" \
--output jsonVerify CPU metrics:
az monitor metrics list \
--resource "$FIRST_VM_ID" \
--metric "Percentage CPU" \
--aggregation Average \
--output jsonPossible causes include:
The rule's evaluation window has not been satisfied.
Autoscaling is disabled.
New VMs cannot be created because of compute quota restrictions.
The requested VM size is unavailable in the selected region or zone.
The VMSS has already reached its maximum capacity.
The observed per-VM metric differs from the aggregate used by autoscale.
Remember that we intentionally disabled autoscale before the automatic repair test.
Scenario 2 — The application is unhealthy after deployment
Check the VM configuration:
az vm run-command invoke \
--resource-group "$RG" \
--name "$VICTIM" \
--command-id RunShellScript \
--scripts \
"systemctl status nginx --no-pager || true" \
"curl -v http://127.0.0.1/health || true" \
--query "value[0].message" \
--output tsvRun this only if the selected VM still exists. If it has already been replaced, select a current VM instead.
Possible causes include:
NGINX did not start.
Cloud-init failed or has not completed.
The health endpoint does not exist.
The health extension is configured with the wrong port or path.
Scenario 3 — Automatic repair does not occur
Inspect the policy:
az vmss show \
--resource-group "$RG" \
--name "$VMSS" \
--query "automaticRepairsPolicy" \
--output jsonInspect the orchestration service:
az vmss get-instance-view \
--resource-group "$RG" \
--name "$VMSS" \
--query "orchestrationServices" \
--output jsonPossible causes include:
The instance is still inside its repair grace period.
The Application Health extension was not installed correctly.
The health endpoint is still returning successful responses.
The repair service was suspended.
The VM failed initial provisioning and therefore never became eligible for application-health-based repair.
Replacement VM creation is blocked by quota or capacity limits.
If repair orchestration is suspended, first resolve the underlying issue. You can then resume it:
az vmss set-orchestration-service-state \
--resource-group "$RG" \
--name "$VMSS" \
--service-name AutomaticRepairs \
--action ResumeScenario 4 — New instances are created without a working application
Verify cloud-init execution on an affected instance:
az vm run-command invoke \
--resource-group "$RG" \
--name "<CURRENT-VM-NAME>" \
--command-id RunShellScript \
--scripts \
"cloud-init status --long" \
"tail -n 50 /var/log/cloud-init-output.log" \
--query "value[0].message" \
--output tsvReplace <CURRENT-VM-NAME> with an existing VM.
Possible causes include package installation failures, missing outbound connectivity, repository availability, or invalid cloud-init commands.
Lab 15 — Cleanup
After completing the workshop, delete the Azure resources to avoid unnecessary charges.
Step 1 — Review deployed resources
az resource list \
--resource-group "$RG" \
--query "[].{Name:name,Type:type}" \
--output tableStep 2 — Delete the resource group
az group delete \
--name "$RG" \
--yesThis removes the lab's VMSS, VMs, disks, virtual network, NSG, public IP addresses, autoscale settings, and other resources contained in the resource group.
Step 3 — Verify deletion
az group exists \
--name "$RG"Expected result:
falseStep 4 — Remove locally generated files
rm -f "${VMSS}-cloud-init.yaml"
rm -f "${VMSS}-health.json"The workshop environment is now cleaned up.
Workshop Review — What Have We Learned?
Let's summarize the main concepts.
Azure feature | What we demonstrated |
Flexible orchestration | Standard Azure VM lifecycle management within a scale set |
Availability Zones | Distribution of VM instances across zones |
Cloud-init | Automatic application provisioning |
Azure Monitor Autoscale | CPU-based scale-out and scale-in |
Azure Monitor Metrics | Observability of VM performance |
Application Health extension | Application-level health monitoring |
Automatic Instance Repairs | Replacement of unhealthy VM instances |
Azure Activity Log | Investigation of VM lifecycle operations |
Azure CLI | End-to-end deployment, troubleshooting, and cleanup |
Flexible vs. Uniform orchestration
One of the most important concepts is the difference between Flexible and Uniform orchestration.
Flexible orchestration manages standard Azure IaaS virtual machines. It offers more flexibility when working with individual VM resources and supports scaling, availability features, and automatic repairs.
Uniform orchestration is designed around a more standardized VM instance model and uses scale-set-specific instance management APIs.
For new deployments, Flexible orchestration is generally Microsoft's recommended option, provided it supports the workload's requirements.
Autoscaling vs. Auto-Healing
These two concepts solve different problems.
Autoscaling adjusts compute capacity in response to demand.
Auto-healing responds to unhealthy instances and attempts to restore their functionality or replace them.
An application can require both.
For example, adding more VMs is not helpful if the new VMs cannot start the application correctly. Likewise, automatically replacing a failed VM does not solve a sustained capacity shortage.
The combination of application-aware monitoring, automated scaling, and instance repairs is what makes this architecture operationally interesting.
Important production considerations
This workshop deliberately simplifies the architecture.
For a production deployment, consider:
Using Azure Load Balancer or Application Gateway as a shared application entry point.
Keeping application VMs private whenever possible.
Using a NAT Gateway or another appropriate controlled outbound connectivity solution.
Using Azure Monitor alerts and Log Analytics for operational visibility.
Implementing controlled deployment and application-update strategies.
Using managed identities instead of embedded credentials.
Treating VMs as replaceable infrastructure and keeping important application data outside the VM's local OS disk.
Designing recovery objectives and considering multi-region disaster recovery separately.
Automatic VM replacement should not be confused with guaranteed uninterrupted application availability. It is an infrastructure recovery mechanism, and a Replace operation does not preserve the original VM's local state.
Final Challenge — Test Your Understanding
Before deleting the environment, try answering these questions:
What is the difference between VM provisioning state and application health?
Why is cloud-init important when Azure automatically creates a replacement VM?
What happens if one instance reaches 100% CPU while the other instances remain idle?
Why might a VMSS fail to scale out even when the CPU threshold has been exceeded?
What is the purpose of the automatic repair grace period?
How does Azure decide whether to replace an unhealthy VM?
Why does a Flexible VMSS require a different approach to individual-instance management than a Uniform VMSS?
Does distributing VMs across three Availability Zones eliminate the need for disaster recovery?
What happens to a VM's local data when Azure uses the Replace repair action?
How would you redesign the lab to expose one highly available public web endpoint?
If you can answer these questions and explain the behavior you observed during the lab, you have gone beyond simply deploying a Virtual Machine Scale Set.
You have started thinking about the operational responsibilities of running scalable workloads in Microsoft Azure.
References
Official Microsoft documentation used for the concepts and CLI configuration:
Workshop: Azure VM Scale Sets Deep Dive — Flexible Orchestration, Autoscaling, Health Monitoring & Self-Healing
Beyond Cloud with Chriz — Azure Workshopz
Build. Break. Troubleshoot. Understand.
Publication note: The commands are based on Microsoft's documented Azure CLI interfaces and intended for Azure Cloud Shell Bash. They have not been executed against a live subscription in this conversation. Validate the deployment in a test subscription before publishing it as a fully tested lab, particularly the Flexible VMSS extension model and automatic repair lifecycle.
Comments