top of page
11 minutes ago
20 min read

Azure VM Scale Sets Deep Dive — Flexible Orchestration, Autoscaling, Health Monitoring & Self-Healing

Build it. Scale it. Break it. Repair it. Verify it.

Azure Workshopz | Beyond Cloud with Chriz

Level: Intermediate to AdvancedPlatform: Microsoft AzureTools: Azure CLI, Azure Cloud Shell (Bash)Deployment: Azure CLI only — no Terraform, Bicep, or Azure Portal requiredOperating System: Ubuntu Server 22.04 LTSTopics: Compute, Availability Zones, Autoscaling, Monitoring, High Availability, Disaster Recovery Concepts

Introduction

Deploying a virtual machine in Microsoft Azure is straightforward. But what happens when your application experiences an unexpected increase in CPU utilization? How do you automatically provision additional compute capacity? And what happens when an instance becomes unhealthy?

This workshop explores Azure Virtual Machine Scale Sets (VMSS) from an operational perspective.

We will deploy a Flexible Virtual Machine Scale Set across Azure Availability Zones, configure automatic scaling based on CPU utilization, monitor application health, deliberately break an application, and observe Azure automatically replacing the unhealthy instance.

The focus is not just deploying resources. It is understanding what happens when infrastructure needs to scale, recover, and remain operational.

What you will learn

By the end of this workshop, you will be able to:

  1. Deploy an Azure Virtual Machine Scale Set using Flexible orchestration.

  2. Distribute instances across Azure Availability Zones.

  3. Deploy a simple NGINX application automatically using cloud-init.

  4. Configure Azure Monitor autoscaling rules.

  5. Generate CPU load and trigger scale-out operations.

  6. Monitor VM metrics, instance states, and scaling actions.

  7. Configure the Azure Application Health extension.

  8. Enable automatic instance repairs.

  9. Simulate an unhealthy application and verify automatic replacement.

  10. Clean up the complete environment using Azure CLI.

Workshop architecture

We will create the following environment:

                  Azure Subscription
                         |
                  Resource Group
                         |
                 Virtual Network
                   10.42.0.0/16
                         |
                  Web Subnet
                   10.42.1.0/24
                         |
              Azure VM Scale Set
              Flexible Orchestration
                         |
             +-----------+-----------+
             |           |           |
          Zone 1      Zone 2      Zone 3
             |           |           |
           VM-01       VM-02       VM-03
           Ubuntu      Ubuntu      Ubuntu
           NGINX       NGINX       NGINX
             |           |           |
         /health     /health     /health
             |           |           |
             +-----------+-----------+
                         |
                Application Health
                    Extension
                         |
                Azure VMSS Repairs
                Unhealthy Instance
                         |
                Replace VM Instance


                 Azure Monitor
                     Autoscale
                         |
              Percentage CPU > 65%
                         |
                    Scale Out
                    +1 Instance
                         |
              Percentage CPU < 30%
                         |
                    Scale In
                    -1 Instance

Architecture note: This lab uses an individual public IP for each VM. There is no shared public application endpoint or load balancer. This makes individual-instance troubleshooting easier and provides explicit outbound connectivity for package installation.

In a production web architecture, use Azure Load Balancer or Application Gateway, controlled inbound access, and a properly designed outbound connectivity solution.

Lab 1 — Prepare the Azure Environment

Step 1 — Open Azure Cloud Shell

Open Azure Cloud Shell at:

Select Bash.

Verify your Azure CLI version and authentication:

az version
az account show \
  --query "{Subscription:name, SubscriptionId:id, TenantId:tenantId}" \
  --output table

If required, select the subscription where you want to deploy the workshop:

az account list --output table
az account set \
  --subscription "<YOUR-SUBSCRIPTION-ID>"

All remaining commands should be executed in the same Bash session.

Prerequisites

You need:

  • An active Azure subscription with permission to create compute, networking, and monitoring resources.

  • VM quota for up to five Standard_D2s_v3 instances (10 vCPUs).

  • A region supporting Availability Zones 1, 2, and 3.

  • Azure CLI with the VMSS and Azure Monitor command groups.

  • Budget to pay for the deployed resources.

This is a billable lab. Running instances, disks, public IPs, and networking resources can generate charges. Delete the resource group when finished.

Step 2 — Configure workshop variables

We will use environment variables to avoid entering resource names repeatedly.

The random suffix allows several participants to run the workshop independently.

# Workshop configuration
export LOCATION="eastus2"
export SUFFIX=$(openssl rand -hex 3)

# Resource names
export RG="rg-vmss-deepdive-$SUFFIX"
export VMSS="vmss-flex-$SUFFIX"
export VNET="vnet-vmss-$SUFFIX"
export SUBNET="snet-web"
export NSG="nsg-vmss-web-$SUFFIX"
export AUTOSCALE="autoscale-vmss-$SUFFIX"

# Compute configuration
export VM_SIZE="Standard_D2s_v3"
export ADMIN_USER="azureuser"
export INITIAL_COUNT=3
export MIN_COUNT=3
export MAX_COUNT=5

# Display configuration
echo "Resource Group: $RG"
echo "VM Scale Set:   $VMSS"
echo "Location:       $LOCATION"
echo "VM Size:        $VM_SIZE"

Expected result: The terminal displays the resource group, VMSS name, region, and VM size that will be used throughout the lab.

Step 3 — Verify VM SKU availability

Before deploying, inspect the VM size and supported zones:

az vm list-skus \
  --location "$LOCATION" \
  --resource-type virtualMachines \
  --size "$VM_SIZE" \
  --all \
  --query "[?name=='${VM_SIZE}'].{SKU:name,Zones:locationInfo[0].zones,Restrictions:restrictions}" \
  --output json

Confirm that zones 1, 2, and 3 are listed and that the selected SKU is available to your subscription.

If the SKU is restricted, choose a suitable alternative and update VM_SIZE. A VM size must be supported in the requested zones for the deployment to succeed.

Step 4 — Create the resource group

az group create \
  --name "$RG" \
  --location "$LOCATION" \
  --tags Environment=Workshop \
         Project=Azure-VMSS-DeepDive

Verify deployment:

az group show \
  --name "$RG" \
  --query "{Name:name,Location:location,State:properties.provisioningState}" \
  --output table

Expected result: The provisioning state is Succeeded.

Lab 2 — Configure Azure Networking

We need a virtual network, a subnet, and a Network Security Group before creating the scale set.

Step 1 — Create the virtual network

az network vnet create \
  --resource-group "$RG" \
  --name "$VNET" \
  --location "$LOCATION" \
  --address-prefixes 10.42.0.0/16 \
  --subnet-name "$SUBNET" \
  --subnet-prefixes 10.42.1.0/24

Inspect the network:

az network vnet subnet list \
  --resource-group "$RG" \
  --vnet-name "$VNET" \
  --query "[].{Subnet:name,AddressPrefix:addressPrefix}" \
  --output table

Step 2 — Create a Network Security Group

az network nsg create \
  --resource-group "$RG" \
  --name "$NSG" \
  --location "$LOCATION"

Allow HTTP traffic to the test application:

az network nsg rule create \
  --resource-group "$RG" \
  --nsg-name "$NSG" \
  --name "Allow-HTTP" \
  --priority 100 \
  --direction Inbound \
  --access Allow \
  --protocol Tcp \
  --source-address-prefixes Internet \
  --destination-address-prefixes "*" \
  --destination-port-ranges 80

Inspect the configured rule:

az network nsg rule list \
  --resource-group "$RG" \
  --nsg-name "$NSG" \
  --query "[].{Name:name,Priority:priority,Access:access,Port:destinationPortRange}" \
  --output table

Security note: Port 80 is exposed for demonstration purposes only. No credentials or sensitive data should be hosted on these VMs. SSH port 22 is not opened; we will administer the Linux instances using Azure Run Command.

Lab 3 — Prepare the Application with Cloud-Init

Every VM instance needs to run the same application.

Instead of installing software manually, we will use cloud-init to configure NGINX during the first boot.

This ensures that newly created instances automatically receive the application.

Step 1 — Create the cloud-init configuration

Execute the following command in Azure Cloud Shell:

cat > "${VMSS}-cloud-init.yaml" <<'EOF'
#cloud-config

package_update: true

packages:
  - nginx

runcmd:
  - [bash, -lc, "hostname > /var/www/html/index.html"]
  - [bash, -lc, "echo ok > /var/www/html/health"]
  - [systemctl, enable, --now, nginx]
EOF

Inspect the file:

cat "${VMSS}-cloud-init.yaml"

The script performs three operations:

  1. Installs NGINX.

  2. Creates a simple website displaying the VM hostname.

  3. Creates /health, an endpoint returning HTTP 200 when NGINX is running.

The health endpoint is especially important because Azure needs an application-level signal to identify unhealthy instances.

Important: VM provisioning success does not necessarily mean cloud-init has finished installing the application. We will verify that separately.

Lab 4 — Deploy a Flexible Virtual Machine Scale Set

Step 1 — Create the scale set

We will deploy three Ubuntu instances across three Availability Zones.

az vmss create \
  --resource-group "$RG" \
  --name "$VMSS" \
  --location "$LOCATION" \
  --orchestration-mode Flexible \
  --image Ubuntu2204 \
  --vm-sku "$VM_SIZE" \
  --instance-count "$INITIAL_COUNT" \
  --zones 1 2 3 \
  --platform-fault-domain-count 1 \
  --admin-username "$ADMIN_USER" \
  --generate-ssh-keys \
  --vnet-name "$VNET" \
  --subnet "$SUBNET" \
  --nsg "$NSG" \
  --public-ip-per-vm \
  --load-balancer "" \
  --public-ip-address "" \
  --custom-data "${VMSS}-cloud-init.yaml" \
  --upgrade-policy-mode Manual

This creates:

  • One Flexible Virtual Machine Scale Set.

  • Three Ubuntu virtual machines.

  • VM network interfaces.

  • Individual VM public IP addresses.

  • Managed operating system disks.

  • VM instances distributed across the selected availability zones on a best-effort basis.

The --platform-fault-domain-count 1 parameter is appropriate for this zone-spanning Flexible deployment.

Step 2 — Inspect the scale set

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "{
    Name:name,
    OrchestrationMode:orchestrationMode,
    Capacity:sku.capacity,
    VMSize:sku.name,
    Zones:zones,
    ProvisioningState:provisioningState
  }" \
  --output json

Expected result: The orchestration mode is Flexible, capacity is 3, and provisioning state is Succeeded.

Step 3 — Retrieve the scale set resource ID

We will use this ID to identify the associated VM instances.

export VMSS_ID=$(az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query id \
  --output tsv)

echo "$VMSS_ID"

Step 4 — List VM instances and availability zones

Flexible orchestration exposes instances as regular Azure virtual machines.

az vm list \
  --resource-group "$RG" \
  --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
    Name:name,
    Zone:zones[0],
    VMSize:hardwareProfile.vmSize,
    State:provisioningState
  }" \
  --output table

The output should show three VMs and their respective zones.

Azure attempts to distribute the VMs across the configured zones, but exact placement should be verified rather than assumed. Capacity availability can affect placement.

Step 5 — List public IP addresses

az network public-ip list \
  --resource-group "$RG" \
  --query "[].{Name:name,IPAddress:ipAddress}" \
  --output table

Record the IP addresses.

When NGINX initialization is complete, you can open an address in your browser:

http://<VM-PUBLIC-IP>

Each VM should display its own hostname.

Lab 5 — Validate the Application

Before configuring health monitoring, we must confirm that every VM has a working application.

Step 1 — Retrieve the VM names

mapfile -t VMS < <(
  az vm list \
    --resource-group "$RG" \
    --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].name" \
    --output tsv
)

printf '%s\n' "${VMS[@]}"

Step 2 — Verify NGINX on every instance

Azure Run Command executes shell commands inside Azure VMs without requiring SSH access.

for VM in "${VMS[@]}"; do
  echo "Checking VM: $VM"

  az vm run-command invoke \
    --resource-group "$RG" \
    --name "$VM" \
    --command-id RunShellScript \
    --scripts \
      "cloud-init status --wait" \
      "systemctl is-active nginx" \
      "curl -fsS http://127.0.0.1/health" \
    --query "value[0].message" \
    --output tsv
done

Expected result:

active
ok

These checks confirm that:

  • Cloud-init has completed.

  • NGINX is running.

  • The local application health endpoint responds successfully.

If NGINX is not active, investigate initialization before proceeding.

Lab 6 — Configure Application Health Monitoring

A VM can report a successful Azure provisioning state while the application inside it is failing.

This is why infrastructure health and application health must be treated separately.

We will install the Azure Application Health extension, which probes the local /health endpoint.

For this workshop, we use version 1.0 with binary health states:

  • Healthy: The HTTP endpoint responds successfully.

  • Unhealthy: The endpoint fails or cannot be reached.

Step 1 — Create the health extension configuration

cat > "${VMSS}-health.json" <<'EOF'
{
  "protocol": "http",
  "port": 80,
  "requestPath": "/health"
}
EOF

Step 2 — Configure the scale set extension model

The scale set model must contain the extension configuration so new instances created from that model can receive it.

az vmss extension set \
  --resource-group "$RG" \
  --vmss-name "$VMSS" \
  --name ApplicationHealthLinux \
  --publisher Microsoft.ManagedServices \
  --version 1.0 \
  --settings "${VMSS}-health.json"

Step 3 — Configure existing Flexible VM instances

With Flexible orchestration, individual instances are managed through the regular Azure VM APIs.

We explicitly configure the application health extension on the existing VMs.

for VM in "${VMS[@]}"; do
  echo "Configuring health extension: $VM"

  az vm extension set \
    --resource-group "$RG" \
    --vm-name "$VM" \
    --name ApplicationHealthLinux \
    --publisher Microsoft.ManagedServices \
    --version 1.0 \
    --settings "${VMSS}-health.json"
done

Step 4 — Verify extension provisioning

for VM in "${VMS[@]}"; do
  echo "VM: $VM"

  az vm extension show \
    --resource-group "$RG" \
    --vm-name "$VM" \
    --name ApplicationHealthLinux \
    --query "{
      Extension:name,
      Publisher:publisher,
      ProvisioningState:provisioningState
    }" \
    --output table
done

Expected result: The extension provisioning state is Succeeded for each instance.

Remember that extension installation success is not the same as the application being healthy. We have already verified that the endpoint responds, and later we will deliberately cause it to fail.

Lab 7 — Configure Azure Monitor Autoscaling

Now we will configure the scale set to increase and decrease its instance count automatically.

Our scaling rules will be:

Condition

Action

Average CPU exceeds 65% for 5 minutes

Add one VM

Average CPU drops below 30% for 5 minutes

Remove one VM

Minimum capacity

3 instances

Default capacity

3 instances

Maximum capacity

5 instances

The scaling limits prevent the lab from continuously creating more VMs.

Step 1 — Create the autoscale setting

az monitor autoscale create \
  --resource-group "$RG" \
  --name "$AUTOSCALE" \
  --resource "$VMSS_ID" \
  --min-count "$MIN_COUNT" \
  --max-count "$MAX_COUNT" \
  --count "$INITIAL_COUNT"

Step 2 — Configure scale-out

When average CPU utilization exceeds 65% for five minutes, Azure Monitor should add one instance.

az monitor autoscale rule create \
  --resource-group "$RG" \
  --autoscale-name "$AUTOSCALE" \
  --condition "Percentage CPU > 65 avg 5m" \
  --scale out 1

Step 3 — Configure scale-in

When CPU utilization drops below 30% for five minutes, Azure Monitor should remove one instance.

az monitor autoscale rule create \
  --resource-group "$RG" \
  --autoscale-name "$AUTOSCALE" \
  --condition "Percentage CPU < 30 avg 5m" \
  --scale in 1

Step 4 — Review the configuration

az monitor autoscale show \
  --resource-group "$RG" \
  --name "$AUTOSCALE" \
  --query "{
    Name:name,
    Enabled:enabled,
    Profiles:profiles
  }" \
  --output json

Verify that both rules exist and the configured capacity limits match our workshop.

How autoscaling works: Azure Monitor evaluates metrics periodically. A threshold crossing does not immediately create or delete a VM. Metric aggregation, evaluation intervals, cooldown behavior, instance provisioning, and available capacity influence how quickly changes occur.

Lab 8 — Simulate High CPU Utilization

This is where the workshop becomes more interesting.

Instead of manually changing the number of instances, we will generate actual CPU load and let Azure Monitor react.

Step 1 — Generate CPU load

We will use the Linux yes command to create sustained CPU activity.

The workload runs for a maximum of 15 minutes and automatically terminates.

for VM in "${VMS[@]}"; do
  echo "Starting CPU load on: $VM"

  az vm run-command invoke \
    --resource-group "$RG" \
    --name "$VM" \
    --command-id RunShellScript \
    --scripts \
      "for i in 1 2 3 4; do nohup timeout 900 yes > /dev/null 2>&1 < /dev/null & done; echo 'CPU load started'" \
    --query "value[0].message" \
    --output tsv
done

This starts four CPU-intensive processes on each initial VM.

The background processes produce continuous CPU load, making it more likely that the average CPU metric will cross the configured threshold.

Step 2 — Monitor the CPU metric

First, identify a VM:

export FIRST_VM="${VMS[0]}"

export FIRST_VM_ID=$(az vm show \
  --resource-group "$RG" \
  --name "$FIRST_VM" \
  --query id \
  --output tsv)

Inspect its CPU metrics:

az monitor metrics list \
  --resource "$FIRST_VM_ID" \
  --metric "Percentage CPU" \
  --interval PT1M \
  --aggregation Average \
  --query "value[0].timeseries[0].data[-10:].{
    Timestamp:timeStamp,
    AverageCPU:average
  }" \
  --output table

Expected behavior: The CPU metric rises while the workload is running. Metrics may take a few minutes to become available.

Step 3 — Inspect the scale set capacity

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "sku.capacity" \
  --output tsv

Initially, the result should be:

3

If the scale-out rule triggers successfully, the capacity should increase to 4 and potentially 5, depending on sustained CPU usage.

Step 4 — Monitor instance creation

az vm list \
  --resource-group "$RG" \
  --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
    Name:name,
    Zone:zones[0],
    State:provisioningState
  }" \
  --output table

Run this command periodically to observe newly created VMs.

You can also compare the scale set capacity before and after the load test.

Step 5 — Verify new instances

New instances should receive the NGINX application from cloud-init and the health extension configuration from the scale set model.

Refresh your instance list:

mapfile -t ALL_VMS < <(
  az vm list \
    --resource-group "$RG" \
    --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].name" \
    --output tsv
)

printf '%s\n' "${ALL_VMS[@]}"

Check the application health extension on every current instance:

for VM in "${ALL_VMS[@]}"; do
  echo "Checking extension on: $VM"

  az vm extension show \
    --resource-group "$RG" \
    --vm-name "$VM" \
    --name ApplicationHealthLinux \
    --query "{Name:name,State:provisioningState}" \
    --output table
done

If a new instance is missing the extension, investigate its provisioning and scale set model rather than assuming automatic repairs can monitor it.

What did we learn?

Autoscaling is not just increasing an instance count.

Successful scale-out requires the new VM to:

  1. Provision successfully.

  2. Configure its operating system.

  3. Install and start the application.

  4. Become healthy.

  5. Be available to handle workload.

A VM that exists but cannot serve traffic does not provide useful additional capacity.

Lab 9 — Test Scale-In

Now we will remove the artificial CPU workload.

Step 1 — Stop CPU load

for VM in "${VMS[@]}"; do
  echo "Stopping CPU load on: $VM"

  az vm run-command invoke \
    --resource-group "$RG" \
    --name "$VM" \
    --command-id RunShellScript \
    --scripts "pkill -x yes || true" \
    --query "value[0].message" \
    --output tsv
done

The load-generation processes have been terminated. Any remaining processes from the original test will also stop when their 15-minute timeout expires.

Step 2 — Observe reduced CPU usage

az monitor metrics list \
  --resource "$FIRST_VM_ID" \
  --metric "Percentage CPU" \
  --interval PT1M \
  --aggregation Average \
  --query "value[0].timeseries[0].data[-10:].{
    Timestamp:timeStamp,
    AverageCPU:average
  }" \
  --output table

After enough low-CPU measurements, Azure Monitor should trigger scale-in.

Step 3 — Monitor scale-in

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "{Name:name,Capacity:sku.capacity}" \
  --output table

The scale set should eventually return to the minimum capacity of three instances, provided the scale-in conditions are satisfied.

Scale-in can take longer than the five-minute metric window because of cooldown periods and Azure Monitor's safeguards against unstable scaling behavior.

Lab 10 — Enable Automatic Instance Repairs

We have demonstrated that the platform can react to demand.

Now we will test whether it can react to an application failure.

Azure Virtual Machine Scale Sets can monitor unhealthy instances and automatically trigger repair actions.

The supported repair actions include:

Repair action

Behavior

Replace

Deletes the unhealthy VM and creates a replacement from the scale set model

Reimage

Recreates the OS environment from the image

Restart

Restarts the unhealthy VM

For this workshop, we will use Replace.

This is important because Replace demonstrates an infrastructure recovery operation rather than simply restarting an application process.

Step 1 — Disable autoscaling for the repair exercise

First, prevent autoscale operations from interfering with the failure simulation.

az monitor autoscale update \
  --resource-group "$RG" \
  --name "$AUTOSCALE" \
  --enabled false

Return the scale set to three instances if necessary:

az vmss scale \
  --resource-group "$RG" \
  --name "$VMSS" \
  --new-capacity 3

Verify the instance count:

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "sku.capacity" \
  --output tsv

Step 2 — Enable automatic repairs

az vmss update \
  --resource-group "$RG" \
  --name "$VMSS" \
  --enable-automatic-repairs true \
  --automatic-repairs-grace-period 30 \
  --automatic-repairs-action Replace

Important: The 30-minute grace period prevents Azure from prematurely repairing an instance immediately after certain VM or scale set state changes. Automatic repair is not an instantaneous failover mechanism.

Step 3 — Verify the repair policy

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "automaticRepairsPolicy" \
  --output json

The configuration should indicate that repairs are enabled, with the configured grace period and Replace action.

Step 4 — Verify the repair orchestration service

az vmss get-instance-view \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "orchestrationServices" \
  --output json

Look for the AutomaticRepairs service.

A running service indicates that automatic repair orchestration is active. If the service is suspended, investigate its state before proceeding.

Lab 11 — Simulate an Unhealthy VM

We will deliberately stop NGINX on one scale set instance.

The VM itself will continue running, but its application will become unavailable.

This demonstrates why application-level health monitoring is essential.

Step 1 — Select a victim instance

export VICTIM=$(az vm list \
  --resource-group "$RG" \
  --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'] | [0].name" \
  --output tsv)

echo "Selected victim: $VICTIM"

Step 2 — Record its VM identity

Before causing the failure, record the current VM resource identity.

export VICTIM_ID=$(az vm show \
  --resource-group "$RG" \
  --name "$VICTIM" \
  --query vmId \
  --output tsv)

echo "Original VM ID: $VICTIM_ID"

This allows us to determine whether Azure subsequently replaced the instance.

Step 3 — Stop NGINX

az vm run-command invoke \
  --resource-group "$RG" \
  --name "$VICTIM" \
  --command-id RunShellScript \
  --scripts \
    "systemctl stop nginx" \
    "systemctl is-active nginx || true" \
    "curl -sS http://127.0.0.1/health || true" \
  --query "value[0].message" \
  --output tsv

Expected behavior:

  • NGINX is no longer active.

  • The /health endpoint is unavailable.

  • The VM operating system continues running.

  • The Application Health extension detects the failing endpoint.

This is an application failure rather than a simulated Azure platform outage.

Step 4 — Check that the VM still exists

az vm get-instance-view \
  --resource-group "$RG" \
  --name "$VICTIM" \
  --query "instanceView.statuses" \
  --output json

The VM can remain in a running state even though its application is unhealthy.

That is the distinction we want to demonstrate.

Lab 12 — Observe Automatic Recovery

Once the Application Health extension reports the instance as unhealthy and the repair grace period has been satisfied, Azure can initiate the configured Replace operation.

Depending on the state of the environment, this process can take a significant amount of time.

Step 1 — Inspect the repair orchestration service

az vmss get-instance-view \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "orchestrationServices" \
  --output json

If automatic repairs are active, the service should report an appropriate running state.

Step 2 — Inspect the Azure Activity Log

az monitor activity-log list \
  --resource-group "$RG" \
  --offset 2h \
  --query "[].{
    Timestamp:eventTimestamp,
    Operation:operationName.localizedValue,
    Status:status.localizedValue
  }" \
  --output table

Look for operations involving VM deletion, creation, or changes to the scale set.

Activity Log entries may be associated with the individual VMs rather than exclusively with the parent scale set.

Step 3 — List current VM instances

az vm list \
  --resource-group "$RG" \
  --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
    Name:name,
    VMId:vmId,
    Zone:zones[0],
    State:provisioningState
  }" \
  --output table

A successful Replace operation should remove the unhealthy instance and create another VM to maintain the desired scale set capacity.

Step 4 — Compare the original VM identity

CURRENT_IDS=$(az vm list \
  --resource-group "$RG" \
  --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].vmId" \
  --output tsv)

if printf '%s\n' "$CURRENT_IDS" | grep -qFx "$VICTIM_ID"; then
  echo "Original VM still exists."
else
  echo "Original VM ID no longer exists in the scale set."
fi

Interpretation:

If the original VM ID is no longer present, Azure has removed that instance. Together with the Activity Log and the appearance of a new VM, this provides evidence of a replacement.

Do not interpret a missing VM ID alone as proof of automatic repair. Confirm that the replacement was initiated by the repair mechanism rather than by manual scaling or another operation.

Step 5 — Verify the final capacity

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "{
    Name:name,
    Capacity:sku.capacity,
    RepairPolicy:automaticRepairsPolicy
  }" \
  --output json

The desired capacity should remain three.

Step 6 — Validate recovered application health

Refresh the current VM names:

mapfile -t ALL_VMS < <(
  az vm list \
    --resource-group "$RG" \
    --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].name" \
    --output tsv
)

Verify the NGINX application on all current instances:

for VM in "${ALL_VMS[@]}"; do
  echo "Validating: $VM"

  az vm run-command invoke \
    --resource-group "$RG" \
    --name "$VM" \
    --command-id RunShellScript \
    --scripts \
      "cloud-init status --wait" \
      "systemctl is-active nginx" \
      "curl -fsS http://127.0.0.1/health" \
    --query "value[0].message" \
    --output tsv
done

Expected result: Every running instance returns a healthy application response.

This is the central objective of the workshop: Azure should recover the required VM capacity, and the replacement VM should deploy the application correctly without manual software installation.

Lab 13 — Explore Availability Zones

Azure Availability Zones provide physically separate locations within an Azure region, each with independent power, cooling, and networking infrastructure.

A zone-spanning VMSS distributes instances across the configured zones to reduce the impact of failures affecting a single zone.

However, Availability Zones are not the same as multi-region disaster recovery.

Step 1 — Review the zone configuration

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "{
    Name:name,
    Zones:zones,
    Capacity:sku.capacity
  }" \
  --output json

Step 2 — Check actual zone placement

az vm list \
  --resource-group "$RG" \
  --query "[?virtualMachineScaleSet.id=='${VMSS_ID}'].{
    VM:name,
    AvailabilityZone:zones[0]
  }" \
  --output table

Compare the placement before and after scale-out and automatic repair.

Operational questions

  • Were all three Availability Zones represented?

  • Did the replacement VM use the same zone as the original?

  • Could a scale-out operation place another VM in an already populated zone?

  • What happens if capacity is unavailable in one zone?

  • Does distributing instances across zones guarantee application availability?

Key lesson: Availability Zone distribution helps improve resilience, but reliable application availability also depends on traffic routing, health-based load balancing, application state, and other dependencies.

Lab 14 — Troubleshooting Scenarios

Now that the workshop is complete, let's examine common operational issues.

Scenario 1 — Autoscaling does not start

Symptoms: CPU is high, but the VM count remains unchanged.

Check the autoscale configuration:

az monitor autoscale show \
  --resource-group "$RG" \
  --name "$AUTOSCALE" \
  --output json

Verify CPU metrics:

az monitor metrics list \
  --resource "$FIRST_VM_ID" \
  --metric "Percentage CPU" \
  --aggregation Average \
  --output json

Possible causes include:

  • The rule's evaluation window has not been satisfied.

  • Autoscaling is disabled.

  • New VMs cannot be created because of compute quota restrictions.

  • The requested VM size is unavailable in the selected region or zone.

  • The VMSS has already reached its maximum capacity.

  • The observed per-VM metric differs from the aggregate used by autoscale.

Remember that we intentionally disabled autoscale before the automatic repair test.

Scenario 2 — The application is unhealthy after deployment

Check the VM configuration:

az vm run-command invoke \
  --resource-group "$RG" \
  --name "$VICTIM" \
  --command-id RunShellScript \
  --scripts \
    "systemctl status nginx --no-pager || true" \
    "curl -v http://127.0.0.1/health || true" \
  --query "value[0].message" \
  --output tsv

Run this only if the selected VM still exists. If it has already been replaced, select a current VM instead.

Possible causes include:

  • NGINX did not start.

  • Cloud-init failed or has not completed.

  • The health endpoint does not exist.

  • The health extension is configured with the wrong port or path.

Scenario 3 — Automatic repair does not occur

Inspect the policy:

az vmss show \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "automaticRepairsPolicy" \
  --output json

Inspect the orchestration service:

az vmss get-instance-view \
  --resource-group "$RG" \
  --name "$VMSS" \
  --query "orchestrationServices" \
  --output json

Possible causes include:

  • The instance is still inside its repair grace period.

  • The Application Health extension was not installed correctly.

  • The health endpoint is still returning successful responses.

  • The repair service was suspended.

  • The VM failed initial provisioning and therefore never became eligible for application-health-based repair.

  • Replacement VM creation is blocked by quota or capacity limits.

If repair orchestration is suspended, first resolve the underlying issue. You can then resume it:

az vmss set-orchestration-service-state \
  --resource-group "$RG" \
  --name "$VMSS" \
  --service-name AutomaticRepairs \
  --action Resume

Scenario 4 — New instances are created without a working application

Verify cloud-init execution on an affected instance:

az vm run-command invoke \
  --resource-group "$RG" \
  --name "<CURRENT-VM-NAME>" \
  --command-id RunShellScript \
  --scripts \
    "cloud-init status --long" \
    "tail -n 50 /var/log/cloud-init-output.log" \
  --query "value[0].message" \
  --output tsv

Replace <CURRENT-VM-NAME> with an existing VM.

Possible causes include package installation failures, missing outbound connectivity, repository availability, or invalid cloud-init commands.

Lab 15 — Cleanup

After completing the workshop, delete the Azure resources to avoid unnecessary charges.

Step 1 — Review deployed resources

az resource list \
  --resource-group "$RG" \
  --query "[].{Name:name,Type:type}" \
  --output table

Step 2 — Delete the resource group

az group delete \
  --name "$RG" \
  --yes

This removes the lab's VMSS, VMs, disks, virtual network, NSG, public IP addresses, autoscale settings, and other resources contained in the resource group.

Step 3 — Verify deletion

az group exists \
  --name "$RG"

Expected result:

false

Step 4 — Remove locally generated files

rm -f "${VMSS}-cloud-init.yaml"
rm -f "${VMSS}-health.json"

The workshop environment is now cleaned up.

Workshop Review — What Have We Learned?

Let's summarize the main concepts.

Azure feature

What we demonstrated

Flexible orchestration

Standard Azure VM lifecycle management within a scale set

Availability Zones

Distribution of VM instances across zones

Cloud-init

Automatic application provisioning

Azure Monitor Autoscale

CPU-based scale-out and scale-in

Azure Monitor Metrics

Observability of VM performance

Application Health extension

Application-level health monitoring

Automatic Instance Repairs

Replacement of unhealthy VM instances

Azure Activity Log

Investigation of VM lifecycle operations

Azure CLI

End-to-end deployment, troubleshooting, and cleanup

Flexible vs. Uniform orchestration

One of the most important concepts is the difference between Flexible and Uniform orchestration.

Flexible orchestration manages standard Azure IaaS virtual machines. It offers more flexibility when working with individual VM resources and supports scaling, availability features, and automatic repairs.

Uniform orchestration is designed around a more standardized VM instance model and uses scale-set-specific instance management APIs.

For new deployments, Flexible orchestration is generally Microsoft's recommended option, provided it supports the workload's requirements.

Autoscaling vs. Auto-Healing

These two concepts solve different problems.

Autoscaling adjusts compute capacity in response to demand.

Auto-healing responds to unhealthy instances and attempts to restore their functionality or replace them.

An application can require both.

For example, adding more VMs is not helpful if the new VMs cannot start the application correctly. Likewise, automatically replacing a failed VM does not solve a sustained capacity shortage.

The combination of application-aware monitoring, automated scaling, and instance repairs is what makes this architecture operationally interesting.

Important production considerations

This workshop deliberately simplifies the architecture.

For a production deployment, consider:

  • Using Azure Load Balancer or Application Gateway as a shared application entry point.

  • Keeping application VMs private whenever possible.

  • Using a NAT Gateway or another appropriate controlled outbound connectivity solution.

  • Using Azure Monitor alerts and Log Analytics for operational visibility.

  • Implementing controlled deployment and application-update strategies.

  • Using managed identities instead of embedded credentials.

  • Treating VMs as replaceable infrastructure and keeping important application data outside the VM's local OS disk.

  • Designing recovery objectives and considering multi-region disaster recovery separately.

Automatic VM replacement should not be confused with guaranteed uninterrupted application availability. It is an infrastructure recovery mechanism, and a Replace operation does not preserve the original VM's local state.

Final Challenge — Test Your Understanding

Before deleting the environment, try answering these questions:

  1. What is the difference between VM provisioning state and application health?

  2. Why is cloud-init important when Azure automatically creates a replacement VM?

  3. What happens if one instance reaches 100% CPU while the other instances remain idle?

  4. Why might a VMSS fail to scale out even when the CPU threshold has been exceeded?

  5. What is the purpose of the automatic repair grace period?

  6. How does Azure decide whether to replace an unhealthy VM?

  7. Why does a Flexible VMSS require a different approach to individual-instance management than a Uniform VMSS?

  8. Does distributing VMs across three Availability Zones eliminate the need for disaster recovery?

  9. What happens to a VM's local data when Azure uses the Replace repair action?

  10. How would you redesign the lab to expose one highly available public web endpoint?

If you can answer these questions and explain the behavior you observed during the lab, you have gone beyond simply deploying a Virtual Machine Scale Set.

You have started thinking about the operational responsibilities of running scalable workloads in Microsoft Azure.

References

Official Microsoft documentation used for the concepts and CLI configuration:

Workshop: Azure VM Scale Sets Deep Dive — Flexible Orchestration, Autoscaling, Health Monitoring & Self-Healing

Beyond Cloud with Chriz — Azure Workshopz

Build. Break. Troubleshoot. Understand.

Publication note: The commands are based on Microsoft's documented Azure CLI interfaces and intended for Azure Cloud Shell Bash. They have not been executed against a live subscription in this conversation. Validate the deployment in a test subscription before publishing it as a fully tested lab, particularly the Flexible VMSS extension model and automatic repair lifecycle.

 
 
 

Comments


bottom of page