top of page
1 day ago
21 min read

Azure Workshopz | Beyond Cloud with Chriz

DEPLOY • SECURE • ROTATE • BREAK • DIAGNOSE • REPAIR • VERIFY • CLEAN UP

Level: Intermediate to AdvancedPlatform: Microsoft AzureTools: Azure CLI, Azure Cloud Shell (Bash)Deployment: Azure CLI only — no Terraform, Bicep, or Azure Portal configuration stepsTechnologies: Azure Key Vault, Managed Identity, Azure RBAC, Azure App Service, Event Grid, Azure Storage Queues, Azure MonitorCost: Low to Medium, depending on App Service runtime and monitoring usage

Introduction

Your application is running successfully.

Its configuration is stored securely in Azure Key Vault, and access is controlled through a managed identity.

Everything looks healthy.

But what happens when your application is still referencing an older secret version after credentials have been rotated?

What happens when a certificate approaches expiration?

And how do you detect and recover from these situations without manually copying credentials into application configuration?

In this workshop, we're going to build a complete Azure Key Vault Secret and Certificate Lifecycle Management environment.

We will create and rotate secrets, generate certificate versions, configure expiration policies, and establish event-driven monitoring.

More importantly, we will connect Azure Key Vault to a real Azure App Service application.

Then we will deliberately introduce a secret-version mismatch, observe the application return an error, investigate the configuration, and restore the service using a versionless Key Vault reference.

Everything will be managed through Azure CLI.

The goal is to understand how secret and certificate rotation affects real applications, not simply how to store a secret.

What you'll learn

By the end of this workshop, you will understand how to:

  1. Deploy Azure Key Vault using Azure RBAC.

  2. Configure least-privilege access for application identities.

  3. Create secrets with expiration dates.

  4. Manage multiple versions of a secret.

  5. Deploy an App Service application using a managed identity.

  6. Configure version-specific and versionless Key Vault references.

  7. Simulate a secret-rotation failure.

  8. Diagnose and repair broken application configuration.

  9. Generate and renew self-signed certificates.

  10. Configure certificate expiration and email notification policies.

  11. Capture Key Vault lifecycle events using Event Grid.

  12. Configure Azure Monitor alerts for application failures.

  13. Verify rotation behavior and clean up the environment.

Workshop Architecture

                      AZURE CLOUD SHELL
                             |
                          Azure CLI
                             |
                             v
               +---------------------------+
               |       AZURE KEY VAULT     |
               |                           |
               |  Secrets                  |
               |  - Version 1              |
               |  - Version 2              |
               |                           |
               |  Certificates             |
               |  - Certificate Version 1  |
               |  - Certificate Version 2  |
               |                           |
               |  Expiration Policies      |
               +------------+--------------+
                            |
              +-------------+-------------+
              |                           |
              v                           v
    +-------------------+       +-------------------+
    |  AZURE APP SERVICE|       |    EVENT GRID     |
    |                   |       |                   |
    |  Managed Identity |       |  Secret Rotation  |
    |  Key Vault Ref    |       |  Cert Rotation    |
    |                   |       |  Near Expiration  |
    |  /health          |       +---------+---------+
    +---------+---------+                 |
              |                           v
              |                 +-------------------+
              |                 | STORAGE QUEUES    |
              |                 |                   |
              |                 | Rotation Events   |
              |                 | Expiration Events |
              |                 +-------------------+
              |
              v
    +-------------------+
    |   AZURE MONITOR   |
    |                   |
    | HTTP 5xx Alerts   |
    | Action Group      |
    +-------------------+

Our failure scenario

Initially, the application receives secret version 1.

We then create version 2.

The application is deliberately configured to require version 2 while its Key Vault reference remains pinned to version 1.

BEFORE ROTATION

App Service
    |
    v
Secret Version 1
    |
    v
HTTP 200 — HEALTHY


AFTER ROTATION

App Service expects Version 2
    |
    v
Pinned reference still uses Version 1
    |
    v
HTTP 503 — UNHEALTHY


AFTER REPAIR

App Service
    |
    v
Versionless Key Vault Reference
    |
    v
Latest Secret Version
    |
    v
HTTP 200 — HEALTHY

This is an intentional application-level compatibility test. Key Vault itself does not automatically invalidate an older secret version when you create a newer one.

Lab 1 — Prepare the Azure Environment

Step 1 — Open Azure Cloud Shell

Open:

Select Bash.

Check the Azure CLI version:

az version

Check the active subscription:

az account show \
  --query "{Name:name,SubscriptionId:id}" \
  --output table

If needed, select your subscription:

az account set \
  --subscription "<YOUR-SUBSCRIPTION-ID>"

Step 2 — Verify the required tools

az --version
jq --version
zip -v | head -n 2

The workshop uses Azure CLI to manage all Azure resources. Bash, jq, zip, and a small Python test application are used locally to prepare payloads and simulate the application dependency.

Step 3 — Configure variables

# General settings
export LOCATION="westeurope"
export SUFFIX=$(openssl rand -hex 3)

# Resource group
export RG="rg-kv-rotation-$SUFFIX"

# Key Vault
export KV="kvrot$SUFFIX"
export SECRET="workshop-api-token"
export CERT="workshop-tls-cert"

# App Service
export PLAN="plan-kv-rotation-$SUFFIX"
export APP="app-kv-rotation-$SUFFIX"

# Event Grid and Storage
export STORAGE="stkvrot$SUFFIX"
export QUEUE_ROTATION="rotation-events"
export QUEUE_EXPIRY="expiry-events"

export SUB_ROTATION="kv-rotation-sub"
export SUB_EXPIRY="kv-expiry-sub"

# Azure Monitor
export ACTION_GROUP="ag-kv-rotation"
export HTTP_ALERT="alert-app-secret-failure"

# Optional notification address
export ALERT_EMAIL=""

If you want to receive email notifications, set ALERT_EMAIL to an address you control before reaching the monitoring section.

Display the resource names:

echo "Resource Group: $RG"
echo "Key Vault:       $KV"
echo "App Service:     $APP"
echo "Storage:         $STORAGE"
echo "Region:          $LOCATION"

Step 4 — Create the resource group

az group create \
  --name "$RG" \
  --location "$LOCATION" \
  --tags Environment=Workshop \
         Project=Azure-KeyVault-Rotation

Verify:

az group show \
  --name "$RG" \
  --query "{Name:name,Location:location,State:properties.provisioningState}" \
  --output table

Expected result: Resource group provisioning succeeds.

Lab 2 — Deploy Azure Key Vault

We will deploy a Key Vault using Azure RBAC authorization.

This means that access to secrets and certificates is controlled through Azure role assignments rather than legacy vault access policies.

Step 1 — Create the vault

az keyvault create \
  --resource-group "$RG" \
  --name "$KV" \
  --location "$LOCATION" \
  --sku standard \
  --enable-rbac-authorization true \
  --retention-days 7

This lab uses the minimum supported soft-delete retention period to reduce the duration of a deleted workshop vault.

For production workloads, enable purge protection according to your organization's security requirements.

Important: Purge protection cannot be disabled after it is enabled.

Step 2 — Verify Key Vault configuration

az keyvault show \
  --resource-group "$RG" \
  --name "$KV" \
  --query "{
    Name:name,
    Location:location,
    RBAC:properties.enableRbacAuthorization,
    SoftDelete:properties.enableSoftDelete,
    PurgeProtection:properties.enablePurgeProtection
  }" \
  --output json

Step 3 — Retrieve the Key Vault resource ID

export KV_ID=$(az keyvault show \
  --resource-group "$RG" \
  --name "$KV" \
  --query id \
  --output tsv)

echo "$KV_ID"

Step 4 — Configure workshop administrator access

Azure RBAC distinguishes between management-plane permissions and Key Vault data-plane permissions.

Being able to create a vault doesn't automatically mean you can create secrets or certificates inside it.

For this lab, assign your signed-in user the Key Vault Administrator role.

export USER_ID=$(az ad signed-in-user show \
  --query id \
  --output tsv)
az role assignment create \
  --assignee-object-id "$USER_ID" \
  --assignee-principal-type User \
  --role "Key Vault Administrator" \
  --scope "$KV_ID"

This step assumes an interactive user login and permission to create role assignments, such as Owner or Role Based Access Control Administrator with suitable scope.

For production systems, separate secret management, certificate management, and secret reading into dedicated roles.

Step 5 — Verify access

az keyvault secret list \
  --vault-name "$KV" \
  --output table

If you receive a Forbidden error immediately after role assignment, retry after RBAC propagation.

Checkpoint: Continue only after the signed-in user can query the vault's data plane.

Lab 3 — Prepare Event Grid Monitoring

Before creating any secrets or certificates, let's configure event collection.

This allows us to observe actual rotation events during the remaining exercises.

We will use two Storage Queues:

  • rotation-events — new secret and certificate versions.

  • expiry-events — near-expiry and expired objects.

Step 1 — Create a Storage Account

az storage account create \
  --resource-group "$RG" \
  --name "$STORAGE" \
  --location "$LOCATION" \
  --sku Standard_LRS \
  --kind StorageV2 \
  --min-tls-version TLS1_2

Step 2 — Retrieve its resource ID

export STORAGE_ID=$(az storage account show \
  --resource-group "$RG" \
  --name "$STORAGE" \
  --query id \
  --output tsv)

Step 3 — Retrieve a temporary storage key

For the lab, we will use a storage account key to create and inspect queue messages.

export STORAGE_KEY=$(az storage account keys list \
  --resource-group "$RG" \
  --account-name "$STORAGE" \
  --query "[0].value" \
  --output tsv)

The key must not be displayed in workshop screenshots or committed to Git.

In production, prefer Microsoft Entra authorization and managed identities for supported queue operations.

Step 4 — Create the queues

az storage queue create \
  --name "$QUEUE_ROTATION" \
  --account-name "$STORAGE" \
  --account-key "$STORAGE_KEY"
az storage queue create \
  --name "$QUEUE_EXPIRY" \
  --account-name "$STORAGE" \
  --account-key "$STORAGE_KEY"

Step 5 — Retrieve queue resource IDs

export ROTATION_QUEUE_ID="${STORAGE_ID}/queueServices/default/queues/${QUEUE_ROTATION}"

export EXPIRY_QUEUE_ID="${STORAGE_ID}/queueServices/default/queues/${QUEUE_EXPIRY}"

Step 6 — Create the rotation event subscription

az eventgrid event-subscription create \
  --name "$SUB_ROTATION" \
  --source-resource-id "$KV_ID" \
  --endpoint-type storagequeue \
  --endpoint "$ROTATION_QUEUE_ID" \
  --included-event-types \
    Microsoft.KeyVault.SecretNewVersionCreated \
    Microsoft.KeyVault.CertificateNewVersionCreated

Step 7 — Create the expiration event subscription

az eventgrid event-subscription create \
  --name "$SUB_EXPIRY" \
  --source-resource-id "$KV_ID" \
  --endpoint-type storagequeue \
  --endpoint "$EXPIRY_QUEUE_ID" \
  --included-event-types \
    Microsoft.KeyVault.SecretNearExpiry \
    Microsoft.KeyVault.SecretExpired \
    Microsoft.KeyVault.CertificateNearExpiry \
    Microsoft.KeyVault.CertificateExpired

Step 8 — Verify subscriptions

az eventgrid event-subscription list \
  --source-resource-id "$KV_ID" \
  --output table

Expected result: Both event subscriptions exist.

This architecture gives us event-driven visibility into secret and certificate lifecycle changes.

An event arriving in the expiry queue is not automatically an email alert. In production, connect the queue to a processing service that sends notifications, opens tickets, or initiates approved rotation workflows.

Lab 4 — Create the First Secret Version

We'll start with a synthetic API token.

Never use actual application passwords or production API keys in a public workshop.

Step 1 — Generate the first test value

export TOKEN_V1="lab-token-v1-$SUFFIX"

export EXPIRY_V1=$(date -u -d "+60 days" \
  +"%Y-%m-%dT%H:%M:%SZ")

Step 2 — Create the secret

az keyvault secret set \
  --vault-name "$KV" \
  --name "$SECRET" \
  --value "$TOKEN_V1" \
  --expires "$EXPIRY_V1" \
  --tags Environment=Workshop \
         Rotation=Version1 \
  --query "{Id:id,Enabled:attributes.enabled,Expires:attributes.expires}" \
  --output json

Step 3 — Retrieve the secret version ID

export SECRET_V1_ID=$(az keyvault secret show \
  --vault-name "$KV" \
  --name "$SECRET" \
  --query id \
  --output tsv)

echo "$SECRET_V1_ID"

A secret version ID looks similar to:

Each new version receives a unique version identifier.

Step 4 — Inspect secret metadata

az keyvault secret show \
  --vault-name "$KV" \
  --name "$SECRET" \
  --query "{
    Id:id,
    Name:name,
    Enabled:attributes.enabled,
    Expires:attributes.expires,
    Created:attributes.created
  }" \
  --output json

Notice that we are inspecting metadata without returning the secret value.

Lab 5 — Deploy a Dependent Azure App Service

Now we'll create an application that uses the Key Vault secret.

This is important because secret rotation can affect real applications.

Our application will compare its resolved secret with a synthetic expected value.

It will never return the secret itself in an HTTP response.

Step 1 — Create a Linux App Service Plan

az appservice plan create \
  --resource-group "$RG" \
  --name "$PLAN" \
  --location "$LOCATION" \
  --sku B1 \
  --is-linux

Step 2 — Create the Web App

az webapp create \
  --resource-group "$RG" \
  --plan "$PLAN" \
  --name "$APP" \
  --runtime "PYTHON:3.12"

Check available Python runtimes if your region or App Service environment does not support the selected version:

az webapp list-runtimes \
  --os linux \
  --output table

Step 3 — Enable managed identity

export APP_PRINCIPAL_ID=$(az webapp identity assign \
  --resource-group "$RG" \
  --name "$APP" \
  --query principalId \
  --output tsv)

echo "$APP_PRINCIPAL_ID"

This creates a system-assigned managed identity for the application.

No client secret or stored service principal password is required.

Step 4 — Grant access to Key Vault secrets

az role assignment create \
  --assignee-object-id "$APP_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$KV_ID"

The application identity can now read secrets from this Key Vault after RBAC propagation.

It does not have permission to create, update, or delete secrets.

Step 5 — Create a small health-check application

We will create a Python application using only the standard library.

mkdir -p kv-rotation-app
cd kv-rotation-app

Create app.py:

cat > app.py <<'PY'
import os
from http.server import HTTPServer, BaseHTTPRequestHandler

class Handler(BaseHTTPRequestHandler):

    def do_GET(self):
        if self.path not in ("/", "/health"):
            self.send_response(404)
            self.end_headers()
            return

        actual = os.environ.get("APP_API_TOKEN", "")
        expected = os.environ.get("APP_EXPECTED_TOKEN", "")

        healthy = (
            bool(actual)
            and bool(expected)
            and actual == expected
        )

        status = 200 if healthy else 503
        message = (
            "HEALTHY: credential matches expected version"
            if healthy
            else "UNHEALTHY: credential version mismatch"
        )

        self.send_response(status)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(message.encode())

port = int(os.environ.get("PORT", "8000"))

HTTPServer(("0.0.0.0", port), Handler).serve_forever()
PY

This is a deliberately simplified credential-compatibility simulator.

In a production application, credentials should be validated by the downstream service rather than compared with another plaintext application setting.

Step 6 — Package the application

zip app.zip app.py

Step 7 — Configure App Service startup

az webapp config set \
  --resource-group "$RG" \
  --name "$APP" \
  --startup-file "python3 app.py"

Step 8 — Configure the Key Vault reference

Initially, pin the application to secret version 1.

az webapp config appsettings set \
  --resource-group "$RG" \
  --name "$APP" \
  --settings \
    "APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$SECRET_V1_ID)" \
    "APP_EXPECTED_TOKEN=$TOKEN_V1"

This is a version-specific reference.

App Service will attempt to resolve the secret through its managed identity.

The Key Vault reference itself is not a plaintext copy of the secret.

Step 9 — Deploy the application

az webapp deploy \
  --resource-group "$RG" \
  --name "$APP" \
  --src-path app.zip \
  --type zip

Step 10 — Test application health

export APP_URL="https://${APP}.azurewebsites.net"
curl -i "${APP_URL}/health"

Expected response:

HTTP/1.1 200 OK

HEALTHY: credential matches expected version

Checkpoint: Confirm HTTP 200 before continuing.

If the application fails at this stage, verify managed identity permissions, Key Vault reference resolution, and the App Service startup command.

Lab 6 — Inspect Key Vault Reference Resolution

Azure App Service exposes the resolution status of Key Vault references through its management API.

This allows us to diagnose a failed reference without displaying the actual secret.

Step 1 — Retrieve the Web App resource ID

export APP_ID=$(az webapp show \
  --resource-group "$RG" \
  --name "$APP" \
  --query id \
  --output tsv)

Step 2 — Inspect reference status

az rest \
  --method get \
  --uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
  --output json

Look for the APP_API_TOKEN configuration reference.

Verify its resolution status, Key Vault name, and referenced secret version.

A healthy reference should report a successful resolution status.

This diagnostic API becomes especially useful when an application cannot access a secret because of missing RBAC permissions, an invalid version, or an unavailable vault.

Lab 7 — ROTATE: Create Secret Version 2

Now we'll perform our first secret rotation.

Step 1 — Generate the new value

export TOKEN_V2="lab-token-v2-$SUFFIX"

export EXPIRY_V2=$(date -u -d "+90 days" \
  +"%Y-%m-%dT%H:%M:%SZ")

Step 2 — Create another version

az keyvault secret set \
  --vault-name "$KV" \
  --name "$SECRET" \
  --value "$TOKEN_V2" \
  --expires "$EXPIRY_V2" \
  --tags Environment=Workshop \
         Rotation=Version2 \
  --query "{Id:id,Expires:attributes.expires}" \
  --output json

Key Vault automatically creates a new version because the secret already exists.

Step 3 — Retrieve version 2

export SECRET_V2_ID=$(az keyvault secret show \
  --vault-name "$KV" \
  --name "$SECRET" \
  --query id \
  --output tsv)

Step 4 — Compare versions

echo "Version 1: $SECRET_V1_ID"
echo "Version 2: $SECRET_V2_ID"

Both identifiers should be different.

Step 5 — List all secret versions

az keyvault secret list-versions \
  --vault-name "$KV" \
  --name "$SECRET" \
  --query "[].{
    Id:id,
    Enabled:attributes.enabled,
    Expires:attributes.expires
  }" \
  --output table

Key concept: Creating version 2 does not automatically disable or revoke version 1.

Older versions remain available unless their attributes or the vault's retention state prevent access.

Lab 8 — BREAK: Simulate a Stale Secret Version

This is the main failure scenario.

We have rotated the secret in Azure Key Vault.

However, our application is still pinned to version 1.

Let's simulate a downstream system requiring version 2.

Step 1 — Update the expected credential

az webapp config appsettings set \
  --resource-group "$RG" \
  --name "$APP" \
  --settings "APP_EXPECTED_TOKEN=$TOKEN_V2"

The application now expects the rotated credential.

But its Key Vault reference still points to version 1.

Step 2 — Test the application

curl -i "${APP_URL}/health"

Expected response:

HTTP/1.1 503 Service Unavailable

UNHEALTHY: credential version mismatch

We have deliberately created an application-level incident.

The Key Vault is still available.

The secret exists.

The managed identity can still read it.

But the application is consuming an outdated version.

Step 3 — Confirm the configured reference

az webapp config appsettings list \
  --resource-group "$RG" \
  --name "$APP" \
  --query "[?name=='APP_API_TOKEN'].name" \
  --output table

Now inspect the reference resolution metadata:

az rest \
  --method get \
  --uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
  --output json

Compare the reported secret version with the current version stored in Key Vault.

Step 4 — Check the latest Key Vault version

az keyvault secret show \
  --vault-name "$KV" \
  --name "$SECRET" \
  --query id \
  --output tsv

The latest version is version 2, while the application remains pinned to version 1.

What happened?

This is not a Key Vault outage.

It is a secret-consumer configuration problem.

Secret rotation requires more than generating a new credential. Applications must also transition to the replacement credential.

Lab 9 — REPAIR: Use a Versionless Reference

The recommended approach for consumers that should automatically follow the latest secret version is to use a versionless Key Vault reference.

Step 1 — Construct the versionless URI

export SECRET_URI="https://${KV}.vault.azure.net/secrets/${SECRET}"

This URI references the secret name rather than one specific version.

Step 2 — Update App Service

az webapp config appsettings set \
  --resource-group "$RG" \
  --name "$APP" \
  --settings \
    "APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$SECRET_URI)"

Step 3 — Refresh Key Vault references

Azure App Service caches resolved Key Vault references.

A versionless reference normally picks up a new secret version within 24 hours.

We can explicitly request a refresh:

az rest \
  --method post \
  --uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings/refresh?api-version=2022-03-01"

Step 4 — Restart the Web App

az webapp restart \
  --resource-group "$RG" \
  --name "$APP"

Step 5 — Verify recovery

curl -i "${APP_URL}/health"

Expected response:

HTTP/1.1 200 OK

HEALTHY: credential matches expected version

The service is restored.

What did we learn?

Version-specific references provide deterministic version selection but require explicit updating when credentials change.

Versionless references allow a consumer to resolve the latest version.

However, the new credential must already be valid in the downstream system before the consuming application switches to it.

In production, a safe rotation usually involves overlapping credential validity, validating the new credential, switching consumers, and only then retiring the previous credential.

Lab 10 — Create a Self-Signed Certificate

Now let's move from secrets to certificates.

Azure Key Vault can manage X.509 certificates, their private keys, metadata, and lifecycle policies.

For this workshop, we'll use a self-signed certificate.

It is suitable for learning but must not be treated as a publicly trusted production TLS certificate.

Step 1 — Generate a default policy

az keyvault certificate get-default-policy \
  > cert-policy.json

Step 2 — Customize the policy

We'll configure:

  • Self-signed certificate.

  • RSA key.

  • 2048-bit key size.

  • Three-month validity.

  • Expiration email notification 30 days before expiry.

jq '
  .issuer.name = "Self" |
  .key_props.kty = "RSA" |
  .key_props.key_size = 2048 |
  .key_props.reuse_key = false |
  .x509_props.subject = "CN=rotation.internal.example" |
  .x509_props.validity_months = 3 |
  .lifetime_actions = [
    {
      "trigger": {
        "days_before_expiry": 30
      },
      "action": {
        "action_type": "EmailContacts"
      }
    }
  ]
' cert-policy.json > cert-policy-custom.json

Step 3 — Inspect the policy

cat cert-policy-custom.json

The EmailContacts action instructs Key Vault to send notifications to configured certificate contacts when its lifecycle trigger is reached.

It does not send an email immediately when the policy is saved.

Step 4 — Create the certificate

az keyvault certificate create \
  --vault-name "$KV" \
  --name "$CERT" \
  --policy "@cert-policy-custom.json"

Step 5 — Inspect the certificate

az keyvault certificate show \
  --vault-name "$KV" \
  --name "$CERT" \
  --query "{
    Id:id,
    Enabled:attributes.enabled,
    Expires:attributes.expires
  }" \
  --output json

Step 6 — Record certificate version 1

export CERT_V1_ID=$(az keyvault certificate show \
  --vault-name "$KV" \
  --name "$CERT" \
  --query id \
  --output tsv)

Checkpoint: Certificate version 1 exists with the expected lifetime policy.

Lab 11 — Configure Certificate Expiration Contacts

Azure Key Vault certificate lifecycle policies can send expiration-related emails to vault certificate contacts.

Step 1 — Add a contact

If you configured ALERT_EMAIL, execute:

if [[ -n "$ALERT_EMAIL" ]]; then
  az keyvault certificate contact add \
    --vault-name "$KV" \
    --email "$ALERT_EMAIL"
fi

Step 2 — Verify contacts

az keyvault certificate contact list \
  --vault-name "$KV" \
  --output table

Step 3 — Inspect the certificate lifecycle policy

az keyvault certificate show \
  --vault-name "$KV" \
  --name "$CERT" \
  --query "policy.lifetime_actions" \
  --output json

Certificate expiration behavior

With our policy:

  • The certificate has a three-month validity period.

  • The lifecycle action is scheduled for 30 days before expiration.

  • Key Vault can email configured certificate contacts when that trigger is reached.

Important: We are configuring EmailContacts, not automatic renewal.

The certificate will not be automatically replaced simply because we configured an email notification policy.

Lab 12 — ROTATE: Generate Certificate Version 2

Now we'll renew the certificate by creating another version.

Step 1 — Create the new certificate version

az keyvault certificate create \
  --vault-name "$KV" \
  --name "$CERT" \
  --policy "@cert-policy-custom.json"

Step 2 — Retrieve the new version ID

export CERT_V2_ID=$(az keyvault certificate show \
  --vault-name "$KV" \
  --name "$CERT" \
  --query id \
  --output tsv)

Step 3 — Compare versions

echo "Certificate Version 1: $CERT_V1_ID"
echo "Certificate Version 2: $CERT_V2_ID"

Step 4 — List certificate versions

az keyvault certificate list-versions \
  --vault-name "$KV" \
  --name "$CERT" \
  --query "[].{
    Id:id,
    Enabled:attributes.enabled,
    Expires:attributes.expires
  }" \
  --output table

Expected result: Two certificate versions with distinct version IDs.

Important certificate lifecycle distinction

Creating a new version in Key Vault does not automatically replace a certificate that an external web server, load balancer, or application has already loaded.

The consumer must support certificate refresh or have a deployment process that installs the new certificate version.

For this workshop, we validate Key Vault certificate issuance and versioning. We are not claiming that the App Service's public HTTPS certificate has been replaced.

Lab 13 — Verify Event Grid Rotation Events

We created Event Grid subscriptions before creating the secrets and certificates.

Let's inspect the rotation queue.

Step 1 — Peek messages

az storage message peek \
  --queue-name "$QUEUE_ROTATION" \
  --account-name "$STORAGE" \
  --account-key "$STORAGE_KEY" \
  --num-messages 10 \
  --output json

Look for events such as:

Microsoft.KeyVault.SecretNewVersionCreated

Microsoft.KeyVault.CertificateNewVersionCreated

A certificate operation can also affect a corresponding Key Vault secret object because Key Vault certificates have associated secret representations.

Step 2 — Inspect the event subscription

az eventgrid event-subscription show \
  --name "$SUB_ROTATION" \
  --source-resource-id "$KV_ID" \
  --output json

Verify:

  • Provisioning state.

  • Event type filters.

  • Storage Queue endpoint.

  • Delivery settings.

Step 3 — Understand the event contents

A Key Vault rotation event identifies the changed object, its version, and the source vault.

It does not include the secret value.

This distinction is important for designing secure event-driven automation.

Lab 14 — Configure Secret Expiration Monitoring

Key Vault can publish events when the current version of a secret is about to expire or has expired.

The built-in SecretNearExpiry event is normally generated 30 days before the configured expiration.

Let's create an additional demonstration secret with a short expiration period and inspect the monitoring configuration.

Step 1 — Create a short-lived secret

export TEST_EXPIRY=$(date -u -d "+10 days" \
  +"%Y-%m-%dT%H:%M:%SZ")
az keyvault secret set \
  --vault-name "$KV" \
  --name "short-lived-demo" \
  --value "synthetic-expiry-test" \
  --expires "$TEST_EXPIRY" \
  --query "{Name:name,Expires:attributes.expires}" \
  --output json

Step 2 — Inspect expiration dates

az keyvault secret list \
  --vault-name "$KV" \
  --query "[].{
    Name:name,
    Expires:attributes.expires,
    Enabled:attributes.enabled
  }" \
  --output table

Step 3 — Inspect the expiration queue

az storage message peek \
  --queue-name "$QUEUE_EXPIRY" \
  --account-name "$STORAGE" \
  --account-key "$STORAGE_KEY" \
  --num-messages 10 \
  --output json

Important event-timing limitation

Creating a secret that expires in 10 days does not guarantee that SecretNearExpiry will be published immediately.

Near-expiry events are generated by Key Vault's lifecycle service according to the documented expiration schedule.

The short-lived secret demonstrates the expiration metadata and event-subscription configuration, not a guaranteed immediate near-expiry event.

For a deterministic workshop test, use the SecretNewVersionCreated event from our previous rotation exercise.

For production expiration monitoring, supplement lifecycle events with periodic inventory checks and a defined notification workflow.

Lab 15 — Configure Azure Monitor Failure Alerts

We now have two monitoring mechanisms:

  1. Key Vault lifecycle notifications through Event Grid.

  2. Application availability monitoring through Azure Monitor.

We'll configure an HTTP 5xx alert for the App Service.

This allows us to detect the application-level failure introduced by our secret-version mismatch.

Step 1 — Create an Action Group

If you supplied an email address, create an email-enabled action group.

if [[ -n "$ALERT_EMAIL" ]]; then
  az monitor action-group create \
    --resource-group "$RG" \
    --name "$ACTION_GROUP" \
    --short-name "KVRotLab" \
    --action email Operations "$ALERT_EMAIL"
fi

Retrieve its ID if it was created:

export ACTION_GROUP_ID=""

if [[ -n "$ALERT_EMAIL" ]]; then
  ACTION_GROUP_ID=$(az monitor action-group show \
    --resource-group "$RG" \
    --name "$ACTION_GROUP" \
    --query id \
    --output tsv)
fi

Step 2 — Configure an HTTP 5xx metric alert

ALERT_ACTION_ARGS=()

if [[ -n "$ACTION_GROUP_ID" ]]; then
  ALERT_ACTION_ARGS=(--action "$ACTION_GROUP_ID")
fi
az monitor metrics alert create \
  --resource-group "$RG" \
  --name "$HTTP_ALERT" \
  --scopes "$APP_ID" \
  --condition "total Http5xx > 0" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 2 \
  --description "App Service secret rotation failure" \
  "${ALERT_ACTION_ARGS[@]}"

Step 3 — Verify the alert rule

az monitor metrics alert show \
  --resource-group "$RG" \
  --name "$HTTP_ALERT" \
  --query "{
    Name:name,
    Enabled:enabled,
    Severity:severity,
    Scopes:scopes
  }" \
  --output json

Step 4 — Review the metric

az monitor metrics list \
  --resource "$APP_ID" \
  --metric "Http5xx" \
  --interval PT1M \
  --aggregation Total \
  --output json

A repeated HTTP 503 response should contribute to the Web App's Http5xx metric once the telemetry is available.

The alert does not fire instantly. Azure Monitor evaluates the metric according to the configured aggregation window and evaluation frequency.

Lab 16 — BREAK AGAIN: Invalid Secret Reference

Let's perform a second failure simulation.

This time, rather than referencing a valid but outdated secret, we'll reference a nonexistent secret version.

Step 1 — Configure an invalid version

export BAD_VERSION="00000000000000000000000000000000"

export BAD_URI="${SECRET_URI}/${BAD_VERSION}"

Step 2 — Apply the invalid reference

az webapp config appsettings set \
  --resource-group "$RG" \
  --name "$APP" \
  --settings \
    "APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$BAD_URI)"

Step 3 — Force a reference refresh

az rest \
  --method post \
  --uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings/refresh?api-version=2022-03-01"

Restart the application:

az webapp restart \
  --resource-group "$RG" \
  --name "$APP"

Step 4 — Inspect resolution failure

az rest \
  --method get \
  --uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
  --output json

Expected behavior: The Key Vault reference reports a resolution problem because the specified version does not exist.

Step 5 — Test application health

curl -i "${APP_URL}/health"

Expected:

HTTP/1.1 503 Service Unavailable

The application cannot obtain the expected credential through the invalid reference.

Step 6 — Inspect HTTP errors

az monitor metrics list \
  --resource "$APP_ID" \
  --metric "Http5xx" \
  --interval PT1M \
  --aggregation Total \
  --output json

This allows us to correlate the configuration failure with application-level monitoring.

Lab 17 — REPAIR: Restore the Correct Reference

Step 1 — Restore the versionless URI

az webapp config appsettings set \
  --resource-group "$RG" \
  --name "$APP" \
  --settings \
    "APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$SECRET_URI)"

Step 2 — Refresh Key Vault references

az rest \
  --method post \
  --uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings/refresh?api-version=2022-03-01"

Step 3 — Restart the application

az webapp restart \
  --resource-group "$RG" \
  --name "$APP"

Step 4 — Verify the application

curl -i "${APP_URL}/health"

Expected:

HTTP/1.1 200 OK

HEALTHY: credential matches expected version

Step 5 — Verify resolution status

az rest \
  --method get \
  --uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
  --output json

The configuration reference should resolve successfully again.

We have restored the dependent application without deleting or recreating Azure Key Vault.

Lab 18 — Final Verification

Before cleanup, let's validate the complete environment.

Step 1 — Verify the Key Vault

az keyvault show \
  --resource-group "$RG" \
  --name "$KV" \
  --query "{Name:name,RBAC:properties.enableRbacAuthorization}" \
  --output table

Step 2 — List secret versions

az keyvault secret list-versions \
  --vault-name "$KV" \
  --name "$SECRET" \
  --output table

Step 3 — List certificate versions

az keyvault certificate list-versions \
  --vault-name "$KV" \
  --name "$CERT" \
  --output table

Step 4 — Verify application identity

az webapp identity show \
  --resource-group "$RG" \
  --name "$APP" \
  --output json

Step 5 — Verify role assignments

az role assignment list \
  --assignee "$APP_PRINCIPAL_ID" \
  --scope "$KV_ID" \
  --output table

Step 6 — Inspect Event Grid subscriptions

az eventgrid event-subscription list \
  --source-resource-id "$KV_ID" \
  --output table

Step 7 — Verify application health

curl -i "${APP_URL}/health"

Expected:

HTTP/1.1 200 OK

Final validation checklist

Component

Expected state

Azure Key Vault

Available

Azure RBAC

Enabled

Secret version 1

Exists

Secret version 2

Exists

Certificate version 1

Exists

Certificate version 2

Exists

Managed identity

Enabled

Key Vault secret access

Authorized

App Service reference

Versionless

Application health

HTTP 200

Rotation Event Grid subscription

Provisioned

Expiry Event Grid subscription

Provisioned

Azure Monitor HTTP 5xx alert

Enabled

Lab 19 — Cleanup

Azure App Service and supporting resources continue generating charges while deployed.

Delete the environment after completing the workshop.

Step 1 — Review deployed resources

az resource list \
  --resource-group "$RG" \
  --query "[].{Name:name,Type:type}" \
  --output table

Step 2 — Delete the resource group

az group delete \
  --name "$RG" \
  --yes

This removes the Web App, App Service Plan, Storage Account, Event Grid subscriptions, monitoring resources, and other resources within the resource group.

The Key Vault enters a soft-deleted state because soft delete is enabled.

Step 3 — Verify resource group deletion

az group exists \
  --name "$RG"

Expected:

false

Step 4 — Inspect the deleted vault

az keyvault list-deleted \
  --query "[?name=='${KV}']" \
  --output table

A soft-deleted Key Vault may remain recoverable for the configured retention period.

This is expected behavior, not an indication that the resource group deletion failed.

Do not permanently purge the vault unless doing so is appropriate for your test environment and you understand the irreversibility of that operation.

Step 5 — Remove locally generated files

cd ..

rm -rf kv-rotation-app
rm -f cert-policy.json
rm -f cert-policy-custom.json

Step 6 — Clear temporary credential variables

unset STORAGE_KEY
unset TOKEN_V1
unset TOKEN_V2

Cleanup complete.

Workshop Review — What Have We Learned?

Azure feature

Learning objective

Azure Key Vault

Secure secret and certificate storage

Azure RBAC

Identity-based authorization

Managed Identity

Application authentication without stored credentials

Secret Versions

Credential lifecycle and controlled rotation

App Service Key Vault References

Secret consumption through application configuration

Versionless References

Following the latest secret version

Certificate Policies

Certificate validity and lifecycle actions

Certificate Contacts

Expiration-related email notifications

Event Grid

Event-driven lifecycle monitoring

Storage Queues

Durable delivery of lifecycle events

Azure Monitor

Application error monitoring

Azure CLI

Deployment, diagnosis, repair, and cleanup

Key lessons

1. Creating a new secret version isn't enough

Rotation is only complete when dependent systems can successfully use the replacement credential.

2. Older versions remain addressable

Key Vault maintains versions rather than automatically deleting the previous value when another version is created.

3. Versionless references simplify rotation

App Service can follow the latest secret version without explicitly updating the version ID.

However, application refresh behavior and credential compatibility still matter.

4. Managed identity removes the need for stored application credentials

The application can retrieve secrets using its Azure identity and appropriately scoped RBAC permissions.

5. Expiration monitoring is part of operations

Certificates and secrets should have documented lifetimes, owners, and notification procedures.

6. Monitoring should cover both the secret store and its consumers

A healthy Key Vault does not guarantee that the application using its secrets is healthy.

Final Challenge — Test Your Knowledge

Before finishing, consider these questions:

  1. What happens when you create another secret with the same name?

  2. What is the difference between version-specific and versionless Key Vault references?

  3. How long can App Service cache a Key Vault secret reference?

  4. Why is a managed identity preferable to storing service principal credentials?

  5. What does the Key Vault Secrets User role allow?

  6. Does creating a new secret version automatically disable previous versions?

  7. Can an application fail even when Key Vault is available?

  8. What is the difference between certificate expiration and certificate renewal?

  9. When does Key Vault publish a SecretNearExpiry event?

  10. Why should an application be monitored during credential rotation?

Bonus challenges

Challenge 1 — RBAC failure injection

Remove the application's Key Vault Secrets User role, allow RBAC and reference caches to update, investigate the authorization failure, and restore permissions.

Challenge 2 — Certificate-backed application

Extend the workshop with a TLS-enabled test workload that consumes an issued certificate. Rotate the certificate and validate that the workload actually reloads the replacement.

Challenge 3 — Automated rotation workflow

Use Event Grid with an Azure Function or Azure Container Apps job to begin an approved rotation workflow.

Challenge 4 — Private Key Vault access

Configure a Private Endpoint, private DNS resolution, and App Service VNet integration. Verify that the application can access Key Vault without public data-plane connectivity.

Challenge 5 — Centralized monitoring

Forward Key Vault audit logs to Log Analytics, build KQL queries for failed access attempts, and define operational alerts for suspicious activity.

Conclusion

Azure Key Vault is more than a secure location to store passwords, API keys, and certificates.

It is a critical component of application security and operational reliability.

Secret rotation, certificate renewal, application identity, and expiration monitoring must work together.

In this workshop, we deployed a Key Vault, configured Azure RBAC, connected an application through a managed identity, generated multiple secret and certificate versions, and established event-driven monitoring.

We also deliberately introduced incorrect secret-version references and observed their impact on a dependent service.

Then we diagnosed the problem and restored the application using a versionless Key Vault reference.

The most important lesson is that rotating a credential is not the same as successfully rotating every application that depends on it.

Effective Azure operations require both secure credential management and reliable recovery procedures.

Build. Secure. Rotate. Break. Diagnose. Repair. Verify.

Official Microsoft Documentation

Azure Workshopz | Beyond Cloud with Chriz

Hands-on Azure workshops for engineers who want to understand deployment, security, troubleshooting, and recovery.

 
 
 

Comments


bottom of page