
Azure Workshopz | Beyond Cloud with Chriz
DEPLOY • SECURE • ROTATE • BREAK • DIAGNOSE • REPAIR • VERIFY • CLEAN UP
Level: Intermediate to AdvancedPlatform: Microsoft AzureTools: Azure CLI, Azure Cloud Shell (Bash)Deployment: Azure CLI only — no Terraform, Bicep, or Azure Portal configuration stepsTechnologies: Azure Key Vault, Managed Identity, Azure RBAC, Azure App Service, Event Grid, Azure Storage Queues, Azure MonitorCost: Low to Medium, depending on App Service runtime and monitoring usage
Introduction
Your application is running successfully.
Its configuration is stored securely in Azure Key Vault, and access is controlled through a managed identity.
Everything looks healthy.
But what happens when your application is still referencing an older secret version after credentials have been rotated?
What happens when a certificate approaches expiration?
And how do you detect and recover from these situations without manually copying credentials into application configuration?
In this workshop, we're going to build a complete Azure Key Vault Secret and Certificate Lifecycle Management environment.
We will create and rotate secrets, generate certificate versions, configure expiration policies, and establish event-driven monitoring.
More importantly, we will connect Azure Key Vault to a real Azure App Service application.
Then we will deliberately introduce a secret-version mismatch, observe the application return an error, investigate the configuration, and restore the service using a versionless Key Vault reference.
Everything will be managed through Azure CLI.
The goal is to understand how secret and certificate rotation affects real applications, not simply how to store a secret.
What you'll learn
By the end of this workshop, you will understand how to:
Deploy Azure Key Vault using Azure RBAC.
Configure least-privilege access for application identities.
Create secrets with expiration dates.
Manage multiple versions of a secret.
Deploy an App Service application using a managed identity.
Configure version-specific and versionless Key Vault references.
Simulate a secret-rotation failure.
Diagnose and repair broken application configuration.
Generate and renew self-signed certificates.
Configure certificate expiration and email notification policies.
Capture Key Vault lifecycle events using Event Grid.
Configure Azure Monitor alerts for application failures.
Verify rotation behavior and clean up the environment.
Workshop Architecture
AZURE CLOUD SHELL
|
Azure CLI
|
v
+---------------------------+
| AZURE KEY VAULT |
| |
| Secrets |
| - Version 1 |
| - Version 2 |
| |
| Certificates |
| - Certificate Version 1 |
| - Certificate Version 2 |
| |
| Expiration Policies |
+------------+--------------+
|
+-------------+-------------+
| |
v v
+-------------------+ +-------------------+
| AZURE APP SERVICE| | EVENT GRID |
| | | |
| Managed Identity | | Secret Rotation |
| Key Vault Ref | | Cert Rotation |
| | | Near Expiration |
| /health | +---------+---------+
+---------+---------+ |
| v
| +-------------------+
| | STORAGE QUEUES |
| | |
| | Rotation Events |
| | Expiration Events |
| +-------------------+
|
v
+-------------------+
| AZURE MONITOR |
| |
| HTTP 5xx Alerts |
| Action Group |
+-------------------+Our failure scenario
Initially, the application receives secret version 1.
We then create version 2.
The application is deliberately configured to require version 2 while its Key Vault reference remains pinned to version 1.
BEFORE ROTATION
App Service
|
v
Secret Version 1
|
v
HTTP 200 — HEALTHY
AFTER ROTATION
App Service expects Version 2
|
v
Pinned reference still uses Version 1
|
v
HTTP 503 — UNHEALTHY
AFTER REPAIR
App Service
|
v
Versionless Key Vault Reference
|
v
Latest Secret Version
|
v
HTTP 200 — HEALTHYThis is an intentional application-level compatibility test. Key Vault itself does not automatically invalidate an older secret version when you create a newer one.
Lab 1 — Prepare the Azure Environment
Step 1 — Open Azure Cloud Shell
Open:
Select Bash.
Check the Azure CLI version:
az versionCheck the active subscription:
az account show \
--query "{Name:name,SubscriptionId:id}" \
--output tableIf needed, select your subscription:
az account set \
--subscription "<YOUR-SUBSCRIPTION-ID>"Step 2 — Verify the required tools
az --version
jq --version
zip -v | head -n 2The workshop uses Azure CLI to manage all Azure resources. Bash, jq, zip, and a small Python test application are used locally to prepare payloads and simulate the application dependency.
Step 3 — Configure variables
# General settings
export LOCATION="westeurope"
export SUFFIX=$(openssl rand -hex 3)
# Resource group
export RG="rg-kv-rotation-$SUFFIX"
# Key Vault
export KV="kvrot$SUFFIX"
export SECRET="workshop-api-token"
export CERT="workshop-tls-cert"
# App Service
export PLAN="plan-kv-rotation-$SUFFIX"
export APP="app-kv-rotation-$SUFFIX"
# Event Grid and Storage
export STORAGE="stkvrot$SUFFIX"
export QUEUE_ROTATION="rotation-events"
export QUEUE_EXPIRY="expiry-events"
export SUB_ROTATION="kv-rotation-sub"
export SUB_EXPIRY="kv-expiry-sub"
# Azure Monitor
export ACTION_GROUP="ag-kv-rotation"
export HTTP_ALERT="alert-app-secret-failure"
# Optional notification address
export ALERT_EMAIL=""If you want to receive email notifications, set ALERT_EMAIL to an address you control before reaching the monitoring section.
Display the resource names:
echo "Resource Group: $RG"
echo "Key Vault: $KV"
echo "App Service: $APP"
echo "Storage: $STORAGE"
echo "Region: $LOCATION"Step 4 — Create the resource group
az group create \
--name "$RG" \
--location "$LOCATION" \
--tags Environment=Workshop \
Project=Azure-KeyVault-RotationVerify:
az group show \
--name "$RG" \
--query "{Name:name,Location:location,State:properties.provisioningState}" \
--output tableExpected result: Resource group provisioning succeeds.
Lab 2 — Deploy Azure Key Vault
We will deploy a Key Vault using Azure RBAC authorization.
This means that access to secrets and certificates is controlled through Azure role assignments rather than legacy vault access policies.
Step 1 — Create the vault
az keyvault create \
--resource-group "$RG" \
--name "$KV" \
--location "$LOCATION" \
--sku standard \
--enable-rbac-authorization true \
--retention-days 7This lab uses the minimum supported soft-delete retention period to reduce the duration of a deleted workshop vault.
For production workloads, enable purge protection according to your organization's security requirements.
Important: Purge protection cannot be disabled after it is enabled.
Step 2 — Verify Key Vault configuration
az keyvault show \
--resource-group "$RG" \
--name "$KV" \
--query "{
Name:name,
Location:location,
RBAC:properties.enableRbacAuthorization,
SoftDelete:properties.enableSoftDelete,
PurgeProtection:properties.enablePurgeProtection
}" \
--output jsonStep 3 — Retrieve the Key Vault resource ID
export KV_ID=$(az keyvault show \
--resource-group "$RG" \
--name "$KV" \
--query id \
--output tsv)
echo "$KV_ID"Step 4 — Configure workshop administrator access
Azure RBAC distinguishes between management-plane permissions and Key Vault data-plane permissions.
Being able to create a vault doesn't automatically mean you can create secrets or certificates inside it.
For this lab, assign your signed-in user the Key Vault Administrator role.
export USER_ID=$(az ad signed-in-user show \
--query id \
--output tsv)az role assignment create \
--assignee-object-id "$USER_ID" \
--assignee-principal-type User \
--role "Key Vault Administrator" \
--scope "$KV_ID"This step assumes an interactive user login and permission to create role assignments, such as Owner or Role Based Access Control Administrator with suitable scope.
For production systems, separate secret management, certificate management, and secret reading into dedicated roles.
Step 5 — Verify access
az keyvault secret list \
--vault-name "$KV" \
--output tableIf you receive a Forbidden error immediately after role assignment, retry after RBAC propagation.
Checkpoint: Continue only after the signed-in user can query the vault's data plane.
Lab 3 — Prepare Event Grid Monitoring
Before creating any secrets or certificates, let's configure event collection.
This allows us to observe actual rotation events during the remaining exercises.
We will use two Storage Queues:
rotation-events — new secret and certificate versions.
expiry-events — near-expiry and expired objects.
Step 1 — Create a Storage Account
az storage account create \
--resource-group "$RG" \
--name "$STORAGE" \
--location "$LOCATION" \
--sku Standard_LRS \
--kind StorageV2 \
--min-tls-version TLS1_2Step 2 — Retrieve its resource ID
export STORAGE_ID=$(az storage account show \
--resource-group "$RG" \
--name "$STORAGE" \
--query id \
--output tsv)Step 3 — Retrieve a temporary storage key
For the lab, we will use a storage account key to create and inspect queue messages.
export STORAGE_KEY=$(az storage account keys list \
--resource-group "$RG" \
--account-name "$STORAGE" \
--query "[0].value" \
--output tsv)The key must not be displayed in workshop screenshots or committed to Git.
In production, prefer Microsoft Entra authorization and managed identities for supported queue operations.
Step 4 — Create the queues
az storage queue create \
--name "$QUEUE_ROTATION" \
--account-name "$STORAGE" \
--account-key "$STORAGE_KEY"az storage queue create \
--name "$QUEUE_EXPIRY" \
--account-name "$STORAGE" \
--account-key "$STORAGE_KEY"Step 5 — Retrieve queue resource IDs
export ROTATION_QUEUE_ID="${STORAGE_ID}/queueServices/default/queues/${QUEUE_ROTATION}"
export EXPIRY_QUEUE_ID="${STORAGE_ID}/queueServices/default/queues/${QUEUE_EXPIRY}"Step 6 — Create the rotation event subscription
az eventgrid event-subscription create \
--name "$SUB_ROTATION" \
--source-resource-id "$KV_ID" \
--endpoint-type storagequeue \
--endpoint "$ROTATION_QUEUE_ID" \
--included-event-types \
Microsoft.KeyVault.SecretNewVersionCreated \
Microsoft.KeyVault.CertificateNewVersionCreatedStep 7 — Create the expiration event subscription
az eventgrid event-subscription create \
--name "$SUB_EXPIRY" \
--source-resource-id "$KV_ID" \
--endpoint-type storagequeue \
--endpoint "$EXPIRY_QUEUE_ID" \
--included-event-types \
Microsoft.KeyVault.SecretNearExpiry \
Microsoft.KeyVault.SecretExpired \
Microsoft.KeyVault.CertificateNearExpiry \
Microsoft.KeyVault.CertificateExpiredStep 8 — Verify subscriptions
az eventgrid event-subscription list \
--source-resource-id "$KV_ID" \
--output tableExpected result: Both event subscriptions exist.
This architecture gives us event-driven visibility into secret and certificate lifecycle changes.
An event arriving in the expiry queue is not automatically an email alert. In production, connect the queue to a processing service that sends notifications, opens tickets, or initiates approved rotation workflows.
Lab 4 — Create the First Secret Version
We'll start with a synthetic API token.
Never use actual application passwords or production API keys in a public workshop.
Step 1 — Generate the first test value
export TOKEN_V1="lab-token-v1-$SUFFIX"
export EXPIRY_V1=$(date -u -d "+60 days" \
+"%Y-%m-%dT%H:%M:%SZ")Step 2 — Create the secret
az keyvault secret set \
--vault-name "$KV" \
--name "$SECRET" \
--value "$TOKEN_V1" \
--expires "$EXPIRY_V1" \
--tags Environment=Workshop \
Rotation=Version1 \
--query "{Id:id,Enabled:attributes.enabled,Expires:attributes.expires}" \
--output jsonStep 3 — Retrieve the secret version ID
export SECRET_V1_ID=$(az keyvault secret show \
--vault-name "$KV" \
--name "$SECRET" \
--query id \
--output tsv)
echo "$SECRET_V1_ID"A secret version ID looks similar to:
Each new version receives a unique version identifier.
Step 4 — Inspect secret metadata
az keyvault secret show \
--vault-name "$KV" \
--name "$SECRET" \
--query "{
Id:id,
Name:name,
Enabled:attributes.enabled,
Expires:attributes.expires,
Created:attributes.created
}" \
--output jsonNotice that we are inspecting metadata without returning the secret value.
Lab 5 — Deploy a Dependent Azure App Service
Now we'll create an application that uses the Key Vault secret.
This is important because secret rotation can affect real applications.
Our application will compare its resolved secret with a synthetic expected value.
It will never return the secret itself in an HTTP response.
Step 1 — Create a Linux App Service Plan
az appservice plan create \
--resource-group "$RG" \
--name "$PLAN" \
--location "$LOCATION" \
--sku B1 \
--is-linuxStep 2 — Create the Web App
az webapp create \
--resource-group "$RG" \
--plan "$PLAN" \
--name "$APP" \
--runtime "PYTHON:3.12"Check available Python runtimes if your region or App Service environment does not support the selected version:
az webapp list-runtimes \
--os linux \
--output tableStep 3 — Enable managed identity
export APP_PRINCIPAL_ID=$(az webapp identity assign \
--resource-group "$RG" \
--name "$APP" \
--query principalId \
--output tsv)
echo "$APP_PRINCIPAL_ID"This creates a system-assigned managed identity for the application.
No client secret or stored service principal password is required.
Step 4 — Grant access to Key Vault secrets
az role assignment create \
--assignee-object-id "$APP_PRINCIPAL_ID" \
--assignee-principal-type ServicePrincipal \
--role "Key Vault Secrets User" \
--scope "$KV_ID"The application identity can now read secrets from this Key Vault after RBAC propagation.
It does not have permission to create, update, or delete secrets.
Step 5 — Create a small health-check application
We will create a Python application using only the standard library.
mkdir -p kv-rotation-app
cd kv-rotation-appCreate app.py:
cat > app.py <<'PY'
import os
from http.server import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path not in ("/", "/health"):
self.send_response(404)
self.end_headers()
return
actual = os.environ.get("APP_API_TOKEN", "")
expected = os.environ.get("APP_EXPECTED_TOKEN", "")
healthy = (
bool(actual)
and bool(expected)
and actual == expected
)
status = 200 if healthy else 503
message = (
"HEALTHY: credential matches expected version"
if healthy
else "UNHEALTHY: credential version mismatch"
)
self.send_response(status)
self.send_header("Content-Type", "text/plain")
self.end_headers()
self.wfile.write(message.encode())
port = int(os.environ.get("PORT", "8000"))
HTTPServer(("0.0.0.0", port), Handler).serve_forever()
PYThis is a deliberately simplified credential-compatibility simulator.
In a production application, credentials should be validated by the downstream service rather than compared with another plaintext application setting.
Step 6 — Package the application
zip app.zip app.pyStep 7 — Configure App Service startup
az webapp config set \
--resource-group "$RG" \
--name "$APP" \
--startup-file "python3 app.py"Step 8 — Configure the Key Vault reference
Initially, pin the application to secret version 1.
az webapp config appsettings set \
--resource-group "$RG" \
--name "$APP" \
--settings \
"APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$SECRET_V1_ID)" \
"APP_EXPECTED_TOKEN=$TOKEN_V1"This is a version-specific reference.
App Service will attempt to resolve the secret through its managed identity.
The Key Vault reference itself is not a plaintext copy of the secret.
Step 9 — Deploy the application
az webapp deploy \
--resource-group "$RG" \
--name "$APP" \
--src-path app.zip \
--type zipStep 10 — Test application health
export APP_URL="https://${APP}.azurewebsites.net"curl -i "${APP_URL}/health"Expected response:
HTTP/1.1 200 OK
HEALTHY: credential matches expected versionCheckpoint: Confirm HTTP 200 before continuing.
If the application fails at this stage, verify managed identity permissions, Key Vault reference resolution, and the App Service startup command.
Lab 6 — Inspect Key Vault Reference Resolution
Azure App Service exposes the resolution status of Key Vault references through its management API.
This allows us to diagnose a failed reference without displaying the actual secret.
Step 1 — Retrieve the Web App resource ID
export APP_ID=$(az webapp show \
--resource-group "$RG" \
--name "$APP" \
--query id \
--output tsv)Step 2 — Inspect reference status
az rest \
--method get \
--uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
--output jsonLook for the APP_API_TOKEN configuration reference.
Verify its resolution status, Key Vault name, and referenced secret version.
A healthy reference should report a successful resolution status.
This diagnostic API becomes especially useful when an application cannot access a secret because of missing RBAC permissions, an invalid version, or an unavailable vault.
Lab 7 — ROTATE: Create Secret Version 2
Now we'll perform our first secret rotation.
Step 1 — Generate the new value
export TOKEN_V2="lab-token-v2-$SUFFIX"
export EXPIRY_V2=$(date -u -d "+90 days" \
+"%Y-%m-%dT%H:%M:%SZ")Step 2 — Create another version
az keyvault secret set \
--vault-name "$KV" \
--name "$SECRET" \
--value "$TOKEN_V2" \
--expires "$EXPIRY_V2" \
--tags Environment=Workshop \
Rotation=Version2 \
--query "{Id:id,Expires:attributes.expires}" \
--output jsonKey Vault automatically creates a new version because the secret already exists.
Step 3 — Retrieve version 2
export SECRET_V2_ID=$(az keyvault secret show \
--vault-name "$KV" \
--name "$SECRET" \
--query id \
--output tsv)Step 4 — Compare versions
echo "Version 1: $SECRET_V1_ID"
echo "Version 2: $SECRET_V2_ID"Both identifiers should be different.
Step 5 — List all secret versions
az keyvault secret list-versions \
--vault-name "$KV" \
--name "$SECRET" \
--query "[].{
Id:id,
Enabled:attributes.enabled,
Expires:attributes.expires
}" \
--output tableKey concept: Creating version 2 does not automatically disable or revoke version 1.
Older versions remain available unless their attributes or the vault's retention state prevent access.
Lab 8 — BREAK: Simulate a Stale Secret Version
This is the main failure scenario.
We have rotated the secret in Azure Key Vault.
However, our application is still pinned to version 1.
Let's simulate a downstream system requiring version 2.
Step 1 — Update the expected credential
az webapp config appsettings set \
--resource-group "$RG" \
--name "$APP" \
--settings "APP_EXPECTED_TOKEN=$TOKEN_V2"The application now expects the rotated credential.
But its Key Vault reference still points to version 1.
Step 2 — Test the application
curl -i "${APP_URL}/health"Expected response:
HTTP/1.1 503 Service Unavailable
UNHEALTHY: credential version mismatchWe have deliberately created an application-level incident.
The Key Vault is still available.
The secret exists.
The managed identity can still read it.
But the application is consuming an outdated version.
Step 3 — Confirm the configured reference
az webapp config appsettings list \
--resource-group "$RG" \
--name "$APP" \
--query "[?name=='APP_API_TOKEN'].name" \
--output tableNow inspect the reference resolution metadata:
az rest \
--method get \
--uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
--output jsonCompare the reported secret version with the current version stored in Key Vault.
Step 4 — Check the latest Key Vault version
az keyvault secret show \
--vault-name "$KV" \
--name "$SECRET" \
--query id \
--output tsvThe latest version is version 2, while the application remains pinned to version 1.
What happened?
This is not a Key Vault outage.
It is a secret-consumer configuration problem.
Secret rotation requires more than generating a new credential. Applications must also transition to the replacement credential.
Lab 9 — REPAIR: Use a Versionless Reference
The recommended approach for consumers that should automatically follow the latest secret version is to use a versionless Key Vault reference.
Step 1 — Construct the versionless URI
export SECRET_URI="https://${KV}.vault.azure.net/secrets/${SECRET}"This URI references the secret name rather than one specific version.
Step 2 — Update App Service
az webapp config appsettings set \
--resource-group "$RG" \
--name "$APP" \
--settings \
"APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$SECRET_URI)"Step 3 — Refresh Key Vault references
Azure App Service caches resolved Key Vault references.
A versionless reference normally picks up a new secret version within 24 hours.
We can explicitly request a refresh:
az rest \
--method post \
--uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings/refresh?api-version=2022-03-01"Step 4 — Restart the Web App
az webapp restart \
--resource-group "$RG" \
--name "$APP"Step 5 — Verify recovery
curl -i "${APP_URL}/health"Expected response:
HTTP/1.1 200 OK
HEALTHY: credential matches expected versionThe service is restored.
What did we learn?
Version-specific references provide deterministic version selection but require explicit updating when credentials change.
Versionless references allow a consumer to resolve the latest version.
However, the new credential must already be valid in the downstream system before the consuming application switches to it.
In production, a safe rotation usually involves overlapping credential validity, validating the new credential, switching consumers, and only then retiring the previous credential.
Lab 10 — Create a Self-Signed Certificate
Now let's move from secrets to certificates.
Azure Key Vault can manage X.509 certificates, their private keys, metadata, and lifecycle policies.
For this workshop, we'll use a self-signed certificate.
It is suitable for learning but must not be treated as a publicly trusted production TLS certificate.
Step 1 — Generate a default policy
az keyvault certificate get-default-policy \
> cert-policy.jsonStep 2 — Customize the policy
We'll configure:
Self-signed certificate.
RSA key.
2048-bit key size.
Three-month validity.
Expiration email notification 30 days before expiry.
jq '
.issuer.name = "Self" |
.key_props.kty = "RSA" |
.key_props.key_size = 2048 |
.key_props.reuse_key = false |
.x509_props.subject = "CN=rotation.internal.example" |
.x509_props.validity_months = 3 |
.lifetime_actions = [
{
"trigger": {
"days_before_expiry": 30
},
"action": {
"action_type": "EmailContacts"
}
}
]
' cert-policy.json > cert-policy-custom.jsonStep 3 — Inspect the policy
cat cert-policy-custom.jsonThe EmailContacts action instructs Key Vault to send notifications to configured certificate contacts when its lifecycle trigger is reached.
It does not send an email immediately when the policy is saved.
Step 4 — Create the certificate
az keyvault certificate create \
--vault-name "$KV" \
--name "$CERT" \
--policy "@cert-policy-custom.json"Step 5 — Inspect the certificate
az keyvault certificate show \
--vault-name "$KV" \
--name "$CERT" \
--query "{
Id:id,
Enabled:attributes.enabled,
Expires:attributes.expires
}" \
--output jsonStep 6 — Record certificate version 1
export CERT_V1_ID=$(az keyvault certificate show \
--vault-name "$KV" \
--name "$CERT" \
--query id \
--output tsv)Checkpoint: Certificate version 1 exists with the expected lifetime policy.
Lab 11 — Configure Certificate Expiration Contacts
Azure Key Vault certificate lifecycle policies can send expiration-related emails to vault certificate contacts.
Step 1 — Add a contact
If you configured ALERT_EMAIL, execute:
if [[ -n "$ALERT_EMAIL" ]]; then
az keyvault certificate contact add \
--vault-name "$KV" \
--email "$ALERT_EMAIL"
fiStep 2 — Verify contacts
az keyvault certificate contact list \
--vault-name "$KV" \
--output tableStep 3 — Inspect the certificate lifecycle policy
az keyvault certificate show \
--vault-name "$KV" \
--name "$CERT" \
--query "policy.lifetime_actions" \
--output jsonCertificate expiration behavior
With our policy:
The certificate has a three-month validity period.
The lifecycle action is scheduled for 30 days before expiration.
Key Vault can email configured certificate contacts when that trigger is reached.
Important: We are configuring EmailContacts, not automatic renewal.
The certificate will not be automatically replaced simply because we configured an email notification policy.
Lab 12 — ROTATE: Generate Certificate Version 2
Now we'll renew the certificate by creating another version.
Step 1 — Create the new certificate version
az keyvault certificate create \
--vault-name "$KV" \
--name "$CERT" \
--policy "@cert-policy-custom.json"Step 2 — Retrieve the new version ID
export CERT_V2_ID=$(az keyvault certificate show \
--vault-name "$KV" \
--name "$CERT" \
--query id \
--output tsv)Step 3 — Compare versions
echo "Certificate Version 1: $CERT_V1_ID"
echo "Certificate Version 2: $CERT_V2_ID"Step 4 — List certificate versions
az keyvault certificate list-versions \
--vault-name "$KV" \
--name "$CERT" \
--query "[].{
Id:id,
Enabled:attributes.enabled,
Expires:attributes.expires
}" \
--output tableExpected result: Two certificate versions with distinct version IDs.
Important certificate lifecycle distinction
Creating a new version in Key Vault does not automatically replace a certificate that an external web server, load balancer, or application has already loaded.
The consumer must support certificate refresh or have a deployment process that installs the new certificate version.
For this workshop, we validate Key Vault certificate issuance and versioning. We are not claiming that the App Service's public HTTPS certificate has been replaced.
Lab 13 — Verify Event Grid Rotation Events
We created Event Grid subscriptions before creating the secrets and certificates.
Let's inspect the rotation queue.
Step 1 — Peek messages
az storage message peek \
--queue-name "$QUEUE_ROTATION" \
--account-name "$STORAGE" \
--account-key "$STORAGE_KEY" \
--num-messages 10 \
--output jsonLook for events such as:
Microsoft.KeyVault.SecretNewVersionCreated
Microsoft.KeyVault.CertificateNewVersionCreatedA certificate operation can also affect a corresponding Key Vault secret object because Key Vault certificates have associated secret representations.
Step 2 — Inspect the event subscription
az eventgrid event-subscription show \
--name "$SUB_ROTATION" \
--source-resource-id "$KV_ID" \
--output jsonVerify:
Provisioning state.
Event type filters.
Storage Queue endpoint.
Delivery settings.
Step 3 — Understand the event contents
A Key Vault rotation event identifies the changed object, its version, and the source vault.
It does not include the secret value.
This distinction is important for designing secure event-driven automation.
Lab 14 — Configure Secret Expiration Monitoring
Key Vault can publish events when the current version of a secret is about to expire or has expired.
The built-in SecretNearExpiry event is normally generated 30 days before the configured expiration.
Let's create an additional demonstration secret with a short expiration period and inspect the monitoring configuration.
Step 1 — Create a short-lived secret
export TEST_EXPIRY=$(date -u -d "+10 days" \
+"%Y-%m-%dT%H:%M:%SZ")az keyvault secret set \
--vault-name "$KV" \
--name "short-lived-demo" \
--value "synthetic-expiry-test" \
--expires "$TEST_EXPIRY" \
--query "{Name:name,Expires:attributes.expires}" \
--output jsonStep 2 — Inspect expiration dates
az keyvault secret list \
--vault-name "$KV" \
--query "[].{
Name:name,
Expires:attributes.expires,
Enabled:attributes.enabled
}" \
--output tableStep 3 — Inspect the expiration queue
az storage message peek \
--queue-name "$QUEUE_EXPIRY" \
--account-name "$STORAGE" \
--account-key "$STORAGE_KEY" \
--num-messages 10 \
--output jsonImportant event-timing limitation
Creating a secret that expires in 10 days does not guarantee that SecretNearExpiry will be published immediately.
Near-expiry events are generated by Key Vault's lifecycle service according to the documented expiration schedule.
The short-lived secret demonstrates the expiration metadata and event-subscription configuration, not a guaranteed immediate near-expiry event.
For a deterministic workshop test, use the SecretNewVersionCreated event from our previous rotation exercise.
For production expiration monitoring, supplement lifecycle events with periodic inventory checks and a defined notification workflow.
Lab 15 — Configure Azure Monitor Failure Alerts
We now have two monitoring mechanisms:
Key Vault lifecycle notifications through Event Grid.
Application availability monitoring through Azure Monitor.
We'll configure an HTTP 5xx alert for the App Service.
This allows us to detect the application-level failure introduced by our secret-version mismatch.
Step 1 — Create an Action Group
If you supplied an email address, create an email-enabled action group.
if [[ -n "$ALERT_EMAIL" ]]; then
az monitor action-group create \
--resource-group "$RG" \
--name "$ACTION_GROUP" \
--short-name "KVRotLab" \
--action email Operations "$ALERT_EMAIL"
fiRetrieve its ID if it was created:
export ACTION_GROUP_ID=""
if [[ -n "$ALERT_EMAIL" ]]; then
ACTION_GROUP_ID=$(az monitor action-group show \
--resource-group "$RG" \
--name "$ACTION_GROUP" \
--query id \
--output tsv)
fiStep 2 — Configure an HTTP 5xx metric alert
ALERT_ACTION_ARGS=()
if [[ -n "$ACTION_GROUP_ID" ]]; then
ALERT_ACTION_ARGS=(--action "$ACTION_GROUP_ID")
fiaz monitor metrics alert create \
--resource-group "$RG" \
--name "$HTTP_ALERT" \
--scopes "$APP_ID" \
--condition "total Http5xx > 0" \
--window-size 5m \
--evaluation-frequency 1m \
--severity 2 \
--description "App Service secret rotation failure" \
"${ALERT_ACTION_ARGS[@]}"Step 3 — Verify the alert rule
az monitor metrics alert show \
--resource-group "$RG" \
--name "$HTTP_ALERT" \
--query "{
Name:name,
Enabled:enabled,
Severity:severity,
Scopes:scopes
}" \
--output jsonStep 4 — Review the metric
az monitor metrics list \
--resource "$APP_ID" \
--metric "Http5xx" \
--interval PT1M \
--aggregation Total \
--output jsonA repeated HTTP 503 response should contribute to the Web App's Http5xx metric once the telemetry is available.
The alert does not fire instantly. Azure Monitor evaluates the metric according to the configured aggregation window and evaluation frequency.
Lab 16 — BREAK AGAIN: Invalid Secret Reference
Let's perform a second failure simulation.
This time, rather than referencing a valid but outdated secret, we'll reference a nonexistent secret version.
Step 1 — Configure an invalid version
export BAD_VERSION="00000000000000000000000000000000"
export BAD_URI="${SECRET_URI}/${BAD_VERSION}"Step 2 — Apply the invalid reference
az webapp config appsettings set \
--resource-group "$RG" \
--name "$APP" \
--settings \
"APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$BAD_URI)"Step 3 — Force a reference refresh
az rest \
--method post \
--uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings/refresh?api-version=2022-03-01"Restart the application:
az webapp restart \
--resource-group "$RG" \
--name "$APP"Step 4 — Inspect resolution failure
az rest \
--method get \
--uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
--output jsonExpected behavior: The Key Vault reference reports a resolution problem because the specified version does not exist.
Step 5 — Test application health
curl -i "${APP_URL}/health"Expected:
HTTP/1.1 503 Service UnavailableThe application cannot obtain the expected credential through the invalid reference.
Step 6 — Inspect HTTP errors
az monitor metrics list \
--resource "$APP_ID" \
--metric "Http5xx" \
--interval PT1M \
--aggregation Total \
--output jsonThis allows us to correlate the configuration failure with application-level monitoring.
Lab 17 — REPAIR: Restore the Correct Reference
Step 1 — Restore the versionless URI
az webapp config appsettings set \
--resource-group "$RG" \
--name "$APP" \
--settings \
"APP_API_TOKEN=@Microsoft.KeyVault(SecretUri=$SECRET_URI)"Step 2 — Refresh Key Vault references
az rest \
--method post \
--uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings/refresh?api-version=2022-03-01"Step 3 — Restart the application
az webapp restart \
--resource-group "$RG" \
--name "$APP"Step 4 — Verify the application
curl -i "${APP_URL}/health"Expected:
HTTP/1.1 200 OK
HEALTHY: credential matches expected versionStep 5 — Verify resolution status
az rest \
--method get \
--uri "https://management.azure.com${APP_ID}/config/configreferences/appsettings?api-version=2022-03-01" \
--output jsonThe configuration reference should resolve successfully again.
We have restored the dependent application without deleting or recreating Azure Key Vault.
Lab 18 — Final Verification
Before cleanup, let's validate the complete environment.
Step 1 — Verify the Key Vault
az keyvault show \
--resource-group "$RG" \
--name "$KV" \
--query "{Name:name,RBAC:properties.enableRbacAuthorization}" \
--output tableStep 2 — List secret versions
az keyvault secret list-versions \
--vault-name "$KV" \
--name "$SECRET" \
--output tableStep 3 — List certificate versions
az keyvault certificate list-versions \
--vault-name "$KV" \
--name "$CERT" \
--output tableStep 4 — Verify application identity
az webapp identity show \
--resource-group "$RG" \
--name "$APP" \
--output jsonStep 5 — Verify role assignments
az role assignment list \
--assignee "$APP_PRINCIPAL_ID" \
--scope "$KV_ID" \
--output tableStep 6 — Inspect Event Grid subscriptions
az eventgrid event-subscription list \
--source-resource-id "$KV_ID" \
--output tableStep 7 — Verify application health
curl -i "${APP_URL}/health"Expected:
HTTP/1.1 200 OKFinal validation checklist
Component | Expected state |
Azure Key Vault | Available |
Azure RBAC | Enabled |
Secret version 1 | Exists |
Secret version 2 | Exists |
Certificate version 1 | Exists |
Certificate version 2 | Exists |
Managed identity | Enabled |
Key Vault secret access | Authorized |
App Service reference | Versionless |
Application health | HTTP 200 |
Rotation Event Grid subscription | Provisioned |
Expiry Event Grid subscription | Provisioned |
Azure Monitor HTTP 5xx alert | Enabled |
Lab 19 — Cleanup
Azure App Service and supporting resources continue generating charges while deployed.
Delete the environment after completing the workshop.
Step 1 — Review deployed resources
az resource list \
--resource-group "$RG" \
--query "[].{Name:name,Type:type}" \
--output tableStep 2 — Delete the resource group
az group delete \
--name "$RG" \
--yesThis removes the Web App, App Service Plan, Storage Account, Event Grid subscriptions, monitoring resources, and other resources within the resource group.
The Key Vault enters a soft-deleted state because soft delete is enabled.
Step 3 — Verify resource group deletion
az group exists \
--name "$RG"Expected:
falseStep 4 — Inspect the deleted vault
az keyvault list-deleted \
--query "[?name=='${KV}']" \
--output tableA soft-deleted Key Vault may remain recoverable for the configured retention period.
This is expected behavior, not an indication that the resource group deletion failed.
Do not permanently purge the vault unless doing so is appropriate for your test environment and you understand the irreversibility of that operation.
Step 5 — Remove locally generated files
cd ..
rm -rf kv-rotation-app
rm -f cert-policy.json
rm -f cert-policy-custom.jsonStep 6 — Clear temporary credential variables
unset STORAGE_KEY
unset TOKEN_V1
unset TOKEN_V2Cleanup complete.
Workshop Review — What Have We Learned?
Azure feature | Learning objective |
Azure Key Vault | Secure secret and certificate storage |
Azure RBAC | Identity-based authorization |
Managed Identity | Application authentication without stored credentials |
Secret Versions | Credential lifecycle and controlled rotation |
App Service Key Vault References | Secret consumption through application configuration |
Versionless References | Following the latest secret version |
Certificate Policies | Certificate validity and lifecycle actions |
Certificate Contacts | Expiration-related email notifications |
Event Grid | Event-driven lifecycle monitoring |
Storage Queues | Durable delivery of lifecycle events |
Azure Monitor | Application error monitoring |
Azure CLI | Deployment, diagnosis, repair, and cleanup |
Key lessons
1. Creating a new secret version isn't enough
Rotation is only complete when dependent systems can successfully use the replacement credential.
2. Older versions remain addressable
Key Vault maintains versions rather than automatically deleting the previous value when another version is created.
3. Versionless references simplify rotation
App Service can follow the latest secret version without explicitly updating the version ID.
However, application refresh behavior and credential compatibility still matter.
4. Managed identity removes the need for stored application credentials
The application can retrieve secrets using its Azure identity and appropriately scoped RBAC permissions.
5. Expiration monitoring is part of operations
Certificates and secrets should have documented lifetimes, owners, and notification procedures.
6. Monitoring should cover both the secret store and its consumers
A healthy Key Vault does not guarantee that the application using its secrets is healthy.
Final Challenge — Test Your Knowledge
Before finishing, consider these questions:
What happens when you create another secret with the same name?
What is the difference between version-specific and versionless Key Vault references?
How long can App Service cache a Key Vault secret reference?
Why is a managed identity preferable to storing service principal credentials?
What does the Key Vault Secrets User role allow?
Does creating a new secret version automatically disable previous versions?
Can an application fail even when Key Vault is available?
What is the difference between certificate expiration and certificate renewal?
When does Key Vault publish a SecretNearExpiry event?
Why should an application be monitored during credential rotation?
Bonus challenges
Challenge 1 — RBAC failure injection
Remove the application's Key Vault Secrets User role, allow RBAC and reference caches to update, investigate the authorization failure, and restore permissions.
Challenge 2 — Certificate-backed application
Extend the workshop with a TLS-enabled test workload that consumes an issued certificate. Rotate the certificate and validate that the workload actually reloads the replacement.
Challenge 3 — Automated rotation workflow
Use Event Grid with an Azure Function or Azure Container Apps job to begin an approved rotation workflow.
Challenge 4 — Private Key Vault access
Configure a Private Endpoint, private DNS resolution, and App Service VNet integration. Verify that the application can access Key Vault without public data-plane connectivity.
Challenge 5 — Centralized monitoring
Forward Key Vault audit logs to Log Analytics, build KQL queries for failed access attempts, and define operational alerts for suspicious activity.
Conclusion
Azure Key Vault is more than a secure location to store passwords, API keys, and certificates.
It is a critical component of application security and operational reliability.
Secret rotation, certificate renewal, application identity, and expiration monitoring must work together.
In this workshop, we deployed a Key Vault, configured Azure RBAC, connected an application through a managed identity, generated multiple secret and certificate versions, and established event-driven monitoring.
We also deliberately introduced incorrect secret-version references and observed their impact on a dependent service.
Then we diagnosed the problem and restored the application using a versionless Key Vault reference.
The most important lesson is that rotating a credential is not the same as successfully rotating every application that depends on it.
Effective Azure operations require both secure credential management and reliable recovery procedures.
Build. Secure. Rotate. Break. Diagnose. Repair. Verify.
Official Microsoft Documentation
Azure Workshopz | Beyond Cloud with Chriz
Hands-on Azure workshops for engineers who want to understand deployment, security, troubleshooting, and recovery.
Comments