Replication Troubleshooting¶

Every replication failure surfaces a stable error code. The code drives the UI message, alert grouping, and the docs link. Use it to find the right fix below.

Quick Index¶

Error Codes¶

source_auth_failed¶

What it means: CertifyClouds couldn't authenticate against the source vault (HTTP 401, or 403 on a secret/certificate read path).

Why it happens

• The identity CertifyClouds runs as has no read role assigned on the source vault
• The vault uses the legacy Access Policies model and the principal is not listed
• A service principal secret expired
• Managed identity is disabled on the host

How to fix it

1. Confirm which identity your deployment uses (Settings > Azure Identity or $AZURE_CLIENT_ID)
2. Check role assignments:

   az role assignment list \
     --assignee $PRINCIPAL_ID \
     --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<source-vault>
   

3. Grant Key Vault Secrets User + Key Vault Certificate User + Key Vault Crypto User on the source vault (or assign the custom source role, see RBAC)
4. Wait up to 5 minutes for RBAC propagation, then re-run Validate connection on the config

When to escalate: After confirming the assignments are present and propagated, if validate still fails with source_auth_failed, open a support ticket with the correlation ID from the response header.

target_auth_failed¶

What it means: Authentication failure on the target vault (HTTP 401, or 403 on a write path that isn't specifically a key backup/restore).

Why it happens

• No write role on the target
• Target vault uses Access Policies and the principal isn't listed
• Vault firewall blocks the egress IP of the CertifyClouds deployment

How to fix it

1. Verify write permissions on the target:

   az role assignment list \
     --assignee $PRINCIPAL_ID \
     --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<target-vault>
   

2. Grant Key Vault Secrets Officer + Key Vault Certificates Officer + Key Vault Crypto Officer on the target (or assign the custom target role)
3. If the vault uses a firewall, allow the CertifyClouds egress subnet or public IP
4. Retry the sync from the records table (Replicate now)

When to escalate: If target_auth_failed persists after both RBAC and firewall checks come back clean.

rbac_backup_key¶

What it means: HTTP 403 specifically on the keys/backup data action against the source vault. The engine can read secrets but can't back up keys.

Why it happens: The source role is missing Microsoft.KeyVault/vaults/keys/backup/action. Common when using a hand-rolled role that only covers secrets and certificates.

How to fix it

Grant one of:

• Key Vault Crypto User role on the source vault, or
• The custom source role (includes keys/read for version checks and keys/backup/action)

az role assignment create \
  --assignee $PRINCIPAL_ID \
  --role "Key Vault Crypto User" \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<source-vault>

Then Replicate now on the blocked key records. If key replication isn't required for this pair, edit the replication and untick Keys. Those records stop matching and become disabled (no_longer_matched) after two checks; their target copies are kept.

When to escalate: Not typically needed. This is a pure RBAC fix.

rbac_restore_key¶

What it means: HTTP 403 on the keys/restore data action against the target vault. The backup was fetched successfully but can't be restored.

Why it happens: Target role missing Microsoft.KeyVault/vaults/keys/restore/action.

How to fix it

Grant one of:

• Key Vault Crypto Officer on the target vault, or
• The custom target role

az role assignment create \
  --assignee $PRINCIPAL_ID \
  --role "Key Vault Crypto Officer" \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<target-vault>

When to escalate: Not typically needed.

source_not_found¶

What it means: The source item no longer exists (HTTP 404 on the source read). The record was valid when the last check ran, but the item has since been deleted or renamed. Removing a matching tag or narrowing a rule can also stop management, but does not itself cause a source-read 404.

Why it happens

• User deleted or renamed the item in the source vault

How to fix it: A record is disabled after two consecutive checks without a match (no_longer_matched). With deletion propagation enabled, only target copies created by CertifyClouds are eligible for removal, and only when the source item is absent from both a complete live listing and the soft-deleted-items listing of its vault. A disabled source item is still present, so it is never marked missing in Azure: its record stays managed with the status Source disabled, syncing pauses until it is re-enabled, and its copy is never removed. A source item that has been soft-deleted in Key Vault is also still present, whether or not deletion propagation is on: the record never accrues a missed check, stays managed with the status Source soft-deleted, keeps its copy, and goes back to active when the item is restored. Only permanent deletion starts the missing-item flow. If the soft-deleted-items listing cannot be read, missed checks are held rather than counted, so a record is never disabled on unproven absence; the sync queue shows Source check unavailable until a later check succeeds. A record that stopped matching while its source item still exists keeps its copy; it is re-checked on every later check and the copy is removed once the source item is permanently deleted. A listing or tenant-routing failure refuses all removals for the config in that run and records the reason in the audit log. While a removal is owed, the sync queue shows Removal pending with its cause. Refused or failed removals are retried on the next check, and the replication failure alert is sent once after three failed attempts: one alert for the whole config when the cause is config-wide, such as a refused source listing.

Narrowing a rule, removing a matching tag, or unticking an item type stops management and keeps the target copy while the source item exists. Exclusions retain existing records as excluded, without missed-check counting or target deletion. Records already disabled for not matching keep no_longer_matched while excluded so they come back correctly later.

When an item matches again, the next check re-enables records that were disabled for not matching. Removing an exclusion returns matching records to active management. Explicitly paused records stay paused.

Check now runs one check. Two consecutive misses are required before a record is disabled; it does not promise immediate cleanup.

When to escalate: If records stay in source_not_found for more than two full checks without being disabled, file a bug with the correlation ID.

target_soft_deleted¶

What it means: The target vault has a soft-deleted item with the same name. Azure blocks writes to a name currently in the soft-deleted state until the item is either purged or recovered.

Why it happens: Someone deleted the item on the target vault in a previous sync cycle or out-of-band, and the vault's soft-delete retention period hasn't expired.

How to fix it

The record shows two actions:

1. Open Azure Portal ↗: deep-link to the target vault's deleted items blade. Options there:
   • Recover: bring the existing item back (replication will still try to overwrite on next sync if it's source of truth)
   • Purge: permanently delete the soft-deleted item, clearing the name for CertifyClouds to write
2. Retry: re-attempts sync. This fails until the soft-deleted item is resolved.

When to escalate: Not applicable; resolution is in your hands.

blocked_conflict¶

What it means: First-write conflict. The target vault already has an active (non-soft-deleted) item with the same name. CertifyClouds blocks the write until you explicitly acknowledge the overwrite.

Why it happens

• Target vault was pre-populated out-of-band
• Two replication configs target the same vault with overlapping rules
• A human wrote to the target vault directly
• An earlier replication to this target vault was deleted. Its copies are kept, so recreating the replication meets them as live items and each one waits for an acknowledgement (or is overwritten straight away with Overwrite target on conflict on)

How to fix it

1. Open the record in the table
2. Click Acknowledge conflict. The modal shows source and target metadata (name, updated times) with an explicit overwrite warning
3. Confirm to set acknowledged_conflict=true. The next sync will overwrite the target.

Or, if this is a systemic choice for the whole config, set overwrite_on_conflict=true in the config's behaviour settings; first-write conflicts will then overwrite immediately without the acknowledge step.

When to escalate: Only if the acknowledge flow itself fails with a different error code.

cross_geography_key¶

What it means: Key replication is blocked because the source and target vaults are in different Azure geographies. Azure's backup_key / restore_key operations are same-geography only, for software keys and HSM keys alike.

This code is also used when CertifyClouds' geography map returns "unknown" for either vault; a conservative block rather than a silent attempt. When the block reason says the Azure region could not be read, the geography is not the problem: the identity lacks Microsoft.KeyVault/vaults/read on the named subscription, so the vault's region was never resolved. Grant it (the built-in Reader role includes it, see RBAC), then select Check now.

Why it happens

• Source in (e.g.) europe, target in north_america
• Vault in a brand-new Azure region the geography map hasn't been taught yet

How to fix it

• Secrets and certificates are unaffected: they replicate across geographies. If keys aren't required for this pair, uncheck keys in the config's What to replicate.
• For same-geography key replication: pick a target vault in the source's geography.
• For cross-geography keys: not supported. There is no BYOK workaround built into the product.
• For unknown geography: wait for a CertifyClouds release that adds the region to the map, or open a support ticket with the vault region so we can prioritise the update.

When to escalate: Only to request a geography map update for a new region.

cross_subscription_key¶

What it means: Key replication is blocked because the source and target vaults are in different Azure subscriptions. Azure's backup_key / restore_key operations restore a key backup only inside the subscription it was taken from, for software keys and HSM keys alike. CertifyClouds blocks the pair up front rather than backing the key up and failing the restore.

This is checked independently of geography: two vaults can share a geography and still be in different subscriptions. The block reason names both subscription IDs.

Why it happens

• Source vault in (e.g.) a production subscription, target vault in a DR or sandbox subscription
• A config whose target was re-pointed at a vault in another subscription

How to fix it

• Secrets and certificates are unaffected: they replicate across subscriptions. If keys aren't required for this pair, uncheck keys in the config's What to replicate.
• For same-subscription key replication: pick a target vault in the source vault's subscription.
• For cross-subscription keys: not supported. There is no BYOK workaround built into the product.

When to escalate: Not applicable. This is an Azure platform limitation, not a CertifyClouds fault.

unknown_key_type¶

What it means: The scanner doesn't yet have a key_type value (RSA, RSA-HSM, EC, etc.) for this key, so CertifyClouds cannot yet check the cross-geography rule.

Why it happens: The scanner hasn't run a full cycle since the key was created.

How to fix it

1. Wait for the next check, which retries and fills in key_type
2. If it persists, trigger a fresh Key Vault scan on the source subscription, then Check now on the config

When to escalate: If unknown_key_type persists after two scan-plus-expand cycles, open a ticket with the correlation ID.

throttled¶

What it means: Azure returned HTTP 429 (rate limited). The engine automatically retries with exponential backoff, respecting the Retry-After header.

Why it happens: High activity against the same vault, bulk replicates, simultaneous Key Vault scans, or other tenants sharing Azure's per-vault limits.

How to fix it: Usually nothing. The engine handles it.

If a record has exhausted retries and surfaced throttled to you:

1. Reduce parallel activity (pause other bulk operations on the vault)
2. Increase the sync interval for this config (5m → 1h or 24h)
3. Manually Retry the record

When to escalate: If throttled appears without a clear traffic spike, or persists across multiple cycles, there may be a quota issue on the vault.

transient_network¶

What it means: HTTP 5xx, timeout, or connection error. The engine automatically retries with exponential backoff.

Why it happens: Azure service blip, network hiccup, DNS lag.

How to fix it: Usually nothing. If records persist in transient_network after retries are exhausted, click Retry on the record or wait for the next automatic cycle.

When to escalate: If many records fail with transient_network at once across multiple configs, check the Azure status page for your region. If Azure is green, gather correlation IDs and open a ticket.

unknown¶

What it means: An exception the engine didn't recognise. Full exc_info is logged backend-side.

Why it happens: A case the error-mapping layer doesn't cover yet. Treat as a bug in CertifyClouds, not in your configuration.

How to fix it

1. Grab the correlation ID from the record's history entry or the X-Correlation-Id response header
2. Open a support ticket with:
   • Correlation ID
   • Config ID
   • Record ID
   • Rough timestamp of the failure
3. While waiting for a fix, if the record is critical, try Retry: intermittent unknowns sometimes clear on the next attempt.

When to escalate: Every unknown should be reported. This code exists specifically to surface gaps in our error mapping.

EXCLUSION_EVALUATION_FAILED¶

An exclusion rule could not be evaluated (for example, a regex timed out or is invalid). CertifyClouds stops the affected sync or check before writing target items or changing replica records. Correct the rule and retry. A failed exclusion check is never treated as permission to replicate.

For a replication record, the failed check is recorded like any other failed sync: a history entry with this code, the next attempt at the record's sync interval rather than on every worker tick, and one alert once the config's repeated-failure threshold is reached.

General Tips¶

Correlation IDs¶

Every sync run generates a correlation ID. Quote it in any support ticket: it lets us pull the exact run from logs, audit events, and WebSocket streams in one query.

Find it in:

• The Record history entries shown in the UI for each replication record
• The X-Correlation-Id response header on any replication API response
• Backend log lines for that run

Alert Flooding¶

Repeated failures on the same record are de-duplicated in the audit log to prevent a single stuck record from flooding alerts. Retry counts and error counts continue to be tracked.

If you're not seeing alerts you expect, confirm alerts.notify_on_failure is enabled on the config's alerts settings.

Check Timeout¶

Config save or check times out? The response is marked truncated: true with expansion_state: 'partial'. The config is still persisted, and the next worker cycle resumes the check from the last checkpointed batch (last_batch_succeeded_at). No data loss. A banner in the UI tells you the check is still running.

Still Stuck?¶

See also:

• RBAC reference for full role requirements
• Support → Contact to open a ticket

Sync skipped¶

A skipped run did not copy an item. The sync dialog shows the reason. If it matches an exclusion, manage the rule under Automation → Replication & Sync → Exclusions and retry. If a key needs a manual update, follow the returned instructions in Azure before retrying; CertifyClouds does not delete keys to replace them.

Excluded items remain visible in the legacy Add-to-sync-queue picker, with an exclusion warning. Each replication check applies exclusions before creating active records. Skips do not count as successful syncs or failures in the success rate.