Deploy, Observe, Update, and Recover

What you will learn

This lesson covers:

• Production deployment
• Confidential containers
• Logging
• Metrics
• Health checks
• Attestation renewal
• Updates
• Incident response
• Disaster recovery

Production is part of the security boundary

A secure prototype can become an insecure production service through:

• Debug configuration
• Incorrect policy
• Leaking logs
• Expired certificates
• Stale attestation collateral
• Unsafe updates
• Unprotected backups
• Manual secret handling
• Inconsistent deployments

Security must cover the complete workload lifecycle.

Deployment pipeline

A production pipeline should include:

1. Source review 2. Dependency verification 3. Reproducible build 4. Image measurement 5. Security tests 6. Side-channel review 7. Release signing 8. Reference-value publication 9. Staged deployment 10. Attestation verification 11. Secret provisioning 12. Runtime monitoring

Confidential-container deployment

Confidential Containers places the workload inside a confidential VM while integrating with container and Kubernetes workflows.

Its Trustee architecture separates:

• Guest attestation agent
• Key Broker Service
• Attestation Service
• Reference Value Provider Service
• Policy

The KBS creates a channel bound to fresh evidence and conditionally releases resources after verification.

A confidential-container deployment should still review:

• Guest kernel
• Guest image
• Container runtime
• Attestation agent
• Workload container
• Kubernetes configuration
• Image registry
• KBS policy

Privacy-safe logging

Logs are often outside the TEE.

Do not log:

• Plaintext user data
• Private keys
• Decryption keys
• Full attestation evidence unless required
• Authentication tokens
• Sensitive model prompts
• Private transaction contents
• Sealed-state plaintext
• Detailed secret-dependent errors

A safe log might contain:

`` request_id workload_version operation_type success_or_failure policy_rule_id duration_bucket non-sensitive error_code ``

Use random request identifiers rather than user identifiers when possible.

Metrics

Metrics can leak behaviour.

Examples:

• Exact request timing
• Exact document size
• Exact output length
• Model token count
• Customer-specific activity
• Error distribution
• Key-use frequency

Consider:

• Aggregation
• Time buckets
• Size buckets
• Delayed reporting
• Minimum group size
• Access control
• Retention limits

Health checks

A health endpoint should not reveal:

• Internal key state
• Exact measurement unless intended
• Secret names
• Customer data
• Detailed policy configuration
• Firmware internals

A basic health response may report:

`` service = ready attestation = valid secrets = provisioned state = synchronized ``

Detailed diagnostic information should require protected administrative access.

Continuous attestation

An initial attestation proves a state at one time.

Afterward:

• Software may update.
• Firmware may change.
• Certificates may expire.
• Policy may change.
• Secrets may rotate.
• The process may restart.

Systems may require:

• Periodic re-attestation
• Attestation on every restart
• Short-lived workload credentials
• Session expiration
• Revocation checks
• Runtime heartbeat

Attestation collateral

Attestation systems may depend on:

• Certificates
• Certificate revocation lists
• TCB information
• Quoting-enclave identities
• Vendor endorsements
• Reference manifests

Missing or expired collateral can break quote verification.

Automata's DCAP documentation describes the operational need to maintain Intel SGX and TDX collateral, including certificate chains, TCB information, identities, and revocation lists.

Even when on-chain verification is not used, the same operational lesson applies:

> Attestation verification has dependencies that expire and change.

Updating a workload

An update may change:

• Binary measurement
• Signer
• Security version
• Guest image
• Kernel
• Runtime
• Container hash
• Attestation policy
• Sealing access

A safe update process should include:

1. Build new version. 2. Review code and dependencies. 3. Generate new measurement. 4. Test state migration. 5. Publish reference value. 6. Deploy new instance. 7. Attest new instance. 8. Migrate state. 9. Transfer or re-provision secrets. 10. Revoke old version. 11. Monitor failures. 12. Keep a controlled recovery plan.

Rollback during failed update

A software rollback may reintroduce a vulnerability.

A failed deployment should not automatically restore a version that is now below the accepted security version.

Recovery policy should define:

• Allowed rollback version
• Emergency version
• State compatibility
• Secret compatibility
• Required approvals
• Expiration

Incident response

A TEE incident may involve:

• Vulnerable platform
• Compromised application
• Compromised signing key
• Incorrect reference value
• Secret leakage
• Attestation failure
• State rollback
• Malicious host behaviour
• Side-channel evidence

Response steps may include:

1. Stop new secret release. 2. Revoke affected workload identity. 3. Preserve non-sensitive evidence. 4. Patch hardware and software. 5. Rotate keys. 6. Restore trusted state. 7. Publish new measurement. 8. Re-attest. 9. Notify affected users. 10. Document remaining risk.

Backups

Backups may contain:

• Sealed state
• Encrypted data
• Wrapped keys
• Reference values
• Certificates
• Policy
• Build metadata

A backup is useful only if the recovery keys and platform migration process remain available.

Test recovery regularly.

Multi-region deployment

A multi-region system must consider:

• Different hardware generations
• Different firmware versions
• Different cloud services
• Attestation-policy consistency
• State synchronization
• Key distribution
• Regional outages
• Duplicate instances

All regions should not silently accept different security policies.

Developer exercise

Create an update plan for a confidential document service.

The update changes:

• Guest kernel
• Application binary
• Model library
• Measurement
• Sealed-state format

Write:

1. Pre-deployment checks 2. Migration sequence 3. Attestation-policy update 4. Rollback rule 5. Key-rotation rule 6. Recovery test 7. Log and monitoring changes

Common mistakes

Key takeaways

• Operations are part of the TEE security model.
• Logs and metrics can leak sensitive metadata.
• Attestation must be maintained over time.
• Updates affect measurements, state, secrets, and policy.
• Incident response should stop secret release first.
• Backup and migration need regular testing.

Check your understanding

1. Why can metrics create privacy leakage? 2. Why is initial attestation not enough for a long-lived service? 3. What is attestation collateral? 4. Why can an automatic software rollback be unsafe? 5. What should happen first after a workload-signing key is compromised?