Memory Isolation, Encryption, and Integrity

Why memory protection has several layers

Sensitive values can appear in:

• CPU registers
• Processor caches
• Main memory
• Shared memory
• Swap storage
• Input buffers
• Output buffers
• Device memory

Protecting "memory" is not one security property. It involves several different controls: access isolation, encryption, integrity, ownership, replay protection, and register protection.

Virtual and physical memory

Applications use virtual addresses. The processor and operating system translate virtual addresses into physical memory locations.

`` Application virtual address ↓ Page-table translation ↓ Physical memory address ↓ System memory ``

In a normal system, the operating system controls the page tables. In a virtualized system, the hypervisor also controls mappings between guest memory and physical memory.

A malicious privileged layer may try to read a page, replace a page, remap a page, reuse an older page, map one page into several environments, or observe which pages are accessed.

TEE hardware adds restrictions below the untrusted operating system or hypervisor.

Memory isolation

Memory isolation determines which execution context may access a memory region.

Examples include:

• Enclave pages assigned to one enclave
• Private pages assigned to a confidential VM
• Secure World memory separated from Normal World memory
• Secure subsystem memory separated from the main processor

Isolation answers: Who is allowed to access this memory?

Memory encryption

Memory encryption protects data when it is stored outside protected processor boundaries.

A simplified path is:

`` Protected processor state ↓ encrypt Encrypted data in external memory ↓ decrypt Protected processor state ``

The encryption key is usually managed by hardware or protected firmware.

AMD SEV uses hardware-managed encryption keys to protect VM memory, with separate keying for protected guests.

Memory encryption can help against direct physical memory inspection, some DMA attacks, host attempts to read protected guest memory, and cross-guest memory disclosure.

Memory encryption is not memory integrity

Encryption hides the content of a page.

It does not automatically prove that the page is current, stored at the correct address, belongs to the correct VM, has not been replaced, has not been replayed, or has not been remapped.

Memory integrity

Memory integrity mechanisms attempt to detect or block unauthorized modification.

Possible mechanisms include:

• Integrity trees
• Message authentication values
• Page ownership tables
• Address binding
• Protected metadata
• Version counters
• Replay controls

AMD SEV-SNP introduced the Reverse Map Table to track page ownership and security attributes. Its design aims to resist hypervisor-controlled remapping, aliasing, and replay attacks against protected guest memory.

Intel describes SGX enclave memory as encrypted and integrity-protected.

These platforms do not use identical mechanisms. Their guarantees should be checked separately.

Register protection

Sensitive data also appears in processor registers.

When execution switches from a protected guest to an untrusted hypervisor, register contents must be protected.

AMD SEV-ES added encrypted guest register state during certain transitions. SEV-SNP added further protections around memory integrity and page ownership.

Register protection answers: Can privileged software inspect protected execution state during a transition?

Shared memory

A TEE still needs to communicate with the outside world. Shared buffers often exist outside protected memory.

Every value received from an untrusted environment should be treated as attacker-controlled.

The TEE should validate:

• Buffer length
• Pointer
• Message structure
• Numeric ranges
• Encoding
• Authentication tag
• Sequence number
• System-call response
• File response
• Network response

Iago attacks

An Iago attack occurs when an untrusted operating system returns a carefully manipulated result to a protected application.

For example, the trusted application asks the operating system to allocate memory. The operating system returns a value designed to break an assumption inside the trusted code.

The attacker does not need to read protected memory directly. The attacker abuses the interface that the trusted code uses to access external services.

This is why TEE interfaces should be small, simple, validated, documented, and tested against malicious inputs.

Side-channel preview

A TEE may block direct memory access while still leaking indirect information.

An attacker may observe:

• Which pages are accessed
• When page faults occur
• Cache behaviour
• Execution time
• Branch behaviour
• Request size
• Response size
• Enclave entry frequency

Blocking direct memory reads does not block every form of observation.

Example: Password comparison

Consider this code:

``` for every character: if input_character != secret_character: return false

return true ```

The function returns as soon as it finds the first mismatch. An observer may measure how long the function runs and estimate how many initial characters were correct.

The password remains inside protected memory. Information still leaks through timing.

A constant-time comparison tries to perform the same pattern of work regardless of where the mismatch occurs.

Key takeaways

• Isolation, encryption, and integrity are different properties.
• Memory encryption does not automatically prevent replay or remapping.
• Registers may also contain sensitive data.
• Shared buffers cross the TEE boundary and must be validated.
• A malicious host can attack trusted code through its external interfaces.
• Side channels may reveal information without direct memory access.