CloudNugget

Envelope Encryption vs Full Encryption: Understanding Data Key Management#

When you first encounter AWS Key Management Service (KMS), the architecture might seem unnecessarily complex. Why not just send your data directly to KMS and have it encrypt everything? The answer lies in a deceptively simple but powerful concept called envelope encryption—a design pattern that underpins how AWS securely and efficiently encrypts data at scale. Understanding this pattern isn’t just academically interesting; it fundamentally changes how you think about managing encryption keys and protecting sensitive information in your applications.

The Problem with Direct Encryption#

Imagine you’re tasked with encrypting a 10 GB video file before storing it in S3. Your first instinct might be to send the entire file to KMS for encryption. After all, KMS is the trusted key management service—surely it can handle the heavy lifting.

But let’s think through the practical implications. KMS is a highly available, regionally distributed service with network latency. Transmitting 10 GB of data over the network to KMS, waiting for it to be encrypted, and receiving the encrypted result back would be slow, costly, and inefficient. You’d be paying for every byte transferred, and the API would become a bottleneck. Furthermore, KMS API calls are metered and priced per request—sending massive payloads would also increase the computational burden on the KMS service itself.

This is where most developers realize: there has to be a better way.

Understanding Envelope Encryption#

Envelope encryption solves this problem through an elegant two-layer approach. Instead of sending your data directly to KMS, you use KMS to encrypt a small, special key called a Data Encryption Key (DEK). You then use that DEK to encrypt your actual data locally, entirely outside of KMS. The result is a clever division of labor: KMS handles the cryptographic security of protecting your master key, while your application handles the actual data encryption.

Here’s the conceptual flow:

  1. Generate a Data Encryption Key: Your application requests a DEK from KMS by calling GenerateDataKey. KMS uses its master key to generate this DEK and returns it to you in both plaintext and encrypted forms.

  2. Encrypt your data locally: You use the plaintext DEK to encrypt your actual data—no network call needed, no KMS involvement. This happens entirely on your server or client.

  3. Store the encrypted DEK with your data: You store the encrypted DEK alongside the encrypted data. This encrypted DEK is useless without KMS’s master key, so it’s safe to store anywhere.

  4. Decrypt later: When you need to access the data, you send the encrypted DEK to KMS, which decrypts it using its master key and returns the plaintext DEK. You then use that plaintext DEK to decrypt your data.

The beauty of this design is that KMS never touches your actual data. It only ever encrypts and decrypts the small DEK—typically 256 bits, regardless of how large your data is.

The Performance Argument#

The performance benefits are substantial. Consider two scenarios:

Direct encryption approach (hypothetical): Encrypting a 5 GB file requires sending 5 GB across the network to KMS, waiting for it to be encrypted, and receiving 5 GB back. Even at high bandwidth, this adds latency and creates a network bottleneck. You’re also limited by KMS’s processing capacity for large payloads.

Envelope encryption approach (actual): You make one API call to KMS requesting a DEK. KMS returns approximately 256 bits of plaintext key material and its encrypted equivalent—a few hundred bytes total. You then encrypt your 5 GB locally using AES-256 in your application or server. The encrypted data and encrypted DEK are stored together. Network overhead is minimal, and encryption happens at local CPU speed.

The difference isn’t marginal—it’s orders of magnitude. Local encryption on modern hardware can process gigabytes per second, whereas sending gigabytes over the network introduces both latency and throughput constraints.

The Cost Argument#

AWS charges for KMS API calls, not for data processed. Each GenerateDataKey call costs money (as do Decrypt calls). Each direct encryption of a different piece of data would require a separate API call to KMS.

With envelope encryption, you generate one DEK per logical data object (a file, a database record, a message) and use it to encrypt that object locally. If you had to send every byte directly to KMS, you’d be making API calls proportional to the volume of data, which would be prohibitively expensive. With envelope encryption, your KMS API charges remain tied to the number of objects you encrypt, not the total bytes.

For a service like S3 where you might store terabytes of data across millions of objects, this distinction becomes critical to operational cost.

Scaling to Arbitrarily Large Objects#

One of the most elegant aspects of envelope encryption is that it allows KMS to secure encryption for objects of any size. A file could be 1 MB, 1 GB, or 1 TB—it doesn’t matter. KMS’s responsibility remains fixed: protect one small DEK per object. The actual encryption happens at the application layer, which can scale horizontally.

This is why services like S3 and DynamoDB can transparently integrate KMS without forcing users to chunk their data or jump through hoops. The object size is irrelevant to the encryption architecture. You’re not constrained by KMS API limitations or costs; you’re limited only by your local encryption speed and network bandwidth for the encrypted data itself.

Security Implications: Keeping the Plaintext DEK Local#

A critical security principle underpins envelope encryption: the plaintext Data Encryption Key should never be transmitted to storage or logged. Once you’ve encrypted your data with the plaintext DEK, that plaintext key should be immediately discarded from memory.

This means that if an attacker gains access to your storage backend—your S3 bucket, your database, your message queue—they only find encrypted data and an encrypted DEK. The encrypted DEK is worthless without access to KMS. And access to KMS is governed by IAM policies, CloudTrail logging, and the secure, auditable nature of the KMS service itself.

Compare this to a scenario where you stored the plaintext DEK alongside the encrypted data. If your storage is compromised, the attacker has both the ciphertext and the key—game over. Envelope encryption prevents this by ensuring the plaintext DEK is transient. It exists only in memory for as long as needed to encrypt or decrypt, and never persists anywhere.

This is why it’s critical to properly implement envelope encryption in your applications: generate the DEK, use it, and forget it. Don’t log it, don’t cache it persistently, and don’t pass it around.

S3 Server-Side Encryption with KMS#

Let’s walk through a concrete example: S3 with server-side encryption using KMS (SSE-KMS).

When you upload an object to S3 with SSE-KMS enabled, S3 handles the entire envelope encryption process transparently. Behind the scenes, here’s what happens:

  1. You upload an object and specify that it should be encrypted with a particular KMS key.

  2. S3 generates a unique DEK by calling GenerateDataKey against your KMS key.

  3. S3 receives the plaintext DEK and uses it to encrypt your object’s data using AES-256.

  4. S3 stores the encrypted object alongside the encrypted DEK in the S3 bucket. The plaintext DEK is never persisted—it’s only held in memory during the encryption operation.

  5. When you later retrieve the object, S3 sends the encrypted DEK to KMS, gets back the plaintext DEK, uses it to decrypt the object, and returns the plaintext object to you.

From your perspective as a developer, you simply set the encryption configuration and upload normally. S3 manages the envelope encryption automatically. The KMS API calls are minimal and fixed—one GenerateDataKey per object uploaded, one Decrypt per object downloaded. The object size is irrelevant.

What makes this efficient is that only the small encrypted DEK crosses the KMS boundary. Your 5 GB object never touches KMS.

DynamoDB Encryption at Rest#

DynamoDB encryption with KMS follows the same envelope encryption pattern, though the mechanics are slightly different since DynamoDB manages the encryption server-side.

When you enable encryption at rest on a DynamoDB table with a customer-managed KMS key, DynamoDB generates a DEK using that key. DynamoDB then uses that DEK to encrypt the actual table data at rest. The encrypted DEK is stored within DynamoDB’s internal systems.

When DynamoDB needs to access the encrypted data, it decrypts the DEK using KMS, then uses the plaintext DEK to decrypt the data. Again, the plaintext DEK is transient—used only during the read operation and then discarded.

From a cost and performance perspective, DynamoDB doesn’t make a KMS API call for every item you read or write. Instead, DynamoDB uses a derived key or maintains the plaintext DEK in memory for a period to amortize the cost of the GenerateDataKey call. This is a practical optimization that balances security with operational efficiency.

SQS Message Encryption#

Amazon SQS also supports envelope encryption when you enable server-side encryption with a customer-managed KMS key. The pattern is consistent:

  1. When you send a message to an encrypted SQS queue, SQS generates (or reuses) a DEK for encrypting messages.

  2. SQS encrypts your message content using the plaintext DEK.

  3. SQS stores the encrypted message and encrypted DEK in the queue.

  4. When a consumer receives the message, SQS decrypts the DEK using KMS, decrypts the message, and delivers the plaintext message to the consumer.

Again, your message payload never travels to KMS. Only the encrypted DEK does, and only when necessary. This allows SQS to encrypt messages of arbitrary size without scalability constraints.

Practical Implementation: When You Manage Envelope Encryption#

While many AWS services handle envelope encryption transparently (S3, DynamoDB, SQS), sometimes you need to implement it yourself. The boto3 library (AWS SDK for Python) makes this straightforward.

Here’s a practical example of encrypting data manually using envelope encryption with KMS:

import boto3
import json
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
import os

kms_client = boto3.client('kms')

def encrypt_with_envelope(plaintext_data, kms_key_id):
    """
    Encrypt data using envelope encryption with KMS.
    Returns a dict with encrypted_data and encrypted_dek.
    """
    
    # Step 1: Generate a Data Encryption Key from KMS
    response = kms_client.generate_data_key(
        KeyId=kms_key_id,
        KeySpec='AES_256'
    )
    
    plaintext_dek = response['Plaintext']  # The key we'll use locally
    encrypted_dek = response['CiphertextBlob']  # Store this with encrypted data
    
    # Step 2: Encrypt data locally using the plaintext DEK
    # Generate a random IV for AES
    iv = os.urandom(16)
    cipher = Cipher(
        algorithms.AES(plaintext_dek),
        modes.CBC(iv),
        backend=default_backend()
    )
    encryptor = cipher.encryptor()
    
    # Add PKCS7 padding
    padding_length = 16 - (len(plaintext_data) % 16)
    padded_data = plaintext_data + bytes([padding_length]) * padding_length
    
    encrypted_data = encryptor.update(padded_data) + encryptor.finalize()
    
    # Step 3: Securely discard the plaintext DEK from memory
    plaintext_dek = None
    
    # Return encrypted data along with encrypted DEK and IV
    return {
        'encrypted_data': encrypted_data,
        'encrypted_dek': encrypted_dek,
        'iv': iv
    }

def decrypt_with_envelope(encrypted_payload, kms_key_id):
    """
    Decrypt data that was encrypted using envelope encryption.
    """
    
    encrypted_dek = encrypted_payload['encrypted_dek']
    encrypted_data = encrypted_payload['encrypted_data']
    iv = encrypted_payload['iv']
    
    # Step 1: Decrypt the DEK using KMS
    response = kms_client.decrypt(CiphertextBlob=encrypted_dek)
    plaintext_dek = response['Plaintext']
    
    # Step 2: Decrypt data locally using the plaintext DEK
    cipher = Cipher(
        algorithms.AES(plaintext_dek),
        modes.CBC(iv),
        backend=default_backend()
    )
    decryptor = cipher.decryptor()
    
    padded_plaintext = decryptor.update(encrypted_data) + decryptor.finalize()
    
    # Remove PKCS7 padding
    padding_length = padded_plaintext[-1]
    plaintext_data = padded_plaintext[:-padding_length]
    
    # Step 3: Discard the plaintext DEK
    plaintext_dek = None
    
    return plaintext_data

This example demonstrates the key principles: you call GenerateDataKey once per object, encrypt locally, and store both the encrypted data and encrypted DEK together. When decrypting, you call Decrypt to recover the plaintext DEK, use it to decrypt, and then discard it.

In production, you’d want to add error handling, potentially use GenerateDataKeyWithoutPlaintext when you only need to encrypt and never decrypt locally, and ensure your DEK handling adheres to your organization’s security practices.

When NOT to Use Envelope Encryption#

It’s worth noting that envelope encryption isn’t always necessary or appropriate. For small secrets—API keys, database passwords, encryption keys themselves—you might use KMS’s direct encryption via Encrypt and Decrypt operations. These operations are appropriate for data smaller than 4 KB.

However, for any substantial data—files, database records, messages—envelope encryption is the standard pattern. It scales, it’s efficient, and it’s the pattern that AWS services themselves use.

Conclusion#

Envelope encryption is not a complicated pattern; it’s a practical solution to the scaling challenges of centralized key management. By separating the concerns of key protection (handled by KMS) from the bulk data encryption (handled locally), AWS architects designed a system that is both highly secure and highly scalable.

The key insight is this: KMS protects your encryption keys with the full weight of the KMS service’s security infrastructure. Your application protects your data by performing the actual encryption locally with those keys. The plaintext DEK never persists, never travels unnecessarily, and never becomes a liability.

Whether you’re working with S3, DynamoDB, SQS, or implementing custom encryption in your applications, this pattern underpins the entire architecture. Understanding why it exists—the performance benefits, the cost benefits, the security benefits—makes you a more effective AWS developer. You’ll design more scalable encryption strategies, troubleshoot KMS-related issues with clarity, and appreciate the elegance of AWS’s security model.

© 2026 CloudNugget