Understanding AWS Instance Metadata Service

Introduction

In AWS, roles have become an essential part of securely granting permissions to services without the need for manual key management. Traditionally, in legacy systems or on-premises environments, services like virtual machines would rely on static access keys or service accounts for authentication. But with AWS roles were introduced to simplify permission management and to ensure better security and scalability.

At the heart of AWS's identity and access management for services like EC2 is the Instance Metadata Service (IMDS). IMDS allows an EC2 instance to securely query metadata about itself, including information about its assigned role. Instead of embedding credentials within your application or instance, IMDS allows instances to retrieve temporary, dynamically generated credentials associated with their role - making it a more secure and manageable approach.

What Happens Behind the Scenes?

When you launch an EC2 instance with an attached role, the instance doesn’t need any access keys stored within it. Instead, it can use IMDS to retrieve metadata, including the temporary credentials associated with its IAM role. This is how it works:

1. The EC2 Instance Role

When you create an EC2 instance, you can assign an IAM role to it. This role will define what AWS services the instance can interact with. For instance, if your EC2 instance needs to access S3, you might assign it a policy that grants read or write permissions to specific S3 buckets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Describe*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": "*"
        }
    ]
}

2. IMDS Endpoint

IMDS operates over the link local IP address 169.254.169.254. This endpoint is only accessible from within the EC2 instance, ensuring that no external parties can query it. Through this endpoint, the instance can retrieve metadata, such as:

Instance ID
Public and private IP addresses
Region
Availability zone
IAM role credentials

3. Fetching Temporary Credentials

When an EC2 instance needs to interact with AWS services (like S3 or DynamoDB), it does not use hardcoded credentials. Instead, it requests temporary credentials from IMDS. Using the AWS CLI or AWS SDKs (like boto3 for Python), the instance automatically queries IMDS.

Step 1: Checking for Credentials

When running an AWS command (e.g., aws s3 ls), the AWS SDKs and CLI check for available credentials in multiple locations:

2024-12-30 16:50:34,631 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env  
2024-12-30 16:50:34,631 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role  
2024-12-30 16:50:34,631 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file  
2024-12-30 16:50:34,632 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role

Step 2: Querying IMDS for Credentials

Once AWS determines that an IAM role is available, it makes a request to IMDS for security credentials:

2024-12-30 16:50:34,634 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/" HTTP/1.1" 200 24  
2024-12-30 16:50:34,636 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/EC2SSMSessionManagerRole HTTP/1.1" 200 1582  
2024-12-30 16:50:34,637 - MainThread - botocore.credentials - DEBUG - Found credentials from IAM Role: EC2SSMSessionManagerRole

Step 3: IMDS Returns Temporary Credentials

IMDS responds with temporary credentials assigned to the IAM role:

{
  "Code": "Success",
  "AccessKeyId": "ASIAMFKOAUSJ7EXAMPLE",
  "SecretAccessKey": "UeEevJGByhEXAMPLEKEY",
  "Token": "TQijaZw==",
  "Expiration": "2024-12-30T22:31:56Z"
}

These credentials include:
✅ Access Key ID
✅ Secret Access Key
✅ Session Token (required for authentication)
✅ Expiration Timestamp (after which the credentials must be refreshed)

4. Using Credentials

Once the instance has retrieved the temporary credentials, it automatically uses them to sign API requests. For example, when listing S3 buckets:

2024-12-30 16:55:43,775 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.us-east-2.amazonaws.com/, headers={'User-Agent': b'aws-cli/2.17.18 md/awscrt#0.19.19 ua/2.0 os/linux#6.1.119-129.201.amzn2023.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/distrib#amzn.2023 md/prompt#off md/command#s3.ls', 'X-Amz-Date': b'20241230T165543Z', 'X-Amz-Security-Token': b'TQijaZw==', 'X-Amz-Content-SHA256': b'e3b0c44206fc1c199afbf4c8996fb92427ae41e4692b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAMJJSIOAJMJ7EXAMPLE/20241230/us-east-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=1b0f7131b8626b61b192f14fdccd5768560b8f3fefca0f60ec9bd2d6cb6b18ad'}>

Here, the request contains:

X-Amz-Security-Token: The session token needed to authenticate API calls
Authorization: A signed request using temporary credentials

Since these credentials expire periodically, the EC2 instance must re-query IMDS to get fresh credentials before making additional API calls.

Why is IMDS Important?

IMDS is a crucial part of how EC2 instances interact securely with AWS services. Without IMDS, you would have to embed access keys within the instance or application, which is error-prone and increases the risk of credentials being exposed. By using IMDS, AWS ensures that credentials are temporary, automatically rotated, and never hard-coded into your applications.

Additionally, since the metadata service is only accessible from the EC2 instance itself (via the link local address 169.254.169.254), this significantly reduces the risk of external attacks trying to access sensitive metadata.

Want to learn how misconfigurations in IMDS can be exploited? Check out my article: