S3 Upload Invoke Lambda Fails – Cross Account Access
Question:
I have a lambda that triggers off an S3 bucket upload (it basically converts a PDF to a dataframe and writes it to a different s3 bucket). Both of these belong to AWS account A. I would like to allow cross-account s3 access to trigger this lambda from another IAM user from account B (Administrator), however I am having issues with the GetObject operation. Here is how my lambda in account A looks:
LOGGER = logging.getLogger(__name__)
logging.basicConfig(level=logging.ERROR)
logging.getLogger(__name__).setLevel(logging.DEBUG)
session = boto3.Session(
aws_access_key_id="XXXX",
aws_secret_access_key="XXXX",
)
s3 = session.resource('s3')
dest_bucket = 'bucket-output'
csv_buffer = StringIO()
def lambda_handler(event,context):
source_bucket = event['Records'][0]['s3']['bucket']['name']
pdf_name = event['Records'][0]['s3']['object']['key']
LOGGER.info('Reading {} from {}'.format(pdf_name, source_bucket))
pdf_obj = s3.Object(source_bucket,pdf_name)
fs = pdf_obj.get()['Body'].read() #### code is failing here
df = convert_bytes_to_df(BytesIO(fs))
df.to_csv(csv_buffer,index=False)
s3.Object(dest_bucket,str(pdf_name.split('.')[0])+".csv").put(Body=csv_buffer.getvalue())
LOGGER.info('Successfully converted {} from {} to {}'.format(pdf_name,source_bucket,dest_bucket))
The lambda is failing with this error:
ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
I’m aware it’s bad practice to have keys in the lambda file but I can’t change things at the moment.
The process works fine if I am uploading to the S3 bucket from within an IAM User in account A itself, but when I expose the S3 buckets to an IAM user from a separate account, the issues above start happening. This is the S3 bucket policy (terraform) allowing cross-account access to an IAM user from account B:
resource "aws_s3_bucket_policy" "cross_account_input_access" {
bucket = aws_s3_bucket.statements_input.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXX:user/Administrator"
},
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::capsphere-input"
]
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXX:user/Administrator"
},
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
}
]
}
And here is the policy attached to an IAM user from another AWS account B which enables Administrator from account B to write a pdf to account A’s s3 bucket programmatically:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket-name"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
}
]
}
I write the file to the bucket from Administrator using aws cli like this:
aws s3 cp filename.pdf s3://bucket-name
I can’t figure out what else needs to change.
Answers:
What if you add Region information to the session:
session = boto3.Session(
aws_access_key_id="XXXX",
aws_secret_access_key="XXXX",
region_name="<REGION>"
)
It appears that your situation is:
Account A contains:
- An AWS Lambda function
- A ‘source’ bucket used to trigger the Lambda function
- A ‘destination’ bucket used by the Lambda function to store output
You want to allow the Administrator IAM User in Account B to upload a file to the source bucket in Account A. This user should be able to retrieve the output from the destination bucket in Account A.
The following would achieve these goals:
- Create an IAM Role in Account A and associate it with the Lambda function. Assign permissions to allow
GetObject from the source bucket and PutObject to the destination bucket. There should be no need to reference any credentials within the Lambda function itself, since any necessary permissions will be provided via this IAM Role. The policy would be:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject"
"Resource": "arn:aws:s3:::source-bucket/*"
},
{
"Effect": "Allow",
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::destination-bucket/*"
}
]
}
- Add a Bucket Policy on the source bucket that permits the
Administrator user in Account B to PutObject into the bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::source-bucket/*"
}
]
}
- Add a Bucket policy on the destination bucket that permits the
Administrator user in Account B to GetObject from the bucket and, I presume, list the bucket and delete objects that have been processed:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::destination-bucket"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::destination-bucket/*"
}
]
}
- Since this is cross-account access, permission must also be granted to the
Administrator IAM User in Account B to let them access the source and destination buckets. This policy is not required if they already have a lot of S3 permissions, such as s3:*, since it would work on any buckets including buckets in different AWS Accounts. This policy would go on the IAM User:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::destination-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::source-bucket/*",
"arn:aws:s3:::destination-bucket/*"
]
}
]
}
Ah! I have it!
While your Lambda function has permission to access the object, the fact that the object was uploaded using credentials from another AWS Account means that the object is still ‘owned’ by the other account.
There are two ways to fix this:
Turn off ACLs on the receiving bucket
From Disabling ACLs for all new buckets and enforcing Object Ownership – Amazon Simple Storage Service:
We recommend that you disable ACLs on your Amazon S3 buckets. You can do this by applying the bucket owner enforced setting for S3 Object Ownership. When you apply this setting, ACLs are disabled and you automatically own and have full control over all objects in your bucket.
Set ownership when uploading
When uploading the object when using credentials from another, you can specify ACL=bucket-owner-full-control (depending on how you perform the upload). This will grant ownership to the AWS Account that owns the receiving bucket.
I have a lambda that triggers off an S3 bucket upload (it basically converts a PDF to a dataframe and writes it to a different s3 bucket). Both of these belong to AWS account A. I would like to allow cross-account s3 access to trigger this lambda from another IAM user from account B (Administrator), however I am having issues with the GetObject operation. Here is how my lambda in account A looks:
LOGGER = logging.getLogger(__name__)
logging.basicConfig(level=logging.ERROR)
logging.getLogger(__name__).setLevel(logging.DEBUG)
session = boto3.Session(
aws_access_key_id="XXXX",
aws_secret_access_key="XXXX",
)
s3 = session.resource('s3')
dest_bucket = 'bucket-output'
csv_buffer = StringIO()
def lambda_handler(event,context):
source_bucket = event['Records'][0]['s3']['bucket']['name']
pdf_name = event['Records'][0]['s3']['object']['key']
LOGGER.info('Reading {} from {}'.format(pdf_name, source_bucket))
pdf_obj = s3.Object(source_bucket,pdf_name)
fs = pdf_obj.get()['Body'].read() #### code is failing here
df = convert_bytes_to_df(BytesIO(fs))
df.to_csv(csv_buffer,index=False)
s3.Object(dest_bucket,str(pdf_name.split('.')[0])+".csv").put(Body=csv_buffer.getvalue())
LOGGER.info('Successfully converted {} from {} to {}'.format(pdf_name,source_bucket,dest_bucket))
The lambda is failing with this error:
ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
I’m aware it’s bad practice to have keys in the lambda file but I can’t change things at the moment.
The process works fine if I am uploading to the S3 bucket from within an IAM User in account A itself, but when I expose the S3 buckets to an IAM user from a separate account, the issues above start happening. This is the S3 bucket policy (terraform) allowing cross-account access to an IAM user from account B:
resource "aws_s3_bucket_policy" "cross_account_input_access" {
bucket = aws_s3_bucket.statements_input.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXX:user/Administrator"
},
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::capsphere-input"
]
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXX:user/Administrator"
},
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
}
]
}
And here is the policy attached to an IAM user from another AWS account B which enables Administrator from account B to write a pdf to account A’s s3 bucket programmatically:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket-name"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
}
]
}
I write the file to the bucket from Administrator using aws cli like this:
aws s3 cp filename.pdf s3://bucket-name
I can’t figure out what else needs to change.
What if you add Region information to the session:
session = boto3.Session(
aws_access_key_id="XXXX",
aws_secret_access_key="XXXX",
region_name="<REGION>"
)
It appears that your situation is:
Account A contains:
- An AWS Lambda function
- A ‘source’ bucket used to trigger the Lambda function
- A ‘destination’ bucket used by the Lambda function to store output
You want to allow the Administrator IAM User in Account B to upload a file to the source bucket in Account A. This user should be able to retrieve the output from the destination bucket in Account A.
The following would achieve these goals:
- Create an IAM Role in Account A and associate it with the Lambda function. Assign permissions to allow
GetObjectfrom the source bucket andPutObjectto the destination bucket. There should be no need to reference any credentials within the Lambda function itself, since any necessary permissions will be provided via this IAM Role. The policy would be:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject"
"Resource": "arn:aws:s3:::source-bucket/*"
},
{
"Effect": "Allow",
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::destination-bucket/*"
}
]
}
- Add a Bucket Policy on the source bucket that permits the
Administratoruser in Account B toPutObjectinto the bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::source-bucket/*"
}
]
}
- Add a Bucket policy on the destination bucket that permits the
Administratoruser in Account B toGetObjectfrom the bucket and, I presume, list the bucket and delete objects that have been processed:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::destination-bucket"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::destination-bucket/*"
}
]
}
- Since this is cross-account access, permission must also be granted to the
AdministratorIAM User in Account B to let them access the source and destination buckets. This policy is not required if they already have a lot of S3 permissions, such ass3:*, since it would work on any buckets including buckets in different AWS Accounts. This policy would go on the IAM User:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::destination-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::source-bucket/*",
"arn:aws:s3:::destination-bucket/*"
]
}
]
}
Ah! I have it!
While your Lambda function has permission to access the object, the fact that the object was uploaded using credentials from another AWS Account means that the object is still ‘owned’ by the other account.
There are two ways to fix this:
Turn off ACLs on the receiving bucket
From Disabling ACLs for all new buckets and enforcing Object Ownership – Amazon Simple Storage Service:
We recommend that you disable ACLs on your Amazon S3 buckets. You can do this by applying the bucket owner enforced setting for S3 Object Ownership. When you apply this setting, ACLs are disabled and you automatically own and have full control over all objects in your bucket.
Set ownership when uploading
When uploading the object when using credentials from another, you can specify ACL=bucket-owner-full-control (depending on how you perform the upload). This will grant ownership to the AWS Account that owns the receiving bucket.