How to return binary data from lambda function in AWS in Python?
Question:
I cannot get python lambda to return binary data. The node-template for thumbnail images works fine but I cannot get a python lambda to work. Below is the relevant lines from my lambda. The print("image_data " + image_64_encode) line prints a base64 encoded image to the logs.
def lambda_handler(event, context):
img_base64 = event.get('base64Image')
if img_base64 is None:
return respond(True, "No base64Image key")
img = base64.decodestring(img_base64)
name = uuid.uuid4()
path = '/tmp/{}.png'.format(name)
print("path " + path)
image_result = open(path, 'wb')
image_result.write(img)
image_result.close()
process_image(path)
image_processed_path = '/tmp/{}-processed.png'.format(name)
print("image_processed_path " + image_processed_path)
image_processed = open(image_processed_path, 'rb')
image_processed_data = image_processed.read()
image_processed.close()
image_64_encode = base64.encodestring(image_processed_data)
print("image_data " + image_64_encode)
return respond(False, image_64_encode)
def respond(err, res):
return {
'statusCode': '400' if err else '200',
'body': res,
'headers': {
'Content-Type': 'image/png',
},
'isBase64Encoded': 'true'
}
Any pointers to what I’m doing wrong?
Answers:
I faced the same problem about 6 months ago. Looks like although there is now binary support (and examples in JS) in API Gateway, Python 2.7 Lambda still does not support valid binary response, not sure about Python 3.6.
Base64 encoded response is having problems because of JSON wrapping. I wrote a custom JS on client side taking the base-64 image out of this JSON manually, but this was also a poor solution.
Upload the result to S3 (behind the CloudFront) and return 301 to CloudFront seems to be a good workaround. Works best for me.
As far as I can tell, this is also the case with Python 3. I’m trying to return a binary data (bytes). It’s not working at all.
I also tried to use base-64 encoding and I have had no success.
This is with API Gateway and Proxy Integration.
[update]
I finally realized how to do this. I enabled binary support for type */* and then returned this:
return({
"isBase64Encoded": True,
"statusCode": 200,
"headers": {
"content-type": "image/jpg",
},
'body': base64.b64encode(open('image.jpg', 'rb').read()).decode('utf-8')
})
I finally figured this out. Returning binary data from a python lambda is doable.
Follow the instructions here:
https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/
Be sure to check the ‘Use Lambda Proxy integration’ when creating a new method.
Also be sure your Python Lambda response returns a base64-encoded body, sets isBase64Encoded to True, and an appropriate content type:
import base64
def lambda_handler(event, context):
# ...
body = base64.b64encode(bin_data)
return {'isBase64Encoded' : True,
'statusCode' : 200,
'headers' : { 'Content-Type': content_type },
'body' : body }
THEN:
For each of your routes/methods issue:
apigateway update-integration-response --rest-api-id <api-id> --resource-id <res-id> --http-method POST --status-code 200 --patch-operations "[{"op" : "replace", "path" : "/contentHandling", "value" : "CONVERT_TO_BINARY"}]"
In the AWS console.
The and can be seen in the API Gateway ‘breadcrumbs’
ex:
<api-id> = zdb7jsoey8
<res-id> = zy2b5g
THEN:
You need to ‘Deploy API’. From what I found only it only worked AFTER deploying the API.
Be sure you setup the ‘Binary Media Types’ before deploying.
Hint:
Nice AWS shell terminal here: https://github.com/awslabs/aws-shell
pip install aws-shell
Following all the steps above didn’t work on my case, because having the binary support for content-type = */* will convert all responses to binary.
My case:
-
Multiple lambda functions returning json (text), just a single lambda returning a binary file. All have lambda proxy enabled.
-
The lambdas are in an API Gateway
-
The API Gateway is behind CloudFront
Hint:
I have notice an important information in the API Gateway -> Settings
Quoting:
API Gateway will look at the Content-Type and Accept HTTP headers to decide how to handle the body.
This means that the Content-Type response header must match Accept request header
Solution:
-
Set Binary Media Types in API gateway to your mime type: image/jpg
-
In your HTTP request set Accept: image/jpg
-
In your HTTP response set Content-Type: image/jpg
{
"isBase64Encoded": True,
"statusCode": 200,
"headers": { "content-type": "image/jpg"},
"body": base64.b64encode(content_bytes).decode("utf-8")
}
- Next we must tell CloudFront to accept the ‘Accept’ header from the request. So, in CloudFront distribution, click on your API Gateway instance (ID is clickable) and once redirected to CloudFront instance go to Behaviour tab, select the path-pattern of your API (example: /api/*) and click on Edit button.
On the new screen, you have to add Accept header to Whitelist.
Note 1: If you have multiple file types, you must add them all to Binary Media Types in the API gateway settings
Note 2: For those coming from serverless and want to set the binary types when deploying your lambdas, then check this post: setting binary media types for API gateway
plugins:
- serverless-apigw-binary
custom:
apigwBinary:
types:
- 'image/jpeg'
The serverless.yml file for cloudfront should contain:
resources:
WebAppCloudFrontDistribution:
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
...
CacheBehaviors:
...
-
#API calls
...
ForwardedValues:
...
Headers:
- Authorization
- Accept
My issue was different – you need to redeploy Lambda after you do changes Binary Media Types.
That wasn’t obvious and I was changing Binary Media Types without any effect and stuck with the issue for days.
For reference .NET core response:
var response = new APIGatewayProxyResponse
{
StatusCode = 200,
Body = base64String,
IsBase64Encoded = true,
Headers = new Dictionary<string, string> {
{"Content-Type", "application/pdf"}
}
};
I cannot get python lambda to return binary data. The node-template for thumbnail images works fine but I cannot get a python lambda to work. Below is the relevant lines from my lambda. The print("image_data " + image_64_encode) line prints a base64 encoded image to the logs.
def lambda_handler(event, context):
img_base64 = event.get('base64Image')
if img_base64 is None:
return respond(True, "No base64Image key")
img = base64.decodestring(img_base64)
name = uuid.uuid4()
path = '/tmp/{}.png'.format(name)
print("path " + path)
image_result = open(path, 'wb')
image_result.write(img)
image_result.close()
process_image(path)
image_processed_path = '/tmp/{}-processed.png'.format(name)
print("image_processed_path " + image_processed_path)
image_processed = open(image_processed_path, 'rb')
image_processed_data = image_processed.read()
image_processed.close()
image_64_encode = base64.encodestring(image_processed_data)
print("image_data " + image_64_encode)
return respond(False, image_64_encode)
def respond(err, res):
return {
'statusCode': '400' if err else '200',
'body': res,
'headers': {
'Content-Type': 'image/png',
},
'isBase64Encoded': 'true'
}
Any pointers to what I’m doing wrong?
I faced the same problem about 6 months ago. Looks like although there is now binary support (and examples in JS) in API Gateway, Python 2.7 Lambda still does not support valid binary response, not sure about Python 3.6.
Base64 encoded response is having problems because of JSON wrapping. I wrote a custom JS on client side taking the base-64 image out of this JSON manually, but this was also a poor solution.
Upload the result to S3 (behind the CloudFront) and return 301 to CloudFront seems to be a good workaround. Works best for me.
As far as I can tell, this is also the case with Python 3. I’m trying to return a binary data (bytes). It’s not working at all.
I also tried to use base-64 encoding and I have had no success.
This is with API Gateway and Proxy Integration.
[update]
I finally realized how to do this. I enabled binary support for type */* and then returned this:
return({
"isBase64Encoded": True,
"statusCode": 200,
"headers": {
"content-type": "image/jpg",
},
'body': base64.b64encode(open('image.jpg', 'rb').read()).decode('utf-8')
})
I finally figured this out. Returning binary data from a python lambda is doable.
Follow the instructions here:
https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/
Be sure to check the ‘Use Lambda Proxy integration’ when creating a new method.
Also be sure your Python Lambda response returns a base64-encoded body, sets isBase64Encoded to True, and an appropriate content type:
import base64
def lambda_handler(event, context):
# ...
body = base64.b64encode(bin_data)
return {'isBase64Encoded' : True,
'statusCode' : 200,
'headers' : { 'Content-Type': content_type },
'body' : body }
THEN:
For each of your routes/methods issue:
apigateway update-integration-response --rest-api-id <api-id> --resource-id <res-id> --http-method POST --status-code 200 --patch-operations "[{"op" : "replace", "path" : "/contentHandling", "value" : "CONVERT_TO_BINARY"}]"
In the AWS console.
The and can be seen in the API Gateway ‘breadcrumbs’
ex:
<api-id> = zdb7jsoey8
<res-id> = zy2b5g
THEN:
You need to ‘Deploy API’. From what I found only it only worked AFTER deploying the API.
Be sure you setup the ‘Binary Media Types’ before deploying.
Hint:
Nice AWS shell terminal here: https://github.com/awslabs/aws-shell
pip install aws-shell
Following all the steps above didn’t work on my case, because having the binary support for content-type = */* will convert all responses to binary.
My case:
-
Multiple lambda functions returning json (text), just a single lambda returning a binary file. All have lambda proxy enabled.
-
The lambdas are in an API Gateway
-
The API Gateway is behind CloudFront
Hint:
I have notice an important information in the API Gateway -> Settings
Quoting:
API Gateway will look at the Content-Type and Accept HTTP headers to decide how to handle the body.
This means that the Content-Type response header must match Accept request header
Solution:
-
Set Binary Media Types in API gateway to your mime type: image/jpg
-
In your HTTP request set
Accept: image/jpg -
In your HTTP response set
Content-Type: image/jpg
{ "isBase64Encoded": True, "statusCode": 200, "headers": { "content-type": "image/jpg"}, "body": base64.b64encode(content_bytes).decode("utf-8") }
- Next we must tell CloudFront to accept the ‘Accept’ header from the request. So, in CloudFront distribution, click on your API Gateway instance (ID is clickable) and once redirected to CloudFront instance go to Behaviour tab, select the path-pattern of your API (example: /api/*) and click on Edit button.
On the new screen, you have to add Accept header to Whitelist.
Note 1: If you have multiple file types, you must add them all to Binary Media Types in the API gateway settings
Note 2: For those coming from serverless and want to set the binary types when deploying your lambdas, then check this post: setting binary media types for API gateway
plugins:
- serverless-apigw-binary
custom:
apigwBinary:
types:
- 'image/jpeg'
The serverless.yml file for cloudfront should contain:
resources:
WebAppCloudFrontDistribution:
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
...
CacheBehaviors:
...
-
#API calls
...
ForwardedValues:
...
Headers:
- Authorization
- Accept
My issue was different – you need to redeploy Lambda after you do changes Binary Media Types.
That wasn’t obvious and I was changing Binary Media Types without any effect and stuck with the issue for days.
For reference .NET core response:
var response = new APIGatewayProxyResponse
{
StatusCode = 200,
Body = base64String,
IsBase64Encoded = true,
Headers = new Dictionary<string, string> {
{"Content-Type", "application/pdf"}
}
};