AWS Glue python shell Job fails with Internal Service error
Question:
I am running a python shell program in AWS Glue but after running for around 10 minutes its failing with error Internal service error. The logs or error logs does not give any information. Most of the time it fails by just saying Internal service error and rarely it runs for 2 days and gets timed out. The code uses pandas for transformations and it looks ok, it runs fine on local machine, necessary changes done so that it works on AWS glue[where it read/write files to s3 location instead of local folder]. What could be wrong here? Any input is appreciated.
Answers:
This issue was figured out. The problem was the job was unable to download the dependent python libraries due to an access issue to the s3 bucket. Once the access issue was resolved the job started running fine.
I am running a python shell program in AWS Glue but after running for around 10 minutes its failing with error Internal service error. The logs or error logs does not give any information. Most of the time it fails by just saying Internal service error and rarely it runs for 2 days and gets timed out. The code uses pandas for transformations and it looks ok, it runs fine on local machine, necessary changes done so that it works on AWS glue[where it read/write files to s3 location instead of local folder]. What could be wrong here? Any input is appreciated.
This issue was figured out. The problem was the job was unable to download the dependent python libraries due to an access issue to the s3 bucket. Once the access issue was resolved the job started running fine.