AWS Glue Job Cloudformation – Values Set in Cloudformation Not Sticking

Question

Cloudformation Setup Below is not behaving as I expected.
The following variables are not being set with the the template below. When the variables are set manually the job runs successfully.

IAM Role
Type
Language

Description: "AWS Glue Job Test"
Resources:
  MyJobRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                - "glue.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        -
          PolicyName: "root"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action: "*"
                Resource: "*"
  MyJob:
    Type: AWS::Glue::Job
    Properties:
      Name: cf-job1
      Command:
        Name: test-etl1
        ScriptLocation: "s3://project_bucket/releases/latest/mixpanel_job.py"
        PythonVersion: "3"
      Description: "Testing setup config"
      ExecutionProperty:
        MaxConcurrentRuns: 2
      MaxRetries: 2
      GlueVersion: "3.0"
      WorkerType: "G.1X"
      NumberOfWorkers: 2
      Timeout: 2880
      DefaultArguments:
        "--class": "GlueApp"
        "--enable-continuous-cloudwatch-log": "true"
        "--enable-job-insights": "true"
        "--enable-metrics": "true"
        "--enable-spark-ui": "true"
        "--extra-jars": "s3://project_bucket/releases/latest/jars/delta-core_2.12-1.0.1.jar"
        "--extra-py-files": "s3://project_bucket/releases/latest/lib.zip"
        "--job-bookmark-option": "job-bookmark-disable"
        "--job-language": "python"
        "--spark-event-logs-path": "s3://project_bucket/logs/"
      Role: !Ref MyJobRole

Result of Stack

Asked By: Nick

||

Source

Answer 1

This is a known issue with AWS Cloudformation on the Glue Team.

A default IAM role to run the job can only be SET with a physical ARN string value pointing to another ROLE

–job-language parameter doesn’t set correctly which is a known issue. This must be done manually via the Glue Console

Type parameter in the console still doesn’t have a cloudformation parameter & must be set manually.

True automation can not be achieved at this time.

Answered By: Nick

Answer 2

I encountered this also.

Change from

      Role: !Ref MyJobRole

to

      Role: !GetAtt MyJobRole.Arn

and it will work fine. 😀

For reference the relevant docs section for the cloudformation resource type is https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html#aws-resource-iam-role-return-values

Answered By: mud

AWS Glue Job Cloudformation – Values Set in Cloudformation Not Sticking

Question:

Answers: