How to tell programmatically that an AWS Step Function execution has been completed?
Question:
I am triggering a Step Function execution via a Python cell in a SageMaker Notebook, like this:
state_machine_arn = 'arn:aws:states:us-west-1:1234567891:stateMachine:alexanderMyPackageStateMachineE3411O13-A1vQWERTP9q9'
sfn = boto3.client('stepfunctions')
..
sfn.start_execution(**kwargs) # Non Blocking Call
run_arn = response['executionArn']
print(f"Started run {run_name}. ARN is {run_arn}.")
and then in order to check that the execution (which might take hours to complete depending on the input) has been completed, before I start doing some custom post-analysis on the results, I manually execute a cell with:
response = sfn.list_executions(
stateMachineArn=state_machine_arn,
maxResults=1
)
print(response)
where I can see from the output the status of the execution, e.g. 'status': 'RUNNING'.
How can I automate this, i.e. trigger the Step Function and continue the execution on my post-analysis custom logic only after the execution has finished? Is there for example a blocking call to start the execution, or a callback method I could use?
I can think of putting a sleep method, so that the Python Notebook cell would periodically call list_executions() and check the status, and only when the execution is completed, continue to rest of the code. I can statistically determine the sleep period, but I was wondering if there is a simpler/more accurate way.
PS: Related: How to avoid simultaneous execution in aws step function, however I would like to avoid creating any new AWS resource, just for this, I would like to do everything from within the Notebook.
PPS: I cannot make any change to MyPackage and the Step Function definition.
Answers:
Based on the comments.
If no new resources are to be created (no CloudWatch Event rules, lambda functions) nor any changes to existing Step Function are allowed, then pooling iteratively list_executions would be the best solution.
AWS CLI and boto3 have implemented similar solutions (not for Step Functions), but for some other services. They are called waiters (e.g. ec2 waiters). So basically you would have to create your own waiter for Step Function, as AWS does not provide one for that. AWS uses 15 seconds sleep time from what I recall for its waiters.
I am triggering a Step Function execution via a Python cell in a SageMaker Notebook, like this:
state_machine_arn = 'arn:aws:states:us-west-1:1234567891:stateMachine:alexanderMyPackageStateMachineE3411O13-A1vQWERTP9q9'
sfn = boto3.client('stepfunctions')
..
sfn.start_execution(**kwargs) # Non Blocking Call
run_arn = response['executionArn']
print(f"Started run {run_name}. ARN is {run_arn}.")
and then in order to check that the execution (which might take hours to complete depending on the input) has been completed, before I start doing some custom post-analysis on the results, I manually execute a cell with:
response = sfn.list_executions(
stateMachineArn=state_machine_arn,
maxResults=1
)
print(response)
where I can see from the output the status of the execution, e.g. 'status': 'RUNNING'.
How can I automate this, i.e. trigger the Step Function and continue the execution on my post-analysis custom logic only after the execution has finished? Is there for example a blocking call to start the execution, or a callback method I could use?
I can think of putting a sleep method, so that the Python Notebook cell would periodically call list_executions() and check the status, and only when the execution is completed, continue to rest of the code. I can statistically determine the sleep period, but I was wondering if there is a simpler/more accurate way.
PS: Related: How to avoid simultaneous execution in aws step function, however I would like to avoid creating any new AWS resource, just for this, I would like to do everything from within the Notebook.
PPS: I cannot make any change to MyPackage and the Step Function definition.
Based on the comments.
If no new resources are to be created (no CloudWatch Event rules, lambda functions) nor any changes to existing Step Function are allowed, then pooling iteratively list_executions would be the best solution.
AWS CLI and boto3 have implemented similar solutions (not for Step Functions), but for some other services. They are called waiters (e.g. ec2 waiters). So basically you would have to create your own waiter for Step Function, as AWS does not provide one for that. AWS uses 15 seconds sleep time from what I recall for its waiters.