AWS ParallelCluster Graviton Rest API Connection Refused
Question:
I am trying to follow a tutorial using AWS ParallelCluster (AWS Research Workshop) and once I get CloudFormation up and running, I cannot connect to the cluster via the Rest API. I am using Slurm 21.08.8 and ParallelCluster 3.1.4. I am able to SSH into the head node and kick off jobs, but when I try to access the Node via SageMaker notebook, I get a connection refused. Below is my configuration YAML for ParallelCluster. Please take note that this is using the Graviton2 ARM processors:
Region: ${REGION}
Image:
Os: alinux2
SharedStorage:
- Name: myebs
StorageType: Ebs
MountDir: /shared
EbsSettings:
VolumeType: gp2
Size: 200
HeadNode:
InstanceType: c6g.medium
Networking:
SubnetId: ${SUBNET_ID}
ElasticIp: true
Ssh:
KeyName: ${KEY_NAME}
CustomActions:
OnNodeConfigured:
Script: ${POST_INSTALL_SCRIPT_LOCATION}
Args:
- ${POST_INSTALL_SCRIPT_ARGS_1}
- ${POST_INSTALL_SCRIPT_ARGS_2}
- ${POST_INSTALL_SCRIPT_ARGS_3}
- ${POST_INSTALL_SCRIPT_ARGS_4}
- ${POST_INSTALL_SCRIPT_ARGS_5}
- ${POST_INSTALL_SCRIPT_ARGS_6}
- ${POST_INSTALL_SCRIPT_ARGS_7}
- ${POST_INSTALL_SCRIPT_ARGS_8}
- ${POST_INSTALL_SCRIPT_ARGS_9}
Iam:
AdditionalIamPolicies:
- Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
S3Access:
- EnableWriteAccess: true
BucketName: '*'
Scheduling:
Scheduler: slurm
SlurmQueues:
- Name: q1
CapacityType: ONDEMAND
ComputeResources:
- Name: cr1
InstanceType: c6g.2xlarge
MinCount: 0
MaxCount: 20
Efa:
Enabled: false
CustomActions:
OnNodeConfigured:
Script: ${POST_INSTALL_SCRIPT_LOCATION}
Args:
- ${POST_INSTALL_SCRIPT_ARGS_1}
- ${POST_INSTALL_SCRIPT_ARGS_2}
- ${POST_INSTALL_SCRIPT_ARGS_3}
- ${POST_INSTALL_SCRIPT_ARGS_4}
- ${POST_INSTALL_SCRIPT_ARGS_5}
- ${POST_INSTALL_SCRIPT_ARGS_6}
- ${POST_INSTALL_SCRIPT_ARGS_7}
- ${POST_INSTALL_SCRIPT_ARGS_8}
- ${POST_INSTALL_SCRIPT_ARGS_9}
Iam:
AdditionalIamPolicies:
- Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
S3Access:
- EnableWriteAccess: true
BucketName: '*'
Networking:
SubnetIds:
- ${SUBNET_ID}
AssignPublicIp: true
PlacementGroup:
Enabled: true
Here is my slurm.conf:
#
# Example slurm.conf file. Please run configurator.html
# (in doc/html) to build a configuration file customized
# for your environment.
#
#
# slurm.conf file generated by configurator.html.
#
# See the slurm.conf man page for more information.
#
# CLUSTER SETTINGS
ClusterName=mypc6g2
SlurmUser=slurm
SlurmctldPort=6820-6829
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurm.state
SlurmdSpoolDir=/var/spool/slurmd
SwitchType=switch/none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ReconfigFlags=KeepPartState
#
# CLOUD CONFIGS OPTIONS
SlurmctldParameters=idle_on_node_suspend,power_save_min_interval=30,cloud_dns
CommunicationParameters=NoAddrCache
SuspendProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
ResumeProgram=/opt/parallelcluster/scripts/slurm/slurm_resume
ResumeFailProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
SuspendTimeout=120
ResumeTimeout=1800
PrivateData=cloud
ResumeRate=0
SuspendRate=0
#
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=180
UnkillableStepTimeout=180
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
MessageTimeout=60
#
# SCHEDULING, JOB, AND NODE SETTINGS
EnforcePartLimits=ALL
SchedulerType=sched/backfill
ProctrackType=proctrack/cgroup
MpiDefault=none
ReturnToService=1
TaskPlugin=task/affinity,task/cgroup
#
# TRES AND GPU CONFIG OPTIONS
GresTypes=gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU
#
# LOGGING
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
JobCompType=jobcomp/none
#
# WARNING!!! The slurm_parallelcluster.conf file included
# get updated by pcluster process, be careful
# when manually editing!
include slurm_parallelcluster.conf
# Enable jwt auth for Slurmrestd
AuthAltTypes=auth/jwt
#
## /opt/slurm/etc/slurm.conf
#
# ACCOUNTING
JobAcctGatherType=jobacct_gather/linux
JobAcctGatherFrequency=30
#
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=<IP ADDRESS> # cluster headnode's DNS
AccountingStorageUser=db_user
AccountingStoragePort=6839
Here is the output of sudo journalctl -u slurmrestd (hiding IP):
-- Logs begin at Thu 2022-05-12 10:46:45 UTC, end at Thu 2022-05-26 03:00:38 UTC. --
May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug: _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug: slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.
Here is the output of sudo systemctl status slurmrestd (hiding IP):
● slurmrestd.service - Slurm restd daemon
Loaded: loaded (/etc/systemd/system/slurmrestd.service; disabled; vendor preset: disabled)
Active: failed (Result: signal) since Wed 2022-05-25 22:30:25 UTC; 4h 31min ago
Main PID: 12872 (code=killed, signal=SEGV)
May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug: _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug: slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.
Here is the output of systemctl status slurmctld:
● slurmctld.service - Slurm controller daemon
Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-05-25 22:30:25 UTC; 4h 33min ago
Main PID: 12899 (slurmctld)
CGroup: /system.slice/slurmctld.service
├─12899 /opt/slurm/sbin/slurmctld -D
└─12903 slurmctld: slurmscriptd
May 26 00:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 03:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
I cannot get the slurmrestd service up and running and am not sure where else to look since this is not super well documented on the Slurm site and the Rest API is not a supported feature of AWS ParallelCluster (according to the AWS Support team).
I am running the follow block of code from the tutorial:
import requests
import json
slurm_openapi_ep = 'http://'+slurm_host+':8082/openapi/v3'
print(slurm_openapi_ep)
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.35'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.36'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.37'
_, get_headers = pcluster_helper.update_header_token()
print(get_headers)
resp_api = requests.get(slurm_openapi_ep, headers=get_headers)
print(resp_api)
if resp_api.status_code != 200:
# This means something went wrong.
print("Error" , resp_api.status_code)
with open('build/slurm_api.json', 'w') as outfile:
json.dump(resp_api.json(), outfile)
print(json.dumps(resp_api.json(), indent=2))
with an print output of:
http://<HEAD NODE IP ADDRESS HERE>:8082/openapi/v3
{'X-SLURM-USER-NAME': 'ec2-user', 'X-SLURM-USER-TOKEN': '<MY TOKEN HERE>', 'Content-type': 'application/x-www-form-urlencoded', 'Accept': 'application/json'}
and an error of:
ConnectionError: HTTPConnectionPool(host='<HEAD NODE IP ADDRESS HERE>', port=8082): Max retries exceeded with url: /openapi/v3 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6151b9940>: Failed to establish a new connection: [Errno 111] Connection refused',))
Thanks in advance for any suggestions or ideas to resolve.
Answers:
For anyone trying to find an answer to this—AWS updated their documentation and tutorial to support the Graviton processors. Works like a charm now.
I am trying to follow a tutorial using AWS ParallelCluster (AWS Research Workshop) and once I get CloudFormation up and running, I cannot connect to the cluster via the Rest API. I am using Slurm 21.08.8 and ParallelCluster 3.1.4. I am able to SSH into the head node and kick off jobs, but when I try to access the Node via SageMaker notebook, I get a connection refused. Below is my configuration YAML for ParallelCluster. Please take note that this is using the Graviton2 ARM processors:
Region: ${REGION}
Image:
Os: alinux2
SharedStorage:
- Name: myebs
StorageType: Ebs
MountDir: /shared
EbsSettings:
VolumeType: gp2
Size: 200
HeadNode:
InstanceType: c6g.medium
Networking:
SubnetId: ${SUBNET_ID}
ElasticIp: true
Ssh:
KeyName: ${KEY_NAME}
CustomActions:
OnNodeConfigured:
Script: ${POST_INSTALL_SCRIPT_LOCATION}
Args:
- ${POST_INSTALL_SCRIPT_ARGS_1}
- ${POST_INSTALL_SCRIPT_ARGS_2}
- ${POST_INSTALL_SCRIPT_ARGS_3}
- ${POST_INSTALL_SCRIPT_ARGS_4}
- ${POST_INSTALL_SCRIPT_ARGS_5}
- ${POST_INSTALL_SCRIPT_ARGS_6}
- ${POST_INSTALL_SCRIPT_ARGS_7}
- ${POST_INSTALL_SCRIPT_ARGS_8}
- ${POST_INSTALL_SCRIPT_ARGS_9}
Iam:
AdditionalIamPolicies:
- Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
S3Access:
- EnableWriteAccess: true
BucketName: '*'
Scheduling:
Scheduler: slurm
SlurmQueues:
- Name: q1
CapacityType: ONDEMAND
ComputeResources:
- Name: cr1
InstanceType: c6g.2xlarge
MinCount: 0
MaxCount: 20
Efa:
Enabled: false
CustomActions:
OnNodeConfigured:
Script: ${POST_INSTALL_SCRIPT_LOCATION}
Args:
- ${POST_INSTALL_SCRIPT_ARGS_1}
- ${POST_INSTALL_SCRIPT_ARGS_2}
- ${POST_INSTALL_SCRIPT_ARGS_3}
- ${POST_INSTALL_SCRIPT_ARGS_4}
- ${POST_INSTALL_SCRIPT_ARGS_5}
- ${POST_INSTALL_SCRIPT_ARGS_6}
- ${POST_INSTALL_SCRIPT_ARGS_7}
- ${POST_INSTALL_SCRIPT_ARGS_8}
- ${POST_INSTALL_SCRIPT_ARGS_9}
Iam:
AdditionalIamPolicies:
- Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
S3Access:
- EnableWriteAccess: true
BucketName: '*'
Networking:
SubnetIds:
- ${SUBNET_ID}
AssignPublicIp: true
PlacementGroup:
Enabled: true
Here is my slurm.conf:
#
# Example slurm.conf file. Please run configurator.html
# (in doc/html) to build a configuration file customized
# for your environment.
#
#
# slurm.conf file generated by configurator.html.
#
# See the slurm.conf man page for more information.
#
# CLUSTER SETTINGS
ClusterName=mypc6g2
SlurmUser=slurm
SlurmctldPort=6820-6829
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurm.state
SlurmdSpoolDir=/var/spool/slurmd
SwitchType=switch/none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ReconfigFlags=KeepPartState
#
# CLOUD CONFIGS OPTIONS
SlurmctldParameters=idle_on_node_suspend,power_save_min_interval=30,cloud_dns
CommunicationParameters=NoAddrCache
SuspendProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
ResumeProgram=/opt/parallelcluster/scripts/slurm/slurm_resume
ResumeFailProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
SuspendTimeout=120
ResumeTimeout=1800
PrivateData=cloud
ResumeRate=0
SuspendRate=0
#
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=180
UnkillableStepTimeout=180
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
MessageTimeout=60
#
# SCHEDULING, JOB, AND NODE SETTINGS
EnforcePartLimits=ALL
SchedulerType=sched/backfill
ProctrackType=proctrack/cgroup
MpiDefault=none
ReturnToService=1
TaskPlugin=task/affinity,task/cgroup
#
# TRES AND GPU CONFIG OPTIONS
GresTypes=gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU
#
# LOGGING
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
JobCompType=jobcomp/none
#
# WARNING!!! The slurm_parallelcluster.conf file included
# get updated by pcluster process, be careful
# when manually editing!
include slurm_parallelcluster.conf
# Enable jwt auth for Slurmrestd
AuthAltTypes=auth/jwt
#
## /opt/slurm/etc/slurm.conf
#
# ACCOUNTING
JobAcctGatherType=jobacct_gather/linux
JobAcctGatherFrequency=30
#
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=<IP ADDRESS> # cluster headnode's DNS
AccountingStorageUser=db_user
AccountingStoragePort=6839
Here is the output of sudo journalctl -u slurmrestd (hiding IP):
-- Logs begin at Thu 2022-05-12 10:46:45 UTC, end at Thu 2022-05-26 03:00:38 UTC. --
May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug: _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug: slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.
Here is the output of sudo systemctl status slurmrestd (hiding IP):
● slurmrestd.service - Slurm restd daemon
Loaded: loaded (/etc/systemd/system/slurmrestd.service; disabled; vendor preset: disabled)
Active: failed (Result: signal) since Wed 2022-05-25 22:30:25 UTC; 4h 31min ago
Main PID: 12872 (code=killed, signal=SEGV)
May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug: _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug: slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.
Here is the output of systemctl status slurmctld:
● slurmctld.service - Slurm controller daemon
Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-05-25 22:30:25 UTC; 4h 33min ago
Main PID: 12899 (slurmctld)
CGroup: /system.slice/slurmctld.service
├─12899 /opt/slurm/sbin/slurmctld -D
└─12903 slurmctld: slurmscriptd
May 26 00:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 03:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
I cannot get the slurmrestd service up and running and am not sure where else to look since this is not super well documented on the Slurm site and the Rest API is not a supported feature of AWS ParallelCluster (according to the AWS Support team).
I am running the follow block of code from the tutorial:
import requests
import json
slurm_openapi_ep = 'http://'+slurm_host+':8082/openapi/v3'
print(slurm_openapi_ep)
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.35'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.36'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.37'
_, get_headers = pcluster_helper.update_header_token()
print(get_headers)
resp_api = requests.get(slurm_openapi_ep, headers=get_headers)
print(resp_api)
if resp_api.status_code != 200:
# This means something went wrong.
print("Error" , resp_api.status_code)
with open('build/slurm_api.json', 'w') as outfile:
json.dump(resp_api.json(), outfile)
print(json.dumps(resp_api.json(), indent=2))
with an print output of:
http://<HEAD NODE IP ADDRESS HERE>:8082/openapi/v3
{'X-SLURM-USER-NAME': 'ec2-user', 'X-SLURM-USER-TOKEN': '<MY TOKEN HERE>', 'Content-type': 'application/x-www-form-urlencoded', 'Accept': 'application/json'}
and an error of:
ConnectionError: HTTPConnectionPool(host='<HEAD NODE IP ADDRESS HERE>', port=8082): Max retries exceeded with url: /openapi/v3 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6151b9940>: Failed to establish a new connection: [Errno 111] Connection refused',))
Thanks in advance for any suggestions or ideas to resolve.
For anyone trying to find an answer to this—AWS updated their documentation and tutorial to support the Graviton processors. Works like a charm now.