Reading command output with Paramiko invoke_shell/send/recv never finishes
Question:
I am trying to use send/recv function in Paramiko. According to what I see, the line throws an exception of timeout
Evaluating: self.shell.recv(1024) did not finish after 3.00 seconds.
tmp = shell.recv(1024)
What is wrong with the function implementation?
My exit condition from while True is an exception, how can I change that to exit without an exception?
Full code:
self.shell = self.SSHConnect(ip, username, password)
def SSHConnect(self, ip, username, passowrd):
ssh = paramiko.SSHClient()
LOGGER.debug(msg="Open SSH Client to :" + str(ip))
try:
ssh.set_missing_host_key_policy(policy=paramiko.AutoAddPolicy())
ssh.connect(ip, port=22, username=username, password=passowrd, allow_agent=False, look_for_keys=True)
if self.device_type != 'linux_host':
session = ssh.invoke_shell()
return session
except Exception as ex:
LOGGER.critical(msg="SSH Client wasn't established! Device name : " + str(self.device_name))
return None
LOGGER.info(msg="Open SSH Client to :" + str(ip) + " established!")
return ssh
def run_command(self,cmd):
# New run command without throwing exception at the end:
LOGGER.debug('Start new Run command with cmd = ' + str(cmd))
try:
#Check the shell is activated before sending command:
LOGGER.debug('Check the shell is activated before sending command: ' + cmd)
if self.shell.get_transport().active:
LOGGER.debug('Shell is Activated ! Running the command ')
if self.device_type == 'linux_host':
stdin, stdout, stderr = self.shell.exec_command(cmd)
else:
try:
#Command for switch of UFMAPL
LOGGER.debug('Sending command to UFMAPL/Switch with send()')
out = ''
self.shell.send(cmd)
while not self.shell.recv_ready():
time.sleep(3)
counter = 1
print('ncommand is : ' + cmd + 'n' )
while True:
try:
print('iteration number is : #' + str(counter))
tmp = self.shell.recv(1024)
counter = counter + 1
if not tmp:
break
except Exception as e:
break
out += tmp.decode("utf-8")
print('After iteration #' + str(counter) + ' out = ' + out + 'nn')
ansi_escape = re.compile(r'x1B[[0-?]*[ -/]*[@-~]')
out = ansi_escape.sub('', out)
print('Printing final value before return : ' + str(out +'n'))
return out
except Exception as e:
LOGGER.error('Exception in send() : ' +str(e) )
return None
else:
LOGGER.critical('Shell is not activated !')
return ""
if stderr.read():
LOGGER.critical('stderr is not empty which means the last command was failed, the command might not exist on host/switch ' )
return stderr.read().decode('utf-8')
out = stdout.read()
if out:
return out.decode("utf-8")
else:
LOGGER.critical('Run command sucussfully but return empty...')
return out.decode("utf-8")
except Exception as e:
LOGGER.error('Exception received in run command : ' + str(e))
Logs (Printing to screen):
IM HEREEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
command is : enable
iteration number is : #1
After iteration #2 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] >
iteration number is : #2
After iteration #3 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > e
iteration number is : #3
After iteration #4 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-a
iteration number is : #4
After iteration #5 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-apl00
iteration number is : #5
After iteration #6 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-apl008-gen2 [ mgmt-sa ] #
iteration number is : #6
As you can see the debugger is stuck on iteration #6 (freeze).
Why does it freeze and doesn’t send output?
Enviroment details:
- Windows 10
- Eclipse latest
I would appreciate any help here. Let me know if you need more details.
Answers:
Your code gets to the point, where the server stops waiting for another input. At the same time you wait for the server to output something. What it never will. That’s called a deadlock.
You might have expected some kind of signal that the first command execution has finished. There’s no such signal. You are using a "shell" (SSHClient.invoke_shell). The shell is a black box with an input and an output. There are no other signals except for the output, which you are reading already.
The "shell" should not be used to automate command execution. For command automation, there’s the "exec" channel (SSHClient.exec_command in Paramiko).
Though I understand that with some special devices, what your server seems to be, you might not have any other option (see Executing command using Paramiko exec_command on device is not working). Moreover I’m not ever sure how the enable command works. Whether it’s a command that has finished, or whether it started a kind of a new shell, which is still running and is waiting for subcommands.
So in the end all you can do is to parse the output, waiting for the command prompt (smg-ib-apl008-gen2 [ mgmt-sa ] #).
Yes, it’s ugly. You are trying to automate something that was not intended for automation. Maybe your server has a better API then enable shell command, that would be nicer to automate. But that’s for another question. From SSH/Python/Paramiko point of view, there’s no better solution, if you need to stick with executing the enable command in the shell.
Related questions:
- How to get each dependent command execution output using Paramiko exec_command
- Execute multiple dependent commands individually with Paramiko and find out when each command finishes
Though they are about more regular servers, like Linux. So they won’t really help. I’m linking them just to provide broader picture and context.
I am trying to use send/recv function in Paramiko. According to what I see, the line throws an exception of timeout
Evaluating: self.shell.recv(1024) did not finish after 3.00 seconds.
tmp = shell.recv(1024)
What is wrong with the function implementation?
My exit condition from while True is an exception, how can I change that to exit without an exception?
Full code:
self.shell = self.SSHConnect(ip, username, password)
def SSHConnect(self, ip, username, passowrd):
ssh = paramiko.SSHClient()
LOGGER.debug(msg="Open SSH Client to :" + str(ip))
try:
ssh.set_missing_host_key_policy(policy=paramiko.AutoAddPolicy())
ssh.connect(ip, port=22, username=username, password=passowrd, allow_agent=False, look_for_keys=True)
if self.device_type != 'linux_host':
session = ssh.invoke_shell()
return session
except Exception as ex:
LOGGER.critical(msg="SSH Client wasn't established! Device name : " + str(self.device_name))
return None
LOGGER.info(msg="Open SSH Client to :" + str(ip) + " established!")
return ssh
def run_command(self,cmd):
# New run command without throwing exception at the end:
LOGGER.debug('Start new Run command with cmd = ' + str(cmd))
try:
#Check the shell is activated before sending command:
LOGGER.debug('Check the shell is activated before sending command: ' + cmd)
if self.shell.get_transport().active:
LOGGER.debug('Shell is Activated ! Running the command ')
if self.device_type == 'linux_host':
stdin, stdout, stderr = self.shell.exec_command(cmd)
else:
try:
#Command for switch of UFMAPL
LOGGER.debug('Sending command to UFMAPL/Switch with send()')
out = ''
self.shell.send(cmd)
while not self.shell.recv_ready():
time.sleep(3)
counter = 1
print('ncommand is : ' + cmd + 'n' )
while True:
try:
print('iteration number is : #' + str(counter))
tmp = self.shell.recv(1024)
counter = counter + 1
if not tmp:
break
except Exception as e:
break
out += tmp.decode("utf-8")
print('After iteration #' + str(counter) + ' out = ' + out + 'nn')
ansi_escape = re.compile(r'x1B[[0-?]*[ -/]*[@-~]')
out = ansi_escape.sub('', out)
print('Printing final value before return : ' + str(out +'n'))
return out
except Exception as e:
LOGGER.error('Exception in send() : ' +str(e) )
return None
else:
LOGGER.critical('Shell is not activated !')
return ""
if stderr.read():
LOGGER.critical('stderr is not empty which means the last command was failed, the command might not exist on host/switch ' )
return stderr.read().decode('utf-8')
out = stdout.read()
if out:
return out.decode("utf-8")
else:
LOGGER.critical('Run command sucussfully but return empty...')
return out.decode("utf-8")
except Exception as e:
LOGGER.error('Exception received in run command : ' + str(e))
Logs (Printing to screen):
IM HEREEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
command is : enable
iteration number is : #1
After iteration #2 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] >
iteration number is : #2
After iteration #3 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > e
iteration number is : #3
After iteration #4 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-a
iteration number is : #4
After iteration #5 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-apl00
iteration number is : #5
After iteration #6 out =
UFM Appliance
UFM is configured as standalone.
UFM mode: Management.
RAID state is: Degraded ( DRIVE1:Online,SpunUp, DRIVE2:Failed )
smg-ib-apl008-gen2 [ mgmt-sa ] > enable
smg-ib-apl008-gen2 [ mgmt-sa ] #
iteration number is : #6
As you can see the debugger is stuck on iteration #6 (freeze).
Why does it freeze and doesn’t send output?
Enviroment details:
- Windows 10
- Eclipse latest
I would appreciate any help here. Let me know if you need more details.
Your code gets to the point, where the server stops waiting for another input. At the same time you wait for the server to output something. What it never will. That’s called a deadlock.
You might have expected some kind of signal that the first command execution has finished. There’s no such signal. You are using a "shell" (SSHClient.invoke_shell). The shell is a black box with an input and an output. There are no other signals except for the output, which you are reading already.
The "shell" should not be used to automate command execution. For command automation, there’s the "exec" channel (SSHClient.exec_command in Paramiko).
Though I understand that with some special devices, what your server seems to be, you might not have any other option (see Executing command using Paramiko exec_command on device is not working). Moreover I’m not ever sure how the enable command works. Whether it’s a command that has finished, or whether it started a kind of a new shell, which is still running and is waiting for subcommands.
So in the end all you can do is to parse the output, waiting for the command prompt (smg-ib-apl008-gen2 [ mgmt-sa ] #).
Yes, it’s ugly. You are trying to automate something that was not intended for automation. Maybe your server has a better API then enable shell command, that would be nicer to automate. But that’s for another question. From SSH/Python/Paramiko point of view, there’s no better solution, if you need to stick with executing the enable command in the shell.
Related questions:
- How to get each dependent command execution output using Paramiko exec_command
- Execute multiple dependent commands individually with Paramiko and find out when each command finishes
Though they are about more regular servers, like Linux. So they won’t really help. I’m linking them just to provide broader picture and context.