Twisted application ignoring a certain UNIX signal – is it possible?

Question

Let’s say we have the following situation:

kill <pid> sends SIGTERM

kill -<SIGNAL> <pid> sends <SIGNAL>

Sometimes, during development, I need to kill my application and restart it, at the moment – using the first type of command. But, if I have a production console opened, I have a chance to kill our production (let’s say, I’ve forgotten about it – THIS HAPPENED RIGHT NOW).

The solution that came into my mind is based on ignoring SIGTERM in production mode, but killing the app gracefully in development mode. This way, if, for some reason, I want to kill our prod, I’ll need to specify a SIGNAL to do it, and it’ll be impossible to be done accidentally.

The app is built on Twisted.

Twisted has a number useful of methods to use signals with it – for example:

reactor.addSystemEventTrigger('before', 'shutdown', shutdown_callback)

But is it possible to make it ignore a certain signal? I need only one, and I don’t want to go this [reactor.run(installSignalHandlers=False)] way (some people say that it doesn’t even work) – it’ll require me to rewrite the whole signal handling by myself, and that’s not what I’m looking for.

Thanks!

Asked By: Leontyev Georgiy

||

Source

Answer 1

Twisted installs some signal handlers by default but the only one it really tightly integrates with is SIGCHLD on POSIX so that it can do child process management correctly.

You can just use the Python signal module to change the signal-handling behavior of any signal you want (even SIGCHLD, just be aware this will probably break reactor.spawnProcess).

Twisted itself doesn’t provide any APIs for customizing signal handling behavior.

Answered By: Jean-Paul Calderone

Answer 2

This is what I’ve done in the end (using twistd module with startReactor() override):

def signal_handler(signum, frame):
    if signum == signal.SIGTERM:
        if is_prod():
            log.critical("Received SIGTERM on PRODUCTION call system, ignoring!")
        else:
            log.critical("Received SIGTERM on DEV call system, shutting down!")
            reactor.stop()
    elif any([
        signum == signal.SIGQUIT,
        signum == signal.SIGINT,
        signum == signal.SIGILL,
        signum == signal.SIGFPE,
        signum == signal.SIGABRT,
        signum == signal.SIGBUS,
        signum == signal.SIGPIPE,
        signum == signal.SIGSYS,
        signum == signal.SIGSEGV,
        signum == signal.SIGHUP
    ]):
        log.critical(f"*Received {signal.Signals(signum).name}, shutting down.*")
        reactor.stop()

def register_signals():
    signal.signal(signal.SIGTERM, signal_handler)
    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGILL, signal_handler)
    signal.signal(signal.SIGFPE, signal_handler)
    signal.signal(signal.SIGABRT, signal_handler)
    signal.signal(signal.SIGBUS, signal_handler)
    signal.signal(signal.SIGPIPE, signal_handler)
    signal.signal(signal.SIGSYS, signal_handler)
    signal.signal(signal.SIGSEGV, signal_handler)
    signal.signal(signal.SIGHUP, signal_handler)

# ...

class ApplicationRunner(twistd._SomeApplicationRunner):
    def startReactor(self, reactor, oldstdout, oldstderr):
        self._exitSignal = None
        from twisted.internet import reactor
        try:
            reactor.run(installSignalHandlers=False)
        except BaseException:
            close = False
            if self.config["nodaemon"]:
                file = oldstdout
            else:
                file = open("TWISTD-CRASH.log", "a")
                close = True
            try:
                traceback.print_exc(file=file)
                file.flush()
            finally:
                if close:
                    file.close()

    def createOrGetApplication(self):
        return application

    def run(self):
        self.preApplication()
        self.application = self.createOrGetApplication()
        self.postApplication()


register_signals()

twistd._SomeApplicationRunner = ApplicationRunner
twistd.run()

Basically, this code gets access to the execution of the inner reactor, adds the required parameter, and takes all signal handling on itself. Not the best solution, but that’s all we have now.

Known bug:
OS kills processes with SIGTERM during the restart, so if the OS triggers the process shutdown, it will send SIGTERM there, and then OS will hang. The solution is to check the following before denying the SIGTERM request:

Existing SSH connections (if there are no connections, no user could make such a mistake, so the process shutdown should proceed).
Bash history for shutdown, reboot, poweroff, and other stuff like that (Poweruser wants to shut down the server, so we should proceed with the process shut down)
Any other system-specific conditions.

Answered By: Leontyev Georgiy

Twisted application ignoring a certain UNIX signal – is it possible?

Question:

Answers: