Airflow signals SIGTERM to subprocesses unexpectedly

I too had similar issue when I was running Python multithreaded code. I was able to resolve the same by joining threads. Airflow then waits till all thread gets executed before sending SIGTERM.

threads = []   #array for threads

 t = Thread(...)
 threads.append(t) #add all threads

 # Start all threads
 for x in threads:
     x.start()

 # Wait for all of them to finish
 for x in threads:
     x.join()

Probably this is way overdue but what solved my issue was increasing the dagrun_timeout parameter. You should check if your execution fits the window.

DAGs that run longer or have more than 10-12 tasks (that may run long) seem to have a higher probability of getting SIGTERM.

PROBABLE SOLUTIONS:
schedule_after_task_execution to False
If True, Task supervisor process runs a "mini scheduler". However, after marking it as success, it will call _run_mini_scheduler_on_child_tasks. And while local_task_job.py will detect the task as success and not running, it will terminate the process which might be still executing under _run_mini_scheduler_on_child_tasks.

The problem may also be because the scheduler health check threshold which is set to be smaller than the scheduler heartbeat interval.
The default  scheduler_health_check_threshold was 30 seconds and scheduler_heartbeat_sec was 60 seconds. During the check for orphaned tasks, the scheduler heartbeat was determined to be older than 30 seconds, which makes sense, because it was only heartbeating every 60 seconds. Thus the scheduler was inferred to be unhealthy and was therefore maybe terminated.

The logs in my case shows that the SIGTERM is associated with tasks being considered as orphaned.

As the  the timestamps coincide closely with the SIGTERM received by the task. It seems that since the SchedulerJob was marked as failed, then the TaskInstance running the actual task was considered an orphan, and thus marked for termination. I changes these values as
scheduler_heartbeat_sec = 60
scheduler_health_check_threshold = 120 (twice that of scheduler_heartbeat_sec)
Some explanations blamed  metadata database being used as 100% cpu while running dags with multiple tasks in parallel, such as the example used in my case.
Increasing the database may solve the issue, but instead increasing the job_heartbeat_sec and the other three configurations actually solved the problem .
job_heartbeat_sec = 60 (roughly based on the time almost taken by the task

This might help if you use a Mac (Not sure if on all versions though. Right now it is MacOS 12.1, Monterey, Python 3.8).

I tried some of the solutions above, to no avail. Then I noticed in the log for the failed task that I was getting the warning:

• WARNING – The recorded hostname
  1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa
  does not match this instance’s hostname grad_mac

Then immediately afterwards the SIGTERM signal being sent by the scheduler:

• INFO – Sending Signals.SIGTERM to GPID 70482

In the airflow.cfg, the python function socket.getfqdn is used by default to infer the hostname (look for line hostname_callable = socket.getfqdn).
I did some digging (here), and turns out this function behaves somewhat differently in Mac.

I replaced the line to use gethostname (hostname_callable = socket.gethostname), restarted the scheduler, et voilá !

In retrospect, the scheduler was trying to contact the task to check its health state (as it’s what a well behaved scheduler usually does), but could not contact it with the originally inferred hostname. It would therefore assume unhealthy state, and send the kill signal…

I was having the same problem after upgrading from Airflow v1.10.15 to v2.2.5 and was seeing the error in long-running DAGs having a fairly high number of tasks.

Apparently, the dagrun_timeout in airflow.models.DAG was not respected in earlier Airflow versions so I noticed that the DAGs I was trying to migrate to the new Airflow instance were running for much longer than the specified dagrun_timeout.

The solution for me was to increase the dagrun_timeout (e.g. dagrun_timeout=datetime.timedelta(minutes=120)).

Note that this variable is effective only for scheduled tasks (in other words with DAGs with a specified schedule_interval).