Restrict/Exclude specific logs from Airflow to Datadog
Question:
We are observing Airflow is sending large amount of logs to Datadog and we want to restrict/Reduce these logs by excluding logs from below tasks:
- pod_manager.py
- base.py
- base_aws.py
- logging_mixin.py
Do we have any configuration settings where I can define this requirement?
We have Airflow-2.0 running on Kubernetes.
Answers:
To send only a specific subset of logs to Datadog, use the log_processing_rules parameter in your configuration file with the exclude_at_match or include_at_match type. Apply this to your schedular which generating the logs. Refer to this Datadog official doc for more information.
We are observing Airflow is sending large amount of logs to Datadog and we want to restrict/Reduce these logs by excluding logs from below tasks:
- pod_manager.py
- base.py
- base_aws.py
- logging_mixin.py
Do we have any configuration settings where I can define this requirement?
We have Airflow-2.0 running on Kubernetes.
To send only a specific subset of logs to Datadog, use the log_processing_rules parameter in your configuration file with the exclude_at_match or include_at_match type. Apply this to your schedular which generating the logs. Refer to this Datadog official doc for more information.