error: Tried to create relationships between tasks that don't have DAGs yet. Set the DAG for at least one task and try again Question: while we are trying to pass dag conf variables as parameters and running in a loop with decorator we are getting below error. is there a way to fix error in …
how to quickly identify if a rule in Snakemake needs an input function Question: I’m following the snakemake tutorial on their documentation page and really got stuck on the concept of input functions https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html#step-3-input-functions Basically they define a config.yaml as follows: samples: A: data/samples/A.fastq B: data/samples/B.fastq and the Snakefile as follows without any input function: …
how to get the list of all tasks along with their status for current dag run in airflow Question: Need help to extract the list of all tasks along with their current status [Success/Failed] for the current dag run. I have a task with a python operator which executes at the end of the workflow. …
How does dask know variable states before it runs map_partitions? Question: In the dask code below I set x with 1 and 2 right before executing two map_partitions. The result seems fine, however I don’t fully understand it. If dask waits to run the two map_partitions only when it finds the compute(), and at the …
Prevent rules from rerunning when intermediate file is updated Question: Let’s say I have two rules in my snakemake file The first rule fetches a remote file and makes a temporary local copy The second rule uses the local file and performs an expensive task Now lets say I ran this pipeline to completion and …
Airflow 2.0 task getting skipped after BranchPython Operator Question: I’m fiddling with branches in Airflow in the new version and no matter what I try, all the tasks after the BranchOperator get skipped. Here is a minimal example of what I’ve been trying to accomplish from airflow.decorators import dag, task from datetime import timedelta, datetime …
Getting Error in airflow DAG unsupported operand type(s) for >>: 'list' and 'list'. Sequential and Parallel execution of tasks Question: I am new to Apache airflow and DAG. There are total 6 tasks in the DAG (task1, task2, task3, task4, task5, task6). But at the time of running the DAG we are getting the error …
How to Trigger a DAG on the success of a another DAG in Airflow using Python? Question: I have a python DAG Parent Job and DAG Child Job. The tasks in the Child Job should be triggered on the successful completion of the Parent Job tasks which are run daily. How can add external job …
Airflow Packaged Dags (zipped) clash when subfolders have same name Question: We’re setting up an Airflow framework in which multiple data scientist teams can orchestrate their data processing pipelines. We’ve developed a Python code-base to help them implement the DAGs, which includes functions and classes (Operator sub-classes as well) in various packages and modules. Every …
How to run Spark code in Airflow? Question: Hello people of the Earth! I’m using Airflow to schedule and run Spark tasks. All I found by this time is python DAGs that Airflow can manage. DAG example: spark_count_lines.py import logging from airflow import DAG from airflow.operators import PythonOperator from datetime import datetime args = { …
