How to deploy locally multiple flows using Prefect 2.0?

Question

I’m reading Prefect documentation and trying to understand how local deployment works. I can deploy a flow locally following the below steps.

First, I build the flow:

prefect deployment build ./log_flow.py:log_flow -n log-simple -q test

Where ./log_flow.py:log_flow are, respectively, the flow’s location and entrypoint. log-simple is the name of deployment and test is the work queue

Second, I start the worker using:

prefect agent start -q 'test'

To apply the deployment, I use python running the below snippet:

from log_flow import log_flow
from prefect.deployments import Deployment

deployment = Deployment.build_from_flow(
    flow=log_flow,
    name="log-simple",
    parameters={"name": "Marvin"},
    infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
    work_queue_name="test",
)

if __name__ == "__main__":
    deployment.apply()

Well, that works fine for a single flow. But how can I deploy several flows at once? I can repeat the above process for every flow, but it looks a bit unpractical to me since each build step generates another YAML file. I think would be more practical if my deployment generates a single YAML file for all flows.

Is there a way to deploy several flows at once in Prefect 2.0?

Asked By: Lucas

||

Source

Answer 1

Yes, absolutely. You could loop over all flows from Python, bash or from CI/CD. Here are a couple of examples showing how you could approach it:

CI/CD: https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/main.yaml#L35
bash script (there are several example bash scripts in this repo for various infra/storage options): https://github.com/anna-geller/prefect-dataplatform/blob/main/deploy_locally.py

Answered By: Anna Geller

Answer 2

You can create a list of deployments and then, use method aply to each one of them. Something like this:

# deployment.py

from my_project.flows import my_flow, my_flow2
from prefect.deployments import Deployment

DEPLOYMENTS = [
Deployment.build_from_flow(
    flow=my_flow, 
    name='deploy_1'
    ),
Deployment.build_from_flow(
    flow=my_flow2, 
    name='deploy_2'
    )
...
]
if __name__ == '__main__':
    for deployment in DEPLOYMENTS:
        deployment.apply()

Then You can simply run this python script

Answered By: Vinícius do Pinho Barbosa

How to deploy locally multiple flows using Prefect 2.0?

Question:

Answers: