DBT – How to insert data twice in the same table using DBT?

Question

I have the scenario, where I need to insert into the same table but in two steps
First, I insert parent rows & hence get the Auto Incremented IDs from the database
Second, I need to insert child data, which will use the **Auto Incremented IDs **generated by database in first step.
I cannot have two models with the same name, & I cannot insert complete data in one go/model.
How do I solve this problem?

I tried making two models with the same name, but that does not work for DBT

Asked By: solomon1994

||

Source

Answer 1

One can use dbt models to insert data into a table in two steps. A dbt model is a SQL query that is executed to create a table in your data warehouse. You can use multiple dbt models to insert data into the same table in multiple steps, and use the ref field to refer to the generated IDs from the first model in the second model.

Here’s an example of how you might do this:

-- dbt model for inserting the parent rows
{{
  config({
    unique_id: "parent_model"
  })
}}

INSERT INTO table (column1, column2, ...) VALUES (value1, value2, ...);

-- dbt model for inserting the child rows, using the generated ID from the parent model
{{
  config({
    unique_id: "child_model"
  })
}}

{% set parent_id = ref('parent_model') %}

INSERT INTO table (column1, column2, parent_id) VALUES (value1, value2, {{ parent_id }});

In this example, the child_model uses the ref field to refer to the parent_model, and fetches the generated ID from that model. It then uses that ID to insert the child rows, using the parent_id column to specify the parent-child relationship.

The first model will insert the parent rows and generate the IDs, and the second model will use those IDs to insert the child rows. You can then run these models in sequence to insert the data into the table in two steps.

Alternatively, one can use dbt’s transaction macro in your models to wrap the two insert statements in a transaction, which will ensure that either both inserts are successful, or neither are. Here’s how that would look:

-- dbt model for inserting the parent and child rows in a single transaction
{{
  config({
    unique_id: "parent_child_model"
  })
}}

{{
  transaction(
    'Insert the parent rows',
    insert('table', [
      {column1: value1, column2: value2, ...},
      ...
    ]),

    'Insert the child rows, using the generated ID from the parent rows',
    {% set parent_id = ref('parent_model') %}
    insert('table', [
      {column1: value1, column2: value2, parent_id: parent_id},
      ...
    ])
  )
}}

Answered By: solomon1994

DBT – How to insert data twice in the same table using DBT?

Question:

Answers: