How we can set table properties for delta table in pyspark using DeltaTable API

Question:

Below is the code that I am trying in PySpark

from delta import DeltaTable

delta_table = DeltaTable.forPath(spark, delta_table_path)
delta_table.logRetentionDuration = "interval 1 days"

After this do we need to save this config or it will be applicable automatically. How we can check what is current logRetentionDuration set for table.
I tried below to get properties info

delta_table.detail()

But it return empty {}

Asked By: YOGESH

||

Answers:

Use Spark SQL:

spark.sql("ALTER TABLE delta.`pathtodeltatable` SET TBLPROPERTIES ('delta.logRetentionDuration'='1 days')")
spark.sql("DESCRIBE DETAIL delta.`pathtodeltatablepath`").show(truncate=False)

Under the "properties" column you can see the log retention duration of the specified delta table.

Note: If the retention period was not set properly, the above SQL command returns {} in the properties column.

Answered By: arudsekaberne
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.