How we can set table properties for delta table in pyspark using DeltaTable API
Question:
Below is the code that I am trying in PySpark
from delta import DeltaTable
delta_table = DeltaTable.forPath(spark, delta_table_path)
delta_table.logRetentionDuration = "interval 1 days"
After this do we need to save this config or it will be applicable automatically. How we can check what is current logRetentionDuration set for table.
I tried below to get properties info
delta_table.detail()
But it return empty {}
Answers:
Use Spark SQL:
spark.sql("ALTER TABLE delta.`pathtodeltatable` SET TBLPROPERTIES ('delta.logRetentionDuration'='1 days')")
spark.sql("DESCRIBE DETAIL delta.`pathtodeltatablepath`").show(truncate=False)
Under the "properties" column you can see the log retention duration of the specified delta table.
Note: If the retention period was not set properly, the above SQL command returns {}
in the properties column.
Below is the code that I am trying in PySpark
from delta import DeltaTable
delta_table = DeltaTable.forPath(spark, delta_table_path)
delta_table.logRetentionDuration = "interval 1 days"
After this do we need to save this config or it will be applicable automatically. How we can check what is current logRetentionDuration set for table.
I tried below to get properties info
delta_table.detail()
But it return empty {}
Use Spark SQL:
spark.sql("ALTER TABLE delta.`pathtodeltatable` SET TBLPROPERTIES ('delta.logRetentionDuration'='1 days')")
spark.sql("DESCRIBE DETAIL delta.`pathtodeltatablepath`").show(truncate=False)
Under the "properties" column you can see the log retention duration of the specified delta table.
Note: If the retention period was not set properly, the above SQL command returns {}
in the properties column.