spark-streaming

How to make sure values are map to the right delta table column? Question: I’m writing a PySpark job to read the Values column from table1. Table1 has two column -> ID, Values Sample data in the Values column: +—-+———————————–+ | ID | values | +—-+———————————–+ | 1 | a=10&b=2&c=13&e=55&d=78&j=98&l=99 | | 2 | l=22&e=67&j=34&a=7&c=9&d=77&b=66 …

How to filter remove null values in spark python Question: I’m trying to filter out the null values in a column and count if its greater than 1. badRows = df.filter($"_corrupt_record".isNotNull) if badRows.count > 0: logger.error("throwing bad rows exception…") schema_mismatch_exception(None, "cdc", item ) I’m getting a syntax error. Also tried to check using : badRows …