How to coalesce multiple pyspark arrays? Question: I have an arbitrary number of arrays of equal length in a PySpark DataFrame. I need to coalesce these, element by element, into a single list. The problem with coalesce is that it doesn’t work by element, but rather selects the entire first non-null array. Any suggestions for …
Fill nulls with values from another column in PySpark Question: I have a dataset col_id col_2 col_3 col_id_b ABC111 shfhs 34775 null ABC112 shfhe 34775 DEF345 ABC112 shfhs 34775 GFR563 ABC112 shfgh 34756 TRS572 ABC113 shfdh 34795 null ABC114 shfhs 34770 null I am trying to create a new column that is identical to col_id_b, …
Joining multiple columns and coalesce and when, then cases ('pace_&YrMn.'- simple variable ) Question: create table pace_&YrMn. as select a.hotel_cd, a.blk_dt, case when a.arr_date <= ’31DEC09’d then coalesce(c.mkt_seg_dsc, m.mkt_seg_dsc1) else coalesce(s.quoteaccountmarketsegment, c.mkt_seg_dsc, m.mkt_seg_dsc1) end as mkt_seg_dsc, case when a.blk_dt-c.definite_dt<=365 then’ 0-365′ when a.blk_dt-c.definite_dt<=730 then’366-730′ else ‘730+ ‘ end as bkg_window format=$7., case when a.blk_dt <= …
