pyspark

How to query for the maximum / highest value in an field with PySpark Question: The following dataframe will produce values 0 to 3. df = DeltaTable.forPath(spark, ‘/mnt/lake/BASE/SQLClassification/cdcTest/dbo/cdcmergetest/1’).history().select(col("version")) Can someone show me how to modify the dataframe such that it only provides the maximum value i.e 3? I have tried df.select("*").max("version") And df.max("version") But no …

Get Geometric Mean Over Window in Pyspark Dataframe Question: I have the following pyspark dataframe Car Time Val1 1 1 3 2 1 6 3 1 8 1 2 10 2 2 21 3 2 33 I want to get the geometric mean of all the cars at each time, resulting df should look like …

spark dataframe convert a few flattened columns to one array of struct column Question: I’d like to have some guidance what functions in spark dataframe together with scala/python code to achieve this transformation. given a dataframe which has below columns columnA, columnB, columnA1, ColumnB1, ColumnA2, ColumnB2 …. ColumnA10, ColumnB10 eg. Fat Value, Fat Measure, Salt …

Datetime object error while inserting datetime in pyspark dataframe Question: I am getting a error while inserting a datetime object into the pyspark data structure from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType, TimestampType, ArrayType, BinaryType filename = ‘Rekorder_2022-08-24_14-12-42.mf4′ match = re.search(r'(d+-d+-d+_d+-d+-d+)’,filename) date_time = datetime.datetime.strptime(match.group(1),"%Y-%m-%d_%H-%M-%S") Now, I am trying to insert the date into the …

How we can set table properties for delta table in pyspark using DeltaTable API Question: Below is the code that I am trying in PySpark from delta import DeltaTable delta_table = DeltaTable.forPath(spark, delta_table_path) delta_table.logRetentionDuration = "interval 1 days" After this do we need to save this config or it will be applicable automatically. How we …

Pyspark: Compare Column Values across different dataframe Question: we are planning to do the following, compare two dataframe, based on comparision add values into first dataframe and then groupby to have combined data. We are using pyspark dataframe and the following are our dataframes. Dataframe1: | Manager | Department | isHospRelated | ——– | ————– …

Calling udf is not working on spark dataframe Question: I have a dictionary and function I defined and registered a udf as a SQL function %%spark d = {‘I’:’Ice’, ‘U’:’UN’, ‘T’:’Tick’} def key_to_val(k): if k in d: return d[k] else: return "Null" spark.udf.register(‘key_to_val’, key_to_val,StringType()) And I have spark dataframe that looks like sdf = +—-+————+————–+ …

Rank does not go in order if the value does not change Question: I have a dataframe: data = [[‘p1’, ‘t1’], [‘p4’, ‘t2’], [‘p2’, ‘t1’],[‘p4’, ‘t3’], [‘p4’, ‘t3’], [‘p3’, ‘t1’],] sdf = spark.createDataFrame(data, schema = [‘id’, ‘text’]) sdf.show() +—+—-+ | id|text| +—+—-+ | p1| t1| | p4| t2| | p2| t1| | p4| t3| | …

How to transform a dataset to basic metrics dataset on various date based rollup in a big dataset using pyspark Question: I have a dataset that looks like this. Date Time Stock-a Stock-b Stock-c 2023-01-01 10:30 10 20 30 2023-01-01 11:30 11 21 31 2023-01-02 01:30 15 19 18 2023-01-02 12:30 6 25 8 I …

How can I have a row-wise rank in a pyspark dataframe Question: I have a dataset for which I am going to find the rank per row. This is a toy example in pandas. import pandas as pd df = pd.DataFrame({"ID":[1,2,3,4], "a":[2,7,9,10], "b":[6,7,4,2], "c":[3,4,8,5]}) print(df) # ID a b c # 0 1 2 6 …