Select random rows from PySpark dataframe

Question

I want to select n random rows (without replacement) from a PySpark dataframe (preferably in the form of a new PySpark dataframe). What is the best way to do this?

Following is an example of a dataframe with ten rows.

+-----+-------------------+-----+
| name|          timestamp|value|
+-----+-------------------+-----+
|name1|2019-01-17 00:00:00|11.23|
|name2|2019-01-17 00:00:00|14.57|
|name3|2019-01-10 00:00:00| 2.21|
|name4|2019-01-10 00:00:00| 8.76|
|name5|2019-01-17 00:00:00|18.71|
|name5|2019-01-10 00:00:00|17.78|
|name4|2019-01-10 00:00:00| 5.52|
|name3|2019-01-10 00:00:00| 9.91|
|name1|2019-01-17 00:00:00| 1.16|
|name2|2019-01-17 00:00:00| 12.0|
+-----+-------------------+-----+

The above given dataframe generated by using the following code:

from pyspark.sql import *

df_Stats = Row("name", "timestamp", "value")

df_stat1 = df_Stats('name1', "2019-01-17 00:00:00", 11.23)
df_stat2 = df_Stats('name2', "2019-01-17 00:00:00", 14.57)
df_stat3 = df_Stats('name3', "2019-01-10 00:00:00", 2.21)
df_stat4 = df_Stats('name4', "2019-01-10 00:00:00", 8.76)
df_stat5 = df_Stats('name5', "2019-01-17 00:00:00", 18.71)
df_stat6 = df_Stats('name5', "2019-01-10 00:00:00", 17.78)
df_stat7 = df_Stats('name4', "2019-01-10 00:00:00", 5.52)
df_stat8 = df_Stats('name3', "2019-01-10 00:00:00", 9.91)
df_stat9 = df_Stats('name1', "2019-01-17 00:00:00", 1.16)
df_stat10 = df_Stats('name2', "2019-01-17 00:00:00", 12.0)

df_stat_lst = [df_stat1 , df_stat2, df_stat3, df_stat4, df_stat5,
               df_stat6, df_stat7, df_stat8, df_stat9, df_stat10]
df = spark.createDataFrame(df_stat_lst)

Asked By: Siddharth Satpathy

||

Source

Answer 1

There is a sample method on a pyspark.sql.DataFrame. The docs here should be helpful.

Usage:

df.sample(withReplacement=False, fraction=desired_fraction)

Answered By: sedavidw

Select random rows from PySpark dataframe

Question:

Answers: