Spark 1.4 increase maxResultSize memory
Question:
I am using Spark 1.4 for my research and struggling with the memory settings. My machine has 16GB of memory so no problem there since the size of my file is only 300MB. Although, when I try to convert Spark RDD to panda dataframe using toPandas()
function I receive the following error:
serialized results of 9 tasks (1096.9 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
I tried to fix this changing the spark-config file and still getting the same error. I’ve heard that this is a problem with spark 1.4 and wondering if you know how to solve this. Any help is much appreciated.
Answers:
You can set spark.driver.maxResultSize
parameter in the SparkConf
object:
from pyspark import SparkConf, SparkContext
# In Jupyter you have to stop the current context first
sc.stop()
# Create new config
conf = (SparkConf()
.set("spark.driver.maxResultSize", "2g"))
# Create new context
sc = SparkContext(conf=conf)
You should probably create a new SQLContext
as well:
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
Looks like you are collecting the RDD, So it will definitely collect all the data to driver node that’s why you are facing this issue.
You have to avoid collect data if not required for a rdd, or if its necessary then specify spark.driver.maxResultSize
. there are two ways of defining this variable
1 – create Spark Config by setting this variable as
conf.set("spark.driver.maxResultSize", "3g")
2 – or set this variable
in spark-defaults.conf
file present in conf folder of spark. like
spark.driver.maxResultSize 3g
and restart the spark.
From the command line, such as with pyspark, --conf spark.driver.maxResultSize=3g
can also be used to increase the max result size.
Tuning spark.driver.maxResultSize
is a good practice considering the running environment. However, it is not the solution to your problem as the amount of data may change time by time. As @Zia-Kayani mentioned, it is better to collect data wisely. So if you have a DataFrame df
, then you can call df.rdd
and do all the magic stuff on the cluster, not in the driver. However, if you need to collect the data, I would suggest:
- Do not turn on
spark.sql.parquet.binaryAsString
. String objects take more space
- Use
spark.rdd.compress
to compress RDDs when you collect them
- Try to collect it using pagination. (code in Scala, from another answer Scala: How to get a range of rows in a dataframe)
long count = df.count()
int limit = 50;
while(count > 0){
df1 = df.limit(limit);
df1.show(); //will print 50, next 50, etc rows
df = df.except(df1);
count = count - limit;
}
There is also a Spark bug
https://issues.apache.org/jira/browse/SPARK-12837
that gives the same error
serialized results of X tasks (Y MB) is bigger than spark.driver.maxResultSize
even though you may not be pulling data to the driver explicitly.
SPARK-12837 addresses a Spark bug that accumulators/broadcast variables prior to Spark 2 were pulled to driver unnecessary causing this problem.
while starting the job or terminal, you can use
--conf spark.driver.maxResultSize="0"
to remove the bottleneck
You can set spark.driver.maxResultSize to 2GB when you start the pyspark shell:
pyspark --conf "spark.driver.maxResultSize=2g"
This is for allowing 2Gb for spark.driver.maxResultSize
I am using Spark 1.4 for my research and struggling with the memory settings. My machine has 16GB of memory so no problem there since the size of my file is only 300MB. Although, when I try to convert Spark RDD to panda dataframe using toPandas()
function I receive the following error:
serialized results of 9 tasks (1096.9 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
I tried to fix this changing the spark-config file and still getting the same error. I’ve heard that this is a problem with spark 1.4 and wondering if you know how to solve this. Any help is much appreciated.
You can set spark.driver.maxResultSize
parameter in the SparkConf
object:
from pyspark import SparkConf, SparkContext
# In Jupyter you have to stop the current context first
sc.stop()
# Create new config
conf = (SparkConf()
.set("spark.driver.maxResultSize", "2g"))
# Create new context
sc = SparkContext(conf=conf)
You should probably create a new SQLContext
as well:
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
Looks like you are collecting the RDD, So it will definitely collect all the data to driver node that’s why you are facing this issue.
You have to avoid collect data if not required for a rdd, or if its necessary then specify spark.driver.maxResultSize
. there are two ways of defining this variable
1 – create Spark Config by setting this variable as
conf.set("spark.driver.maxResultSize", "3g")
2 – or set this variable
inspark-defaults.conf
file present in conf folder of spark. like
spark.driver.maxResultSize 3g
and restart the spark.
From the command line, such as with pyspark, --conf spark.driver.maxResultSize=3g
can also be used to increase the max result size.
Tuning spark.driver.maxResultSize
is a good practice considering the running environment. However, it is not the solution to your problem as the amount of data may change time by time. As @Zia-Kayani mentioned, it is better to collect data wisely. So if you have a DataFrame df
, then you can call df.rdd
and do all the magic stuff on the cluster, not in the driver. However, if you need to collect the data, I would suggest:
- Do not turn on
spark.sql.parquet.binaryAsString
. String objects take more space - Use
spark.rdd.compress
to compress RDDs when you collect them - Try to collect it using pagination. (code in Scala, from another answer Scala: How to get a range of rows in a dataframe)
long count = df.count()
int limit = 50;
while(count > 0){
df1 = df.limit(limit);
df1.show(); //will print 50, next 50, etc rows
df = df.except(df1);
count = count - limit;
}
There is also a Spark bug
https://issues.apache.org/jira/browse/SPARK-12837
that gives the same error
serialized results of X tasks (Y MB) is bigger than spark.driver.maxResultSize
even though you may not be pulling data to the driver explicitly.
SPARK-12837 addresses a Spark bug that accumulators/broadcast variables prior to Spark 2 were pulled to driver unnecessary causing this problem.
while starting the job or terminal, you can use
--conf spark.driver.maxResultSize="0"
to remove the bottleneck
You can set spark.driver.maxResultSize to 2GB when you start the pyspark shell:
pyspark --conf "spark.driver.maxResultSize=2g"
This is for allowing 2Gb for spark.driver.maxResultSize