key not found: _PYSPARK_DRIVER_CALLBACK_HOST

Question

I’m trying to run this code:

import pyspark
from pyspark.sql import SparkSession

spark = SparkSession.builder 
        .master("local") 
        .appName("Word Count") 
        .getOrCreate()

df = spark.createDataFrame([
    (1, 144.5, 5.9, 33, 'M'),
    (2, 167.2, 5.4, 45, 'M'),
    (3, 124.1, 5.2, 23, 'F'),
    (4, 144.5, 5.9, 33, 'M'),
    (5, 133.2, 5.7, 54, 'F'),
    (3, 124.1, 5.2, 23, 'F'),
    (5, 129.2, 5.3, 42, 'M'),
   ], ['id', 'weight', 'height', 'age', 'gender'])

df.show()
print('Count of Rows: {0}'.format(df.count()))
print('Count of distinct Rows: {0}'.format((df.distinct().count())))

spark.stop()

And getting an error

18/06/22 11:58:39 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]
java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST
    ...
Exception: Java gateway process exited before sending its port number

I’m using PyCharm and MacOS, Python 3.6, Spark 2.3.1

What is the possible reason of this error?

Asked By: bboy

||

Source

Answer 1

This error is a result of a version mismatch. Environment variable which is referenced in the traceback (_PYSPARK_DRIVER_CALLBACK_HOST) has been removed during update Py4j dependency to 0.10.7 and backported to 2.3 branch in 2.3.1.

Considering version information:

I’m using PyCharm and MacOS, Python 3.6, Spark 2.3.1

it looks like you have 2.3.1 package installed, but SPARK_HOME points to an older (2.3.0 or earlier) installation.

Answered By: Alper t. Turker

Answer 2

This resolution that I’m about to render also takes care of the “key not found: _PYSPARK_DRIVER_CALLBACK_HOST/Java Gateway/PySpark 2.3.1” error!! Add to your bashrc or /etc/environment or /etc/profile

export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

That should do the doobie right there. You may thank me in advance. #thumbsup 🙂

Answered By: SCOTT McNEIL

Answer 3

the env var in .bash_profile or /etc/profile may not be accessed by your code ,put them in your code directly.

import os
import sys

os.environ['SPARK_HOME'] = "/opt/cloudera/parcels/SPARK2/lib/spark2"
os.environ['PYSPARK_SUBMIT_ARGS'] = "--master yarn pyspark-shell" 




sys.path.append(os.path.join(os.environ['SPARK_HOME'], "python"))
sys.path.append(os.path.join(os.environ['SPARK_HOME'], "python/lib/py4j-0.10.6-src.zip"))


try:
    from pyspark import SparkContext
    from pyspark.sql import SparkSession

    from pyspark import SparkConf

    print("success")

except ImportError as e:
    print("error importing spark modules", e)
    sys.exit(1)

Answered By: xfly

Answer 4

I have got the similar errors:
java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST and
Exception: Java gateway process exited before sending its port number

Running the command “export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH” or setting this to .bashrc resolved the issue.

Please also check if the mapr credentails are setup.

Answered By: jakkula saketh

Answer 5

I had same issue and all above settings did not work for me. Actually i had SPARK_HOME already set. Finally the issue was because i simply installed pyspark using pip install pyspark without verifying the version.
After a lot of debugging inside the code , figured out that the _PYSPARK_DRIVER_CALLBACK_HOST inside

anaconda3/lib/python3.7/site-packages/pyspark/java_gateway.py

did not have this variable whereas older versions of pyspark has it (I am using anaconda hence this file path location. The exact location of the file might be different for others)

Finally came to the conclusion that it was due to version mismatch. Seems very stupid enough , but i guess it might help others a lot of debugging time.

solution is to find out the spark version that is installed for eg 2.3.0 and then ensure to install pyspark of same version pip install pyspark==2.3.0 . After this it worked like a charm.

Note : this issue occurs only if i call SparkSession.builder.appName within python . It was working fine even with the version mismatch for pyspark and spark-submit commands , and that’s why it easily skipped my mind that it can be due to version mismatch.

Answered By: niths4u

Answer 6

pip install pyspark==2.3.0

is ok

Answered By: user9225422

Answer 7

I got the same error key not found: _PYSPARK_DRIVER_CALLBACK_HOST while upgrading to Spark 3.1.1.

What worked for me was upgrading pyspark via pip install pyspark==3.1.1, installing findspark, and then running the following lines before starting the SparkSession:

import findspark
findspark.init()

If you have multiple version of Spark on system use path

  import findspark
  findspark.init("C:spark-2.3.0")

Answered By: Camila A. González Williamson

key not found: _PYSPARK_DRIVER_CALLBACK_HOST

Question:

Answers: