pyspark datastax cassandra connector keeps connecting to localhost
Question:
I am trying to connect pyspark to cassandra using datastax driver
conf = SparkConf()
.setAppName('Test')
.setMaster('local[4]')
.set("spark.cassandra.connection.host", "192.168.0.150")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
df = sqlContext.read.format("org.apache.spark.sql.cassandra").
options(table="test", keyspace="test_keyspace").load()
For some reason it keeps connecting to 127.0.0.1:9042 instead of 192.168.0.150
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query
failed (tried: localhost/127.0.0.1:9042
(com.datastax.driver.core.exceptions.TransportException:
[localhost/127.0.0.1] Cannot connect))
I am using spark 2.10 and running program as below
spark-submit --packages datastax:spark-cassan
dra-connector:2.0.0-RC1-s_2.11 test.py
Answers:
Figure out the issue in Spark 2.10 cassandra configuration is set as option in SqlContext.
code below works
sqlContext.read.format("org.apache.spark.sql.cassandra").
option("spark.cassandra.connection.host", "192.168.0.150").
option("spark.cassandra.auth.username", "user").
option("spark.cassandra.auth.password", "password").
options(table="test_table", keyspace="test_space").load()
I am trying to connect pyspark to cassandra using datastax driver
conf = SparkConf()
.setAppName('Test')
.setMaster('local[4]')
.set("spark.cassandra.connection.host", "192.168.0.150")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
df = sqlContext.read.format("org.apache.spark.sql.cassandra").
options(table="test", keyspace="test_keyspace").load()
For some reason it keeps connecting to 127.0.0.1:9042 instead of 192.168.0.150
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query
failed (tried: localhost/127.0.0.1:9042
(com.datastax.driver.core.exceptions.TransportException:
[localhost/127.0.0.1] Cannot connect))
I am using spark 2.10 and running program as below
spark-submit --packages datastax:spark-cassan
dra-connector:2.0.0-RC1-s_2.11 test.py
Figure out the issue in Spark 2.10 cassandra configuration is set as option in SqlContext.
code below works
sqlContext.read.format("org.apache.spark.sql.cassandra").
option("spark.cassandra.connection.host", "192.168.0.150").
option("spark.cassandra.auth.username", "user").
option("spark.cassandra.auth.password", "password").
options(table="test_table", keyspace="test_space").load()