Copy PySpark Dataframe to MySQL
Question:
I am trying to load a dataframe created with PySpark in DataBricks to MySql but it tells me:
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
df_videojuegos.select("Nombre", "Plataforma", "Año", "Genero", "Editorial", "Ventas NA", "Ventas EU", "Ventas JP", "Ventas Otros", "Ventas Global")
.write
.format("jdbc")
.option("url", "jdbc:mysql://127.0.0.1:3306/dezyre_db&useUnicode=true&characterEncoding=UTF-8&useSSL=false")
.option("driver", "com.mysql.jdbc.Driver").option("dbtable", "reportes")
.option("user", "root").option("password", "root")
.save()
I already have the connector installed in the cluster for MySql and these imported libraries:
from pyspark.sql import SparkSession
from pyspark.sql import Row
spark = SparkSession.builder
.config("spark.jars", "/home/hduser/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar")
.master("local")
.appName("PySpark_MySQL_test2")
.getOrCreate()
I don’t know if the url that I have declared is really correct, or if I have the syntax wrong.
Install the connectors, create the connection with MySQL but when loading the data it gives me an error
Answers:
Ok, the error was a newbie, I was trying to connect from DataBricks to a localhost! :C
I am trying to load a dataframe created with PySpark in DataBricks to MySql but it tells me:
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
df_videojuegos.select("Nombre", "Plataforma", "Año", "Genero", "Editorial", "Ventas NA", "Ventas EU", "Ventas JP", "Ventas Otros", "Ventas Global")
.write
.format("jdbc")
.option("url", "jdbc:mysql://127.0.0.1:3306/dezyre_db&useUnicode=true&characterEncoding=UTF-8&useSSL=false")
.option("driver", "com.mysql.jdbc.Driver").option("dbtable", "reportes")
.option("user", "root").option("password", "root")
.save()
I already have the connector installed in the cluster for MySql and these imported libraries:
from pyspark.sql import SparkSession
from pyspark.sql import Row
spark = SparkSession.builder
.config("spark.jars", "/home/hduser/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar")
.master("local")
.appName("PySpark_MySQL_test2")
.getOrCreate()
I don’t know if the url that I have declared is really correct, or if I have the syntax wrong.
Install the connectors, create the connection with MySQL but when loading the data it gives me an error
Ok, the error was a newbie, I was trying to connect from DataBricks to a localhost! :C