您好,登錄后才能下訂單哦!
這篇文章主要介紹“spark sql怎么連接和使用mysql數(shù)據(jù)源”,在日常操作中,相信很多人在spark sql怎么連接和使用mysql數(shù)據(jù)源問題上存在疑惑,小編查閱了各式資料,整理出簡單好用的操作方法,希望對大家解答”spark sql怎么連接和使用mysql數(shù)據(jù)源”的疑惑有所幫助!接下來,請跟著小編一起來學(xué)習(xí)吧!
spark sql 可以通過標(biāo)準(zhǔn)的jdbc連接數(shù)據(jù)庫,獲得數(shù)據(jù)源
public class SparkSql {public static SimpleDateFormat sdf = new SimpleDateFormat("_yyyyMMdd_HH_mm_ss"); private static final String appName = "spark sql test"; private static final String master = "spark://192.168.1.21:7077"; private static final String JDBCURL = "jdbc:mysql://192.168.1.18:3306/lng?user=root&password=123456"; public static void main(String[] avgs){ SparkContext context = new SparkContext(master, appName); SQLContext sqlContext = new SQLContext(context); // Creates a DataFrame based on a table named "people" // stored in a MySQL database. DataFrame df = sqlContext .read() .format("jdbc") .option("url", JDBCURL) .option("dbtable", "tsys_user") .load(); // Looks the schema of this DataFrame. df.printSchema(); // Counts people by age DataFrame countsByAge = df.groupBy("customStyle").count(); countsByAge.show(); // Saves countsByAge to S3 in the JSON format. countsByAge.write().format("json").save("hdfs://192.168.1.17:9000/administrator/sql-result" + sdf.format(new Date())); } }
如果沒有包含mysql的驅(qū)動程序,需要參考http://stackoverflow.com/questions/34764505/no-suitable-driver-found-for-jdbc-in-spark
You might want to assembly you application with your build manager (Maven,SBT) thus you'll not need to add the dependecies in your spark-submit
cli. (意思就是把mysql的驅(qū)動程序打包到提交到spark的jar包里)
You can use the following option in your spark-submit
cli :(改成下面,經(jīng)測試,可行,或者加入export SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/local/spark-1.6.1-bin-hadoop2.6/conf/driverLib/mysql-connector-java-5.1.36.jar 到conf/spark-env.sh)
spark-submit --driver-class-path /usr/local/spark-1.6.1-bin-hadoop2.6/conf/driverLib/mysql-connector-java-5.1.36.jar --class com.xxx.SparkSql /usr/local/spark.jar
Explanation : Supposing that you have all your jars in a lib
directory in your project root, this will read all the libraries and add them to the application submit.
You can also try to configure these 2 variables : spark.driver.extraClassPath
and spark.executor.extraClassPath
in SPARK_HOME/conf/spark-default.conf
file and specify the value of these variables as the path of the jar file. Ensure that the same path exists on workernodes.(經(jīng)測,不行)
到此,關(guān)于“spark sql怎么連接和使用mysql數(shù)據(jù)源”的學(xué)習(xí)就結(jié)束了,希望能夠解決大家的疑惑。理論與實踐的搭配能更好的幫助大家學(xué)習(xí),快去試試吧!若想繼續(xù)學(xué)習(xí)更多相關(guān)知識,請繼續(xù)關(guān)注億速云網(wǎng)站,小編會繼續(xù)努力為大家?guī)砀鄬嵱玫奈恼拢?/p>
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進行舉報,并提供相關(guān)證據(jù),一經(jīng)查實,將立刻刪除涉嫌侵權(quán)內(nèi)容。