溫馨提示×

您好，登錄后才能下訂單哦！

密碼登錄×
登錄注冊(cè)×
其他方式登錄
點(diǎn)擊 登錄注冊(cè) 即表示同意《億速云用戶服務(wù)條款》

spark怎么讀取hbase表

發(fā)布時(shí)間：2022-01-17 09:38:52 來(lái)源：億速云 閱讀：194 作者：iii 欄目：開發(fā)技術(shù)

這篇文章主要介紹了spark怎么讀取hbase表的相關(guān)知識(shí)，內(nèi)容詳細(xì)易懂，操作簡(jiǎn)單快捷，具有一定借鑒價(jià)值，相信大家閱讀完這篇spark怎么讀取hbase表文章都會(huì)有所收獲，下面我們一起來(lái)看看吧。

一.場(chǎng)景:
    spark通過phoenix 讀取hbase表，其實(shí)說(shuō)白了先要去Zookeeper建立connection。

二.代碼:

  1. val zkUrl = "192.168.100.39,192.168.100.40,192.168.100.41:2181"

  2. val formatStr = "org.apache.phoenix.spark"

  3. val oms_orderinfoDF = spark.read.format(formatStr)

  4.   .options(Map("table" -> "oms_orderinfo", "zkUrl" -> zkUrl))

  5.   .load


三.查看SparkJob日志:

  1. 17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop40/192.168.100.40:2181. Will not attempt to authenticate using SASL (unknown error)

  2. 17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.100.48:35952, server: hadoop40/192.168.100.40:2181

  3. 17/10/24 03:25:25 WARN zookeeper.ClientCnxn: Session 0x0 for server hadoop40/192.168.100.40:2181, unexpected error, closing socket connection and attempting reconnect

  4. java.io.IOException: Connection reset by peer

  5.     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

  6.     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

  7.     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)

  8.     at sun.nio.ch.IOUtil.read(IOUtil.java:192)

  9.     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)

  10.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)

  11.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)

  12.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

  13. 17/10/24 03:25:25 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://nameservice1/user/hdfs/.sparkStaging/application_1507703377455_4854

  14. 17/10/24 03:25:25 INFO util.ShutdownHookManager: Shutdown hook called


四.查看Zookeeper日志:

  1. 2017-10-24 03:25:22,498 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

  2. 2017-10-24 03:25:25,588 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

  3. 2017-10-24 03:25:25,819 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

  4. 2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception

  5. EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee09897d, likely client has closed socket

  6.     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)

  7.     at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

  8.     at java.lang.Thread.run(Thread.java:745)

  9. 2017-10-24 03:25:26,092 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47084 which had sessionid 0x15ed091ee09897d

  10. 2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception

  11. EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee098981, likely client has closed socket

  12.     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)

  13.     at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

  14.     at java.lang.Thread.run(Thread.java:745)

  15. 2017-10-24 03:25:26,093 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47304 which had sessionid 0x15ed091ee098981

  16. 2017-10-24 03:25:26,093 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception


五.解決方法:
undefinedmaxClientCnxns調(diào)大到1000，重啟zookeeper服務(wù)生效配置即可。
spark怎么讀取hbase表

關(guān)于“spark怎么讀取hbase表”這篇文章的內(nèi)容就介紹到這里，感謝各位的閱讀！相信大家對(duì)“spark怎么讀取hbase表”知識(shí)都有一定的了解，大家如果還想學(xué)習(xí)更多知識(shí)，歡迎關(guān)注億速云行業(yè)資訊頻道。

向AI問一下細(xì)節(jié)

免責(zé)聲明：本站發(fā)布的內(nèi)容（圖片、視頻和文字）以原創(chuàng)、轉(zhuǎn)載和分享為主，文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱：is@yisu.com進(jìn)行舉報(bào)，并提供相關(guān)證據(jù)，一經(jīng)查實(shí)，將立刻刪除涉嫌侵權(quán)內(nèi)容。

AI
助
手