溫馨提示×

溫馨提示×

您好,登錄后才能下訂單哦!

密碼登錄×
登錄注冊×
其他方式登錄
點擊 登錄注冊 即表示同意《億速云用戶服務條款》

大數據之Apache Beam 使用Flink Runner管道參數

發(fā)布時間:2020-07-29 13:51:27 來源:網絡 閱讀:1425 作者:小涴熊 欄目:大數據

例子

Flink集群

mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--runner=FlinkRunner --flinkMaster=<flink master> --filesToStage=target/word-count-beam-bundled-0.1.jar --inputFile=/path/to/quickstart/pom.xml --output=/tmp/counts" -Pflink-runner

管道參數說明


FieldDescriptionDefault Value
runnerThe pipeline runner to use. This option allows you to determine the pipeline runner at runtime.Set to FlinkRunnerto run using Flink.
streamingWhether streaming mode is enabled or disabled; true if enabled. Set to true if running pipelines with unbounded PCollections.false
flinkMasterThe url of the Flink JobManager on which to execute pipelines. This can either be the address of a cluster JobManager, in the form "host:port" or one of the special Strings "[local]" or "[auto]""[local]" will start a local Flink Cluster in the JVM while "[auto]" will let the system decide where to execute the pipeline based on the environment.[auto]
filesToStageJar Files to send to all workers and put on the classpath. Here you have to put the fat jar that contains your program along with all dependencies.empty
parallelismThe degree of parallelism to be used when distributing operations onto workers.1
checkpointingIntervalThe interval between consecutive checkpoints (i.e. snapshots of the current pipeline state used for fault tolerance).-1L, i.e. disabled
numberOfExecutionRetriesSets the number of times that failed tasks are re-executed. A value of 0 effectively disables fault tolerance. A value of -1 indicates that the system default value (as defined in the configuration) should be used.-1
executionRetryDelaySets the delay between executions. A value of -1 indicates that the default value should be used.-1
stateBackendSets the state backend to use in streaming mode. The default is to read this setting from the Flink config.empty, i.e. read from Flink config



向AI問一下細節(jié)

免責聲明:本站發(fā)布的內容(圖片、視頻和文字)以原創(chuàng)、轉載和分享為主,文章觀點不代表本網站立場,如果涉及侵權請聯系站長郵箱:is@yisu.com進行舉報,并提供相關證據,一經查實,將立刻刪除涉嫌侵權內容。

AI