忽略独立群集中通过REST作业提交进行的spark.master配置忽略

我在HA模式下有一个独立的spark群集(2个主控),并且有几个工作人员在那里注册。忽略独立群集中通过REST作业提交进行的spark.master配置忽略

我通过REST接口提交的火花与工作细节之后,

{ 

"sparkProperties": {

"spark.app.name": "TeraGen3",

"spark.default.parallelism": "40",

"spark.executor.memory": "512m",

"spark.driver.memory": "512m",

"spark.task.maxFailures": "3",

"spark.jars": "file:///tmp//test//spark-terasort-1.1-SNAPSHOT-jar-with-dependencies.jar",

"spark.eventLog.enabled": "false",

"spark.submit.deployMode": "cluster",

"spark.driver.supervise": "true",

"spark.master": "spark://spark-hn0:7077,spark-hn1:7077"

},

"mainClass": "com.github.ehiggs.spark.terasort.TeraGen",

"environmentVariables": {

"SPARK_ENV_LOADED": "1"

},

"action": "CreateSubmissionRequest",

"appArgs": ["4g", "file:///tmp/data/teradata4g/"],

"appResource": "file:///tmp//test//spark-terasort-1.1-SNAPSHOT-jar-with-dependencies.jar",

"clientSparkVersion": "2.1.1"

}

这个请求是通过REST接口(http://spark-hn1:6066/v1/submissions/create)提交到活动星火法师。

当驱动程序启动时,将-Dspark.master设置为“spark:// spark-hn1:7077”而不是在sparkProperties中传递的值,即spark:// spark-hn0:7077,spark -hn1:7077" 。从工作节点

日志,其中驱动程序正在运行

17/12/18 13:29:49 INFO worker.DriverRunner: Launch Command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-Dhdp.version=2.6.99.200-0" "-cp" "/usr/hdp/current/spark2-client/conf/:/usr/hdp/current/spark2-client/jars/*:/etc/hadoop/conf/" "-Xmx512M" "-Dspark.driver.memory=51 

2m" "-Dspark.master=spark://spark-hn1:7077" "-Dspark.executor.memory=512m" "-Dspark.submit.deployMode=cluster" "-Dspark.app.name=TeraGen3" "-Dspark.default.parallelism=40" "-Dspark.jars=file:///tmp//test//spark-terasort-1.1-SNAPSHOT-jar-with-dependencies.jar" "-Dspark.ta

sk.maxFailures=3" "-Dspark.driver.supervise=true" "-Dspark.eventLog.enabled=false" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:40803" "/var/spark/work/driver-20171218132949-0001/spark-terasort-1.1-SNAPSHOT-jar-with-dependencies.jar" "com.git

hub.ehiggs.spark.terasort.TeraGen" "4g" "file:///tmp/data/teradata4g/"

这是造成问题,我当活动的主任务执行和其他主成为活跃在下降。由于司机只知道一个主人(旧人),因此无法达到新主人并继续执行工作(因为spark.driver.supervise = true)

传递多个主要网址的正确方法是什么Spark REST接口。

回答:

看起来像这是RestServer实现中的一个错误,其中spark.master被替换。 https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala#L147

我们仍然可以同时通过REST接口提交作业如下

"sparkProperties": { 

"spark.app.name": "TeraGen3",

...

"spark.driver.extraJavaOptions": "-Dspark.master=spark://spark-hn0:7077,spark-hn1:7077"

}

这为我工作在spark.driver.extraJavaOptions设置spark.master解决此。

以上是 忽略独立群集中通过REST作业提交进行的spark.master配置忽略 的全部内容, 来源链接: utcz.com/qa/263829.html

回到顶部