![cover](https://img-blog.csdnimg.cn/cda6ec5e63474cf7923800f0c01629fd.png)
Spark on Kubernetes
一、目标运行 spark 任务到 kubernetes,spark on kubernetes。二、步骤
Spark on Yarn 和 K8S 对比
Yarn 和 K8S 调度 Spark 作业的对比
http://www.fanyilun.me/2022/06/02/YARN%E5%92%8CK8s%E8%B0%83%E5%BA%A6Spark%E4%BD%9C%E4%B8%9A%E7%9A%84%E5%AF%B9%E6%AF%94/
Spark on Kubernetes 作业执行流程
http://www.fanyilun.me/2021/08/22/Spark%20on%20Kubernetes%E4%BD%9C%E4%B8%9A%E6%89%A7%E8%A1%8C%E6%B5%81%E7%A8%8B/
Spark on K8S 示意图
![在这里插入图片描述](https://i-blog.csdnimg.cn/blog_migrate/04d8c355b079c69e1d596f79f59e9143.png#pic_center)
一、目标
运行 spark 任务到 kubernetes,spark on kubernetes。
参考 https://spark.apache.org/docs/3.2.0/running-on-kubernetes.html
二、步骤
0.前置步骤
1. 部署 docker 环境。
2. 部署 k8s 集群。
3. 准备 spark,spark-3.2.0-bin-hadoop2.7.tgz。
1.制作镜像
cd /root/spark-3.2.0-bin-hadoop2.7/bin/
ll docker-image-tool.sh
ll /root/spark-3.2.0-bin-hadoop2.7/kubernetes/dockerfiles/spark/Dockerfile
ll /root/spark-3.2.0-bin-hadoop2.7/kubernetes/dockerfiles/spark/entrypoint.sh
spark 自带脚本,制作镜像,网络超时
./docker-image-tool.sh -r harbor.dev-XXX.com/k8s/sre -t v4.0.0 build
更换原始镜像,重新制作镜像
docker pull harbor.dev-XXX.com/library/centos7-oracle-jdk1.8:1.0
cd /root/spark-3.2.0-bin-hadoop2.7/kubernetes/dockerfiles/spark/
docker build --network host --no-cache -t harbor.dev-XXX.com/k8s/sre/spark:v4.0.0 -f ./Dockerfile .
docker image ls
推送到镜像仓库
docker push harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
2.提交 spark 任务到 k8s
cd /root/spark-3.2.0-bin-hadoop2.7/bin/; ls -lrt
./spark-submit \
--master k8s://https://K8S_API_IP:6443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=harbor.dev-XXX.com/k8s/sre/spark:v4.0.0 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar
kubectl get pods -A
kubectl logs -f spark-pi-f0ae047d619e134a-driver
查看任务状态
./spark-submit --status spark:spark-pi-f0ae047d619e134a-driver --master k8s://https://K8S_API_IP:6443
三、问题及解决
3.1 问题1
## 问题
Exception in thread "main" org.apache.spark.SparkException: Could not load KUBERNETES classes. This copy of Spark may not have been compiled with KUBERNETES support.
https://issues.apache.org/jira/browse/SPARK-33340
## 解决
--conf spark.kubernetes.container.image 指定正确的 spark 镜像。
3.2 问题2
## 问题
尝试不自己 build spark 的镜像源,尝试使用 hub.docker 上的镜像源。
https://hub.docker.com/r/newfrontdocker/spark
https://hub.docker.com/r/bitnami/spark
harbor.dev-XXX.com/k8s/sre/spark:v1.0.0 -- bitnami/spark:latest
harbor.dev-XXX.com/k8s/sre/spark:v2.0.0 -- newfrontdocker/spark:v3.0.1-j14
1. bitnami/spark 运行日志
/opt/bitnami/scripts/spark/entrypoint.sh: line 32: exec: driver: not found
镜像中没有 build 对应的程序。这个镜像不能用。
2. newfrontdocker/spark 运行日志
WARN DependencyUtils: Local jar /opt/spark/examples/jars/*.jar does not exist, skipping
Error: Failed to load class org.apache.spark.examples.SparkPi.
从 https://hub.docker.com/r/bitnami/spark 上的镜像说明中,没找到 examples 下 spark 的版本号,而 spark-submit 提交命令时,需要指定 spark 的版本号。这个镜像不能用。
## 解决
不使用 hub.docker 上的镜像源,自己 build spark 的镜像包。
3.3 问题3
## 问题
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/pods/spark-pi-a275a77d6134ec82-driver. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-a275a77d6134ec82-driver" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default".
## 解决
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
spark-submit 提交任务时,增加 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
四、其他记录
4.1 社区链接
https://github.com/bitnami/bitnami-docker-spark/issues/56
https://issues.apache.org/jira/browse/SPARK-33340
4.2 修改 spark 自带的 Dockerfile,如下截图
4.3 spark on k8s 涉及到的文件
4.4 spark 的其他运行方式
## local
./spark-submit \
--class org.apache.spark.examples.SparkPi \
--master local[4] \
local:///home/biadmin/XXXhadoop_client/spark-3.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.2.0.jar 100
## yarn
./spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores 1 \
--queue lx_etl \
--jars /home/biadmin/XXXhadoop_client/spark-3.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.2.0.jar \
local:///home/biadmin/XXXhadoop_client/spark-3.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.2.0.jar 100
4.5 编译 spark
cd /var/lib/docker/bigdata/workspace/spark
export MAVEN_OPTS="-Xss1g -Xmx12g -XX:ReservedCodeCacheSize=4g"
for i in {1..3}; do date; sleep 1; done
for i in {1..10}; do ./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes; sleep 1; done
for i in {1..10}; do date; ./dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes; done
下载超时,手动上传的 jar
/your/local/.m2/repository/org/scalanlp/breeze_2.12/1.2
/your/local/.m2/repository/org/apache/parquet/parquet-jackson/1.12.2
/your/local/.m2/repository/org/apache/hive/hive-llap-common/2.3.9
4.6 执行 spark-submit 的日志
21/11/28 09:09:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/11/28 09:09:43 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
21/11/28 09:09:44 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
21/11/28 09:09:45 INFO LoggingPodStatusWatcherImpl: State changed, new state:
pod name: spark-pi-cf8ea77d6417381c-driver
namespace: default
labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
creation time: 2021-11-28T01:09:44Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
node name: k8s-node1
start time: 2021-11-28T01:09:44Z
phase: Pending
container status:
container name: spark-kubernetes-driver
container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
container state: waiting
pending reason: ContainerCreating
21/11/28 09:09:45 INFO LoggingPodStatusWatcherImpl: State changed, new state:
pod name: spark-pi-cf8ea77d6417381c-driver
namespace: default
labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
creation time: 2021-11-28T01:09:44Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
node name: k8s-node1
start time: 2021-11-28T01:09:44Z
phase: Pending
container status:
container name: spark-kubernetes-driver
container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
container state: waiting
pending reason: ContainerCreating
21/11/28 09:09:45 INFO LoggingPodStatusWatcherImpl: Waiting for application spark-pi with submission ID default:spark-pi-cf8ea77d6417381c-driver to finish...
21/11/28 09:09:46 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Pending)
21/11/28 09:09:46 INFO LoggingPodStatusWatcherImpl: State changed, new state:
pod name: spark-pi-cf8ea77d6417381c-driver
namespace: default
labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
creation time: 2021-11-28T01:09:44Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
node name: k8s-node1
start time: 2021-11-28T01:09:44Z
phase: Pending
container status:
container name: spark-kubernetes-driver
container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
container state: waiting
pending reason: ContainerCreating
21/11/28 09:09:47 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Pending)
21/11/28 09:09:47 INFO LoggingPodStatusWatcherImpl: State changed, new state:
pod name: spark-pi-cf8ea77d6417381c-driver
namespace: default
labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
creation time: 2021-11-28T01:09:44Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
node name: k8s-node1
start time: 2021-11-28T01:09:44Z
phase: Running
container status:
container name: spark-kubernetes-driver
container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
container state: running
container started at: 2021-11-28T01:09:46Z
21/11/28 09:09:48 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:49 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:50 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:51 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:52 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:53 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:54 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:55 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:56 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:58 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:59 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: State changed, new state:
pod name: spark-pi-cf8ea77d6417381c-driver
namespace: default
labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
creation time: 2021-11-28T01:09:44Z
service account name: spark
volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
node name: k8s-node1
start time: 2021-11-28T01:09:44Z
phase: Succeeded
container status:
container name: spark-kubernetes-driver
container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
container state: terminated
container started at: 2021-11-28T01:09:46Z
container finished at: 2021-11-28T01:09:59Z
exit code: 0
termination reason: Completed
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Succeeded)
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Container final statuses:
container name: spark-kubernetes-driver
container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
container state: terminated
container started at: 2021-11-28T01:09:46Z
container finished at: 2021-11-28T01:09:59Z
exit code: 0
termination reason: Completed
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Application spark-pi with submission ID default:spark-pi-cf8ea77d6417381c-driver finished
21/11/28 09:10:00 INFO ShutdownHookManager: Shutdown hook called
21/11/28 09:10:00 INFO ShutdownHookManager: Deleting directory /tmp/spark-fc84bdad-8fca-4773-8a9d-c81d7e260eea
4.7 执行 kubectl logs -f spark-pi-cf8ea77d6417381c-driver 的日志
++ id -u
+ myuid=185
++ id -g
+ mygid=0
+ set +e
++ getent passwd 185
+ uidentry=
+ set -e
+ '[' -z '' ']'
+ '[' -w /etc/passwd ']'
+ echo '185:x:185:0:anonymous uid:/opt/spark:/bin/false'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z x ']'
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'
+ case "$1" in
+ shift 1
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.36.108 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar
21/11/28 09:09:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/11/28 09:09:48 INFO SparkContext: Running Spark version 3.2.0
21/11/28 09:09:48 INFO ResourceUtils: ==============================================================
21/11/28 09:09:48 INFO ResourceUtils: No custom resources configured for spark.driver.
21/11/28 09:09:48 INFO ResourceUtils: ==============================================================
21/11/28 09:09:48 INFO SparkContext: Submitted application: Spark Pi
21/11/28 09:09:49 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
21/11/28 09:09:49 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
21/11/28 09:09:49 INFO ResourceProfileManager: Added ResourceProfile id: 0
21/11/28 09:09:49 INFO SecurityManager: Changing view acls to: 185,root
21/11/28 09:09:49 INFO SecurityManager: Changing modify acls to: 185,root
21/11/28 09:09:49 INFO SecurityManager: Changing view acls groups to:
21/11/28 09:09:49 INFO SecurityManager: Changing modify acls groups to:
21/11/28 09:09:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(185, root); groups with view permissions: Set(); users with modify permissions: Set(185, root); groups with modify permissions: Set()
21/11/28 09:09:49 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
21/11/28 09:09:49 INFO SparkEnv: Registering MapOutputTracker
21/11/28 09:09:49 INFO SparkEnv: Registering BlockManagerMaster
21/11/28 09:09:49 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/11/28 09:09:49 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/11/28 09:09:49 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
21/11/28 09:09:49 INFO DiskBlockManager: Created local directory at /var/data/spark-6ecc5286-05a5-478b-ad0c-143361509542/blockmgr-77aa2d40-7383-486c-9345-0c67d595f390
21/11/28 09:09:49 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB
21/11/28 09:09:49 INFO SparkEnv: Registering OutputCommitCoordinator
21/11/28 09:09:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/11/28 09:09:50 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://spark-pi-cf8ea77d6417381c-driver-svc.default.svc:4040
21/11/28 09:09:50 INFO SparkContext: Added JAR local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar at file:/opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar with timestamp 1638061788937
21/11/28 09:09:50 INFO SparkContext: The JAR local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar at file:/opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar has been added already. Overwriting of added jar is not supported in the current version.
21/11/28 09:09:50 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
21/11/28 09:09:51 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5, known: 0, sharedSlotFromPendingPods: 2147483647.
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
21/11/28 09:09:51 INFO NettyBlockTransferService: Server created on spark-pi-cf8ea77d6417381c-driver-svc.default.svc:7079
21/11/28 09:09:51 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/11/28 09:09:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BlockManagerMasterEndpoint: Registering block manager spark-pi-cf8ea77d6417381c-driver-svc.default.svc:7079 with 366.3 MiB RAM, BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.124:37132) with ID 3, ResourceProfileId 0
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.122:58850) with ID 1, ResourceProfileId 0
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.118:44710) with ID 4, ResourceProfileId 0
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.124:45414 with 408.9 MiB RAM, BlockManagerId(3, 10.244.36.124, 45414, None)
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.122:44299 with 408.9 MiB RAM, BlockManagerId(1, 10.244.36.122, 44299, None)
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.118:41553 with 408.9 MiB RAM, BlockManagerId(4, 10.244.36.118, 41553, None)
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.117:35256) with ID 2, ResourceProfileId 0
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.117:45961 with 408.9 MiB RAM, BlockManagerId(2, 10.244.36.117, 45961, None)
21/11/28 09:09:57 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.125:52560) with ID 5, ResourceProfileId 0
21/11/28 09:09:57 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.125:33322 with 408.9 MiB RAM, BlockManagerId(5, 10.244.36.125, 33322, None)
21/11/28 09:09:57 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
21/11/28 09:09:57 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
21/11/28 09:09:57 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
21/11/28 09:09:57 INFO DAGScheduler: Parents of final stage: List()
21/11/28 09:09:57 INFO DAGScheduler: Missing parents: List()
21/11/28 09:09:57 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
21/11/28 09:09:57 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.0 KiB, free 366.3 MiB)
21/11/28 09:09:57 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.3 KiB, free 366.3 MiB)
21/11/28 09:09:57 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on spark-pi-cf8ea77d6417381c-driver-svc.default.svc:7079 (size: 2.3 KiB, free: 366.3 MiB)
21/11/28 09:09:57 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1427
21/11/28 09:09:57 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
21/11/28 09:09:57 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0
21/11/28 09:09:57 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.244.36.124, executor 3, partition 0, PROCESS_LOCAL, 4597 bytes) taskResourceAssignments Map()
21/11/28 09:09:57 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (10.244.36.125, executor 5, partition 1, PROCESS_LOCAL, 4597 bytes) taskResourceAssignments Map()
21/11/28 09:09:58 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.244.36.125:33322 (size: 2.3 KiB, free: 408.9 MiB)
21/11/28 09:09:58 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.244.36.124:45414 (size: 2.3 KiB, free: 408.9 MiB)
21/11/28 09:09:58 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 966 ms on 10.244.36.124 (executor 3) (1/2)
21/11/28 09:09:58 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 945 ms on 10.244.36.125 (executor 5) (2/2)
21/11/28 09:09:58 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
21/11/28 09:09:58 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 1.188 s
21/11/28 09:09:58 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
21/11/28 09:09:58 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
21/11/28 09:09:58 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.243353 s
Pi is roughly 3.1410157050785252
21/11/28 09:09:58 INFO SparkUI: Stopped Spark web UI at http://spark-pi-cf8ea77d6417381c-driver-svc.default.svc:4040
21/11/28 09:09:58 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
21/11/28 09:09:58 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
21/11/28 09:09:58 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
21/11/28 09:09:59 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/11/28 09:09:59 INFO MemoryStore: MemoryStore cleared
21/11/28 09:09:59 INFO BlockManager: BlockManager stopped
21/11/28 09:09:59 INFO BlockManagerMaster: BlockManagerMaster stopped
21/11/28 09:09:59 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/11/28 09:09:59 INFO SparkContext: Successfully stopped SparkContext
21/11/28 09:09:59 INFO ShutdownHookManager: Shutdown hook called
21/11/28 09:09:59 INFO ShutdownHookManager: Deleting directory /tmp/spark-f2b506ed-ab8f-41b8-bd36-eb90576f93ce
21/11/28 09:09:59 INFO ShutdownHookManager: Deleting directory /var/data/spark-6ecc5286-05a5-478b-ad0c-143361509542/spark-a5cd7f8a-883c-4943-843e-0a18d912e15a
----------
可以看到上面日志中,有一行成功运行的结果 "Pi is roughly 3.1410157050785252"。
更多推荐
所有评论(0)