Spark Command: /opt/java/openjdk/bin/java -cp /opt/spark/conf:/opt/spark/jars/slf4j-api-2.0.17.jar:/opt/spark/jars/* -Dspark.master.ui.title=Prod Spark Cluster -Dspark.kubernetes.namespace=spark -Dspark.master.rest.enabled=true -Dspark.master.rest.host=0.0.0.0 -Dspark.ui.reverseProxy=true -Dspark.kubernetes.container.image=apache/spark:4.1.1-scala -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dio.netty.allocator.type=pooled -Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE --enable-native-access=ALL-UNNAMED org.apache.spark.deploy.master.Master --host prod-master-0 --port 7077 --webui-port 8080
========================================
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/03/12 12:20:20 INFO Master: Started daemon with process name: 20@prod-master-0
26/03/12 12:20:21 INFO SignalUtils: Registering signal handler for TERM
26/03/12 12:20:21 INFO SignalUtils: Registering signal handler for HUP
26/03/12 12:20:21 INFO SignalUtils: Registering signal handler for INT
26/03/12 12:20:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/03/12 12:20:33 INFO SecurityManager: Changing view acls to: spark
26/03/12 12:20:33 INFO SecurityManager: Changing modify acls to: spark
26/03/12 12:20:33 INFO SecurityManager: Changing view acls groups to: spark
26/03/12 12:20:33 INFO SecurityManager: Changing modify acls groups to: spark
26/03/12 12:20:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY; RPC SSL disabled
26/03/12 12:20:37 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
26/03/12 12:20:37 INFO Master: Starting Spark master at spark://prod-master-0:7077
26/03/12 12:20:37 INFO Master: Running Spark version 4.1.1
26/03/12 12:20:39 INFO JettyUtils: Start Jetty 0.0.0.0:8080 for MasterUI
26/03/12 12:20:42 INFO Utils: Successfully started service 'MasterUI' on port 8080.
26/03/12 12:20:42 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://prod-master-0:8080
26/03/12 12:20:47 INFO Master: Spark Master is acting as a reverse proxy. Master, Workers and Applications UIs are available at http://prod-master-0:8080
26/03/12 12:20:47 INFO Utils: Successfully started service on port 6066.
26/03/12 12:20:47 INFO StandaloneRestServer: Started REST server for submitting applications on 0.0.0.0 with port 6066
26/03/12 12:20:47 INFO Master: Registering worker 192.168.139.145:41301 with 4 cores, 6.8 GiB RAM
26/03/12 12:20:47 INFO Master: Registering worker 192.168.139.191:42665 with 4 cores, 6.8 GiB RAM
26/03/12 12:20:47 INFO Master: Registering worker 192.168.139.141:45593 with 4 cores, 6.8 GiB RAM
26/03/12 12:20:47 INFO Master: I have been elected leader! New state: ALIVE
26/03/12 12:20:51 INFO Master: Registering worker 192.168.139.141:45593 with 4 cores, 6.8 GiB RAM
26/03/12 12:20:51 INFO Master: Registering worker 192.168.139.191:42665 with 4 cores, 6.8 GiB RAM
26/03/12 12:20:52 INFO Master: Registering worker 192.168.139.145:41301 with 4 cores, 6.8 GiB RAM
26/03/12 12:25:15 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
26/03/12 12:25:15 INFO Master: Launching driver driver-20260312122515-0000 on worker worker-20260312122037-192.168.139.145-41301
26/03/12 12:25:22 INFO Master: Registering app Spark Pi
26/03/12 12:25:22 INFO Master: Registered app Spark Pi with ID app-20260312122522-0000
26/03/12 12:25:23 INFO Master: Start scheduling for app app-20260312122522-0000 with rpId: 0
26/03/12 12:25:23 INFO Master: Launching executor app-20260312122522-0000/0 on worker worker-20260312122037-192.168.139.191-42665
26/03/12 12:25:23 INFO Master: Launching executor app-20260312122522-0000/1 on worker worker-20260312122037-192.168.139.141-45593
26/03/12 12:25:23 INFO Master: Start scheduling for app app-20260312122522-0000 with rpId: 0
26/03/12 12:25:23 INFO Master: Start scheduling for app app-20260312122522-0000 with rpId: 0
26/03/12 12:25:50 INFO Master: Received unregister request from application app-20260312122522-0000
26/03/12 12:25:50 INFO Master: Removing app app-20260312122522-0000
26/03/12 12:25:50 INFO Master: 192.168.139.145:36628 got disassociated, removing it.
26/03/12 12:25:50 INFO Master: prod-worker-2.prod-worker-svc.spark.svc.cluster.local:42235 got disassociated, removing it.
26/03/12 12:25:50 WARN Master: Got status update for unknown executor app-20260312122522-0000/1
26/03/12 12:25:50 WARN Master: Got status update for unknown executor app-20260312122522-0000/0
26/03/12 12:25:50 INFO Master: Removing driver: driver-20260312122515-0000 (FINISHED, Elapsed time: 34834 ms)