logo
CHEATSHEETSTOOLSABOUT中文

Spark on Kubernetes

Running Apache Spark on Kubernetes (K8s) allows you to use Kubernetes as a native cluster manager—replacing traditional managers like Standalone, YARN, or Mesos.

Spark achieved full, production-ready support for Kubernetes in version 3.1. Instead of needing a dedicated Spark cluster, Spark workloads can share CPU, memory, and GPU resources directly alongside your microservices and databases on a standard Kubernetes cluster.

Core Architecture & Flow of a Spark Job on K8s

When you run a Spark application on Kubernetes, the Spark Driver runs inside a K8s Pod and talks directly to the Kubernetes API Server to request and manage Executor Pods.

  1. Submission: You submit an application using spark-submit (configured with --master k8s://https://<k8s-api-server>) or via a Kubernetes Custom Resource (CRD) using the Spark Operator.
  2. Driver Pod Creation: The Kubernetes API Server creates the Spark Driver Pod.
  3. Executor Request: Once initialized, the Driver Pod uses its internal KubernetesClusterSchedulerBackend to authenticate with the K8s API Server (using a assigned ServiceAccount) and requests executor pods based on your configuration (spark.executor.instances).
  4. Executor Scheduling: Kubernetes schedules and launches the Executor Pods.
  5. Execution & Communication: The Executor Pods connect directly back to the Driver Pod over the internal Pod network to receive tasks, stream heartbeat metrics, and exchange shuffle/broadcast data.
  6. Teardown: When the Spark job finishes, the Executor Pods are terminated and cleaned up automatically. The Driver Pod remains in a Completed state (keeping logs accessible) until explicitly deleted or cleaned up by a Garbage Collection policy.

Key Components Under the Hood

1. Dynamic Allocation (Scaling Up & Down)

Spark on K8s supports Dynamic Resource Allocation. When enabled (spark.dynamicAllocation.enabled=true), the Driver monitors task queue backlogs:

  • If tasks accumulate, it requests additional Executor Pods from the K8s API.
  • If executors sit idle, the Driver requests the K8s API to terminate them, returning resources back to the cluster.

2. Local Storage & Shuffle Data Handling

Shuffle data (intermediate map outputs) needs fast disk access. On K8s, this is handled via:

  • emptyDir volumes: Uses node-local ephemeral storage or RAM-backed tmpfs.
  • PersistentVolumeClaims (PVCs): Mounts network storage (EBS, Persistent Disk, NFS) to executors.
  • External Shuffle Service / Push-Based Shuffle: Modern Spark on K8s uses Push-Based Shuffle (e.g., Remote Shuffle Service or Apache Uniffle) to prevent shuffle data loss when Executor Pods are preempted or dynamically scaled down.

3. Pod Templates & Customization

You can customize the driver and executor pods (adding sidecars, custom environment variables, annotations, or security contexts) using Pod Templates:

# pod-template.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: spark-workload
spec:
  containers:
    - name: spark-kubernetes-executor
      resources:
        limits:
          nvidia.com/gpu: '1'

You pass these to Spark via spark.kubernetes.executor.podTemplateFile.

Two Ways to Deploy Spark on K8s

Deployment Method How It Works Best Used For
spark-submit CLI Direct CLI call pointing to k8s://<api-server-url>. Ad-hoc jobs, local testing, or traditional CI/CD pipelines.
Spark Operator Uses a K8s Custom Resource (SparkApplication / ScheduledSparkApplication) managed by an active in-cluster controller. Production declarative deployments (GitOps, ArgoCD), scheduled cron jobs, and native K8s monitoring.

Example: SparkApplication Custom Resource (Spark Operator)

apiVersion: 'sparkoperator.k8s.io/v1beta2'
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: spark-jobs
spec:
  type: Scala
  mode: cluster
  image: 'apache/spark:v3.5.0'
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: 'local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar'
  sparkVersion: '3.5.0'
  driver:
    cores: 1
    memory: '512m'
    serviceAccount: spark-sa
  executor:
    cores: 1
    instances: 2
    memory: '512m'

Major Advantages vs. Challenges

Advantages

  • Unified Infrastructure: Consolidates web apps, microservices, and big data ML pipelines on the same K8s clusters (EKS, GKE, AKS, or on-prem).
  • Isolation & Multi-Tenancy: Each Spark application can use its own container image, Python version, and dependency isolation without library conflicts.
  • Cost Efficiency with Spot Instances: Uses K8s node affinity, tolerations, and preemption handling to run executor pods on cheap Spot/Preemptible instances.

Challenges

  • Node Startup Overhead: Spinning up new K8s pods adds a few seconds of cold-start latency compared to long-running YARN or Standalone daemons.
  • Shuffle Data Loss on Preemption: If a Spot instance terminates an executor node, its local shuffle data is lost unless using remote/push-based shuffle services.
Loading...