Spark Trouble Shooting - NoClassDefFoundError

Last Updated: 2021-11-19

Error

java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
 at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:45)
 at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:163)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:150)
 at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
 at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:149)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
 at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)
 at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
 at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
 at scala.Option.getOrElse(Option.scala:121)
 at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
 at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101)
 … 47 elided
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
 at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 … 60 more
<console>:14: error: not found: value spark
 import spark.implicits._
 ^
<console>:14: error: not found: value spark
 import spark.sql
^

Cause

Spark 2.0 is upgraded to jersey 2, but somehow jars in jersey 1 may still be required.

Solution

Download jersey-bundle-1.17.1.jar, and copy to $SPARK_HOME/jars, or use --jars when calling spark-shell or spark-submit:

$ spark-shell --jars extra_jars/jersey-bundle-1.17.1.jar

or set spark.hadoop.yarn.timeline-service.enabled to false