Spark Trouble Shooting - InvalidProtocolBufferException

Last Updated: 2021-11-19

ERROR

ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: “<local.hostname>/xxx.xxx.xxx.xxx"; destination host is: “<dest.hostname>":50070;

Solution

dest.hostname is wrong. To find the correct hostname for HDFS:

$ hdfs getconf -confKey fs.default.name
hdfs://foo

Then use that in the configurations, e.g. in $SPARK_CONF_DIR/spark-defaults.conf or $SPARK_HOME/conf/spark-defaults.conf:

spark.eventLog.dir  hdfs://foo/spark-logs