I just came across this error ‘sparkDriver’ failed after 16 retries!’ while starting the spark shell.
From the above screenshot, I came to know the error occurred due to the SparkUI port was not available. For this scenario I used Cloudera sandbox using Oracle VirtualBox.
I just solved this issue by the following steps
- Go to /usr/lib/spark/bin (path for cloudera sandbox)
- Open load-spark-env.sh and add this export SPARK_LOCAL_IP=’127.0.0.1′ (you need to be root user)
- Save the file and type ‘spark-shell’ to get sparkcontext initialized
Thanks for reading this post. Subscribe my blog for some awesome posts.