from pyspark import SparkContext
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('ose')
.master("local[*]")
.config("spark.executor.extraJavaOptions", "??")
.config('spark.driver.extraJavaOptions',"??")
.config('spark.executor.memory', '8g')
.config('spark.driver.memory', '8g')
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.getOrCreate()


I am not sure what values I should pass to the ?? arguments in the jupyter notebook to increase spark stack size to prevent the java.lang.StackOverflowError ?

??

java.lang.StackOverflowError

Does the spark.executor.memory clash with spark.executor.extraJavaOptions ?

spark.executor.memory

spark.executor.extraJavaOptions

In the documentation for spark.driver.extraJavaOptions

Note: In client mode, this config must not be set through the
SparkConf directly in your application, because the driver JVM has
already started at that point. Instead, please set this through the
--driver-java-options command line option or in your default properties file.

and for spark.executor.extraJavaOptions

spark.executor.extraJavaOptions

A string of extra JVM options to pass to executors. For instance, GC
settings or other logging. Note that it is illegal to set Spark
properties or maximum heap size (-Xmx) settings with this option.
Spark properties should be set using a SparkConf object or the
spark-defaults.conf file used with the spark-submit script. Maximum
heap size settings can be set with spark.executor.memory.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.