2021년/ERROR
21/04/06 15:20:45 ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks
위지원
2021. 4. 6. 15:22
pyspark 실행할 대 --master 설정을 해줘야한다... 하...
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.1.1
/_/
Using Python version 3.8.1 (v3.8.1:1b293b6006, Dec 18 2019 14:08:53)
Spark context Web UI available at http://jiwon.local:4042
Spark context available as 'sc' (master = spark://jiwon.local:7077, app id = app-20210406152023-0003).
SparkSession available as 'spark'.
>>> numbers = sc.range(0, 1000000, 1, 2)
>>> evens = numbers.filter(lambda x: x % 2)
>>> listfelements = evens.collect()
21/04/06 15:20:45 ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks
java.io.IOException: Failed to connect to /172.16.115.136:42373
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:287)