当前位置:网站首页>Error in running test pyspark in idea2020
Error in running test pyspark in idea2020
2022-07-02 07:13:00 【wuzd】
Project environment :
Windows: Idea2020
JDK: java version "1.8.0_231"
Python: 3.8.3
spark:spark-3.2.1-bin-hadoop2.7.tgz
There used to be hadoop2.7 Environment , Therefore, the Spark3 Is based on hadoop2.7 Of .
Problem description
IDEA Medium test PySpark Of Map operator , error Message
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times,
most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (XXXX.mshome.net executor driver):
java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the specified file
Detailed error information :
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (E490.mshome.net executor driver):
java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the specified file .
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:166)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:108)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:121)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:162)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the specified file .
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:444)
at java.lang.ProcessImpl.start(ProcessImpl.java:140)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 15 more
countermeasures
Python Copy from the installation directory of python.exe, And changed his name to python3.exe
Spark Of Map Operator runs successfully .
边栏推荐
猜你喜欢
随机推荐
2021-07-05c /cad secondary development create arc (4)
php中根据数字月份返回月份的英文缩写
Check log4j problems using stain analysis
SQL注入闭合判断
Sqli-labs customs clearance (less6-less14)
2021-07-17C#/CAD二次开发创建圆(5)
Go package name
Oracle segment advisor, how to deal with row link row migration, reduce high water level
RMAN incremental recovery example (1) - without unbacked archive logs
Anti shake and throttling of JS
ORACLE EBS DATAGUARD 搭建
CRP implementation methodology
Sqli-labs customs clearance (less15-less17)
CAD二次开发 对象
CAD secondary development object
中年人的认知科普
JS create a custom JSON array
ssm垃圾分类管理系统
Analysis of MapReduce and yarn principles
如何高效开发一款微信小程序