当前位置:网站首页>Error in running test pyspark in idea2020
Error in running test pyspark in idea2020
2022-07-02 07:13:00 【wuzd】
Project environment :
Windows: Idea2020
JDK: java version "1.8.0_231"
Python: 3.8.3
spark:spark-3.2.1-bin-hadoop2.7.tgz
There used to be hadoop2.7 Environment , Therefore, the Spark3 Is based on hadoop2.7 Of .
Problem description
IDEA Medium test PySpark Of Map operator , error Message
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times,
most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (XXXX.mshome.net executor driver):
java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the specified file

Detailed error information :
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (E490.mshome.net executor driver):
java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the specified file .
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:166)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:108)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:121)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:162)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the specified file .
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:444)
at java.lang.ProcessImpl.start(ProcessImpl.java:140)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 15 more
countermeasures
Python Copy from the installation directory of python.exe, And changed his name to python3.exe

Spark Of Map Operator runs successfully .

边栏推荐
- MySQL无order by的排序规则因素
- Go common compilation fails
- ORACLE EBS 和 APEX 集成登录及原理分析
- JSP智能小区物业管理系统
- 叮咚,Redis OM对象映射框架来了
- 2021-07-05C#/CAD二次开发创建圆弧(4)
- JSP intelligent community property management system
- sqli-labs通关汇总-page1
- [leetcode question brushing day 35] 1060 Missing element in ordered array, 1901 Find the peak element, 1380 Lucky number in matrix
- UEditor .Net版本任意文件上传漏洞复现
猜你喜欢
随机推荐
MapReduce与YARN原理解析
Explanation of suffix of Oracle EBS standard table
php中生成随机的6位邀请码
ARP attack
SSM学生成绩信息管理系统
2021-07-17c /cad secondary development creation circle (5)
MySQL组合索引加不加ID
php中计算两个日期之前相差多少天、月、年
Spark的原理解析
CAD二次开发 对象
php中时间戳转换为毫秒以及格式化时间
In depth study of JVM bottom layer (II): hotspot virtual machine object
ORACLE EBS中消息队列fnd_msg_pub、fnd_message在PL/SQL中的应用
Pyspark build temporary report error
Oracle general ledger balance table GL for foreign currency bookkeeping_ Balance change (Part 1)
sqli-labs通关汇总-page2
JSP智能小区物业管理系统
SQLI-LABS通关(less18-less20)
Sqli labs customs clearance summary-page4
Anti shake and throttling of JS







