当前位置：网站首页>dolphinscheduler 2.0.5 任务测试（spark task）报错：Container exited with a non-zero exit code 1

dolphinscheduler 2.0.5 任务测试（spark task）报错：Container exited with a non-zero exit code 1

2022-06-23 03:52:00 【韧小钊】

Container exited with a non-zero exit code 1

昨天在dolphinscheduler涉及HDFS功能测试（三）spark task中遇到的额问题，一直没有解决，今天再来瞅一眼，就一眼
在这里插入图片描述

本地浏览器访问虚拟机域名配置

每次查看页面还要把虚拟机域名host1替换为具体IP,浏览器才能正常访问，太麻烦了

配置方法

将本机的C:\Windows\System32\drivers\etc'\hosts配置成和虚拟机/etc/hosts地址一致

登录验证

在这里插入图片描述

查看日志

查看输出日志
在这里插入图片描述

stderr（没发现有用信息）
stdout（总算摸到门路了）

具体日志：

Tools
/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/context.py:77: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/dataframe.py:138: FutureWarning: Deprecated in 2.0, use createOrReplaceTempView instead.
Traceback (most recent call last):
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/sparktasktest.py", line 42, in <module>
    df_result.coalesce(1).write.json(sys.argv[2])
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 846, in json
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/py4j-0.10.9.3-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o56.json.
: java.io.IOException: Incomplete HDFS URI, no host: hdfs:///test/softresult
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
	at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:556)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
	at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
	at org.apache.spark.sql.DataFrameWriter.json(DataFrameWriter.scala:763)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)

环境问题

ModuleNotFoundError: No module named ‘py4j’

昨晚空跑脚本是可以成功的，现在看来应该还是脚本执行报错了，单独执行确实报模块不存在
在这里插入图片描述

pyspark重装

在线安装，昨晚直接离线安装的，可能下载的包有问题

 sudo /usr/local/python3/bin/pip3 install pyspark

在这里插入图片描述

再次执行脚本不再报模块缺失错
手动安装pyspark步骤（解压安装包执行sudo python3 setup.py install）

spark-submit再次验证

错误依旧：
在这里插入图片描述

报错:Incomplete HDFS URI, no host: hdfs:///test/softresult

网上说可能是没引入hadoop配置文件，结果一检查，发现配置的还真有问题（配的hadoop_home的地址）
softresult

索性HADOOP_HOME也配置上吧（conf/spark-env.sh）

export JAVA_HOME=/usr/local/java/jdk1.8.0_151
export HADOOP_HOME=/home/dolphinscheduler/app/hadoop-2.7.3
export HADOOP_CONF_DIR=/home/dolphinscheduler/app/hadoop-2.7.3/etc/hadoop
export SPARK_PYTHON=/usr/local/bin/python3