当前位置：网站首页>Dolphin scheduler 2.0.5 task test (spark task) reported an error: container exited with a non zero exit code 1

Dolphin scheduler 2.0.5 task test (spark task) reported an error: container exited with a non zero exit code 1

2022-06-23 05:03:00 【Zhun Xiaozhao】

Catalog

Container exited with a non-zero exit code 1

Container exited with a non-zero exit code 1

Yesterday in dolphinscheduler involve HDFS A functional test （ 3、 ... and ）spark task Problems encountered in , It hasn't been solved , Take another look today , At a glance
Insert picture description here

Local browser access virtual machine domain name configuration

Every time you view the page, you need to add the virtual machine domain name host1 Replace with specific IP, The browser can access it normally , It's too troublesome

Configuration method

Will the machine C:\Windows\System32\drivers\etc'\hosts Configure to and virtual machine /etc/hosts The address is consistent

validate logon

Insert picture description here

Check the log

Check the output log
Insert picture description here

stderr（ No useful information ）
stdout（ Finally, I have found a way ）

Specific log ：

Tools
/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/context.py:77: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/dataframe.py:138: FutureWarning: Deprecated in 2.0, use createOrReplaceTempView instead.
Traceback (most recent call last):
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/sparktasktest.py", line 42, in <module>
    df_result.coalesce(1).write.json(sys.argv[2])
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 846, in json
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
  File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/py4j-0.10.9.3-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o56.json.
: java.io.IOException: Incomplete HDFS URI, no host: hdfs:///test/softresult
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
	at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:556)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
	at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
	at org.apache.spark.sql.DataFrameWriter.json(DataFrameWriter.scala:763)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)

Environmental problems

ModuleNotFoundError: No module named ‘py4j’

The last night's runaway script can be successful , Now it seems that the script execution reports an error , The module of independent execution does not exist
Insert picture description here

pyspark reinstall

Online installation , It was installed offline last night , There may be a problem with the downloaded package

 sudo /usr/local/python3/bin/pip3 install pyspark

Insert picture description here

Execute the script again and no module missing error will be reported
Manual installation pyspark step （ Unzip the installation package to execute sudo python3 setup.py install）

spark-submit To verify again

The mistake remains ：
Insert picture description here

Report errors :Incomplete HDFS URI, no host: hdfs:///test/softresult

It is said on the Internet that it may not be introduced hadoop The configuration file , Result 1 inspection , It is found that there is a real problem with the configuration （ Matched hadoop_home The address of ）
softresult

Simply HADOOP_HOME Configure it （conf/spark-env.sh）

export JAVA_HOME=/usr/local/java/jdk1.8.0_151
export HADOOP_HOME=/home/dolphinscheduler/app/hadoop-2.7.3
export HADOOP_CONF_DIR=/home/dolphinscheduler/app/hadoop-2.7.3/etc/hadoop
export SPARK_PYTHON=/usr/local/bin/python3

Report errors :path hdfs://192.168.56.10:8020/test/softresult already exists.

The problem should have been solved , Before no host: hdfs:///test/softresult, It is suspected that the directory does not exist , Manually created , So the report directory already exists , Specify a new directory to view again
Insert picture description here

SUCCEEDED

It's a success
The verification results , Still a little imperfect , The data is not written in , It must be a matter of procedure （ The next step is to study python Well ？ The more you study, the more ignorant you become ）