spark read local file

1 Background

Based on spark development program

The data is placed in a local file in text format

The language case of this article is java

2 steps

2.1 Construction Project

  ​​​​​​​ build a spark project based on maven _this is a book blog-CSDN blog

2.2 Code Example

 SparkConf conf = new SparkConf().setAppName("demo").setMaster("local[*]");SparkSession sparkSession = SparkSession.builder().config(conf).getOrCreate();JavaSparkContext javaSparkContext = new JavaSparkContext(sparkSession.sparkContext());JavaRDD rdd = javaSparkContext.textFile(INPUT_PATH);rdd.foreach(new VoidFunction() {@Overridepublic void call(String s) throws Exception {System.out.println(s);}}); 

3 Notes

If it is placed on a linux server and running on spark, the file path needs to be prefixed with this

file:///data/demo.text

The reason is that spark is generally used in combination with hdms, and a prefix needs to be added to read a local file to indicate that a local file is read