Cluster usage specification

1. Try to use the database table of the application layer , Reduce access to the source layer library table ; Or build your own application layer table and update it regularly ;

2."spark Inquire about ,pandas Handle 、 Splicing " and "spark Inquire about 、 Handle 、 Splicing " Balance the calculation speed 、 Cluster resources occupy both ;

3. Avoid spark.dataframe turn pd.dataframe and to_csv; take spark.dataframe Directly to csv Save to hdfs：

192.168.101.11 yes master Of ip, Default web UI Interface port 50070, File upload and download port 8020

3.1 Select the cluster location to store ： for example hdfs://192.168.101.11:8020/tmp/123.csv

3.2 preservation
spark_df.write.format(“csv”).options(header=‘true’, inferschema=‘true’).save(‘hdfs://192.168.101.11:8020/tmp/’ + “123.csv”)

3.3 see
df = od.spark.read.format(‘com.databricks.spark.csv’).options(header=‘true’, inferschema=‘true’).load(‘hdfs://192.168.101.11:8020/tmp/123.csv’)