Pyspark changes the column order and saves it into iceberg database

Create an environment , Appoint catalog

def get_spark():
    os.environ.setdefault('HADOOP_USER_NAME', 'root')
    # total size of serialized results of tasks is bigger than spark.driver.maxResultSize
    # ERROR DataWritingSparkTask: Aborting commit for partition 2 (task 2, attempt 0, stage 0.0)  Out of memory 
    spark = SparkSession.builder \
        .config('spark.sql.debug.maxToStringFields', 2000) \
        .config('spark.debug.maxToStringFields', 2000) \
        .config('spark.driver.memory', '16g') \
        .config('spark.executor.memory', '16g') \
        .config('spark.driver.maxResultSize', '4g') \
        .config('spark.network.timeout', 180) \
        .getOrCreate()
    spark.conf.set("spark.sql.catalog.iceberg",
                   "org.apache.iceberg.spark.SparkCatalog")
    spark.conf.set("spark.sql.catalog.iceberg.type", "hive")
    spark.conf.set("spark.sql.catalog.iceberg.uri",
                   "thrift://192.168.x.xx:9083")
    spark.conf.set("spark.sql.session.timeZone", "GMT+8") 
    spark.conf.set("spark.sql.iceberg.handle-timestamp-without-timezone", True)
    return spark

Column A B To B A Store in database , If you do not change the order, you will report wrong columns and mismatches
1.spark dataframe Create a temporary view , And then to spark dataframe

spark = get_Spark()
spark_df = spark.sql('select A from ***')
spark_df = spark_df.withColumn("B", )
spark_df.createOrReplaceTempView("test")
DF = spark.sql("select B A from test")
DF.write.insertInto('xxx',  True)

2. establish schema( Not recommended )
2.1 direct spark.createDataFrame(df) There may be a mistake schema ValueError: Some of types cannot be determined after inferring

There are fields spark Its type cannot be inferred ,

2.2 Specify the type of each field
Data conversion may report errors IntegerType can not accept object 2.0 in type <class 'float'>

schema = StructType([
                         StructField("xx", StringType(), True),
                         StructField("xx", IntegerType(), True)
                     ])