Add an empty column to spark dataframe - add an empty column to spark dataframe

problem ：

As mentioned in manyother locations on the web, adding a new column to an existing DataFrame is not straightforward. As on the Internet many Other places Described , To the present DataFrame Adding new columns is not easy .Unfortunately it is important to have this functionality (even though it is inefficient in a distributed environment) especially when trying to concatenate two DataFrame s using unionAll . Unfortunately , Try to connect two, especially when there is such a function （ Even if it is inefficient in a distributed environment ） It's important DataFrame S Use unionAll .

What is the most elegant workaround for adding a null column to a DataFrame to facilitate a unionAll ? take null Columns added to DataFrame In order to promote unionAll The most elegant solution is unionAll ？

My version goes like this: My version is like this ：

from pyspark.sql.types import StringTypefrom pyspark.sql.functions import UserDefinedFunctionto_none = UserDefinedFunction(lambda x: None, StringType())new_df = old_df.withColumn('new_column', to_none(df_old['any_col_from_old']))

---

Solution ：

Reference resources ： https://stackoom.com/en/question/2Ecs2