Spark's projection

Projection

Projection is a contract of Scala functions that produce an internal binary row for a given internal row.

What is a projection in Spark?

Projection refers to the selected columns. For example: If your filters pass only 5% of the rows, only 5% of the table will be passed from the storage to Spark instead of the full table.

What is projection pruning in Spark?

The Projection Pushdown feature allows the minimization of data transfer between the file system/database and the Spark engine by eliminating unnecessary fields from the table scanning process. It is primarily useful when a dataset contains too many columns.