Explanation of spark operator

1 Background

An important feature of writing spark programs is based on operator development

Each operator has its own characteristics and uses

A certain type of operator is the performance bottleneck

2 Categories

There are two main categories of spark operators

3 Transformation

3.1 Introduction

Used to transform RDDs

Every Transformation operation will generate a new RDD

This operation is delayed in execution

4 Action

4.1 Introduction

Used to trigger the calculation of RDD; get relevant calculation results or save the results in an external system

5 Summary of the characteristics of the two

Transformation is asynchronous and does not affect performance

Returns the result int, double, and set (will not return a new RDD). When encountering an Action operation, it starts from the source of the lineage and performs physical conversion operations, so the operator of Action consumes performance

/p>