Use and cases of partitions

The use of partitioned tables

Why use partitions

In the use of select When querying data , If there are no restrictions, the entire table will be scanned , But there are many data that are not what we want , It takes a lot of time .

Partition table corresponds to hdfs It is a separate folder （ There will be cases later ）, When searching for data, we only need to scan the data in this folder , Greatly improve the efficiency of the query . Partitions are divided into static partitions and dynamic partitions , Later, we will focus on the differences between the two .

Let's learn about partitions from a simple example ：

Static partitioning

1. Create a partition table ：

    CREATE table student(id int, name string, age int)
    PARTITIONED by(dt string)
    ROW format delimited fields terminated by ','
    STORED as textfile;

2. insert data

1,zs,18
2,li,19
3,ws,20

load data local inpath '/opt/data/student.txt' overwrite into table student partition(dt='2022-07-20');
insert into table student partition(dt='2022-07-21') values(4,'qq',21);
insert into table student partition(dt='2022-07-22') values(5,'ww',22);
insert into table student partition(dt='2022-07-23') values(6,'ee',23);

3. Query data in table

hive > select * from student;
OK
student.id	student.name	student.age	student.dt
1	zs	18	2022-07-20
2	li	19	2022-07-20
3	ws	20	2022-07-20
4	qq	21	2022-07-21
5	ww	22	2022-07-22
6	ee	23	2022-07-23
Time taken: 0.293 seconds, Fetched: 6 row(s)

​ Query the data in a partition ：

hive > select * from student where dt = '2022-07-20';
OK
student.id	student.name	student.age	student.dt
1	zs	18	2022-07-20
2	li	19	2022-07-20
3	ws	20	2022-07-20
Time taken: 0.361 seconds, Fetched: 3 row(s)

4.（1） View the partitions in the table

hive > show partitions student;
OK
partition
dt=2022-07-20
dt=2022-07-21
dt=2022-07-22
dt=2022-07-23
Time taken: 0.104 seconds, Fetched: 4 row(s)

（2） see hdfs Corresponding directory structure

You can see different partitions in hdfs Corresponding to different directories

Dynamic partitioning ：

Create a partition table ：

  create table student_dync(
    id int,
    name string,
    age int
  ) partitioned by(dt string) 
  row format delimited fields terminated by ','
  stored as textfile;

Insert data into table ：

set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table student_dync partition (dt)
   select 
     id,
     name,
     age,
     dt 
    from student_tmp;

No partition is specified when inserting data into the table , Then look at the partitions in the following table ：

hive > show partitions student_dync;
OK
partition
dt=2022-08-01
dt=2022-08-02
dt=2022-08-03
dt=2022-08-04
Time taken: 0.058 seconds, Fetched: 4 row(s)

You can also see that different partitions correspond to different directories .

summary : The difference between dynamic partition and static partition

1. Static partition columns are determined at compile time by manually specifying partitions , Dynamic partitioning is in sql Only when it is implemented can it be determined .

2. Because the partition needs to be manually specified in use , Therefore, static partitions are suitable for a small number of partitions 、 Data with clear partition name

3. Dynamic partition is suitable for the situation with a large number of partitions , The partition field position of dynamic partition is at the end , Multiple partition fields are placed in partition order

4. When inserting data , Static partition support load and insert Two ways , Dynamic partitioning only supports insert The way