[Presto] details of the new version upgrade of Presto

One 、 Preface

Upgraded version :
The old version ：0.220 → The new version : 0.269

Purpose of upgrade ：

1. The new version does a lot of optimization for the old version , Improve service stability .
2. The new version needs to be compatible with the functions of the old version , And transparent to the business side .
3. Introduce a stricter memory judgment strategy .

New version upgrade ：

1. Added a lot of functions
2. query.initial-hash-partitions By default 8 Change it to 100
3. about hive Of orc Format data has better support
4. hive Support for weighing and splitting according to file size , This allows deeper workers to split queues when files are smaller
5. Fixed the underlying data format as parquet Inconsistent query results
6. Fixed a lot of bug
7. For more details about the upgrade, see :https://prestodb.io/docs/current/release/release-0.269.html

The details of the secondary development of the old version are as follows ：

1. presto Support recursive query hive View （presto Not supported by default hive View function ）
2. presto password Authentication plug-in （ Self developed plug-ins , be used for python Code client authentication , Only for certification , Permission is not involved ）

For compatibility with older versions , And improve stability The new version changes as follows ：

	function	Purpose
1	Support recursive query hive View function	Compatible with older versions
2	Native password Authentication plug-in	Replace self-developed plug-ins
3	Support for older versions password Authentication plug-in	Compatible with older versions
4	modify general memory pool Memory shortage judgment strategy	Improve stability
5	modify log4j Version of , prevent jni Inject	Security
6	Use jdk11	Improve stability

Upgrade plan ：

This upgrade cannot adopt rolling upgrade （ New and old services are incompatible ）, Set up a new version service , take presto Hang the soft connection to the new version , Then stop the old version cluster , The new version of the cluster follows the configuration of the old cluster .

Upgrade time ：

Because rolling upgrade cannot be adopted , Therefore, the upgrade time is selected in the low peak period

Upgrade steps ：

1. Bulk copy presto The installation package to /usr/local Under the table of contents => /usr/local/presto-server-0.269
2. Bulk copy jdk11 The installation package to /home/rong Under the table of contents => /home/rong/jdk-11.0.14
3. Add soft connection ln -s /home/rong/jdk-11.0.14 jdk11
4. Update and issue worker Node config.properties The configuration file
5. Update and issue condition node config.properties The configuration file
6. Initialize the node configuration information script , The purpose is to update worker Node node.properties In the configuration file host Information
7. Stop the old cluster service
8. Modify the soft connection to the new version service
9. start-up & Verify the new version of the service

Upgrade impact ：

redash Theory has no effect
tableau Theory has no effect
python Client code Theory has no effect

The fallback program ：

Reconnect the soft connection to the old version , Restart the old cluster .

Two 、 Support HIVE View

Refer to the following article , Combined with the new version for transplantation and adaptation （done）

• https://github.com/prestodb/presto/pull/9031
• https://blog.csdn.net/hjw199089/article/details/81415444

3、 ... and 、 Insufficient memory judgment strategy

1 When does the cluster run out of memory ？

remarks : because RESERVED_POOL The... Is not enabled in the configuration file , So the judgment condition is only GENERAL_POOL Whether there are blocking nodes

2 What conditions determine whether the node is blocked （ Out of memory ）？

if (poolInfo.getFreeBytes() + poolInfo.getReservedRevocableBytes() <= 0) {
    
    blockedNodes++;
}
  

Purpose ： In order to trigger the memory protection strategy faster .

modify ：
Turned out to be poolInfo.getFreeBytes <= 0, Change to less than pool MaxByte Of 80%, Trigger the memory protection policy in advance

This function is aimed at GENERAL_POOL The changes to the , It can't be solved completely OOM The problem of , Because part of the memory is Presto Can't be monitored , So there are still nodes OOM The risk of .

Reference resources ：

• http://armsword.com/2020/02/18/presto-memory-kill-policy/

• http://armsword.com/2018/05/22/the-memory-management-and-tuning-experience-of-presto/

• http://armsword.com/2019/11/13/the-configuration-settings-of-presto-memory-management/

Four 、password-file-authenticators plug-in unit

This plug-in is a custom plug-in , It is used for identity authentication . Do not do permission management .

This plug-in has the same functions as password-authenticators The functions of official plug-ins are basically the same .

difference ：
password-file-authenticators The user name and password of the plug-in are clear text

password-authenticators The password of the plug-in is ciphertext

Custom developed plug-ins have also been ported and adapted in the new version , At present, both plug-ins can be used . The function is the same .