HDFS centralized cache management

Hadoop from 2.3.0 Version starting support HDFS Caching mechanisms ,HDFS Allows users to cache some directories or files in HDFS among ,NameNode Will notify the DataNodes Cache it in DataNode In the memory of

Centralized caching Management has many significant advantages ：

1. Prevent frequently used data from being cleared from memory
2. because DataNode The cache of is provided by NameNode To manage ,applications You can query the cached list when scheduling tasks , Using a cached block copy can improve read performance
3. When the block is DataNode After caching , The client can use a new 、 efficient 、zero-copy The reading of API, Because the data in the cache has been calculated checksum, When using new API when , The client is basically zero overhead
4. Can improve colony Memory utilization of . When using the operating system cache , Repeated reading of a block will cause all copies to be put into the buffer , When using Centralized caching when , Users can specify n One copy of m Will be cached , You can save n-m Of memory

usage

Centralized caching It is very useful for frequently accessed files , for example hive Often used in fact Tables are ideal for caching ; On the other hand , Caching query results for a year may not be so useful , Because this result may only be viewed once

It also helps to improve the efficiency of mixed type operations SLA performance , Caching high priority data ensures that it does not compete with low priority data IO

Use ：

[[email protected] hadoop]$ hdfs cacheadmin -addPool admin    【 Add one pool】

Successfully added cache pool admin.

[[email protected] hadoop]$ hdfs cacheadmin -listPools       【 see pool list 】

Found 1 result.

NAME  OWNER  GROUP  MODE            LIMIT  MAXTTL

admin  hadoop  hadoop  rwxr-xr-x  unlimited  never

[[email protected] hadoop]$ hdfs cacheadmin -addDirective -path /input -pool admin 【 Add a directory to pool】

Added cache directive 1

[[email protected] hadoop]$ hdfs cacheadmin -listDirectives      【 View cached data 】

Found 2 entries

 ID POOL    REPL EXPIRY  PATH     

  1 admin      1 never  /input   

  2 test      1 never  /output/0

[[email protected] hadoop]$ hdfs cacheadmin -removeDirective 1    【 Press id Clear cache 】

Removed cached directive 1

[[email protected] hadoop]$ hdfs cacheadmin -listDirectives     

Found 1 entry

 ID POOL  REPL EXPIRY  PATH     

  2 test      1 never  /output/0

[[email protected] hadoop]$ hdfs cacheadmin -removeDirectives -path /output/0 【 Press path Clear cache 】

Removed cache directive 2

Removed every cache directive with path /output/0

[[email protected] hadoop]$ hdfs cacheadmin -listDirectives

Found 0 entries