Automatically refresh CDN cache in real time based on cos log

Problem scenario

background ： Many companies now use object storage services to store static files / Pictures, etc , Make full use of cloud functions to realize data processing （ Data Vientiane ）, Such as COS Zoom in / Cutting and other functions ; At the same time through CDN The ability to accelerate , Improve access quality .

problem ： File deletion , Possible complaints from third parties ; It may come from the violation ; It may come from the company's own business needs, etc ;COS File deletion ,CDN Cache refresh required ; If data processing capability is used , Like Vientiane , May produce the same key, Several URL, Need to refresh the cache .

solve ：

single key- single path scene , Such as key by /aaa/bbb.jpg, The only cache deleted is /aaa/bbb.jpg Scene , At present Function calculation Function automatic refresh function ;

single key- Scenes with parameters or styles , Such as key by /aaa/bbb.jpg, Deleted cache URL May be /aaa/bbb.jpg, It could be /aaa/bbb.jpg?aaaa=1233&a=b, It could be /aaa/bbb.jpg/yangshi, At present, this scenario has not been covered by function calculation , Need to pass through COS The real-time log of is CLS The log service To automatically implement the refresh function , It also supports the coverage of the first scenario .

Architecture Topology

Implementation process

Prerequisite ：

• First open the corresponding cos bucket Log retrieval function of

• Get the corresponding log set and log topic information

• Get account number API Key information

• obtain CLS Public domain name information corresponding to different regions

Available geographical Links

Create function services

Basic configuration and function code ：

• How it was created ： choice " Custom creation "
• Function name ： Customize it , such as refresh_cdn_cache
• regional ： Need and cos bucket In the same region
• Running environment ： choice "python3.6"
• function code ： choice " Upload locally zip package "

Local zip Please use the attachment of this article for the package ：

refresh_cdn_cache_150b4751-1e47-47fe-bfbd-fb6986ec9020.zip

Advanced configuration - Environment configuration ：

cls_day_num=7                                   #  The cycle of retrieving logs 
cls_domain=ap-shanghai.cls.tencentcs.com        #  Corresponding area cls The public domain name of the log service 
cos_bucket_name=tinkerli-cls-1251956900         #  Corresponding cos bucket Name 
hosts=cls-cos.xxxxxxx.cn                        #  Corresponding to those that need to be clearly cached CDN domain name , Support multiple , With ';' Division 
logset_id=49fc1a43-a340-49c8-91be-fcb4abd106a4  #  Corresponding log set id
secret_id=AxxxxD7nxxxxxxxxxxLDsiN7B8p  #  Corresponding API secret key id
secret_key=d1AHFJDtdDLjxxxxxxxx    #  Corresponding API secret key key
topic_ids=0b82160f-9c46-4b95-a668-63874b5cf37f  #  Corresponding log topics id

Trigger configuration ：

1. Trigger mode ： choice COS Trigger
2. Event type ： Select delete file

functional verification

COS File deletion ：

• Both batch deletion and single file deletion are allowed , Such as through cos Console batch delete

see SCF Run log ：

• You can see batch deletion 3 File , Triggered three cloud function calls , All successful .

verification CDN Brush result ：

• To CDN In the console - View the refresh results

You can see each URL Both are refreshed successfully , Whether styled or spliced URL Of

Additional explanation

• Tencent cloud The log service Tencent cloud has gradually supported various cloud product log access , Such as CLB,CDN,COS,TKE And so on ;
• COS Delete files in bulk , The delete event of multiple files will be triggered ;
• This article is only aimed at the problem solving of one scenario , More log based games are welcome .