Use lambda function URL + cloudfront to realize S3 image back to source

1. The background that

Amazon cloud technology S3 Is an object storage service , Provide industry-leading scalability 、 Data availability 、 Safety and performance . Customers of all sizes and industries can S3 Store and protect any amount of data . To improve data security , Many customers will choose to migrate their existing data to S3.

When migrating data stored in a third party to S3 when , Sometimes you need to use “ Mirror back to source ” To reduce business interruption time .“ Mirror back to source ” Usually refers to from S3 When the obtained file does not exist , This file needs to be obtained from a third-party source and stored in S3. because S3 Itself does not provide “ Mirror back to source ” The function of , Therefore, we need to consider self implementation .

Generally, it is stored in S3 The document will pass CloudFront For external use , and CloudFront The source group supports failover . A feasible solution is to use CloudFront When the source group fails over, use the standby source to synchronize files from a third party and save them to S3 And back to the client .

stay Lambda function URL Before release , It is usually necessary to EC2 Deploy a simple application as an alternate source . It needs to be right “ Mirror back to source ” The amount of data is accurately estimated, otherwise it will cause back to source anomalies or EC2 Waste of resources , With the help of Lambda Can effectively avoid these problems .

Lambda function URL yes Lambda stay 4 A new feature released in January , Allow direct passage through functions URL Breakpoint call Lambda function . Borrow this feature , Can be used without API Gateway perhaps ALB In case of use Lambda Complete something similar webhook、 Simple functions such as form verification , And there's no extra Lambda cost . The following article will introduce how to pass Lambda function URL+CloudFront To achieve “ Mirror back to source ” This function .

The overall architecture of the scheme is as follows .

Its working principle is described as follows ：

1） Terminal equipment to CloudFront Request file

2）CloudFront towards S3（ Main source station ） Request file

3） If the file does not exist, return 403 to CloudFront

4）CloudFront Try the request according to the origin group configuration Lambda function URL（ Standby origin ）

5）Lambda Get files from third-party sources

6）Lambda Save file to S3 bucket

7）Lambda return 302 to CloudFront, The attached address is the original file, so that the client requests the file again

8）CloudFront Again to S3（ Main source station ） Request file

9）S3 Return file to CloudFront,CloudFront The edge node configures the cache file according to the cache policy

10）CloudFront Return the file to the terminal device

2. Configuration guidelines

The overall configuration process of this scheme is ：

1. establish S3 bucket

2. Create and configure Lambda function

3. Create and configure CloudFront distribution

S3 Bucket creation is relatively simple , Omitted in this paper ,Lambda function 、CloudFront The distribution configuration instructions are shown below .

2.1 Lambda Function configuration

1） Get into Lambda Console , Click on “ Create a function ” Start to create Lambda function

2） Set as shown in the screenshot Lambda Function name 、 Running environment

3） an “ Advanced settings ”, Select enable function URL, For the convenience of demonstration, the authorization type is set to “NONE”, The actual production environment can be configured as needed

4） The function code can be copied from the following address . Be careful , The example code in the link uses Python Requests library , So it needs to be Lambda Add layers to contain the Library , The operation mode is shown in the following figure

https://github.com/ensean/lambda_sync_from_3rd_origin/blob/master/lambda_hanlder.py

5） When adding layers, the layer type is “ Appoint ARN”, At the same time specified ARN value .ARN You can follow the link below according to the area 、Python Library type query

https://github.com/keithrozario/Klayers/tree/master/deployments/python3.9

6） Get into Lambda Function configuration interface , Edit general configuration . For example, adjust the timeout according to the file size that needs to be synchronized

7） Select the left permission menu , Click execute role , Configure for roles S3 Bucket access

8） Enter the role permission configuration page , Click on “ Add permissions ” → “ Additional strategies ”

9） For the convenience of demonstration, select according to the search in the pop-up strategy “AmazonS3FullAccess” Strategy , The subsequent production environment can be adjusted as needed

10） Select the environment variables menu , Add environment variables as shown in the figure external_endpoint, s3_bucket, service_endpoint

11） Selection function URL menu , Copy function URL spare

2.2 CloudFront To configure

1） establish CloudFront distribution , Source domain selection S3 bucket , And configuration CloudFront adopt OAI visit S3 bucket . The others remain the default configuration , establish CloudFront distribution

2） Get into CloudFront Distribution configuration , Switch to “ Source ” TAB , Click on “ Create source ”

3） Fill in the source domain configuration interface Lambda function URL domain name , The rest remain in the default configuration , Click create source

4） Click on “ Create source group ”

5） Select according to the prompt S3 Source based ,Lambda function URL Source for backup , Failover condition selection 403、404

6） stay CloudFront Select the left side of the console “ Strategy ” menu , stay “ cache ” Click on the tab “ Create a cache policy ”

7） Enter the cache policy name in the cache policy configuration interface , At the same time, adjust the shortest TTL to 0, rest TTL Value can remain the default , Then create a cache policy

8） Get into CloudFront distribution “ Behavior ” TAB , Select the default behavior and edit

9） Set in the behavior editing interface “ Source and source group ” For step 5） Created source group , At the same time, adjust the cache strategy as a step 7） Cache policy created , Then save the changes

10） adopt CloudFront Access files similar to the following , Observe the browser console network request . If a result similar to the following figure appears , And the file is normally saved to S3 bucket “file/2022/03/18/ea27cb786ffb474682a8feb5.jpg” It indicates that the configuration of mirroring back to source is successful https://d281txxxxxxxx.cloudfront.net/file/2022/03/18/ea27cb786ffb474682a8feb5.jpg

3. summary

Lambda function URL As a newly released feature , Allow users to access directly through the Internet Lambda, And there is no extra charge for this function . This article provides examples with CloudFront The failover function of the source group and Lambda function URL It can be realized quickly S3“ Mirror back to source ” function . Readers can also pass according to their own business needs Lambda function URL To realize some lightweight microservice applications .

4. Reference material

https://docs.aws.amazon.com/zh_cn/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html

https://aws.amazon.com/blogs/aws/announcing-aws-lambda-function-urls-built-in-https-endpoints-for-single-function-microservices/

https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html

Author of this article

      Li Yinxiang

Amazon cloud technology solution architect

Responsible for consulting and designing cloud computing solution architecture based on Amazon cloud technology , Promote Amazon cloud technology, cloud platform technology and various solutions in China . Worked in IBM, Responsible for enterprise private cloud solution consulting and architecture design , Rich experience in infrastructure .

hear , Click below 4 Button

You won't encounter bug 了 ！