How Facebook moves instagram from AWS to its own server

When Instagram stay 2012 To join in Facebook, We have quickly established a large number of Facebook Infrastructure integration point , To accelerate product development , Make the community safer . At first we used ad-hoc The endpoint is Facebook web Effective delivery between services to build these integrations . However, we found that this method may be a little clumsy , It also limits our use of internal Facebook Ability to serve .

2013 At the beginning of April , We started to put Instagram Back end of from Amazon Web Services(AWS） towards Facebook Massive data center migration . This will moderate with other internal Facebook The system integrates and allows us to make full use of it for managing large-scale The server Deploy built tools . The main goal of migration is to maintain the complete service of the website during the transition , Avoid affecting feature deployment , Minimize infrastructure level changes to avoid operational complexity .

At first, migration seemed simple : stay Amazon Of Elastic Compute Cloud(EC2) and Facebook Build a secure connection between data centers , Migrate services piece by piece . Simple .

More than that . The main obstacle to this simple migration is Facebook Private ownership of IP Space and EC2 Private ownership of IP Space conflict . We have only one way to go : First migrate to Amazon Of Virtual Private Cloud（VPC), Subsequent use Amazon Direct Connect Migrate to Facebook.Amazon Of VPC Provides the necessary scalable addressing to avoid and Facebook Private network conflict .

We are deterred from this task ; stay EC2 Thousands of instances are running on , There are also new instances that appear every day . To minimize downtime and operational complexity , Running on the EC2 and VPC Instances in must appear to be from the same network .AWS There is no way to share security groups , There's no private EC2 and VPC Network bridging . The only way for these two private networks to communicate is to use the public address space .

So we use Python Developed Neti—— A dynamic IP Packet filtering system daemon , from Hadoop The formal subproject of ZooKeeper Provide support .Neti The security group function is provided , And for running in EC2 and VPC Each instance in provides a separate address . It manages thousands of local NAT And the filtering rules of each instance , Allow independent 、 Flat " overlap "("overlay") Address space for secure communication .NAT The rule selects the most efficient path between the source instance and the target instance .VPC and EC2 The instance communication between uses the public network , Internal communication uses private network . This is transparent to our applications and back-end systems , because Neti In each case, the appropriate IP Packet filtering system .

constitute Instagram The various components of the stack range from EC2 To VPC The migration of the environment took less than three weeks , This makes us believe that if there is no Neti, A lot longer . In the process , There has been no major service downtime , At the same time, we soon realize that this is the fastest in history VPC Scale migration .

With VPC Completion of migration , Our example runs in a compatible address space ,Instagram Ready to begin to finish Facebook Data center migration .

Around one EC2 Built toolsets have been around for years , It manages. Instagram Our product system , Including configuration management scripts , Used to supply Chef（" Chef ”）, From application deployment to database master Used for a wide range of operational tasks such as promotion Fabric. This tool set is right EC2 The assumptions made are Facebook It is no longer applicable in the environment .

In order to make our supply tools more portable ,Instagram Specific software is now running Facebook Data Center The server On the one Linux In the container (LXC).Facebook Supply tools are used to build basic systems ,Chef Run install and configure in container Instagram Specific software . To provide a leap EC2 and Facebook Infrastructure of data center , Our current Chef recipes（" Chef's recipe “） Whether to allow them to support Facebook For internal use CentOS The platform is supported in EC2 Used in Ubuntu The new logic of .

For basic tasks EC2 Specific command line tools , For example, enumerate running hosts and Chef“knife" Supply support within the tool , Replaced by the same tool . This tool is designed as an abstraction layer , towards EC2 The tools used in provide similar workflows , It reduces people's pressure , Eased the technological transition to the new environment .

We finished within two weeks after the tools and environment were in place Instagram Our product infrastructure starts from VPC To Facebook Data center migration .

This phased work has achieved the main objectives set at the beginning of the project , It was a great success . Besides , In the stage of planning and executing migration , The team delivered nearly twice as many as Instagram Direct And our user base . We adhere to the objective original intention of minimizing change , So the transition is almost transparent to our engineering team .

Review some key points of this one-year-old project (key takeaways):

• Plan to support minimal changes to the new environment , avoid “while we’re here.” The temptation of .
• Sometimes , Crazy ideas are useful ——Neti Is a proof .
• Devote yourself to building your tools ; Perform such a large-scale migration , What you need most is an unexpected curve ball . 
• Reuse concepts and workflows familiar to the team to avoid complicating communication changes to the team .

This is the collaboration of multiple teams and individual contributors . In the next few weeks , We will provide a more in-depth introduction to this migration , Always pay attention to this space .