Distributed file storage system fastdfs hands on how to do it

6.FastDFS Installation and use

6.1 General introduction to installation

• 1. First install libfastcommon (fastdfs Dependent function libraries )
• 2. next install fastdfs( It's just a way of client , tracker ,storage The installation is complete )
• 3. Next install fastdfs-nginx-module (fastdfs Provide right nginx Dependent modules for )
• 4. Last install nginx( If previously installed , There is no need to reinstall )

  Be careful : Installation is divided into online installation ( Good network is required ) Or install offline ( Download relevant software packages and install them offline , The installation information is in the information \fastdfs\ Installation package )

• 5. Before installation , Create two directories

     /usr/local/soft: The directory where the installation package is located , Carry out orders : mkdir -p /usr/local/soft

     /data/fastdfs: The directory where the data is located (fastdfs The directory where the uploaded files and logs are located ) Carry out orders : mkdir -p /data/fastdfs

6.2 install libfastcommon ( Choose one of two installation methods )

• Online installation

   Step one :   git clone https://github.com/happyfish100/libfastcommon.git --depth 1
   Step two :   cd libfastcommon/
   Step three :# Compilation and installation 
  ./make.sh && ./make.sh install
  

• Offline installation

   Step one :   unzip libfastcommon-master.zip
   Step two :   cd libfastcommon-master
   Step three :   ./make.sh && ./make.sh install
  

6.3 install fastdfs ( Choose one of two installation methods )

• Online installation

  cd ../ 															# Return to the previous directory  , go back to soft directory 
  git clone https://github.com/happyfish100/fastdfs.git --depth 1 # download fastdfs.git
  cd fastdfs/ 													# Get into fastdfs
  ./make.sh && ./make.sh install 									# Compilation and installation 
  # for nginx Access to use 
  cp /usr/local/soft/fastdfs/conf/http.conf /etc/fdfs/			
  cp /usr/local/soft/fastdfs/conf/mime.types /etc/fdfs/
  

• Offline installation

   Step one :  unzip fastdfs-master.zip  # First switch to soft Catalog 
   Step two :  cd fastdfs-master
   Step three :  ./make.sh && ./make.sh install
  
  # for nginx Access to use 
  cp /usr/local/soft/fastdfs-master/conf/http.conf /etc/fdfs/			
  cp /usr/local/soft/fastdfs-master/conf/mime.types /etc/fdfs/
  

6.4 install fastdfs-nginx-module ( Choose one of two installation methods )

• Online installation

  cd ../ # Return to the previous directory 
  git clone https://github.com/happyfish100/fastdfs-nginx-module.git --depth 1
  cp /usr/local/soft/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs
  

• Offline installation

   Step one :   unzip fastdfs-nginx-module-master.zip 
   Step two :   cp /usr/local/soft/fastdfs-nginx-module-master/src/mod_fastdfs.conf  /etc/fdfs
  

6.5 install nginx ( Choose one of two installation methods )

• Online installation

  wget http://nginx.org/download/nginx-1.15.4.tar.gz # download nginx Compressed package 
  tar -zxvf nginx-1.15.4.tar.gz # decompression 
  cd nginx-1.15.4/
  # add to fastdfs-nginx-module modular 
  ./configure --add-module=/usr/local/soft/fastdfs-nginx-module/src/
  make && make install # Compilation and installation 
  

• Offline installation

   Step one : tar -zxvf nginx-1.8.1.tar.gz   # Switch to /usr/local/soft Below directory 
   Step two :  install nginx rely on 
  		sudo apt-get install openssl libssl-dev
  		sudo apt-get install libpcre3 libpcre3-dev
  		sudo apt-get install zlib1g-dev
   Step three :  Switch in nginx Catalog 
  		cd /usr/local/soft/nginx-1.8.1
   Step four :  take fastdfs Provided nginx Dependency configuration to nginx Inside 
  
  		sudo ./configure --prefix=/usr/local/nginx/ --add-module=fastdfs-nginx-module-master/src( The corresponding absolute path )
   Step five :  make &&make install
  
   If step 5 is wrong , Then first delete nginx-1.8.1/objs/Makefile This file ,
   The order is : rm -rf /usr/local/soft/nginx-1.8.1/objs/Makefile
   Then perform step 5 again 
  
  

7. Single deployment fastdfs

• Pay attention !!!

  Due to offline installation and online installation , The generated profile names are inconsistent ,

  If it's an offline installation , Then you need to modify the name of the configuration file

  # Execute the following command 
  cd /etc/fdfs
  mv  tracker.conf.sample  tracker.conf
  mv  storage.conf.sample  storage.conf
  mv  client.conf.sample  client.conf
  
  

• Step one : Configure the virtual host

   such as :
   Edit virtual machine's host file :  vim  /etc/hosts
   Add this paragraph :  192.168.200.13 fastdfs.com
   Be careful : ip Write your own mainframe ip, fastdfs.com domain name   You can start at will .
  
   If you pass windows The browser accesses the virtual machine fastdfs, At the same time, you need to configure C:\Windows\System32\drivers\etc\hosts file 
   Add this paragraph :  192.168.200.131 www.fastdfs.com
  
  

• Step two : To configure tracker

  vim /etc/fdfs/tracker.conf
  # What needs to be revised is as follows 
  port=22122 # tracker Server port （ Default 22122, Generally, it is not modified ）
  base_path=/data/fastdfs #  The root directory where logs and data are stored 
  

• Step three : To configure storage

  vim /etc/fdfs/storage.conf
  # What needs to be revised is as follows 
  port=23000 # storage Service port （ Default 23000, Generally, it is not modified ）
  base_path=/data/fastdfs #  Data and log file storage root 
  store_path0=/data/fastdfs #  The first storage directory 
  tracker_server=fastdfs.com:22122 # tracker The server IP And port 
  http.server_port=80 # http Port to access files ( Default 8888, Modify according to the situation , and nginx Is consistent in the port of )
  

• Step four : Start service test

  1. Start the service first

  #  Permanently shut down firewall 
  systemctl disable firewalld.service
  # start-up tracker
  /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
  # start-up storage
  /usr/bin/fdfs_storaged /etc/fdfs/storage.conf
  #  restart storage
  /usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
  

  2. client test

     vim /etc/fdfs/client.conf
     # What needs to be revised is as follows 
     base_path=/data/fastdfs
     tracker_server=fastdfs.com:22122 #tracker The server IP And port 
     # Test after saving , return ID It means success   Such as ：group1/M00/00/00/wKgDDWDtRu6AMPhBARBlpcz7xUs146.jpg
     #  Upload files 
     fdfs_upload_file /etc/fdfs/client.conf /usr/local/soft/fox.jpg
     #  Delete file 
     fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgDCmD1LHaADXrMAAW3EDO1WQ106.jpg
     
     

     

• Step five : To configure nginx visit （client）

  vim /etc/fdfs/mod_fastdfs.conf
          # What needs to be revised is as follows 
          tracker_server=fastdfs.com:22122 #tracker The server IP And port 
          url_have_group_name=true
          store_path0=/data/fastdfs
  # To configure nginx.config
  		vim /usr/local/nginx/conf/nginx.conf
  # Add the following configuration 
  server {
      listen 80; ##  The port is storage.conf Medium http.server_port identical 
      server_name localhost;
      location ~/group[0-9]/ {
      		ngx_fastdfs_module;
      } 
      error_page 500 502 503 504 /50x.html;
      location = /50x.html {
     	 root html;
      }
  }
  

  Nginx test :

  1. Start the service

  /usr/local/nginx/sbin/nginx # start-up nginx
  /usr/local/nginx/sbin/nginx -s reload # restart nginx
  /usr/local/nginx/sbin/nginx -s stop # stop it nginx
  

  2. test , If you can see the picture, it means success **( The following picture is your own picture name )**

  http://fastdfs.com:8888/group1/M00/00/00/wKgDDWDtRu6AMPhBARBlpcz7xUs146.jpg

8. Cluster deployment fastdfs

• Statement : Cluster deployment fastdfs when ,

  Server planning ：

  ​ Tracking server 1【 host 】（Tracker Server）：192.168.200.131

  ​ Tracking server 2【 Standby machine 】（Tracker Server）：192.100.200.132

  ​ Storage server 1（Storage Server）：192.168.200.x1

​ Storage server 2（Storage Server）：192.168.200.x2

​ Storage server 3（Storage Server）：192.168.200.x3

​ Storage server 4（Storage Server）：192.168.200.x4

​ best : Guarantee tracker and storage Installed in different locations ip On a host of , Avoid reporting mistakes .

​ The specific process is as follows :

• Step one : Get ready 3 Servers ( At least 2 platform )

  *  Storage server :192.168.200.131( It is also a scheduler )
  
  ​  Storage server :192.168.200.132
  
  ​  Storage server :192.168.200.133                    
  

  Configure the virtual host at the same time :

   such as :
   Edit virtual machine's host file :  vim  /etc/hosts
   Add this paragraph :  192.168.200.131 fastdfs.com
  			 192.168.200.132 fastdfs2.com
  			 192.168.200.133 fastdfs3.com
   Be careful : ip Write your own mainframe ip, fastdfs.com domain name   You can start at will .
  
   If you pass windows The browser accesses the virtual machine fastdfs, At the same time, you need to configure C:\Windows\System32\drivers\etc\hosts file 
   Add this paragraph :  192.168.200.131 www.fastdfs.com
  			 192.168.200.132 www.fastdfs2.com
  			 192.168.200.133 www.fastdfs3.com
  

• Step two : tracker To configure

  If configured 3 On the mainframe , So it's configuration 3 platform tracker

  vim /etc/fdfs/tracker.conf
  # What needs to be revised is as follows 
  port=22122 # tracker Server port （ Default 22122, Generally, it is not modified ）
  base_path=/data/fastdfs #  The root directory where logs and data are stored 
  

• Step three : storage To configure

  If configured 3 On the mainframe , So it's configuration 3 platform storage

  vim /etc/fdfs/storage.conf
  # What needs to be revised is as follows 
  port=23000 # storage Service port （ Default 23000, Generally, it is not modified ）
  base_path=/data/fastdfs #  Data and log file storage root 
  store_path0=/data/fastdfs #  The first storage directory 
  tracker_server=fastdfs.com:22122 #  The server 1
  tracker_server=fastdfs2.com:22122 #  The server 2
  tracker_server=fastdfs3.com:22122 #  The server 3
  http.server_port=80 # http Port to access files ( Default 8888, Modify according to the situation , and nginx Consistent in )
  

• Step four : client test

  • 1. Start the service first

    #  Permanently shut down firewall 
    systemctl disable firewalld.service
    # start-up tracker
    /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
    # start-up storage
    /usr/bin/fdfs_storaged /etc/fdfs/storage.conf
    #  restart storage
    /usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
    

  • Query whether the startup is successful

    

  • 2. Configure client tests :

    *  To configure client.conf
    vim /etc/fdfs/client.conf
    # What needs to be revised is as follows 
    base_path=/data/fastdfs
    tracker_server=fastdfs.com:22122 #  The server 1
    tracker_server=fastdfs2.com:22122 #  The server 2
    tracker_server=fastdfs3.com:22122 #  The server 3
    
    # Test after saving , return ID It means success   Such as ：group1/M00/00/00/wKgDDWDtRu6AMPhBARBlpcz7xUs146.jpg
    fdfs_upload_file /etc/fdfs/client.conf  /usr/a.jpg   ( File directory )
    

• Step five : To configure nginx

  vim /etc/fdfs/mod_fastdfs.conf
  # What needs to be revised is as follows 
  tracker_server=fastdfs.com:22122 #  The server 1
  tracker_server=fastdfs2.com:22122 #  The server 2
  tracker_server=fastdfs3.com:22122 #  The server 3
  url_have_group_name=true
  store_path0=/data/fastdfs
  # To configure nginx.config
  vim /usr/local/nginx/conf/nginx.conf
  # Add the following configuration 
  server {
  	listen 80; ##  The port is storage.conf Medium http.server_port Same detection cluster 
  	server_name localhost;
  	location ~/group[0-9]/ {
  		ngx_fastdfs_module;
  	} 
  	error_page 500 502 503 504 /50x.html;
  	location = /50x.html {
  		root html;
  	}
  }
  
  

  Detection cluster :

  /usr/bin/fdfs_monitor /etc/fdfs/storage.conf
  #  It will show how many servers there will be   Yes 3 The stage will   Show  Storage 1-Storage 3 Details of 
  

• Operation related command

  Startup and shutdown tracker and storage

  sudo service fdfs_trackerd start		[stop]
  sudo service fdfs_storaged start		[stop]
  

  Startup and shutdown nginx

   cd  /usr/local/nginx/sbin/  Folder 
  
   sudo ./nginx  # start-up 
   sudo ./nginx -s stop # close 
   sudo ./nginx -s reload # Reload 
  

• Query whether the service is started

  ps -ef | grep tracker
  ps -ef | grep storage
  ps -ef | grep nginx
  

• Detection cluster

  /usr/bin/fdfs_monitor /etc/fdfs/storage.conf
  #  It will show how many servers there will be   Yes 3 The stage will   Show  Storage 1-Storage 3 Details of 
  

9.java Integrate fastDFS

• fastDFS The configuration file

  fdfs_client.conf

  connect_timeout = 60  # Connection timeout 
  network_timeout = 60  # Network timeout 
  charset = UTF-8
  http.tracker_http_port = 8080  #tracker  Access port 
  tracker_server = 192.168.200.128:22122 #traker storage  Installed on the host ip
  

• fastdfs-client-java Use

  fastdfs-client-java Use
  https://github.com/happyfish100/fastdfs-client-java

  Step one : Introduce dependencies

  <dependency>
  <groupId>com.taoyuanx</groupId>
  <artifactId>fastdfs-client-java</artifactId>
  <version>1.29-SNAPSHOT</version>
  </dependency
      
  

  Step two : java Code

  // Long transmission document 
  public class UploadDemo {
        
  	public static void main(String[] args) throws IOException, MyException {
        
          //1、 Load profile 
          ClientGlobal.init("fdfs_client.conf");
          //2、 Create a TrackerClient object .
          TrackerClient trackerClient = new TrackerClient();
          //3、 Use TrackerClient Object to obtain trackerserver object .
          TrackerServer trackerServer = trackerClient.getTrackerServer();
          //4、 Create a StorageClient object .trackerserver、StorageServer Two parameters .
          StorageClient storageClient = new StorageClient(trackerServer, null);
          String path = System.getProperty("user.dir")+ File.separator+"1.jpg";
          //5、 Use StorageClient Object upload file .
          String[] strings = storageClient.upload_file(path, "jpg", null);
          // The returned array contains two elements, each similar to ：
          //group1
          //M00/00/00/wKgDCmDtis2AAsj9AAW3ED-O1WQ151.jpg
          for (String string : strings) {
        
          System.out.println(string);
          }
  	}
  }
  
  // Download the file 
  public class DownloadDemo {
        
      public static void main(String[] args) {
        
              try {
        
              //1、 Load profile 
              ClientGlobal.init("fdfs_client.conf");
              //2、 Create a TrackerClient object .
              TrackerClient trackerClient = new TrackerClient();
              //3、 Use TrackerClient Object to obtain trackerserver object .
              TrackerServer trackerServer = trackerClient.getTrackerServer();
              //4、 Create a StorageClient object .trackerserver、StorageServer Two parameters .
              StorageClient storageClient = new StorageClient(trackerServer,
              null);
              String path = System.getProperty("user.dir")+
              File.separator+"fox.jpg";
              //5、 Use StorageClient Object Download File .
              storageClient.download_file("group1",
              "M00/00/00/wKgDDWDtRu6AMPhBARBlpcz7xUs146.jpg",path);
              } catch (IOException e) {
        
              e.printStackTrace();
              } catch (MyException e) {
        
              e.printStackTrace();
              }
      }
  }
  

• fastDFS_Client Use
  In the original author YuQing And yuqih released java A lot of refactoring work has been done on the basis of the client , Easy Java Workers learn and read .
  The main features
  \1. Unit tests have been added to key parts of the code , It is easy to understand the interface transaction with the server , Improve interface quality
  \2. Put the previous on byte The hard parsing style is refactored to use object + annotation In the form of , Try to enhance the readability of the code
  \3. Support connection pool management on the server (commons-pool2)
  \4. Support to check the image format when uploading images , And automatically generate thumbnails
  \5. stay SpringBoot Auto import dependency

  https://github.com/tobato/FastDFS_Client

• Upload and download pictures

  • Step one : Introduce dependencies

    <
    dependency>
    <groupId>com.github.tobato</groupId>
    <artifactId>fastdfs-client</artifactId>
    <version>1.27.2</version>
    </dependency>
    

  • Step two : stay yml To configure

    #  distributed file system FDFS To configure 
    fdfs:
    so-timeout: 1500 #  Read time 
    connect-timeout: 600 #  Connection timeout 
    thumb-image: # Thumbnail generation parameters 
    width: 150
    height: 150
    tracker-list: #Tracker Service configuration address list 
    - 192.168.3.10:22122
    - 192.168.3.12:22122
    - 192.168.3.13:22122
    pool:
    # The maximum number of objects loaned from the pool （ Configure to -1 Means unrestricted ）
    max-total: -1
    # Maximum number of milliseconds to wait for a connection ( Default configuration is 5 second )
    max-wait-millis: 5000
    # Every key maximum connection 
    max-total-per-key: 50
    # Every key The maximum number of free connections in the corresponding connection pool 
    max-idle-per-key: 10
    # Every key The minimum number of free connections in the corresponding connection pool 
    min-idle-per-key: 5
    upload:
    base-url: http://fastdfs.com:8888/
    

    

    Solution : add to config To configure

    @Configuration
    public class FileConfig {
            
    /** *  Solve the problem that the uploaded file is too large and causes exceptions . * the request was rejected because its size (170982031) exceeds the configured * @return */
    @Bean
    public MultipartConfigElement multipartConfigElement() {
            
        MultipartConfigFactory factory = new MultipartConfigFactory();
        factory.setMaxRequestSize(DataSize.ofBytes(200*1048576L));
        factory.setMaxFileSize(DataSize.ofBytes(200*1048576L));
        return factory.createMultipartConfig();
    }
    

  • Step three : web layer -service Layer code

    @RestController
    public class UploadController {
            
        @Autowired
        private UploadService uploadService;
        @RequestMapping("/upload")
        @ResponseBody
        public Map<String,Object> upload(MultipartFile file){
            
            Map<String, Object> map =new HashMap<>();
            String path = this.uploadService.uploadImage(file);
            map.put("path",path);
            return map;
        }
    }
    
    
    @Service
    @Slf4j
    public class UploadService {
            
        @Autowired
        private FastFileStorageClient fastFileStorageClient;
        @Value("${upload.base-url}")
        private String baseUrl;
        /** *  To upload pictures  * @param file * @return */
        public String uploadImage(MultipartFile file) {
            
        	try {
            
                // Verify File 
                BufferedImage image = ImageIO.read(file.getInputStream());
              	if (image == null || image.getWidth() == 0 || image.getHeight() ==0) {
            
                	throw new RuntimeException(" The uploaded file is not a picture ");
                }
            } catch (IOException e) {
            
                log.error(" Failed to verify file content ....{}", e);
                throw new RuntimeException(" Failed to verify file content " + e.getMessage());
            } 
            try {
            
                //  Get extension 
                String extension =
                StringUtils.substringAfterLast(file.getOriginalFilename(), ".");
          
                StorePath storePath =
                fastFileStorageClient.uploadFile(file.getInputStream(), file.getSize(),
                extension, null);
                //  Return path 
                return baseUrl+storePath.getFullPath();
              } catch (IOException e) {
            
                log.error("【 Upload files 】 Failed to upload file ！....{}", e);
                throw new RuntimeException("【 Upload files 】 Failed to upload file ！" + e.getMessage());
               
              }
    	}
    }
    

  • Upload thumbnail

    // Upload thumbnails 
    StorePath storePath =
    fastFileStorageClient.uploadImageAndCrtThumbImage(file.getInputStream(),
    file.getSize(), extension, null)
    

    The thumbnail is the upload file name + Thumbnail suffix ( Default _150x150)
    Such as : After uploading the source image, the path is xxx.jpg, The thumbnail is xxx_150x150.jpg

  • test

    • The test page index.html , The code is as follows :

      <form action="/upload" enctype="multipart/form-data">
          <input type="file" name="file" />
          <input type="submit" value=" To upload pictures ">
      </form>