Minio distributed file system learning notes

docker Stand alone installation

Server side

docker run -p 9000:9000 -p 50000:50000 --name minio \
-d --restart=always \
-e "MINIO_ROOT_USER=admin" \
-e "[email protected]" \
-v  /opt/software/minio/data:/data \
-v  /opt/software/config:/root/.minio \
minio/minio server --console-address ":50000" /data

client

wget http://dl.minio.org.cn/client/mc/release/linux-amd64/mc
#  Executive authority 
chmod +x mc
./mc --help
mv mc /usr/local/sbin/

Reference resources : https://juejin.cn/post/6988340287559073799

Distributed file system application scenario

• The storage requirements of massive unstructured data on the Internet
• E-commerce website ： Massive commodity pictures
• Video website ： Massive video files
• Network disk : Massive files
• Social networking sites ： Lots of pictures

Introduce

MinIO It's based on Apache License v2.0 Object storage service of open source protocol . It's Amazon compatible S3 Cloud storage service interface , Not It is often suitable for storing large amounts of unstructured data , For example, pictures 、 video 、 Log files 、 Backup data and containers / Virtual machine image, etc , and An object file can be any size , From several kb To maximum 5T Unequal . MinIO It's a very lightweight service , It can be easily combined with other applications , similar NodeJS, Redis perhaps MySQL.

Minio advantage

• Simple deployment : One single Binary files are everything , It can also support various platforms .
• minio Support mass storage , May press zone Expand ( primary zone Not affected by anything ), Supports the maximum size of a single object 5TB;
• compatible Amazon S3 Interface , Fully consider the needs and experience of developers ;
• Low redundancy and high tolerance of disk damage , The standard and highest data redundancy factor is 2( That is to store a 1M Data objects for , Actual occupancy
  Disk space is 2M). But at any time n/2 block disk Data can still be read out in case of damage (n For a set of erasure codes (Erasure
  Coding Set) Medium disk Number ). And this damage recovery is based on a single object , Not based on the entire storage volume .
  Excellent reading and writing performance

MinIO The basic concept of

• Object： Store in Minio The basic object of , Such as file 、 Byte stream ,Anything…
• Bucket： Used to store Object Logical space . Every Bucket The data is isolated from each other . For clients , It is equivalent to a top-level folder for storing files .
• Drive： The disk on which data is stored , stay MinIO Startup time , Pass in... As a parameter .Minio All object data in will be stored in Drive in .
• Set ： It's a group Drive Set , Distributed deployment automatically divides one or more clusters according to the cluster size Set , Every Set Medium Drive Distributed in different locations . An object is stored in a Set On （For example: {1…64} is divided into 4sets each of size 16.）
• An object is stored in a Set On
• A cluster is divided into multiple Set
• One Set Contains Drive The quantity is fixed , By default, it is automatically calculated by the system according to the cluster size
• One SET Medium Drive Distributed on different nodes as much as possible

command

• ls List files and folders .
• mb Create a bucket or folder .
• cat Show file and object content .
• pipe Will a STDIN Redirect to an object or file or STDOUT.
• share Generate... For sharing URL.
• cp Copy files and objects .
• mirror Mirror buckets and folders .
• find Find files based on parameters .
• diff Compare the differences between two folders or buckets .
• rm Delete files and objects .
• events Management object notification .
• watch Monitoring events for files and objects .
• policy Manage access policies .
• config management mc The configuration file .
• update Check for software updates .
• version Output version information .

To configure mc operation minio service

#  Inquire about mc host To configure 
mc config host ls
#  add to minio service 
mc config host add minio-server http://192.168.62.130:9000 admin [email protected]
#  Delete host
mc config host remove minio-server

Upload and download

#  Inquire about minio All on the service buckets( Documents and folders )
mc ls minio-server
#  Download the file 
mc cp minio-server/temp/xzb/1.jpg /tmp/
# Delete file 
mc rm minio-server/temp/xzb/1.jpg
# Upload files 
mc cp abc.text  minio-server/xzb/

Bucket management

#  establish bucket
mc mb minio-server/bucket01
#  Delete bucket
mc rb minio-server/bucket02
# bucket Not empty , You can force the deletion of   Use with caution 
mc rb --force minio-server/bucket01
# Inquire about bucket03 Disk usage 
mc du minio-server/bucket03

mc admin Use

MinIO Client（mc） Provides “ admin” Subcommands to your MinIO Deploy and perform administrative tasks .

service  The service restarts and stops all MinIO The server 
update  Update all MinIO The server 
info  Information display MinIO server information 
user  User management user 
group  Team management team 
policy MinIO Policy management policies defined in the server 
config  Configuration Management MinIO Server configuration 
heal  Repair MinIO Disks on the server , Buckets and objects 
profile  Profiles generate profile data for debugging 
top  Provided at the top MinIO Top statistics for 
trace  Tracking shows MinIO Server's http track 
console  Console display MinIO The server's console log 
prometheus Prometheus management Prometheus To configure 
kms kms perform KMS Manage operations 

User management

mc admin user --help
# A new user 
mc admin user add minio-server xzb
mc admin user add minio-server xzb 12345678
# To view the user 
mc admin user list minio-server
# Disable users 
mc admin user disable minio-server xzb
# Enable users 
mc admin user disable minio-server xzb
# View user information 
mc admin user info minio-server xzb
# Delete user 
mc admin user remove minio-server xzb

Strategic management

policy command , Used to add , Delete , List policies , Get information about the policy and for MinIO User settings policy on the server .

mc admin policy --help
# List MinIO All fixed policies on 
mc admin policy list minio-server
#  see plicy Information 
mc admin policy info minio-server readwrite

Add a new policy

Write a policy file /opt/software/minio/xzbTemp.json

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"s3:GetBucketLocation",
			"s3:GetObject"
		],
		"Resource": [
			"arn:aws:s3:::xzbTemp"
		]
	}, {
		"Effect": "Allow",
		"Action": [
			"s3:*"
		],
		"Resource": [
			"arn:aws:s3:::xzbTemp/*"
		]
	}]
}

xzbTemp.json Add to policy database

# Add a new policy , bucket01 This bucket of resources xzb It can be authorized to use

mc admin policy add minio-server bucket01 /opt/software/minio/xzbTemp.json
mc admin policy list minio-server
mc admin user add minio-server xzb 12345678
# Set the user's access policy
mc admin policy set minio-server bucket01 user=xzb

Minio Java Client Use

MinIO Java Client SDK Provide simple API To access any and Amazon S3 Compatible object storage services .
official demo: https://github.com/minio/minio-java
Official documents ：https://docs.min.io/docs/java-client-api-reference.html

 rely on : 
<dependency>
	<groupId>io.minio</groupId>
	<artifactId>minio</artifactId>
	<version>8.3.0</version>
</dependency>
<dependency>
	<groupId>me.tongfei</groupId>
	<artifactId>progressbar</artifactId>
	<version>0.5.3</version>
</dependency>
<dependency>
	<groupId>com.squareup.okhttp3</groupId>
	<artifactId>okhttp</artifactId>
	<version>4.8.1</version>
</dependency>

Upload files

public class FileUploader {
    
	public static void main(String[] args)
	throws IOException, NoSuchAlgorithmException, InvalidKeyException {
    
		try {
    
			// Create a minioClient with the MinIO server playground, its access key
			and secret key. MinioClient minioClient =
			MinioClient.builder()
			.endpoint("http://192.168.62.130:9000")
			.credentials("admin", "[email protected]")
			.build();
			//  establish bucket
			String bucketName = "bucket01";
			Boolean exists =
			minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
			if (!exists) {
    
				//  non-existent , establish bucket
				minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
			}
			//  Upload files 
			minioClient.uploadObject(
			UploadObjectArgs.builder()
			.bucket(bucketName)
			.object("1.txt") //  Upload object name 
			.filename("D:\1.txt") //  Uploaded local files 
			.build());
			System.out.println(" Upload file successful ");
		}
		catch (MinioException e) {
    
			System.out.println("Error occurred: " + e);
			System.out.println("HTTP trace: " + e.httpTrace());
		}
	}
}

File download

public class DownLoadDemo {
    
	public static void main(String[] args) {
    
		// Create a minioClient with the MinIO server playground, its access key
		and secret key. MinioClient minioClient =
		MinioClient.builder()
		.endpoint("http://192.168.62.130:9000")
		.credentials("admin", "[email protected]")
		.build();
		// Download object given the bucket, object name and output file name
		try {
    
			minioClient.downloadObject(
			DownloadObjectArgs.builder()
			.bucket("bucket01")
			.object("temp/1.jpg")
			.filename("1.jpg")
			.build());
		}
		catch (Exception e) {
    
			e.printStackTrace();
		}
	}
}

3.3 Spring boot Integrate minio

structure MinioClient object , And to spring management

//yml
minio:
endpoint: http://192.168.62.130:9000
accesskey: admin
secretKey: [email protected]

@Data
@Component
@ConfigurationProperties(prefix = "minio")
public class MinioProperties {
    
	private String endpoint;
	private String accessKey;
	private String secretKey;
}

    
@Configuration
public class MinioConfig {
    
	@Autowired
	private MinioProperties minioProperties;
    @Bean
    public MinioClient minioClient(){
    
        MinioClient minioClient =
        MinioClient.builder()
        .endpoint(minioProperties.getEndpoint())
        .credentials(minioProperties.getAccessKey(),
        minioProperties.getSecretKey())
        .build();
        return minioClient;
    }
}

File upload , download , Delete operation

@RestController
@Slf4j
public class MinioController {
    
	@Autowired
	private MinioClient minioClient;
	@Value("${minio.bucketName}")
	private String bucketName;
	@GetMapping("/list")
	public List<Object> list() throws Exception {
    
		// obtain bucket list 
		Iterable<Result<Item>> myObjects = minioClient.listObjects(
		ListObjectsArgs.builder().bucket(bucketName).build());
		Iterator<Result<Item>> iterator = myObjects.iterator();
		List<Object> items = new ArrayList<>();
		String format = "{'fileName':'%s','fileSize':'%s'}";
		while (iterator.hasNext()) {
    
			Item item = iterator.next().get();
			items.add(JSON.parse(String.format(format, item.objectName(),
			formatFileSize(item.size()))));
		}
		return items;
	}
    
	@PostMapping("/upload")
	public Res upload(@RequestParam(name = "file", required = false)
	MultipartFile[] file) {
    
		if (file == null || file.length == 0) {
    
			return Res.error(" Upload file cannot be empty ");
		}
		List<String> orgfileNameList = new ArrayList<>(file.length);
		for (MultipartFile multipartFile : file) {
    
			String orgfileName = multipartFile.getOriginalFilename();
			orgfileNameList.add(orgfileName);
			try {
    
				// Upload files 
				InputStream in = multipartFile.getInputStream();
				minioClient.putObject(
				PutObjectArgs.builder().bucket(bucketName).object(orgfileName).stream(
				in, multipartFile.getSize(), -1)
				.contentType(multipartFile.getContentType())
				.build());
				in.close();
			}
			catch (Exception e) {
    
				log.error(e.getMessage());
				return Res.error(" Upload failed ");
			}
		}
		Map<String, Object> data = new HashMap<String, Object>();
		data.put("bucketName", bucketName);
		data.put("fileName", orgfileNameList);
		return Res.ok(" Upload successful ",data);
	}
	@RequestMapping("/download/{fileName}")
	public void download(HttpServletResponse response, @PathVariable("fileName")
	String fileName) {
    
		InputStream in = null;
		try {
    
			//  Get object information 
			StatObjectResponse stat = minioClient.statObject(
			StatObjectArgs.builder().bucket(bucketName).object(fileName).build());
			response.setContentType(stat.contentType());
			response.setHeader("Content-Disposition", "attachment;filename=" +
			URLEncoder.encode(fileName, "UTF-8"));
			// File download 
			in = minioClient.getObject(GetObjectArgs.builder()
			.bucket(bucketName)
			.object(fileName)
			.build());
			IOUtils.copy(in, response.getOutputStream());
		}
		catch (Exception e) {
    
			log.error(e.getMessage());
		}
		finally {
    
			if (in != null) {
    
				try {
    
					in.close();
				}
				catch (IOException e) {
    
					log.error(e.getMessage());
				}
			}
		}
	}
	@DeleteMapping("/delete/{fileName}")
	public Res delete(@PathVariable("fileName") String fileName) {
    
		try {
    
			minioClient.removeObject(
			RemoveObjectArgs.builder().bucket(bucketName).object(fileName).build());
		}
		catch (Exception e) {
    
			log.error(e.getMessage());
			return Res.error(" Delete failed ");
		}
		return Res.ok(" Delete successful ",null);
	}
	private static String formatFileSize(long fileS) {
    
		DecimalFormat df = new DecimalFormat("#.00");
		String fileSizeString = "";
		String wrongSize = "0B";
		if (fileS == 0) {
    
			return wrongSize;
		}
		if (fileS < 1024) {
    
			fileSizeString = df.format((double) fileS) + " B";
		} else if (fileS < 1048576) {
    
			fileSizeString = df.format((double) fileS / 1024) + " KB";
		} else if (fileS < 1073741824) {
    
			fileSizeString = df.format((double) fileS / 1048576) + " MB";
		} else {
    
			fileSizeString = df.format((double) fileS / 1073741824) + " GB";
		}
		return fileSizeString;
	}
}