Tencent cloud harbor private warehouse deployment practice

Harbor By VMware Open source enterprise level Docker Registry Management project , comparison docker The authorities have more authority and perfect architecture design , Applicable to large scale docker Cluster deployment provides warehouse services , And to provide UI Interface . commonly harbor Can pass helm perhaps docker-compose install , This article takes compose For example, installation , Introduce harbor How to configure Tencent cloud object storage COS As a private image warehouse storage address .

One Deployment process

1 Get ready ：

install docker-compose

curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

2 restart docker

systemctl daemon-reload

3 download habor Unpacking harbor.v2.1.0.tar.gz

3.1 Configuration item note ：

• hostname Modify host address , That is to access the domain name
• https increase HTTPS Certificate configuration , Note that if used CLB, Need to be in CLB Configure certificates at the same time
• storage_service Middle configuration COS Information , Be careful harbor Support aws S3, Can be in s3 Middle configuration COS Bucket information , The corresponding bucket information needs to be created in advance in the object storage
• Be careful harbor Log path ： Default /var/log/harbor/registryctl.log

# harbor.v2.1.0.tar.gz
vi harbor.yml
-----------harbor.yml---------
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.yourset.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/key/harbor.yourset.com.crt
  private_key: /data/key/harbor.yourset.xyz.key

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harborxxx

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 50
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 1000

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
storage_service:
  s3:
     accesskey: xxxxxx
     secretkey: xxxxxxx
     regionendpoint: cos.ap-guangzhou.myqcloud.com
     region: ap-guangzhou
     bucket: harborgz-xxxxx
     secure: true
   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.

#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

###....

4 start-up , Enter the installation directory

$ docker-compose down -v

#  After modifying the configuration , The operating environment needs to be rechecked 
$ ./prepare

#  start-up harbor service 
$ docker-compose up -d

5 test

5.1 Web Testing , if necessary https visit , You need to apply for a certificate and configure it in CLB And harbor Server

Configure in harbor.yml Of https in ：

-----------------
  https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/key/harbor.yourset.com.crt
  private_key: /data/key/harbor.yourset.xyz.key

5.2 Local testing , Input password

5.3 Test push image

matters needing attention ： Need to be in harbor Create the project first , Otherwise, the push will fail

[[email protected] ~/tmp]# docker tag hello-world 127.0.0.1/s3/hello-world:v1.0.0 
[[email protected] ~/tmp]# docker push 127.0.0.1/s3/hello-world:v1.0.0
The push refers to repository [127.0.0.1/s3/hello-world]
f22b99068db9: Preparing 
unauthorized: project not found, name: s3: project not found, name: s3
[[email protected] ~/tmp]# docker push 127.0.0.1/s3/hello-world:v1.0.0
The push refers to repository [127.0.0.1/s3/hello-world]
f22b99068db9: Pushed 
v1.0.0: digest: sha256:1b26826f602946860c279fce65829b57792 size: 525

5.4 Related files are also generated in the object store ：

Two Record on pit

2.1 A pit ：S3 Configuration of stepping pit

Articles searched on the Internet and other user feedback S3 Part of the configuration is as follows ：

s3:
      region: ap-xxx
      bucket: xx-sigp-xxxxxxx
      accesskey: xxxxxxx
      secretkey: xxxxx
      endpoint: cos.ap-singapore.myqcloud.com
      secure: true

After using this configuration , start-up harbor There will always be harbor-registryclt Wait for the container to restart , The image cannot be pushed or pulled ：

Check the error log ：

tail -f /var/log/harbor/registryctl.log 

Aug  3 15:32:31 172.30.0.1 registryctl[28778]: 2021-08-03T07:32:31Z [ERROR] [/registryctl/config/config.go:63]: failed to load storage driver, err:No region parameter provided
Aug  3 15:32:31 172.30.0.1 registryctl[28778]: 2021-08-03T07:32:31Z [FATAL] [/registryctl/main.go:78]: Failed to load configurations with error: No region parameter provided

The key information ：

Ask several colleagues, but there is no conclusion , stay google After checking for a long time, I couldn't find the relevant documents , I had to decide to start with the source code , First check registryctl/main.go Source code ：

Keep looking at config.go:63

https://github.com/goharbor/harbor/blob/9e117539492b9e54658b8c4dd240af231c351cb5/src/registryctl/config/config.go#L71

see setStorageDriver()

find storagedriver in s3 Related code

see s3 part ：

https://github.com/distribution/distribution/blob/01f589cf8726565aa3c5c053be12873bafedbedc/registry/storage/driver/s3-aws/s3.go#L109

Find a special place ：

When "regionendpoint" It's empty time , The program will go aws The official validRegins Query the availability zone in the list , This time, we will configure Tencent cloud COS Address , Of course. aws Of region There is no... In the list , So it will prompt err:No region parameter provided.

So you need to pass in "regionendpoint" Of key To avoid queries aws Their own region list（ The articles on the Internet are misleading ）, Instead of passing in "endpoint"

, So you need to be in harbor.yml Change the configuration to ：

s3:
      region: ap-xxx
      bucket: xx-sigp-xxxxxxx
      accesskey: xxxxxxx
      secretkey: xxxxx
      regionendpoint: cos.ap-singapore.myqcloud.com
      secure: true

Reload after modification harbor Successful launch , The push-pull image is normal .

2.2 Pit two ：COS Strong consistent configuration step pit

After users deploy according to the above , New errors found , The phenomenon is through docker push File successfully , however harbor Always return to 500 Report errors ：

Process and COS Team communication , Mainly list Strong consistency problem , namely put Direct after file list Not necessarily list To the file , because list It's ultimately consistent , need COS After the operation and maintenance colleagues issue strong and consistent configurations , Will ensure that list come out .

The customer account number shall be provided for the distribution configuration appid、 Availability zone 、 Bucket name information （ You can submit work orders to Tencent cloud ）, After publishing the configuration, the error can be solved .

3、 ... and summary

1 The deployment process records its own operation process in time , Follow the log

2 If there is no ready-made answer on the Internet , Ask the experts around you

3 It can't be solved , Check the source code , All the logic has been written in the code