亲宝软件园·资讯

展开

部署 harbor 私有仓库

高枫_henu 人气:2

安装下载依赖包

  1. 安装docker-compose

    从 docker compose 发布页面下载最新的 docker-compose 二进制文件,本文以1.25.4为例

    cd /opt/k8s/work/harbor/
    
    wget https://github.comhttps://img.qb5200.com/download-x/docker/compose/releaseshttps://img.qb5200.com/download-x/download/1.25.4https://img.qb5200.com/download-x/docker-compose-Linux-x86_64
    
    mv docker-compose-Linux-x86_64 /opt/k8s/binhttps://img.qb5200.com/download-x/docker-compose
    chmod a+x  /opt/k8s/binhttps://img.qb5200.com/download-x/docker-compose
    export PATH=/opt/k8s/bin:$PATH
    
  2. 下载harbor二进制文件

    从 harbor 发布页面下载最新的 harbor 离线安装包,本文以v1.9.4为例

    cd /opt/k8s/work/harbor/
    
    wget https://github.com/goharbor/harbor/releaseshttps://img.qb5200.com/download-x/download/v1.9.4/harbor-offline-installer-v1.9.4.tgz
    
    tar -xzvf harbor-offline-installer-v1.5.1.tgz
    
  3. 导入 docker images
    导入离线安装包中 harbor 相关的 docker images

    cd /opt/k8s/work/harbor/harbor
    docker load -i harbor.v1.9.4.tar.gz
    

以http形式启动

  1. 修改harbor.yml文件
    修改 hostname、data_volume属性值

    cd /opt/k8s/work/harbor/harbor
    
    cp harbor.yml harbor.yml.bak
    
    vim harbor.yml
    

    修改信息如下

    diff harbor.yml harbor.yml.bak
    5c5
    < hostname: 192.168.0.107
    ---
    > hostname: reg.mydomain.com
    40c40
    < data_volume: https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data
    ---
    > data_volume: https://img.qb5200.com/download-x/data
    
  2. 加载和启动 harbor 镜像

    cd /opt/k8s/work/harbor/harbor
    mkdir -p https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data
    chmod 777 /var/runhttps://img.qb5200.com/download-x/docker.sock https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data
    ./install.sh
    

    启动日志

    
    Note: docker version: 18.09.6
    
    Note: docker-compose version: 1.25.4
    # Configuration file of Harbor
    
    [Step 1]: loading Harbor images ...
    Loaded image: goharbor/harbor-core:v1.9.4
    Loaded image: goharbor/clair-photon:v2.1.0-v1.9.4
    Loaded image: goharbor/harbor-portal:v1.9.4
    Loaded image: goharbor/nginx-photon:v1.9.4
    Loaded image: goharbor/chartmuseum-photon:v0.9.0-v1.9.4
    Loaded image: goharbor/prepare:v1.9.4
    Loaded image: goharbor/redis-photon:v1.9.4
    Loaded image: goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.4
    Loaded image: goharbor/notary-server-photon:v0.6.1-v1.9.4
    Loaded image: goharbor/harbor-log:v1.9.4
    Loaded image: goharbor/harbor-db:v1.9.4
    Loaded image: goharbor/harbor-jobservice:v1.9.4
    Loaded image: goharbor/harbor-registryctl:v1.9.4
    Loaded image: goharbor/notary-signer-photon:v0.6.1-v1.9.4
    Loaded image: goharbor/harbor-migrator:v1.9.4
    
    
    [Step 2]: preparing environment ...
    prepare base dir is set to /opt/k8s/work/harbor/harbor
    Generated configuration file: /config/log/logrotate.conf
    Generated configuration file: /config/log/rsyslog_docker.conf
    Generated configuration file: /config/nginx/nginx.conf
    Generated configuration file: /config/core/env
    Generated configuration file: /config/core/app.conf
    Generated configuration file: /config/registry/config.yml
    Generated configuration file: /config/registryctl/env
    Generated configuration file: /confighttps://img.qb5200.com/download-x/db/env
    Generated configuration file: /config/jobservice/env
    Generated configuration file: /config/jobservice/config.yml
    Generated and saved secret to file: /secret/keys/secretkey
    Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt
    Generated configuration file: /compose_locationhttps://img.qb5200.com/download-x/docker-compose.yml
    Clean up the input dir
    
    
    
    [Step 3]: starting Harbor ...
    Creating network "harbor_harbor" with the default driver
    Creating harbor-log ... done
    Creating harbor-db     ... done
    Creating redis         ... done
    Creating harbor-portal ... done
    Creating registry      ... done
    Creating registryctl   ... done
    Creating harbor-core   ... done
    Creating harbor-jobservice ... done
    Creating nginx             ... done
    
    ✔ ----Harbor has been installed and started successfully.----
    
    Now you should be able to visit the admin portal at http://192.168.0.107.
    For more details, please visit https://github.com/goharbor/harbor .
    

    查看启动状态

    root@master:/opt/k8s/work/harbor/harbor# docker-compose ps
        Name                     Command                  State                 Ports
    ---------------------------------------------------------------------------------------------
    harbor-core         /harbor/harbor_core              Up (healthy)
    harbor-db           https://img.qb5200.com/download-x/docker-entrypoint.sh            Up (healthy)   5432/tcp
    harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
    harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
    harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
    nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp
    redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
    registry            /entrypoint.sh /etc/regist ...   Up (healthy)   5000/tcp
    registryctl         /harbor/start.sh                 Up (healthy)                            
    
    • 如果有不是healthy状态的对象,到/var/log/harbor/目录下查看对应对象的日志
    • harbor官方提供的启动文件中容器的日志类型是syslog,不支持用docker logs查看

      
      ls /var/log/harbor/
      core.log  jobservice.log  portal.log  postgresql.log  proxy.log  redis.log  registryctl.log  registry.log
      

    浏览器访问 http://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。

  3. 创建一个新的项目

  4. docker命令拉取和上传镜像
    1. 执行login

      root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
      WARNING! Using --password via the CLI is insecure. Use --password-stdin.
      Error response from daemon: Get https://192.168.0.107/v2/: dial tcp 192.168.0.107:443: connect: connection refused
      
    2. 因为docker命令默认采用https和API交互,而我们的harbor是http的,所以不能执行,需要在/etchttps://img.qb5200.com/download-x/dockerhttps://img.qb5200.com/download-x/daemon.json中追加insecure-registries配置

      {
          "registry-mirrors": ["https:/https://img.qb5200.com/download-x/docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
          "insecure-registries": ["192.168.0.107"],
          "max-concurrent-downloads": 20,
          "live-restore": true,
          "max-concurrent-uploads": 10,
          "data-root": "https://img.qb5200.com/download-x/data/k8shttps://img.qb5200.com/download-x/dockerhttps://img.qb5200.com/download-x/data",
          "log-opts": {
            "max-size": "100m",
            "max-file": "5"
          }
      }
      
    3. 重新启动docker服务

      root@slave:~# systemctl restart docker
      
      root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
      WARNING! Using --password via the CLI is insecure. Use --password-stdin.
      WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
      Configure a credential helper to remove this warning. See
      https:/https://img.qb5200.com/download-x/docs.docker.com/engine/reference/commandline/login/#credentials-store
      
      Login Succeeded
      
    4. 向harbor中刚创建的私有项目里面上传镜像

      root@slave:~# docker tag k8s.gcr.io/pause:3.1 192.168.0.107/k8s/pause:3.1
      root@slave:~# docker push 192.168.0.107/k8s/pause:3.1
      The push refers to repository [192.168.0.107/k8s/pause]
      e17133b79956: Pushed
      3.1: digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d size: 527
      
    5. 浏览器查看

    6. 从harbor中下载镜像

      root@slave:~# docker rmi  192.168.0.107/k8s/pause:3.1
      Untagged: 192.168.0.107/k8s/pause:3.1
      Untagged: 192.168.0.107/k8s/pause@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      root@slave:~# docker pull  192.168.0.107/k8s/pause:3.1
      3.1: Pulling from k8s/pause
      Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      Status: Downloaded newer image for 192.168.0.107/k8s/pause:3.1
      

以https形式启动

下列操作的工作目录/opt/k8s/work/harbor/harbor是解压harbor离线安装文件后 生成的 harbor 目录

  1. 停止harbor(如果没有启动过,则跳过该步骤)

    cd /opt/k8s/work/harbor/harbor
    
    root@master:/opt/k8s/work/harbor/harbor# COMPOSE_HTTP_TIMEOUT=200 docker-compose down -v
    Stopping nginx             ... done
    Stopping harbor-jobservice ... done
    Stopping harbor-core       ... done
    Stopping registryctl       ... done
    Stopping redis             ... done
    Stopping harbor-portal     ... done
    Stopping harbor-db         ... done
    Stopping registry          ... done
    Stopping harbor-log        ... done
    Removing harbor-log        ... done
    Removing network harbor_harbor
    
    • 执行docker-compose down -v后,有可能对应的容器还在,可以执行强制停掉容器

      docker ps | grep busybox_busybox | awk '{print $1}' | xargs -I {} docker rm -f {}
      
  2. 创建 harbor nginx 服务器使用的 x509 证书,本文档采用cfssl工具生成,也可以利用openssl生成,具体步骤参考harbor-openssl

    1. 创建harbor用的CA根证书

      1. 创建配置文件

        cd /opt/k8s/work/harbor/
        cat > harbor-ca-config.json <<EOF
        {
          "signing": {
            "default": {
              "expiry": "87600h"
            },
            "profiles": {
              "harbor": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "87600h"
              }
            }
          }
        }
        EOF
        
      2. 创建证书签名请求文件

        cd /opt/k8s/work/harbor/
        
        cat > harbor-ca-csr.json <<EOF
        {
          "CN": "harbor-ca",
          "key": {
            "algo": "rsa",
            "size": 2048
          },
          "names": [
            {
              "C": "CN",
              "ST": "NanJing",
              "L": "NanJing",
              "O": "k8s",
              "OU": "system"
            }
          ],
          "ca": {
            "expiry": "87600h"
         }
        }
        EOF
        
      3. 生成自签名的根证书,分发证书

        cd /opt/k8s/work/harbor/
        
        cfssl gencert -initca harbor-ca-csr.json | cfssljson -bare harbor-ca
        
        ls harbor-ca*pem
        
        mkdir -p /etc/harbor/cert
        
        mv harbor-ca*pem harbor-ca-config.json  /etc/harbor/cert
        
    2. 创建harbor用的证书

      1. 生成证书请求文件

        cd /opt/k8s/work/harbor/
        
        cat > harbor-server-csr.json <<EOF
        {
          "CN": "harbor",
          "hosts": [
            "127.0.0.1",
            "192.168.0.107"
          ],
          "key": {
            "algo": "rsa",
            "size": 2048
          },
          "names": [
            {
              "C": "CN",
              "ST": "NanJing",
              "L": "NanJing",
              "O": "k8s",
              "OU": "system"
            }
          ]
        }
        EOF
        
        • hosts 字段指定授权使用该证书的当前部署节点 IP
      2. 生成证书

        
        cfssl gencert -ca=/etc/harbor/cert/harbor-ca.pem \
         -ca-key=/etc/harbor/cert/harbor-ca-key.pem \
         -config=/etc/harbor/cert/harbor-ca-config.json \
         -profile=harbor harbor-server-csr.json | cfssljson -bare harbor-server
        
        ls harbor-server*pem
        
        mv harbor-server*pem /etc/harbor/cert
        
  3. 编辑配置文件

    修改 hostname、data_volume属性值,去除默认的http方式,追加https的配置

    cd /opt/k8s/work/harbor/harbor
    
    vim harbor.yml
    

    修改内容如下

    diff harbor.yml harbor.yml.bak
    5c5
    < hostname: 192.168.0.107
    ---
    > hostname: reg.mydomain.com
    8c8
    < #http:
    ---
    > http:
    10c10
    <   #  port: 80
    ---
    >   port: 80
    13c13
    < https:
    ---
    > # https:
    15c15
    <    port: 443
    ---
    > #   port: 443
    17,18c17,18
    <    certificate: /etc/harbor/cert/harbor-server.pem
    <    private_key: /etc/harbor/cert/harbor-server-key.pem
    ---
    > #   certificate: /your/certificate/path
    > #   private_key: /your/private/key/path
    40c40
    < data_volume: https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data
    ---
    > data_volume: https://img.qb5200.com/download-x/data
    
  4. 生成配置文件(如果没有启动过,则跳过该步骤)

    cd /opt/k8s/work/harbor/harbor
    
    ./prepare
    

    执行日志

    prepare base dir is set to /opt/k8s/work/harbor/harbor
    Clearing the configuration file: /config/log/rsyslog_docker.conf
    Clearing the configuration file: /config/log/logrotate.conf
    Clearing the configuration file: /config/jobservice/config.yml
    Clearing the configuration file: /config/jobservice/env
    Clearing the configuration file: /confighttps://img.qb5200.com/download-x/db/env
    Clearing the configuration file: /config/registryctl/config.yml
    Clearing the configuration file: /config/registryctl/env
    Clearing the configuration file: /config/nginx/nginx.conf
    Clearing the configuration file: /config/registry/config.yml
    Clearing the configuration file: /config/registry/root.crt
    Clearing the configuration file: /config/core/app.conf
    Clearing the configuration file: /config/core/env
    Generated configuration file: /config/log/logrotate.conf
    Generated configuration file: /config/log/rsyslog_docker.conf
    Generated configuration file: /config/nginx/nginx.conf
    Generated configuration file: /config/core/env
    Generated configuration file: /config/core/app.conf
    Generated configuration file: /config/registry/config.yml
    Generated configuration file: /config/registryctl/env
    Generated configuration file: /confighttps://img.qb5200.com/download-x/db/env
    Generated configuration file: /config/jobservice/env
    Generated configuration file: /config/jobservice/config.yml
    loaded secret from file: /secret/keys/secretkey
    Generated configuration file: /compose_locationhttps://img.qb5200.com/download-x/docker-compose.yml
    Clean up the input dir
    

    修改生成文件的访问权限

    cd /opt/k8s/work/harbor/harbor
    chmod -R 777 common
    
  5. 启动harbor

    如果是首次启动 执行

    cd /opt/k8s/work/harbor/harbor
    
    mkdir -p https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data
    chmod 777 /var/runhttps://img.qb5200.com/download-x/docker.sock https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data
    ./install.sh
    

    否则执行

    docker-compose up -d
    
  6. 查看启动状态

    root@master:/opt/k8s/work/harbor/harbor# docker-compose ps
      Name                     Command                  State                          Ports
    ---------------------------------------------------------------------------------------------------------------
    harbor-core         /harbor/harbor_core              Up (healthy)
    harbor-db           https://img.qb5200.com/download-x/docker-entrypoint.sh            Up (healthy)   5432/tcp
    harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
    harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
    harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
    nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
    redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
    registry            /entrypoint.sh /etc/regist ...   Up (healthy)   5000/tcp
    registryctl         /harbor/start.sh                 Up (healthy)    
    
    • nginx 暴露了两个端口http的80和https的443,使用http访问80会自动重定向到https
  7. 浏览器访问 https://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。

    • 其中的k8s是在http模式下创建的项目,直接启动https的可参照上面创建
  8. docker命令拉取和上传镜像

    如果之前已经执行过docker login,配置修改后需要重新docker login,需要删除 /root/.docker/config.json文件

    如果在/etchttps://img.qb5200.com/download-x/dockerhttps://img.qb5200.com/download-x/daemon.json中配置过insecure-registries,需要去掉,并重新启动docker服务
    1. 执行login

      root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
      WARNING! Using --password via the CLI is insecure. Use --password-stdin.
      Error response from daemon: Get https://192.168.0.107/v2/: x509: certificate signed by unknown authority
      

      错误是说我们用的证书是自签名的证书,签发证书机构未经认证
      解决方法是将签署 harbor-server 证书的 CA 证书拷贝到 /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107 目录下(需要在所有要访问harbor的节点上都执行此操作)

      root@slave:~# mkdir -p /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107
      root@slave:~# scp root@192.168.0.107:/etc/harbor/cert/harbor-ca.pem /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107/ca.crt
      root@192.168.0.107's password:
      harbor-ca.pem                        100% 1306   283.7KB/s   00:00
      root@slave:~# ls /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107/
      ca.crt
      

      重新执行login

      root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
      WARNING! Using --password via the CLI is insecure. Use --password-stdin.
      WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
      Configure a credential helper to remove this warning. See
      https:/https://img.qb5200.com/download-x/docs.docker.com/engine/reference/commandline/login/#credentials-store
      
      Login Succeeded
      
  9. 上传镜像

    
    root@slave:~# docker images
    REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
    kubernetesui/metrics-scraper   v1.0.3              3327f0dbcb4a        2 weeks ago         40.1MB
    busybox                        latest              6d5fcfe5ff17        7 weeks ago         1.22MB
    coredns/coredns                1.6.6               cc4d8e8c6169        2 months ago        40.8MB
    k8s.gcr.io/pause               3.1                 da86e6ba6ca1        2 years ago         742kB
    192.168.0.107/k8s/pause        3.1                 da86e6ba6ca1        2 years ago         742kB
    kubeimage/pause                3.1                 da86e6ba6ca1        2 years ago         742kB
    nginx                          1.9.1               94ec7e53edfc        4 years ago         133MB
    root@slave:~# docker tag busybox:latest 192.168.0.107/k8s/busybox:latest
    root@slave:~# docker push 192.168.0.107/k8s/busybox:latest
    The push refers to repository [192.168.0.107/k8s/busybox]
    195be5f8be1d: Pushed
    latest: digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 size: 527
  10. 通过浏览器查看刚上传的镜像

  11. 从harbor下载镜像

    root@slave:~# docker rmi 192.168.0.107/k8s/busybox:latest
    Untagged: 192.168.0.107/k8s/busybox:latest
    Untagged: 192.168.0.107/k8s/busybox@sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6
    root@slave:~# docker pull 192.168.0.107/k8s/busybox:latest
    latest: Pulling from k8s/busybox
    Digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6
    Status: Downloaded newer image for 192.168.0.107/k8s/busybox:latest
    

kubernetes集群中使用私有的harbor镜像仓库

  1. kubelet配置

    kubernetes以pod为管理单元,而不是docker容器,在创建pod时,会先启动一个名为k8s.gcr.io/pause:3.1的镜像。

    该镜像存在于google的镜像仓库中,国内不能直接访问,需要FQ下载,如果集群新加节点,或者节点上这个镜像被清理掉,将导致pod不能启动

    一般做法是搭建私有镜像仓库,把这个镜像push到我们的私有镜像仓库,再配置kubelet使用我们私有仓库中的pause镜像

    具体做法是修改kubelet的配置文件,配置文件的位置通过/etc/systemd/system/kubelet.service中的--config属性指定
    例如

    --config=/etc/kubernetes/kubelet-config.yaml 
    

    编辑配置文件/etc/kubernetes/kubelet-config.yaml ,追加--pod-infra-container-image属性值

    --pod-infra-container-image: 192.168.0.107/k8s/pause:3.1
    

    重启kubelet服务

    systemctl restart kubelet
    
  2. 创建拉取镜像的secret
    如果宿主机上执行过docker login,成功后会生成认证文件/root/.docker/config.json,可用如下命令生成secret

    kubectl create secret generic registrykey \
    --from-file=.dockerconfigjson=/root/.docker/config.json \
    --type=kubernetes.iohttps://img.qb5200.com/download-x/dockerconfigjson
    
    • registrykey是生成secret的名称

    也可以直接在 kubectl中传入用户名和密码直接生成secret

    kubectl create secret docker-registry registrykey --docker-server=192.168.0.107 --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@harbor.com
    
    • registrykey是生成secret的名称
    • --docker-server 指定harbor的地址
    • --docker-username 登陆harbor的用户名
    • --docker-password 登陆harbor的密码
  3. 在pod中使用创建的secret

    cat > private-busybox.yml << EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: private-busybox
    spec:
      containers:
      - name: private-busybox
        image: 192.168.0.107/k8s/busybox:latest
        command:
          - sleep
          - "3600"
      imagePullSecrets:
      - name: registrykey
    EOF
    

    创建pod

    root@master:/opt/k8s/yml# kubectl create -f private-busybox.yml
    pod/private-busybox created
    root@master:/opt/k8s/yml# kubectl get pod
    NAME                                READY   STATUS    RESTARTS   AGE
    busybox                             1/1     Running   56         2d9h
    nginx-deployment-56f8998dbc-6b6qm   1/1     Running   0          2d9h
    private-busybox                     1/1     Running   0          5s
    

加载全部内容

相关教程
猜你喜欢
用户评论