Configure GitLab Runner with Ceph S3

2021-03-26

When building frontend projects with npm, it’s common for dependency downloads to take a long time, and reusing artifacts or caches across different jobs is also challenging. Whether using artifacts or cache, we ultimately need persistent reuse of files. Here, we’ll use cache as an example.

Note: The GitLab Runner is deployed to the Kubernetes cluster via Helm chart (deployment details are omitted). You must prepare a Ceph S3 key pair in advance for configuring accesskey and secretkey.

Automated ECS Creation with Terraform

wanzi

2021-02-26

Quickly Create an Alibaba Cloud ECS Instance

Specify Terraform Version

Here, we specify the Alibaba Cloud provider version and set the required Terraform version.

# mkdir aliyun-ecs-one && cd aliyun-ecs-one
# touch versions.tf
# vim versions.tf
terraform {
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud"
      version = "1.115.1"
    }
  }

  required_version = ">= 0.12"
}

Configure Variables

Here we define key pairs, cloud region, ECS account, and image information.

Terraform Installation and Command Reference

wanzi

2021-02-25

Installing Terraform

Installing on Mac

1
2

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Installing on Linux

Ubuntu installation

1
2
3

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform

CentOS installation

1
2
3

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

Verifying Installation

# terraform -v
Terraform v0.14.3

Your version of Terraform is out of date! The latest version
is 0.14.7. You can update by downloading from https://www.terraform.io/downloads.html
# terraform
Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

All other commands:
  console       Try Terraform expressions at an interactive command prompt
  fmt           Reformat your configuration in the standard style
  force-unlock  Release a stuck lock on the current workspace
  get           Install or upgrade remote Terraform modules
  graph         Generate a Graphviz graph of the steps in an operation
  import        Associate existing infrastructure with a Terraform resource
  login         Obtain and save credentials for a remote host
  logout        Remove locally-stored credentials for a remote host
  output        Show output values from your root module
  providers     Show the providers required for this configuration
  refresh       Update the state to match remote systems
  show          Show the current state or a saved plan
  state         Advanced state management
  taint         Mark a resource instance as not fully functional
  untaint       Remove the 'tainted' state from a resource instance
  version       Show the current Terraform version
  workspace     Workspace management

Global options (use these before the subcommand, if any):
  -chdir=DIR    Switch to a different working directory before executing the
                given subcommand.
  -help         Show this help output, or the help for a specified subcommand.
  -version      An alias for the "version" subcommand.

Terraform Commands for Resource Management

Initializing Resources

For a Terraform project, I created three basic files: main.tf (entry file), variables.tf (variable definitions), and versions.tf (version information).

Introduction to the Automation Orchestration Tool Terraform

wanzi

2021-02-24

What is Terraform?

Terraform is an open-source infrastructure orchestration tool introduced by HashiCorp around 2014. It is now supported by nearly all major cloud service providers, including Alibaba Cloud, Tencent Cloud, Huawei Cloud, AWS, Azure, Baidu Cloud, and more. Many companies today build their infrastructure using Terraform.

Background: In traditional operations, launching a business required multiple preparatory steps such as hardware procurement, server rack mounting, network setup, and system installation. With the rise of cloud computing, major public cloud providers offer user-friendly graphical interfaces—users can purchase various cloud resources via a browser and quickly set up their architecture. However, as business architectures expand, the scale and variety of cloud resource procurement continue to grow. When users need to rapidly acquire large numbers of diverse cloud resources, the numerous interactive operations across cloud management consoles actually reduce procurement efficiency. For example, initializing a classic VPC network on the Alibaba Cloud console—from creating the VPC and VSwitches to setting up NAT gateways, elastic IPs, and routing configurations—can take 20 minutes or even longer. Moreover, the non-reproducible nature of manual work leads to redundant efforts when operating across regions or multi-cloud environments.

Automated Application Deployment with ArgoCD and Jenkins Pipeline

wanzi

2020-07-29

Create Helm Repository

First, create a basic Helm template repository:

`1`	`helm create template .`

For actual deployments, you’ll need to customize the Helm template according to your business requirements. Here, we directly use an internal custom generic template for rapid deployment. Alternatively, you can refer to Bitnami’s maintained Helm charts: https://github.com/bitnami/charts/tree/master/bitnami

Jenkins Credential Configuration: ArgoCD Token

argocd and jenkins

Configure Jenkins Pipeline

We’ll use the gotest project (https://code.test.cn/hqliang/gotest) as an example.

When deploying a Deployment in Argo CD, you encounter the error: `no space left on device`.

wanzi

2020-05-18

Failure Phenomenon

This morning, I deployed several business applications via ArgoCD. After successfully deploying two applications, subsequent deployments from a third-party source consistently failed—despite using identical configurations, only the target cluster differed. Why would this happen?

I checked the logs and found the following:

1
2
3

  Warning  Failed     1m                kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/ba37165607862efb350093e5e287207e2547759fd81dc4e5e356a86ac5e28324-init/merged: no space left on device
  Warning  Failed     1m                kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/f69b62f360fc2a94487aca041b08d0929810beab0602e0ec8b90c94b2e893337-init/merged: no space left on device
  Warning  Failed     48s               kubelet, 172.16.25.13  Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/a8d20a44183b39ae989eee8a442960124ff23844482f726ea7ab39a292aecbb3-init/merged: no space left on device

Solution

Check disk space—no issues found:

1
2
3

root@gpu613:~# df -Th /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda2      ext4  1.8T  359G  1.3T  22% /

After Googling, I discovered this might be due to exhausted inotify watches.

Check current limit:

Add multiple clusters to ArgoCD

wanzi

2020-05-05

Generate Argo CD Management User Token

Log in to the dashboard, go to Settings → Accounts → admin → Generate New.
After generation, record the token information, which looks like this:

fyJhbGciOiJ3UzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI2OWI0M2M0Mi01MmZiLTRlZmItODIxOC0yOWU3NGM5MWI0NDIiLCJpYXQiOjE1OTUzMTEx3zQsImlzcyI6ImFyZ29jZCIsIm5iZiI6MTU5NTMxMTE3NCwic3ViIjoib3duZXIifQ.9u4XzArEeaz7G2Q2TWusnTkakEmq9BYDAUHr3dC6wG5

Configure Argo CD Config

For Argo CD with HTTPS enabled, adding clusters becomes cumbersome—it requires logging into the server pod for configuration. Follow these steps:

# cat ~/.argocd/config
contexts:
- name: argocd-server.argocd
  server: qacd.test.cn
  user: argocd-server.argocd
current-context: argocd-server.argocd
servers:
- grpc-web-root-path: ""
  insecure: true
  server: qacd.test.cn
users:
- auth-token: xxxxxx # This is the token generated in step 1
  name: argocd-server.argocd

Configure kubeconfig

Skip detailed configuration here—refer to previous documentation. Ensure you can access the cluster and have cluster administrator privileges. Set the CONTEXT to idc-bj-k8s.

ArgoCD Installation and Deployment

wanzi

2020-05-01

Installation and Deployment

Deploying ArgoCD is straightforward. Use the official high-availability (HA) deployment method:

1
2

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.5.2/manifests/ha/install.yaml

You can customize the deployment file as needed. After the pods are successfully started:

# kubectl -n argocd get pod
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-66fbf66657-ghf2c   1/1     Running   0          6d17h
argocd-application-controller-66fbf66657-gpm7d   1/1     Running   0          6d17h
argocd-application-controller-66fbf66657-tr5kd   1/1     Running   0          6d17h
argocd-dex-server-5c5f986596-c8ftv               1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-2fxd6         1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-mksg2         1/1     Running   0          9d
argocd-redis-ha-haproxy-69c6df79c6-wq57f         1/1     Running   0          9d
argocd-redis-ha-server-0                         2/2     Running   0          9d
argocd-redis-ha-server-1                         2/2     Running   0          9d
argocd-redis-ha-server-2                         2/2     Running   0          9d
argocd-repo-server-76bbb56cc7-d8fp5              1/1     Running   0          7d
argocd-repo-server-76bbb56cc7-qvl5z              1/1     Running   0          7d
argocd-repo-server-76bbb56cc7-xqrfn              1/1     Running   0          7d
argocd-server-6464c7bcd-fgktr                    1/1     Running   0          6d19h
argocd-server-6464c7bcd-jkqdb                    1/1     Running   0          6d19h
argocd-server-6464c7bcd-nfdwn                    1/1     Running   0          6d19h

Configure Ingress for ArgoCD Access

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/redirect-entry-point: https
spec:
  rules:
    - host: cd.testcn
      http:
        paths:
        - backend:
            serviceName: argocd-server
            servicePort: https
          path: /

Access ArgoCD via https://cd.test.cn/. The default username is admin, and the password is the name of one of the pods. Retrieve the password using: