Improving Release Management in GitOps

Safe and reliable software delivery process and release process always have been hard problem to tackle. Even though it's becoming better, but it's still problematic. As it involves state change and human/machine coordination.

Problems with Release Management

We thru our clients started facing issues with existing software delivery release management process as it was built atop on CI system pipelines.

Main issues were:

• Lack of visibility deployed artifacts between environments in multiproduct and application setup;

• Pipeline maintenance become burden as orchestartion complexity grew;

A Little Bit of Theory

GitOps software delivery process can be split in 3 large parts:

1. Building artifact

2. Artifact release/promotion

3. Delivery of runtime and continuous apply.

During the last 10 years, significant progress has been made in 1 and 3 areas. CI systems have made building/packaging much easier and runtime delivery/continuous apply by systems like ArgoCD. These two aspects have become standardized and commoditization

However, artifact promotion and release management tooling haven't seen such evolution.

Options What We Had on Hand

Release management in GitOps so far has had the following options:

1. Use internally developed solutions based on CI systems, like promotion pipelines or a fully custom release management solution. Caveats include that CI systems are not a natural fit for release management. Or full responsibility for additional software solution, in case of custom built solution.

2. Legacy release management tools. Caveats: partly or fully not suitable for GitOps process and most cases quite hefty entry costs from licensing.

3. SaaS platforms from vendors like Codefresh etc. Does the job, but sometimes too big for the job as these are platforms and includes many other tools/solutions.

4. Kargo

We decided to go with Kargo.

Kargo

From Akuity is the same team behind ArgoCD comes Kargo. It is release management tool or multi-stage GitOps continuous promotion. It plugs the gap in promotion tooling space for native GitOps promotions. Kargo it self is open source, there also is commrecial offering into Akuity platform.

Why Kargo for Us?

Following reasons were considered why we choose to go with the Kargo:

1. Proper visualization of artifact lifecycle and location, state change history;

2. Ability to integrate in our existing toolset (JIRA);

3. Ability do post deployment smokechecks after deployment;

4. Good configuration management, same principles as ArgoCD

5. Active product development and community

6. Kargo can be used for promoting not just

Workflow

Our current workflow for delivery is to:

1. Build artifact and store in container

2. Use Kargo to do promotion activities within Git repository

3. Sync with ArgoCD specific stage

4. Do post deployment smoke tests

5. If smoke tests pass, update issue tracking software with updated release info

Promotion actions are being initiated by Kargo, and we are working on fthe uture expansion of the solution

Deployment Model

We strongly believe in Conway's Law, that systems reflect the organization. From our experience, it has been true in 9 of 10 cases. That's why we started with on centralized Kargo that manages multiple products/argocd installs. If needed, we can easily expand to multiple instances.

Practical Example

For a practical example, we will create a simple, two-tier environment promotion scheme:

• Two environments: preprod and prod

• There is requirements to deploy before prod to preprod

• Preprod (with auto promotion enabled when new artifact created in artifact repository)

• Prod with manual promotion

• Container image tag format is YYYY-MM-HHMM_git_hash

• Deployment is based on helm repo

Steps to Create

1. Create project definition, in this case, we add auto promotion if new artifact created in the container registry. This will create a project object and a namespace named "sandbox-example". Kargo projects are namespaced objects

---
apiVersion: kargo.akuity.io/v1alpha1
kind: Project
metadata:
  name: sandbox-example
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  promotionPolicies:
    - stage: preprod
      autoPromotionEnabled: true
    - stage: prod
      autoPromotionEnabled: false

2. We create a warehouse object, which is either a configuration repo or a container registry to track new artifacts or configuration versions:

---
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: example-container
  namespace: sandbox-example
spec:
  subscriptions:
  - image:
      repoURL: us-east1-docker.pkg.dev/container/main-containers/example-container
      imageSelectionStrategy: Lexical
      allowTags: ^\d{4}-\d{2}-\d{2}_\d{4}-\w{7}$

3. Promotion task, what will act as a workflow for promotion

apiVersion: kargo.akuity.io/v1alpha1
kind: PromotionTask
metadata:
  name: promote
  namespace: sandbox-example
spec:
  vars:
  - name: image
    value: us-east1-docker.pkg.dev/container/main-containers/example-container
  - name: repoURL
    value: https://github.com/test/sandbox.git
  - name: branch
    value: master
  - name: envPath
  steps:
  - uses: git-clone
    config:
      repoURL: ${{ vars.repoURL }}
      checkout:
        - branch: ${{ vars.branch }}
          path: ./src
  - uses: yaml-update
    as: update-image
    config:
      path: ./src/${{ vars.envPath }}/values.yaml
      updates:
      - key: general.lmsImageTag
        value: ${{ imageFrom( vars.image ).Tag }}
  - uses: git-commit
    as: commit
    config:
      path: ./src
      message: ${{ task.outputs['update-image'].commitMessage }}
  - uses: git-push
    config:
      path: ./src							

4. Setup "preprod" and "prod" stages, referencing correct promotion task workflow and warehouse

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: preprod
  namespace: sandbox-example
  annotations:
    kargo.akuity.io/color: green
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: example-container
    sources:
      direct: true
  promotionTemplate:
    spec:
      steps:
      - task:
          name: promote
        vars:
        - name: envPath
          value: preprod
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: prod
  namespace: sandbox-example
  annotations:
    kargo.akuity.io/color: blue
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: example-container
    sources:
      stages:
        - preprod
  promotionTemplate:
    spec:
      steps:
      - task:
          name: promote
        vars:
        - name: envPath
          value: prod

5. That's it, you have got a working promotion in Kargo

   
   

Closing Thoughts

Kargo is solving the real problem of the release process. Its still in active development, but already usable and can replace many custom solutions. If ArgoCD is used as a GitOps tool, this is a considerable replacement for custom promotion pipelines.

Cons are that project is in active development, many features changes across the releases pretty rapidly