# Use Sealer Image Offline

# Motivations

It's common that some k8s clusters have their own private image registry, and they don't want to pull images from other registry for some reasons. This page is about how to integrate kyverno into k8s cluster, which will redirect image pull request to Specified registry.

# How to use Kyverno BaseImage

We provide official BaseImage which integrates kyverno into cluster.

Note that these contains no docker images other than those necessary to run a k8s cluster, so if you want to use these cloud image, and you also need other docker images(such as nginx) to run a container, you need to cache the docker images to your private registry.

Of course sealer can help you do this,use nginx and kubernetes-kyverno:v1-22-15-sealerio-1 as an example.

# step1: create the build content

We prepare the following nginx.yaml file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: my-nginx
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
        - name: my-nginx
          image: docker.io/library/nginx:latest
          ports:
            - containerPort: 80

Note: The address of the image to be cached needs to be filled in completely, image address such as nginx:latest are temporarily not supported now.

Create a kyvernoNginxBuild directory with four files: the nginx.yaml and a Kubefile which content is following:

FROM docker.io/sealerio/kubernetes-kyverno:v1-22-15-sealerio-1
APP nginx local://nginx.yaml
APPCMDS kyverno ["sh redirect-registry.sh -r sea.hub:5000"]
LAUNCH ["calico", "kyverno", "nginx"]

# step2: build the image

Supposing you are at the kyvernoNginxBuild directory, please execute sealer build --no-cache -t kubernetes-kyverno-nginx:v1 .

# step3: run the image

Use the sealer run command to deploy:

sealer run localhost/kubernetes-kyverno-nginx:v1 -m 10.0.0.238 -n 10.0.0.239 10.0.0.240 -p Sealer.io@123

# How to use Kyverno AppImage

We also provide official Kyverno AppImage, you can use Kyverno AppImage when your k8s cluster don't integrate with kyverno.

Due to kyverno's compatibility requirements with kubernetes (shown below), we need to pick the corresponding version of kyverno.

You can find a more detailed compatibility matrix here.

* Due to a known issue with Kubernetes 1.23.0-1.23.2, support for 1.23 begins at 1.23.3.

Note that these contains no docker images other than kyverno, so if you want to use these, and you also need other docker images(such as nginx) to run a container, you need to cache the docker images to your private registry.

Of course sealer can help you do this,use nginx and docker.io/sealerio/scratch-kyverno:v1-7-5-sealerio-1 as an example.

# step1: create the build content

We prepare the following nginx.yaml file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: my-nginx
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
        - name: my-nginx
          image: docker.io/library/nginx:latest
          ports:
            - containerPort: 80

Note: The address of the image to be cached needs to be filled in completely, image address such as nginx:latest are temporarily not supported now.

Create a kyvernoNginxBuild directory with four files: the nginx.yaml and a Kubefile which content is following:

FROM docker.io/sealerio/scratch-kyverno:v1-7-5-sealerio-1
APP nginx local://nginx.yaml
APPCMDS kyverno ["sh redirect-registry.sh -r sea.hub:5000"]
LAUNCH ["kyverno", "nginx"]

# step2: build the image

Supposing you are at the kyvernoNginxBuild directory, please execute sealer build --type=app-installer --no-cache -t kyverno-nginx:v1 .

# step3: run the image

Note: If your cluster already has kyverno installed, please uninstall the previous kyverno first.

Use the sealer run command to deploy:

sealer run localhost/kyverno-nginx:v1

# How to build Kyverno BaseImage

The following is a sequence steps of building kyverno build-in sealer image.

# step1: choose a base image

Choose a base image which can create a k8s cluster with at least one master node and one work node. You can find the Official docker version of the image to build the kyverno base image: here

To demonstrate the workflow, we use docker.io/sealerio/kubernetes:v1.22.15-sealerio-2-official-docker-1. You can get the same image by executing sealer pull docker.io/sealerio/kubernetes:v1.22.15-sealerio-2-official-docker-1.

# step2: get the kyverno install yaml

Due to kyverno's compatibility requirements with kubernetes (shown below), we need to pick the corresponding version of kyverno based on the version of the base image we have selected.

You can find a more detailed compatibility matrix here.

* Due to a known issue with Kubernetes 1.23.0-1.23.2, support for 1.23 begins at 1.23.3.

1.6.x: https://raw.githubusercontent.com/kyverno/kyverno/release-1.6/config/release/install.yaml

1.7.x: https://raw.githubusercontent.com/kyverno/kyverno/release-1.7/config/release/install.yaml

1.8.5: https://github.com/kyverno/kyverno/releases/download/v1.8.5/install.yaml

You can also find the install.yaml for the version of kyverno you want here: https://github.com/kyverno/kyverno/releases

Because we are using a base image with Kubernetes version v1.22.15, we can use 1.6.x or 1.7.x versions of kyverno.

# step3: create a ClusterPolicy

Create a yaml with the following content:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: replace-image-registry
spec:
  background: false
  rules:
    - name: replace-image-registry-pod-containers
      match:
        any:
        - resources:
            kinds:
            - Pod
      preconditions:
        all:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
      mutate:
        foreach:
        - list: "request.object.spec.containers"
          patchStrategicMerge:
            spec:
              containers:
              - name: "{{ element.name }}"
                image: "{{ regex_replace_all_literal('^[^/]+', '{{element.image}}', 'sea.hub:5000' )}}"
    - name: replace-image-registry-pod-initcontainers
      match:
        any:
        - resources:
            kinds:
            - Pod
      preconditions:
        all:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
        - key: "{{ request.object.spec.initContainers[] || '' | length(@) }}"
          operator: GreaterThanOrEquals
          value: 1
      mutate:
        foreach:
        - list: "request.object.spec.initContainers"
          patchStrategicMerge:
            spec:
              initContainers:
              - name: "{{ element.name }}"
                image: "{{ regex_replace_all_literal('^[^/]+', '{{element.image}}', 'sea.hub:5000' )}}"

This ClusterPolicy will redirect image pull request to private registry sea.hub:5000, and we can name this file as redirect-registry.

# step4: create a shell script

we use this script to change the address of the mirror repository in install.yaml and redirect-registry, install kyverno and create policy.

#!/bin/bash

function usage {
   cat << EOF
Usage: redirect-registry.sh -r <your private registry>
EOF
   exit 1
}

registry="sea.hub:5000"

if [ $# -gt 2 ]; then
  usage;
elif [ $# -eq 2 ]; then
  if [ "$1" != "-r" ]; then
    usage
  fi
  registry=$2
fi

sed -i "s/ghcr.io/${registry}/g" ./install.yaml
sed -i "s/sea.hub:5000/${registry}/g" ./redirect-registry
cat >>install.yaml<<EOF
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: In
                values:
                - ""
EOF

kubectl create -f install.yaml
kubectl create -f redirect-registry

echo "[kyverno-start]: Waiting for the kyverno to be ready..."

while true
do
    clusterPolicyStatus=`kubectl get cpol -o go-template --template={{range.items}}{{.status.ready}}{{end}}`;
    if [ "$clusterPolicyStatus" == "true" ];then
        break;
    fi
    sleep 1
done

echo "kyverno is running"

I named this file redirect-registry.sh.

# step5: create the build content

Create a kyvernoBuild directory with four files: the install.yaml in step 2, redirect-registry in step 3, redirect-registry.sh in step4 and a Kubefile which content is following:

FROM docker.io/sealerio/kubernetes:v1.22.15-sealerio-2-official-docker-1
LABEL "app.alpha.sealer.io/image-policy-plugin"="kyverno"
COPY install.yaml application/apps/kyverno/
COPY redirect-registry application/apps/kyverno/
APP kyverno local://redirect-registry.sh
APPCMDS kyverno ["sh redirect-registry.sh -r sea.hub:5000"]
LAUNCH ["calico", "kyverno"]

If the private repository you want to use is not sea.hub:5000, you will also need to change sea.hub:5000 to the private repository address you want to use.

# step6: build the image

Supposing you are at the kyvernoBuild directory, please execute sealer build --no-cache -t kubernetes-kyverno:v1.22.15 .

# How to build Kyverno AppImage

# Step1: get the kyverno install yaml

Due to kyverno's compatibility requirements with kubernetes (shown below), we need to pick the corresponding version of kyverno based on the version of the base image we have selected.

* Due to a known issue with Kubernetes 1.23.0-1.23.2, support for 1.23 begins at 1.23.3.

1.6.x: https://raw.githubusercontent.com/kyverno/kyverno/release-1.6/config/release/install.yaml

1.7.x: https://raw.githubusercontent.com/kyverno/kyverno/release-1.7/config/release/install.yaml

1.8.5: https://github.com/kyverno/kyverno/releases/download/v1.8.5/install.yaml

You can also find the install.yaml for the version of kyverno you want here: https://github.com/kyverno/kyverno/releases

Because we are using a base image with Kubernetes version v1.22.15, we can use 1.6.x or 1.7.x versions of kyverno.

# Step2: create a ClusterPolicy

Create a yaml with the following content:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: replace-image-registry
spec:
  background: false
  rules:
    - name: replace-image-registry-pod-containers
      match:
        any:
        - resources:
            kinds:
            - Pod
      preconditions:
        all:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
      mutate:
        foreach:
        - list: "request.object.spec.containers"
          patchStrategicMerge:
            spec:
              containers:
              - name: "{{ element.name }}"
                image: "{{ regex_replace_all_literal('^[^/]+', '{{element.image}}', 'sea.hub:5000' )}}"
    - name: replace-image-registry-pod-initcontainers
      match:
        any:
        - resources:
            kinds:
            - Pod
      preconditions:
        all:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
        - key: "{{ request.object.spec.initContainers[] || '' | length(@) }}"
          operator: GreaterThanOrEquals
          value: 1
      mutate:
        foreach:
        - list: "request.object.spec.initContainers"
          patchStrategicMerge:
            spec:
              initContainers:
              - name: "{{ element.name }}"
                image: "{{ regex_replace_all_literal('^[^/]+', '{{element.image}}', 'sea.hub:5000' )}}"

This ClusterPolicy will redirect image pull request to private registry sea.hub:5000, and we can name this file as redirect-registry.

# Step3: create a shell script

we use this script to change the address of the mirror repository in install.yaml and redirect-registry, install kyverno and create policy.

#!/bin/bash

function usage {
   cat << EOF
Usage: redirect-registry.sh -r <your private registry>
EOF
   exit 1
}

registry="sea.hub:5000"

if [ $# -gt 2 ]; then
  usage;
elif [ $# -eq 2 ]; then
  if [ "$1" != "-r" ]; then
    usage
  fi
  registry=$2
fi

sed -i "s/ghcr.io/${registry}/g" ./install.yaml
sed -i "s/sea.hub:5000/${registry}/g" ./redirect-registry
cat >>install.yaml<<EOF
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: In
                values:
                - ""
EOF

kubectl create -f install.yaml
kubectl create -f redirect-registry

echo "[kyverno-start]: Waiting for the kyverno to be ready..."

while true
do
    clusterPolicyStatus=`kubectl get cpol -o go-template --template={{range.items}}{{.status.ready}}{{end}}`;
    if [ "$clusterPolicyStatus" == "true" ];then
        break;
    fi
    sleep 1
done

echo "kyverno is running"

I named this file redirect-registry.sh.

# Step4: create the build content

Create a kyvernoAppImageBuild directory with four files: the install.yaml in step 1, redirect-registry in step 2, redirect-registry.sh in step3 and a Kubefile which content is following:

FROM scratch
LABEL "app.alpha.sealer.io/image-policy-plugin"="kyverno"
COPY install.yaml application/apps/kyverno/
COPY redirect-registry application/apps/kyverno/
APP kyverno local://redirect-registry.sh
APPCMDS kyverno ["sh redirect-registry.sh -r sea.hub:5000"]
LAUNCH ["kyverno"]

If the private repository you want to use is not sea.hub:5000, you will also need to change sea.hub:5000 to the private repository address you want to use.

# Step5: build the image

Supposing you are at the kyvernoAppImageBuild directory, please execute sealer build --no-cache --type=app-installer -t scratch-kyverno:v1.7.5 .

# How to build Kyverno BaseImage to make Kyverno highly available

coming soon...