How to save/sync multi-architecture container Docker images

This article was last updated on: July 24, 2024 am

preface

With the further development of container and chip technologies, as well as the requirements of green, energy-saving, and information creation, multi-CPU architecture scenarios are becoming more and more common. Typical application scenarios include:

  1. Xinchuang: x86 server + Kunpeng ARM and other Xinchuang servers;
  2. Personal computer: Apple Mac M1 + Windows computer (or old Intel chip Mac);
  3. Edge: The data center uses x86 servers, and the edge edge uses low-power arm edge devices (such as Raspberry Pi).

Container cloud native technology is very good in this regard, but there will be some problems with the details in actual use, for example:How to save/sync multi-architecture container Docker images

This time, let’s take the example of synchronizing the image of Docker Hub to the local image repository.

glossary

English Chinese Description
multi-arch image Multi-architecture image
variant Variants Different variations refer to Redis images arm/v5 and arm/v7 Two variants
manifest Manifest
manifest-list List of lists
layer (mirror) layer
image index Image Index OCI proper noun with the same meaning as manifest-list
manifest digest Manifest Summary

How container images support multiple architectures

A multi-arch image is a container image that can combine variants of different architectures (such as amd64 and arm) and sometimes variants of different operating systems (such as Windows and Linux). When you run an image that supports multiple architectures, the container client automatically selects an image variant that matches your OS and architecture.

Multi-architecture mirroring is implemented based on image manifests and manifest lists.

Manifests

Each container image is represented by a “manifest”. A manifest is a JSON file that uniquely identifies an image and references its layer and its corresponding size.

hello-world Linux The basic manifest for an image resembles the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1510,
      "digest": "sha256:fbf289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
    },
  "layers": [
      {
        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
        "size": 977,
        "digest": "sha256:2c930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced"
      }
    ]
}

Manifest-lists

Multi-architecture mirroringList of manifests(Often referred to as OCI mirroring.) The mirror index of ) is a collection (index) of mirrors that you can create by specifying one or more image names. It includes details about each image, such as supported operating systems and architectures, size, and manifest digest. How the manifest list is used is the same as docker pull and docker run The image name in command is the same.

docker CLI use docker manifestCommand management manifests and manifest lists.

🐾 Warning:

Currently, the order docker manifest And subcommands are experimental. For more information about using experimental commands, see the Docker documentation.

✍️ Author’s note: Probably becauseExperimentalThe reason is that several multi-architecture images have encountered strange problems during use.

You can use this command docker manifest inspect Review the list of inventory. The following are multi-architecture imageshello-world:latest output, which has three manifests: two for the Linux operating system architecture and one for the Windows architecture.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 524,
      "digest": "sha256:83c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 525,
      "digest": "sha256:873612c5503f3f1674f315c67089dee577d8cc6afc18565e0b4183ae355fb343",
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1124,
      "digest": "sha256:b791ad98d505abb8c9618868fc43c74aa94d08f1d7afe37d19647c0030905cae",
      "platform": {
        "architecture": "amd64",
        "os": "windows",
        "os.version": "10.0.17763.1697"
      }
    }
  ]
}

use docker manifest Save the multi-architecture image

Here are the steps to push a multi-architecture image to the local image repository:

  1. Tag and push each architecture-specific image to your container registry. The following example assumes two Linux architectures: arm64 and amd64.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    docker tag myimage:arm64 \
      192.168.2.23:5000/multi-arch-samples/myimage:arm64

    docker push 192.168.2.23:5000/multi-arch-samples/myimage:arm64

    docker tag myimage:amd64 \
      192.168.2.23:5000/multi-arch-samples/myimage:amd64

    docker push 192.168.2.23:5000/multi-arch-samples/myimage:amd64

  2. run docker manifest create to create an inventory list to merge the preceding images into a multi-architecture image.

    1
    2
    3
    docker manifest create 192.168.2.23:5000/multi-arch-samples/myimage:multi \
     192.168.2.23:5000/multi-arch-samples/myimage:arm64 \
     192.168.2.23:5000/multi-arch-samples/myimage:amd64

  3. Use the following commanddocker manifest pushTo push the manifest to the image repository:

    1
    docker manifest push 192.168.2.23:5000/multi-arch-samples/myimage:multi

  4. Use commandsdocker manifest inspectReview the list of inventory. The previous section shows an example of command output.

After you push a multi-architecture manifest to an image repository, you can use multi-architecture images in the same way as single-architecture images. For example, use docker pull Pull the image.

Save/Synchronize Multi-Schema Image Utility Script One - Based docker manifest

Scenario one

Already have a multi-architecture package You need to load the package and upload the multi-architecture image to the local image repository

Taking K3s as an example, the official has released multi-architecture offline image compressed packages at the time of release, which are:

  • k3s-airgap-images-amd64.tar.gz
  • k3s-airgap-images-arm.tar.gz
  • k3s-airgap-images-arm64.tar.gz
  • …

These packages are downloaded and uploaded to the client/user’s offline environment. You now need to load the package and upload the multi-architecture image to the local image repository

Approximate steps

  1. docker load Compressed package
  2. The images in it are tagged one by one, and changed to <本地镜像仓库地址>/.../...:<tag>-<arch>
  3. Push image
  4. Repeat the above steps 3 times, with the tag included-amd64 -arm -arm64are pushed to the local image repository
  5. Mirror one by one docker manifest create to create an inventory list
  6. Use the following commanddocker manifest pushPush the manifests to the image repository one by one

The full script is as follows:

🐾 Warning:

Due to my limited ability, when using the 8*3 offline images of k3s v1.21.7+k3s1 for testing, 5 are always successful, and the other 3 appear that the arch and manifest of the manifest list are not matched.
Not sure if it’s my script issue or not docker manifest The command is experimental.
If you have experience, please help take a look. Thank you~

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
#!/bin/bash
amd64_images="k3s-airgap-images-amd64.tar.gz"
arm64_images="k3s-airgap-images-arm64.tar.gz"
arm_images=""
list="k3s-images.txt"

usage() {
    echo "USAGE: $0 [--amd64-images k3s-airgap-images-amd64.tar.gz] [--arm64-images k3s-airgap-images-arm64.tar.gz] [---arm-images k3s-airgap-images-arm.tar.gz] --registry my.registry.com:5000"
    echo "  [-l|--image-list path] text file with list of images; one image per line."
    echo "  [-x|--amd64-images path] amd64 arch tar.gz generated by docker save."
    echo "  [-a|--arm64-images path] arm64 arch tar.gz generated by docker save."
    echo "  [---arm-images path] arm arch tar.gz generated by docker save."
    echo "  [-r|--registry registry:port] target private registry:port."
    echo "  [-h|--help] Usage message"
}

push_manifest() {
    export DOCKER_CLI_EXPERIMENTAL=enabled
    manifest_list=()
    for i_arch in "${arch_list[@]}"; do
        manifest_list+=("$1-${i_arch}")
    done

    echo "Preparing manifest $1, list[${arch_list[@]}]"
    docker manifest create "$1" "${manifest_list[@]}" --insecure
    docker manifest push "$1" --purge --insecure
}

while [[ $# -gt 0 ]]; do
    key="$1"
    case $key in
    -r | --registry)
        reg="$2"
        shift # past argument
        shift # past value
        ;;
    -l | --image-list)
        list="$2"
        shift # past argument
        shift # past value
        ;;
    -x | --amd64-images)
        amd64_images="$2"
        shift # past argument
        shift # past value
        ;;
    -a | --arm64-images)
        arm64_images="$2"
        shift # past argument
        shift # past value
        ;;
    --arm-images)
        arm_images="$2"
        shift # past argument
        shift # past value
        ;;
    -h | --help)
        help="true"
        shift
        ;;
    *)
        usage
        exit 1
        ;;
    esac
done

if [[ -z $reg ]]; then
    usage
    exit 1
fi

if [[ $help ]]; then
    usage
    exit 0
fi

arch_list=()
if [[ -n "${amd64_images}" ]]; then
    arch_list+=("amd64")
fi
if [[ -n "${arm64_images}" ]]; then
    arch_list+=("arm64")
fi
if [[ -n "${arm_images}" ]]; then
    arch_list+=("arm")
fi

image_list=()
while IFS= read -r i; do
    [ -z "${i}" ] && continue
    image_list+=("${i}")
done <"${list}"

for arch in "${arch_list[@]}"; do
    [ -z "${arch}" ] && continue

    case $arch in
    amd64)
        docker load --input ${amd64_images}
        ;;
    arm64)
        docker load --input ${arm64_images}
        ;;
    arm)
        docker load --input ${arm_images}
        ;;
    esac

    for i in "${image_list[@]}"; do
        [ -z "${i}" ] && continue

        case $i in
        */*)
            image_name="${reg}/${i}"
            ;;
        *)
            image_name="${reg}/library/${i}"
            ;;
        esac

        docker tag "${i}" "${image_name}-${arch}"
        docker rmi -f "${i}"
        docker push "${image_name}-${arch}"
    done
done

for i in "${image_list[@]}"; do
    [ -z "${i}" ] && continue

    case $i in
    */*)
        image_name="${reg}/${i}"
        ;;
    *)
        image_name="${reg}/library/${i}"
        ;;
    esac
    push_manifest "${image_name}"
done

How to use:

1
./load-images-multi-arch.sh --registry 192.168.2.23:5000 --arm-images k3s-airgap-images-arm.tar.gz

The log output is as follows:

$ ./load-images-multi-arch.sh --registry 192.168.2.23:5000 --arm_images k3s-airgap-images-arm.tar.gz
# docker load 镜像第一轮，是 amd64 架构的
67f770da229b: Loading layer [==================================================>]   1.45MB/1.45MB
Loaded image: rancher/library-busybox:1.32.1
...

# 打 tag 并 delete 原 tag 镜像，并 push
Untagged: rancher/coredns-coredns:1.8.3
The push refers to repository [192.168.2.23:5000/rancher/coredns-coredns]
85c53e1bd74e: Pushed
225df95e717c: Pushed
1.8.3-amd64: digest: sha256:db4f1c57978d7372b50f416d1058beb60cebff9a0d5b8bee02bfe70302e1cb2f size: 739
...

# docker load 镜像第二轮，是 arm64 架构的
...
32626eb1fe89: Loading layer [==================================================>]  526.8kB/526.8kB
Loaded image: rancher/pause:3.1

...
Untagged: rancher/pause:3.1
The push refers to repository [192.168.2.23:5000/rancher/pause]
32626eb1fe89: Pushed
3.1-arm64: digest: sha256:2aac966ece8906a535395f92bb25f0e8e21dac737df75b381e8f9bdd3ed56528 size: 527

# docker load 镜像第二轮，是 arm 架构的
8e322dc9c333: Loading layer [==================================================>]  5.045MB/5.045MB
efed3cfd1b26: Loading layer [==================================================>]  1.623MB/1.623MB
a46153382f22: Loading layer [==================================================>]  3.584kB/3.584kB
Loaded image: rancher/klipper-lb:v0.3.4
...

Untagged: rancher/coredns-coredns:1.8.3
The push refers to repository [192.168.2.23:5000/rancher/coredns-coredns]
9f4a0b0fd8b2: Pushed
225df95e717c: Layer already exists
1.8.3-arm: digest: sha256:dfc241eae22da74dd378535b69d7927f897acf48424cdcb90991b33f412cb7ae size: 739

# docker manifest create
Preparing manifest 192.168.2.23:5000/rancher/coredns-coredns:1.8.3, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/coredns-coredns:1.8.3
sha256:dc76fece93e42f05e7013e159097a0d426734fd268467f242d5b155dd49b0221
Preparing manifest 192.168.2.23:5000/rancher/klipper-helm:v0.6.6-build20211022, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/klipper-helm:v0.6.6-build20211022
sha256:e1c6842554ea37e66443cfab9a2422231bf8390b4c69711a74eb4cccde9d3dba
Preparing manifest 192.168.2.23:5000/rancher/klipper-lb:v0.3.4, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/klipper-lb:v0.3.4
sha256:98842bae8630a2aab1a94960185e152745ecf16ca69cf1eefdb53848cbc41063
Preparing manifest 192.168.2.23:5000/rancher/library-busybox:1.32.1, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/library-busybox:1.32.1
sha256:0b93c11bfd89ee5c971deaf9f312d115b2e1d797f79a7f68a266baecfb09a99f
Preparing manifest 192.168.2.23:5000/rancher/library-traefik:2.4.8, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/library-traefik:2.4.8
sha256:58464dda10504d271a17855541ed8d31a787ea25eb751ecce90e14256f23eb24
Preparing manifest 192.168.2.23:5000/rancher/local-path-provisioner:v0.0.19, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/local-path-provisioner:v0.0.19
sha256:0c797ef85540a4934ea84a9471f4f5a10c93f749ee668d92527361c61bbe98c3
Preparing manifest 192.168.2.23:5000/rancher/metrics-server:v0.3.6, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/metrics-server:v0.3.6
sha256:742595f61320bcaead987c5aafc3eb64b9a9151edb02b9e4d27f8abcae26d92e
Preparing manifest 192.168.2.23:5000/rancher/pause:3.1, list[amd64 arm64 arm]
Created manifest list 192.168.2.23:5000/rancher/pause:3.1
sha256:f3ef3cbaf2ea466a0c2a2cf3db0d9fbc30f4c24e57a79603aa0fa8999d4813b0

About Skopeo

Recently the Skopeo version has been updated v1.8Recent versions have added a bit withMulti-architectureAbout flags, make pass skopeo It is easier to save/synchronize multi-architecture images.

📝 Notes:

Currently regarding multiple architectures, there are only 3 options, none of which have the ability to select source mirroring several of multiple architectures, but are under development.
See this issue for details: feature: Support list of archs for sync command · Issue #1694 · containers/skopeo (github.com)

Here are some relevant flags:

  • skopeo
    • --override-arch <arch>:use arch Select the image instead of the architecture of the machine.
    • --override-os <os>:use os Select the image instead of the machine’s OS.
    • --override-variant <variant>:use variant Run a variation of the schema to select the image. (Different variants refer to Redis images.) arm/v5 and arm/v7 Two variants)
  • skopeo copy
    • --all, -a: If source-image refers to a list of images, thenNoOnly images that match the current operating system and architecture (depending on the global.)--override-os、--override-archand--override-variantoption), instead try to copy all the images in the list, as well as the list itself.
    • --multi-arch: If the source image references a multi-architecture image, control what is replicated. The default setting issystem。
      • system: Only images that match the system architecture are copied
      • all: Copy a complete multi-architecture image
      • index-only: Copy image index only. (index-onlyOptions typically fail unless the mirror referenced by each schema already exists in the target, or the target registry supports sparse indexes. )
  • skopeo sync
    • --all, -a:ditto

📝 Notes:

according to skopeo copy --multi-arch index-only Description,Scenario one Another implementation is:

  1. docker manifest The previous steps, leave it as it is
  2. will docker manifest create and docker manifest push to be replaced with skopeo copy --multi-arch index-only

Save/Synchronize Multi-Schema Image Utility Script II - Based skopeo copy

Scenario two

Synchronize images directly from docker.io to the local image repository

For example, for a version of K3s, the image list is:

  • rancher/coredns-coredns:1.8.3
  • rancher/klipper-helm:v0.6.6-build20211022
  • rancher/klipper-lb:v0.3.4
  • rancher/library-busybox:1.32.1
  • rancher/library-traefik:2.4.8
  • rancher/local-path-provisioner:v0.0.19
  • rancher/metrics-server:v0.3.6
  • rancher/pause:3.1

Here directly based on Mirror porter skopeo The provided script is modified and modified as follows:

📝 Notes:

Because of newer versions skopeo There is a series of flags mentioned above, and my Ubuntu apt is installed skopeo Still staying v1.5 version, without the above features. So go straight through docker run mode operation
Added to that --multi-arch all Options.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/bin/bash
GREEN_COL="\\033[32;1m"
RED_COL="\\033[1;31m"
NORMAL_COL="\\033[0;39m"
SOURCE_REGISTRY=$1
TARGET_REGISTRY=$2
IMAGES_LIST_FILE=$3
: ${IMAGES_LIST_FILE:="k3s-images.txt"}
: ${TARGET_REGISTRY:="192.168.2.23:5000"}
: ${SOURCE_REGISTRY:="docker.io"}

set -eo pipefail

CURRENT_NUM=0
ALL_IMAGES="$(sed -n '/#/d;s/:/:/p' ${IMAGES_LIST_FILE} | sort -u)"
TOTAL_NUMS=$(echo "${ALL_IMAGES}" | wc -l)

skopeo_copy() {
    if docker run -it quay.io/skopeo/stable:latest copy --insecure-policy --src-tls-verify=false --dest-tls-verify=false \
        --src-creds caseycui:xxxxxxxxxxxxxxxxxxxxx --multi-arch all --override-os linux -q docker://$1 docker://$2; then
        echo -e "$GREEN_COL Progress: ${CURRENT_NUM}/${TOTAL_NUMS} sync $1 to $2 successful $NORMAL_COL"
    else
        echo -e "$RED_COL Progress: ${CURRENT_NUM}/${TOTAL_NUMS} sync $1 to $2 failed $NORMAL_COL"
        exit 2
    fi
}

for image in ${ALL_IMAGES}; do
    let CURRENT_N192.168.2.23:5000UM=${CURRENT_NUM}+1
    skopeo_copy ${SOURCE_REGISTRY}/${image} ${TARGET_REGISTRY}/${image}
done

The operation effect is as follows:

$ bash sync.sh
 Progress: 1/8 sync docker.io/rancher/coredns-coredns:1.8.3 to 192.168.2.23:5000/rancher/coredns-coredns:1.8.3 successful
 Progress: 2/8 sync docker.io/rancher/klipper-helm:v0.6.6-build20211022 to 192.168.2.23:5000/rancher/klipper-helm:v0.6.6-build20211022 successful
 Progress: 3/8 sync docker.io/rancher/klipper-lb:v0.3.4 to 192.168.2.23:5000/rancher/klipper-lb:v0.3.4 successful
 Progress: 4/8 sync docker.io/rancher/library-busybox:1.32.1 to 192.168.2.23:5000/rancher/library-busybox:1.32.1 successful
 Progress: 5/8 sync docker.io/rancher/library-traefik:2.4.8 to 192.168.2.23:5000/rancher/library-traefik:2.4.8 successful
 Progress: 6/8 sync docker.io/rancher/local-path-provisioner:v0.0.19 to 192.168.2.23:5000/rancher/local-path-provisioner:v0.0.19 successful
 Progress: 7/8 sync docker.io/rancher/metrics-server:v0.3.6 to 192.168.2.23:5000/rancher/metrics-server:v0.3.6 successful
 Progress: 8/8 sync docker.io/rancher/pause:3.1 to 192.168.2.23:5000/rancher/pause:3.1 successful

The final effect

The final local image effect is as follows:

Docker Registry 中的多架构镜像

🎉🎉🎉

📚️ Reference

CloudNative
#Docker #Container #Linux #Skopeo #Architecture #Arch #ARM
How to save/sync multi-architecture container Docker images
https://e-whisper.com/posts/52241/
Author
east4ming
Posted on
July 4, 2022
Licensed under