Skopeo is one of the specialized tool that performs various operations on container images and image repositories.

Skopeo can perform operations which consist of:

• Copying an image from and to various storage mechanisms. For example you can copy images from one registry to another, without requiring privilege.
• Inspecting a remote image showing its properties including its layers, without requiring you to pull the image to the host.
• Deleting an image from an image repository.
• Syncing an external image repository to an internal registry for air-gapped (aka offline) deployments.
• When required by the repository, skopeo can pass the appropriate credentials and certificates for authentication.

Installing Skopeo

Skopeo is available for general linux package repositories already. So obtaining is generally as simply as asking your package manager to grab and install it. For detailed instructions wrt to your linux distribution, you can read official instructions.

For the purpose of this blog post, we’ll be using an official Fedora-based container image at quay.io/skopeo/stable. With the entrypoint set to /usr/bin/skopeo, skopeo is invoked by default when you run the container. We can then supply the specific arguments and options to container, in order to perform specific tasks.

We can pull down the image using podman pull quay.io/skopeo/stable :

# pull down skopeo image
[cloud_user@81f87084a51c ~]$ sudo podman pull quay.io/skopeo/stable
[sudo] password for cloud_user: 
Trying to pull quay.io/skopeo/stable:latest...
Getting image source signatures
Copying blob 51001f99c005 done  
Copying blob c86e6518116e done  
Copying blob 05d390cf73b7 done  
Copying blob 85a74b04b5b8 done  
Copying blob ee818e90ed5d done  
Copying blob ba367e1aec3a done  
Copying config 0346fd8ddb done  
Writing manifest to image destination
Storing signatures
0346fd8ddb3b1933d68cc26f2b289891f7d0e70bdbf055ee10147cb7891bf1c4

# list image
[cloud_user@81f87084a51c ~]$ sudo podman images
REPOSITORY             TAG     IMAGE ID      CREATED     SIZE
quay.io/skopeo/stable  latest  0346fd8ddb3b  3 days ago  400 MB

Running skopeo inside a container, comes with all the benefits and caveats of running containers.

Inspecting an Image with Skopeo

Skopeo was born with desire to inspect container images located on remote registries, without pulling the image to local machine. It can be used to inspect images on local, private and remote registries. Inspecting container image returns data similar to docker inspect command. To inspect a repository/image, we can use skopeo inspect command.

[cloud_user@81f87084a51c ~]$ sudo podman run --rm quay.io/skopeo/stable inspect docker://docker.io/nginx
{
    "Name": "docker.io/library/nginx",
    "Digest": "sha256:b0ea179ab61c789ce759dbe491cc534e293428ad232d00df83ce44bf86261179",
    "RepoTags": [
        "1-alpine-perl",
        "1-alpine",
        "1-perl",
        "1.10-alpine",
        "1.10.0-alpine",

...
...
        "1.9.8",
        "1.9.9",
        "1.9",
        "1",
        "alpine-perl",
        "alpine",
        "latest",
        "mainline-alpine-perl",
        "mainline-alpine",
        "mainline-perl",
        "mainline",
        "perl",
        "stable-alpine-perl",
        "stable-alpine",
        "stable-perl",
        "stable"
    ],
    "Created": "2021-03-27T06:50:35.609627369Z",
    "DockerVersion": "19.03.12",
    "Labels": {
        "maintainer": "NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:ac2522cc72690febc428fb46fb39a4efc5e0a721c3ad15d9992b01515f2fad1a",
        "sha256:09de04de3c750e3ebcce85cc5a312e7ce2d29891740129ea6007bec7bfbb72a7",
        "sha256:b0c8a51e66287c43c855b9538e067be5c245bfe14e96f5166b9128997dcd3045",
        "sha256:08b11a3d692c1a2e15ae840f2c15c18308dcb079aa5320e15d46b62015c0f6f3",
        "sha256:a0e0e6bcfd2cd773311862c3fb2e6da7ab49d3c127faa2007bba37b274734b4a",
        "sha256:4fcb23e29ba19bf305d0d4b35412625fea51e82292ec7312f9be724cb6e31ffd"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "NGINX_VERSION=1.19.8",
        "NJS_VERSION=0.5.2",
        "PKG_RELEASE=1~buster"
    ]
}

Working with Private and Insecure Registries

Most of the organizations will have their own registries setup and running, for security. Insecure registries are registries running with either non-https setup or private certs issued by their internal CAs. So in order to work with such registries, clients will need to provide either specific certs with --cert-dir parameter or set --tls-verify accordingly.

As mentioned in the container image specification, a custom TLS configuration for a container registry can be configured by creating a directory under $HOME/.config/containers/certs.d or /etc/containers/certs.d. The name of the directory must correspond to the host:port of the registry (e.g., my-registry.com:5000).

A certs directory can contain one or more files with the following extensions:

• *.crt files with this extensions will be interpreted as CA certificates
• *.cert files with this extensions will be interpreted as client certificates
• *.key files with this extensions will be interpreted as client keys

However, do note that when we are running Skopeo in a container, we need to volume mount certs directory with --volume $LOCALPATH:/etc/containers/certs.d. So our solution looks like below:

# Running Skopeo at local command line
$ skopeo inspect --cert-dir /localpath docker://$DOCKER-IMAGE

# Alternatively - using skopeo in a container
$ sudo podman run --rm quay.io/skopeo/stable inspect --volume $LOCALPATH:/etc/containers/certs.d docker://$DOCKER-IMAGE

Working with Credentials to Access Registries

Clients may also need to pass specific credentials in order to access the registry images. We can specify credentials on the command line via the --creds USERNAME[:PASSWORD] parameter, if needed. However this may seem like too much work at times. In such a case, we can use --authfile to supply the authentication file.

However, do note that when we are running skopeo in a container, we need to volume mount authfile with --volume $AUTHFILE:/auth.json. So our solution looks like below:

# Passing credentials for authentication 
$ sudo podman run --rm quay.io/skopeo/stable inspect --creds $USER:$PASSWORD docker://$DOCKER-IMAGE

# Alternatively - using auth file
$ sudo podman run --rm quay.io/skopeo/stable inspect --volume $AUTHFILE:/auth.json docker://$DOCKER-IMAGE

Other Skopeo commands works in a similar manner.

Copy Container Images to/from Registries

skopeo copy command can be used to copy an image (manifest, filesystem layers, signatures) from one location to another. Location in this context means:

• Container registries
  • The Quay, Docker Hub, OpenShift, GCR, Artifactory, ACR, ECR…
• Container Storage backends
  • github.com/containers/storage (Backend for Podman, CRI-O, Buildah and friends)
  • Docker daemon storage
• Local directories
• Local OCI-layout directories

It also provides lots of arguments and options for this functionality.

Copy Image between Registries

Since skopeo copy can copy registry from one registry to another, it is pretty useful feature for various scenarios. For example, you might want to use this functionality to populate internal repository with images from external registries, or sync images registries in two different locations, etc.

To copy an image from one registry to another:

# running skopeo natively on command line
skopeo copy docker://quay.io/skopeo/stable:latest docker://registry.example.com/skopeo:latest

# running skopeo in a container
$ sudo podman run --rm quay.io/skopeo/stable copy docker://quay.io/skopeo/stable:latest docker://registry.example.com/skopeo:latest

Copy Image from Registry to local

Lets say we want to copy layers of nginx:latest image to local directory /var/lib/images/nginx/. We can do so by running following commands:

[cloud_user@81f87084a51c ~]$ sudo mkdir -p /var/lib/images/nginx

[cloud_user@81f87084a51c ~]$ sudo skopeo copy docker://docker.io/nginx:latest dir:/var/lib/images/nginx
Getting image source signatures
Copying blob ac2522cc7269 done  
Copying blob 09de04de3c75 done  
Copying blob b0c8a51e6628 done  
Copying blob 08b11a3d692c done  
Copying blob a0e0e6bcfd2c done  
Copying blob 4fcb23e29ba1 done  
Copying config b8cf2cbeab done  
Writing manifest to image destination
Storing signatures

[cloud_user@81f87084a51c ~]$ ls /var/lib/images/nginx
08b11a3d692c1a2e15ae840f2c15c18308dcb079aa5320e15d46b62015c0f6f3  b0c8a51e66287c43c855b9538e067be5c245bfe14e96f5166b9128997dcd3045
09de04de3c750e3ebcce85cc5a312e7ce2d29891740129ea6007bec7bfbb72a7  b8cf2cbeabb915843204ceb7ef0055fecadd55c2b0c58ac030e01fe75235885a
4fcb23e29ba19bf305d0d4b35412625fea51e82292ec7312f9be724cb6e31ffd  manifest.json
a0e0e6bcfd2cd773311862c3fb2e6da7ab49d3c127faa2007bba37b274734b4a  version
ac2522cc72690febc428fb46fb39a4efc5e0a721c3ad15d9992b01515f2fad1a

If we are running skopeo in a container, then first we need to volume mount host’s file system into container. For the skopeo image we are using, this path inside container varies between root (/var/lib/containers/storage) and non-root users ($HOME/.local/share/containers/storage).

[cloud_user@81f87084a51c ~]$ sudo mkdir -p /var/lib/images/nginx-s

[cloud_user@81f87084a51c ~]$ sudo podman run --privileged --rm -v /var/lib/images/nginx-s:/var/lib/containers/storage quay.io/skopeo/stable copy docker://docker.io/nginx:latest dir:/var/lib/containers/storage
Getting image source signatures
Copying blob sha256:ac2522cc72690febc428fb46fb39a4efc5e0a721c3ad15d9992b01515f2fad1a
Copying blob sha256:09de04de3c750e3ebcce85cc5a312e7ce2d29891740129ea6007bec7bfbb72a7
Copying blob sha256:b0c8a51e66287c43c855b9538e067be5c245bfe14e96f5166b9128997dcd3045
Copying blob sha256:08b11a3d692c1a2e15ae840f2c15c18308dcb079aa5320e15d46b62015c0f6f3
Copying blob sha256:a0e0e6bcfd2cd773311862c3fb2e6da7ab49d3c127faa2007bba37b274734b4a
Copying blob sha256:4fcb23e29ba19bf305d0d4b35412625fea51e82292ec7312f9be724cb6e31ffd
Copying config sha256:b8cf2cbeabb915843204ceb7ef0055fecadd55c2b0c58ac030e01fe75235885a
Writing manifest to image destination
Storing signatures

[cloud_user@81f87084a51c ~]$ ls /var/lib/images/nginx-s
08b11a3d692c1a2e15ae840f2c15c18308dcb079aa5320e15d46b62015c0f6f3  b0c8a51e66287c43c855b9538e067be5c245bfe14e96f5166b9128997dcd3045
09de04de3c750e3ebcce85cc5a312e7ce2d29891740129ea6007bec7bfbb72a7  b8cf2cbeabb915843204ceb7ef0055fecadd55c2b0c58ac030e01fe75235885a
4fcb23e29ba19bf305d0d4b35412625fea51e82292ec7312f9be724cb6e31ffd  manifest.json
a0e0e6bcfd2cd773311862c3fb2e6da7ab49d3c127faa2007bba37b274734b4a  version
ac2522cc72690febc428fb46fb39a4efc5e0a721c3ad15d9992b01515f2fad1a

Note that the use of --privileged flag disables all kinds of security features and container isolations that would stop the Skopeo container from accessing the mounted container storage.

Same way we can upload images from local directory to remote registry. There are few options to sign and encrypt the image, using trust policy etc. as well.

Deleting Images with skopeo-delete

skopeo-delete can be used to mark the specified container image for later deletion by the registry’s garbage collector.

# running skopeo natively on command line
skopeo delete --force docker://registry.example.com/example/pause:latest

# running skopeo in a container
$ sudo podman run --rm quay.io/skopeo/stable delete --force docker://registry.example.com/example/pause:latest