Received an error running the following under Docker Desktop (3.5.2) for MacOS X (11.5.1) on Apple Silicon (MacBook Pro 13-inch, M1, 2020).

docker run --privileged quay.io/singularity/singularity:v3.8.1 run docker://hello-world

ERROR  : Failed to spawn stage 1

However, have been successful with the following.

docker run quay.io/singularity/singularity:v3.8.1 --version

singularity-ce version 3.8.1

Is anyone else able to reproduce this error?

Version of Singularity

$ singularity version
3.8.0

Describe the bug Singularity 3.8.0 (both CE and forks) fail to catch & handle the error case where a .sif fails to push to Harbor.

sif images continue to be pushed using oras (see below ticket)

Relates to https://github.com/hpcng/singularity/issues/5691

To Reproduce Steps to reproduce the behavior:

Have Harbor 2.2+ with OIDC (not username+password auth)

rm -rf $HOME/.singularity

singularity login -u <user> -p <harbor CLI secret> oras://your.harbor.instance

singularity -d push $HOME/alpine_latest.sif oras://your.harbor.instance/test/alpine:latest
DEBUG   [U=1000,P=18264]   persistentPreRun()            Singularity version: 3.8.0
DEBUG   [U=1000,P=18264]   persistentPreRun()            Parsing configuration file /usr/local/etc/singularity/singularity.conf
DEBUG   [U=1000,P=18264]   handleConfDir()               /home/dsouthwi/.singularity already exists. Not creating.
DEBUG   [U=1000,P=18264]   handleRemoteConf()            Ensuring file permission of 0600 on /home/dsouthwi/.singularity/remote.yaml
DEBUG   [U=1000,P=18264]   Init()                        Image format detection
DEBUG   [U=1000,P=18264]   Init()                        Check for sandbox image format
DEBUG   [U=1000,P=18264]   Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=1000,P=18264]   Init()                        Check for sif image format
DEBUG   [U=1000,P=18264]   Init()                        sif image format detected
DEBUG   [U=1000,P=18264]   UploadImage()                 ORAS push not accepted, retrying without config for registry compatibility

<wait 1-1000 minutes>
 
^CDEBUG   [U=1000,P=18264]   func2()                       User requested cancellation with interrupt
^C^C^C^C^C^C^C^C^C^C^C^C

program has froze/locked & unresponsive to interrupts. Reboot your session or machine.

Expected behavior Ideally, pushing to oras harbor registry successfully. If you do fail, at least timeout or something. Anything but locking up the session.

OS / Linux Distribution Which Linux distribution are you using? fails on centos/ubuntu/rhel in the same way.

Installation Method bug present for both source build and RPM (EPEL, fedora, etc)

Description of the Pull Request (PR):

Add test certificate chain. Add X.509 flags and environment variables to singularity verify command.

This fixes or addresses the following GitHub issues:

• Closes #1095

Version of Singularity What version of Singularity are you using? Run:

$ singularity --version
singularity-ce version 3.10.0

Describe the bug singularity installed as unprivileged user to NFS share on CentOS7 fails while running post install test.

edit: However, when installed as root into local filesystem the test succeeds.

To Reproduce Steps to reproduce the behavior:

singularity -d exec library://alpine cat /etc/alpine-release
...
INFO    [U=1232,P=26117]   extractImage()                Converting SIF file to temporary sandbox...
DEBUG   [U=1232,P=26117]   findFromConfigOrPath()        Using "unsquashfs" at "/n/projects/mec/local/miniconda3/envs/singularity-ce/bin/unsquashfs" (from singularity.conf)
DEBUG   [U=1232,P=26117]   findFromConfigOrPath()        Using "unsquashfs" at "/n/projects/mec/local/miniconda3/envs/singularity-ce/bin/unsquashfs" (from singularity.conf)
DEBUG   [U=1232,P=26117]   extract()                     Excluding /dev directory during root filesystem extraction (non root user)
DEBUG   [U=1232,P=26117]   extract()                     Trying unsquashfs options: [-user-xattrs -r]
DEBUG   [U=1232,P=26117]   unsquashfsSandboxCmd()        Calling wrapped unsquashfs: singularity [-q exec --no-home --no-nv --no-rocm -C --no-init --writable -B /tmp/rootfs-328367774:/image -B /n/projects/mec/local/miniconda3/envs/singularity-ce/bin/unsquashfs:/n/projects/mec/local/miniconda3/envs/singularity-ce/bin/unsquashfs:ro -B /usr/lib64/libpthread.so.0:/usr/lib64/libpthread.so.0:ro -B /usr/lib64/libm.so.6:/usr/lib64/libm.so.6:ro -B /n/projects/mec/local/miniconda3/envs/singularity-ce/bin/../lib/libz.so.1:/n/projects/mec/local/miniconda3/envs/singularity-ce/lib/libz.so.1:ro -B /n/projects/mec/local/miniconda3/envs/singularity-ce/bin/../lib/liblzma.so.5:/n/projects/mec/local/miniconda3/envs/singularity-ce/lib/liblzma.so.5:ro -B /n/projects/mec/local/miniconda3/envs/singularity-ce/bin/../lib/liblzo2.so.2:/n/projects/mec/local/miniconda3/envs/singularity-ce/lib/liblzo2.so.2:ro -B /n/projects/mec/local/miniconda3/envs/singularity-ce/bin/../lib/liblz4.so.1:/n/projects/mec/local/miniconda3/envs/singularity-ce/lib/liblz4.so.1:ro -B /n/projects/mec/local/miniconda3/envs/singularity-ce/bin/../lib/libzstd.so.1:/n/projects/mec/local/miniconda3/envs/singularity-ce/lib/libzstd.so.1:ro -B /usr/lib64/libc.so.6:/usr/lib64/libc.so.6:ro -B /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2:ro -B /usr/lib64/librt.so.1:/usr/lib64/librt.so.1:ro /tmp/rootfs-328367774/tmp-rootfs-613646387 /n/projects/mec/local/miniconda3/envs/singularity-ce/bin/unsquashfs -user-xattrs -r -d /image/root /image/archive-1028146708 ^(.{0}[^d]|.{1}[^e]|.{2}[^v]|.{3}[^\x2f]).*$]
FATAL   [U=1232,P=26117]   execStarter()                 while handling /home/mec/.singularity/cache/library/sha256.77dea042db93316b7ecc65fc39f7e19e7c209e01c38a0a87418d4be753fb9d37: while extracting image: %!w(<nil>)

Expected behavior the test to succeed

OS / Linux Distribution Which Linux distribution are you using?

$ cat /etc/os-release
 NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Installation Method

   make -C ./builddir install 

Write here how you installed SingularityCE. Eg. RPM, source.

From source following: https://docs.sylabs.io/guides/3.10/admin-guide/installation.html#installation-on-linux with exceptions:

compiled --without-suid --prefix=/some/nfsmounted/network/share

Since rpm from upstream repos is to old, compilation and testing was performed in a conda environment in which squashfs 4.4 tools was installed, viz:

/n/projects/mec/local/miniconda3/envs/singularity-ce/bin/unsquashfs -v
unsquashfs version 4.4 (2019/08/29)

make install was performed by non-root user into NFS mounted filesystem

Additional context Anything else which might be relevant. E.g. if the bug only occurs on a specific filesystem, or kernel version etc.

Plenty of disk space available.

singularity buildcfg
PACKAGE_NAME=singularity-ce
PACKAGE_VERSION=3.10.0
BUILDDIR=/n/projects/mec/local/singularity-ce-3.10.0/builddir
PREFIX=/n/projects/mec/local/singularity-ce
EXECPREFIX=/n/projects/mec/local/singularity-ce
BINDIR=/n/projects/mec/local/singularity-ce/bin
SBINDIR=/n/projects/mec/local/singularity-ce/sbin
LIBEXECDIR=/n/projects/mec/local/singularity-ce/libexec
DATAROOTDIR=/n/projects/mec/local/singularity-ce/share
DATADIR=/n/projects/mec/local/singularity-ce/share
SYSCONFDIR=/n/projects/mec/local/singularity-ce/etc
SHAREDSTATEDIR=/n/projects/mec/local/singularity-ce/com
LOCALSTATEDIR=/n/projects/mec/local/singularity-ce/var
RUNSTATEDIR=/n/projects/mec/local/singularity-ce/var/run
INCLUDEDIR=/n/projects/mec/local/singularity-ce/include
DOCDIR=/n/projects/mec/local/singularity-ce/share/doc/singularity-ce
INFODIR=/n/projects/mec/local/singularity-ce/share/info
LIBDIR=/n/projects/mec/local/singularity-ce/lib
LOCALEDIR=/n/projects/mec/local/singularity-ce/share/locale
MANDIR=/n/projects/mec/local/singularity-ce/share/man
SINGULARITY_CONFDIR=/n/projects/mec/local/singularity-ce/etc/singularity
SESSIONDIR=/n/projects/mec/local/singularity-ce/var/singularity/mnt/session
PLUGIN_ROOTDIR=/n/projects/mec/local/singularity-ce/libexec/singularity/plugin
SINGULARITY_CONF_FILE=/n/projects/mec/local/singularity-ce/etc/singularity/singularity.conf
SINGULARITY_SUID_INSTALL=0

Description of the Pull Request (PR):

There was a bug reported in slack that --env is truncating variables at commas:

Hello, I am trying to use --env to specify an environment variable (--env clustering_threshold='100,97') but I am running into an issue with

WARNING: Ignore environment variable "97": '=' is missing .

I have tried to use a backslash to escape the comma but that doesn't seem to work. I am currently using singularity-ce version 3.9.0

And I think I've seen this before? https://github.com/google/go-containerregistry/pull/1178. So because I love Go and don't get to work on enough Go projects (and this seemed like maybe a good way to engage with Singularity again) I decided to try for a fix! The bug looks to be the same here - there are just a few more layers added on top of it with the custom cmdline package provided here. To go through the changes:

• we ultimately need to use StringToStringVarP/StringToStringVar that can handle taking a varA=varB and parsing into map[string]string. We need to do this because StringArrayVarP considers a comma an ending delimiter to separate something out into a different string. There could be a way to tweak this particular parsing, but for other projects I've found a more straight forward fix is to use a map[string]string.
• This means that we cannot just parse the SIngularityEnvFile into SingularityEnv by appending to the list. Instead we parse the list of env directly from the file, and then check if it's already defined (by --env) or not. We issue a warning if the flag is also provided with --env and do not overwrite (--env takes precedence!)
• Finally, when we go over the list we only skip a variable when the envName is not defined. We can technically allow an empty value for envValue because the user might be trying to unset something.

This same type can be used for any other variables that might come up to truncate on commas!

Apologies in advance for anything I did wrong - I'll fix things as the CI brings up errors!

Description

Singularity currently supports PGP/GPG web of trust for digital signatures. It would be useful to also support hierarchical chain of trust with X.509 certificates. This would provide advantages for users and system administrators having Certificate Authority (CA) certificates already in place in the OS, since they could trust signed containers without additional steps for verification of the signer.

Desired Solution The ability to sign containers with X.509 certificates. The ability to verify containers against CA certificates already in place in the host OS.

Describe alternatives you've considered PGP/GPG key management servers can provide another mechanism for trusting certificates. However, some organizations already have established CA cert. processes and lack PGP/GPG key management servers. It would be useful to support the organizations existing CA certs.

Additional context Certificate providers such as IdenTrust provide code signing certificates. It would be useful to be consistent with the processes used for code signing and verification established by such providers and implemented in other software execution environments.

Description of the Pull Request (PR):

Non-root users can now build from a definition file, on systems that do not support --fakeroot. This requires the statically built proot command (https://proot-me.github.io/) to be available on the user PATH.

These builds:

• Do not support arch / debootstrap / yum / zypper bootstraps. Use localimage, library, oras, or one of the docker/oci sources.
• Do not support %pre and %setup sections.
• Run the %post sections of a build in the container as an emulated root user.
• Run the %test section of a build as the non-root user, like singularity test.
• Are subject to any restrictions imposed in singularity.conf.
• Incur a performance penalty due to proot's ptrace based interception of syscalls.
• May fail if the %post and %test scripts require privileged operations that proot cannot emulate.

This fixes or addresses the following GitHub issues:

• Fixes #880

Before submitting a PR, make sure you have done the following:

• Read the Guidelines for Contributing, and this PR conforms to the stated requirements.
• Added changes to the CHANGELOG if necessary according to the Contribution Guidelines
• Added tests to validate this PR, linted with make check and tested this PR locally with a make test, and make testall if possible (see CONTRIBUTING.md).
• Based this PR against the appropriate branch according to the Contribution Guidelines
• Added myself as a contributor to the Contributors File

Version of Singularity What version of Singularity are you using? Run:

$ singularity --version
singularity-ce version 3.9.1

Describe the bug Environment variables are passed to the container when it's run interactively, but not when run with singularity exec python testEnv.py. I can give a concise example:

I have a singularity image that's based on a docker image. In the docker image, there is an env variable set:

I have no name!@382eaaff1b1f:~$ echo $LMDFIT_BUILD_PATH/
/mnt/work/LuminosityFit/build/

On the system I'm running this image, this variable is set to something else:

roklasen@login23:~$ echo $LMDFIT_BUILD_PATH
/home/roklasen/LuminosityFit/build

Which is what I want. I noticed Singularity passes these environment variables to the container automatically, which is okay (I want this here too):

roklasen@login23:~$ singularity run lmdfit-mini.sif
INFO:    Converting SIF file to temporary sandbox...
Singularity> echo $LMDFIT_BUILD_PATH/
/home/roklasen/LuminosityFit/build/
Singularity>

So far so good. Now, I want to read that environment variable with python. I have this very basic python script:

#!/usr/bin/env python
import os
lmd_build_path = os.environ["LMDFIT_BUILD_PATH"]
print(f'lmd build path: {lmd_build_path}')
print(f'AND: {os.system("echo $LMDFIT_BUILD_PATH")}')

When I enter the container manually and execute this script, the variable is set correctly:

roklasen@login23:~$ singularity run lmdfit-mini.sif
INFO:    Converting SIF file to temporary sandbox...
Singularity> python testEnv.py
lmd build path: /home/roklasen/LuminosityFit/build
/home/roklasen/LuminosityFit/build
AND: 0
Singularity> exit

But when I run this script via singularity directly, the variable is NOT set to the current one, but instead the very old one from the first docker container, which I thought was overridden:

roklasen@login23:~$ singularity run lmdfit-mini.sif python testEnv.py
INFO:    Converting SIF file to temporary sandbox...
lmd build path: /mnt/work/LuminosityFit/build
/mnt/work/LuminosityFit/build
AND: 0
INFO:    Cleaning up image...

Expected behavior

So it seems if I run singularity exec python testEnv.py, the environment variables are NOT correctly passed to the container.

*OS / Linux Distribution

Which Linux distribution are you using?

The host system is

$ cat /etc/os-release
roklasen@login23:~$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.5 (Arctic Sphynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.5 (Arctic Sphynx)"

The guest system is the Debian 10 docker image with python3 installed.

Installation Method Write here how you installed SingularityCE. Eg. RPM, source.

It is provided to me on a cluster system by the HPC team, I don't know how they installed it.

Additional context

I can build some minimal docker image and singularity image to reproduce this, if that's required. But I don't think this issue is from my image or python. Maybe I'm missing how python handles environment variables?

Version of Singularity What version of Singularity are you using? Run:

$ singularity version
singularity-ce version 3.8.2

Describe the bug I cannot run the example from the documentation, i.e.

sudo singularity build lolcow.sif library://lolcow

the output:

INFO:    Starting build...
INFO:    Using cached image
INFO:    Verifying bootstrap image /root/.singularity/cache/library/sha256.2175ac94c763b1b3e0982cff4d0c9f73e8586d9e226c5311faa43629578f57b8
ERROR:   unpackSIF failed: root filesystem extraction failed: extract command failed: WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
/usr/local/bin/unsquashfs: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
: exit status 127
FATAL:   While performing build: packer failed to pack: root filesystem extraction failed: extract command failed: WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
/usr/local/bin/unsquashfs: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
: exit status 127

To Reproduce sudo singularity build lolcow.sif library://lolcow

OS / Linux Distribution Which Linux distribution are you using?

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Installation Method Installation from source, following this: https://sylabs.io/guides/3.8/user-guide/quick_start.html#download-singularityce-from-a-release

$ ./mconfig && \
    make -C builddir && \
    sudo make -C builddir install

Additional context

I tried:

• [x] sudo apt-get install zlib1g; the library was already installed
• [x] ( unset LD_LIBRARY_PATH ; sudo singularity build lolcow.sif library://sylabs-jms/testing/lolcow ) (following issue https://github.com/hpcng/singularity/issues/5666)
• [x] Changing LD_LIBRARY_PATH to use another location for libz.so.1.
• [x] Testing different versions of unsquashfs (4.5, 4.4)

I also opened an issue here → https://github.com/hpcng/singularity/issues/6139

Description of the Pull Request (PR):

A testing cleanup.

This pull request replaces ioutil.TempDir with t.TempDir. We can use the T.TempDir function from the testing package to create temporary directory. The directory created by T.TempDir is automatically removed when the test and all its subtests complete.

This saves us at least 2 lines (error check, and cleanup) on every instance, or in some cases adds cleanup that we forgot.

Reference: https://pkg.go.dev/testing#T.TempDir

func TestFoo(t *testing.T) {
	// before
	tmpDir, err := ioutil.TempDir("", "")
	if err != nil {
		t.Fatal(err)
	}
	defer os.RemoveAll(tmpDir)

	// now
	tmpDir := t.TempDir()
}

This fixes or addresses the following GitHub issues:

Before submitting a PR, make sure you have done the following:

• Read the Guidelines for Contributing, and this PR conforms to the stated requirements.
• Added changes to the CHANGELOG if necessary according to the Contribution Guidelines
• Added tests to validate this PR, linted with make check and tested this PR locally with a make test, and make testall if possible (see CONTRIBUTING.md).
• Based this PR against the appropriate branch according to the Contribution Guidelines
• Added myself as a contributor to the Contributors File

Description of the Pull Request (PR):

This seems to be the basic to add support for a custom docker host as an envar or command line option. Some questions I have:

1. How/where to test?
2. Should the default value be set to something instead of empty string? If we use the dockerclient library we can just set the default to dockerclient.DockerDaemonHost which will populate based on the host-type (here is unix https://github.com/moby/moby/blob/a6919e12b103aab6eb95a67450b2a487bfec1097/client/client_unix.go) - but I figured it would be OK to test with the default not set since the struct would do the same if we didn't define it!

Signed-off-by: vsoch [email protected]

This fixes or addresses the following GitHub issues:

• Fixes #805
• Fixes #878

Note that I decided to open these together because full support for DOCKER_HOST would not work without the second.

Description of the Pull Request (PR):

When the cache is enabled, and --oci mode is used, extract the image directly from the cache oci-layout, rather than copying into a temporary oci-layout first.

In addition, within the transparent caching functions, check if an image manifest is present before performing a copy into the cache. Although the copy is effectively a no-op if the image is present, this avoids some overhead.

This fixes or addresses the following GitHub issues:

• Fixes #1221

Before submitting a PR, make sure you have done the following:

• Read the Guidelines for Contributing, and this PR conforms to the stated requirements.
• Added changes to the CHANGELOG if necessary according to the Contribution Guidelines
• Added tests to validate this PR, linted with make check and tested this PR locally with a make test, and make testall if possible (see CONTRIBUTING.md).
• Based this PR against the appropriate branch according to the Contribution Guidelines
• Added myself as a contributor to the Contributors File

At present, when an OCI image is used in --oci mode, layer blobs are pulled and cached, before an oci-layout is created which will then be extracted. We always have 2 instances of a containers/image Copy operation.

~~OCI -> SIF conversion caches a converted image. We could do simple caching of an oci-archive for an image with a specific hash in order to remove one of the Copy operations and speed things up a bit.~~

Because the OCI cache is a transparent wrapper, we can just pull through it when the cache is enabled. No need for any caching of an archive etc.

The --oci runtime lanucher should receive systemContext vars from the CLI layer, to enable OCI registry auth, access to insecure registries, a specified Docker Daemon host, etc.

https://github.com/sylabs/singularity/blob/2255816ac94ef3f8261e05aef03dce4947476323/internal/pkg/runtime/launcher/oci/launcher_linux.go#L324

Signed-off-by: Fotis Nikolaidis [email protected]

Description of the Pull Request (PR):

Add support for Online Revocation Checks using the OCSP protocols.

This fixes or addresses the following GitHub issues:

• Fixes #https://github.com/sylabs/singularity/issues/1152

Testing & Limitations

Since there is no officially signed certificate chain for Singularity, the validation of OCSP is done:

1. Using the self-signed certificates. However, e2e testing fails because the root certificate is signed using Ed25519 (see the issue discussion)

2. Using third-party certificate chain (AKAMAI). OCSP successfully validates the certificate, but the signature verification fails (since they are not useful for signing the image).

In order to allow unprivileged overlay from images, following the pattern where --sif-fuse performs mounts prior to the invocation of the singularity runtime, we need to:

• At the CLI app level, i.e. before starter is exec'd, identify any SIF images provided as overlay mount sources.
• Mount these onto a temporary directory, using squashfuse or fuse2fs (depending on overlay partition type in the SIF file).
• Instruct the engine to perform an unpriv overlay mount based on the fuse mounted dir, rather than the original image.
• Clean up by unmounting the fuse mount when the container terminates.

Unfortunately, I think that we are blocked until https://github.com/tytso/e2fsprogs/pull/124 is addressed. I was hoping this would have been reviewed / merged, but looks like we'll have to wait longer.

fuse2fs doesn't currently support mounting from an offset in a file, so we can't fuse mount an ext partition out of a SIF. Most overlays would be ext. A squashfs read-only overlay would be a rare thing. I doubt that supporting only that is really worthwile?

Apptainer has a workaround for this blocker via an LD_PRELOAD - https://github.com/apptainer/apptainer/blob/main/tools/offsetpreload.c

I'm not particularly keen to build, bundle, and use an LD_PRELOAD here. Perhaps we'd consider it if it doesn't look like an offset patch will be merged, or we don't find an alternative?

We are still introducing the ability to use a directory overlay unprivileged in 3.11 - as long as the host kernel supports unprivileged overlay (we are not enabling fuse-overlayfs).