[lxc-users] unprivileged container with systemd?

[Dirk Geschke]

Hi all,

I just to follow

https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/

once more to install a new container and it fails. First of all it
was a problem with the access to the directory

~/.local/share/lxc/jessie1

The owner changed to a mapped one -> 100000 and then there was no
access for the lxcuser, which has uid 1001. I fixed this via setting
write access for the users group.

But then I installed a download template:

lxc-create -t download -n jessie1 -- -d debian -r jessie -a amd64

which worked without problems (except warnings regarding reopen tty).

If I try to start the container it ends up with:

~$ lxc-start -n jessie1
lxc_container: Permission denied - Unable to create /dev/.lxc for autodev
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted

Here it ends, nothing more happens and only a kill -9 works...

And yes, /sbin/init in the container is now a link to systemd:

/sbin/init -> /lib/systemd/systemd

I suspect, this does not work at all without cgroup namespace support
in the kernel? Or am I missing something else?

Best regards

Dirk

[Serge Hallyn]

Post by Dirk Geschke
Hi all,
I just to follow
https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/
once more to install a new container and it fails. First of all it
was a problem with the access to the directory
~/.local/share/lxc/jessie1
The owner changed to a mapped one -> 100000 and then there was no
access for the lxcuser, which has uid 1001. I fixed this via setting
write access for the users group.
lxc-create -t download -n jessie1 -- -d debian -r jessie -a amd64
which worked without problems (except warnings regarding reopen tty).
~$ lxc-start -n jessie1
lxc_container: Permission denied - Unable to create /dev/.lxc for autodev
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
Here it ends, nothing more happens and only a kill -9 works...
/sbin/init -> /lib/systemd/systemd
I suspect, this does not work at all without cgroup namespace support
in the kernel? Or am I missing something else?

There's something else you're missing, but I'm not sure what. What is
your environment (os/release and any custom installs)? Try 1.1.0, and
make sure to re-create the container as the new config file should be
more correct for systemd backed containers.

[Dirk Geschke]

Hi Serge,

Post by Serge Hallyn

Post by Dirk Geschke
I just to follow
https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/
once more to install a new container and it fails. First of all it
was a problem with the access to the directory
~/.local/share/lxc/jessie1
The owner changed to a mapped one -> 100000 and then there was no
access for the lxcuser, which has uid 1001. I fixed this via setting
write access for the users group.
lxc-create -t download -n jessie1 -- -d debian -r jessie -a amd64
which worked without problems (except warnings regarding reopen tty).
~$ lxc-start -n jessie1
lxc_container: Permission denied - Unable to create /dev/.lxc for autodev
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
Here it ends, nothing more happens and only a kill -9 works...
/sbin/init -> /lib/systemd/systemd
I suspect, this does not work at all without cgroup namespace support
in the kernel? Or am I missing something else?

There's something else you're missing, but I'm not sure what. What is
your environment (os/release and any custom installs)? Try 1.1.0, and
make sure to re-create the container as the new config file should be
more correct for systemd backed containers.

the host is Debian wheezy, kernel 3.18.4 and a backported shadow
package for newuidmap & Co.

For LXC I have now lxc-1.1, cgmanager-0.35 and lxcfs-0.5. This works
fine with an debian-container for wheezy and jessie. But if I switch
jessie to systemd, it fails. I just tried an upgrade to experimental,
but this fails, too.

Now I started with a fresh template download of ubuntu vivid. This
works fine, too, but with upstart. If I install systemd an put a
lxc.init_cmd=/bin/systemd, it fails too:

$ lxc-start -n ubuntu -F -l DEBUG
WARN: could not reopen tty: Permission denied
systemd 218 running in system mode. (+PAM +AUDIT +SELINUX +IMA
+APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL
+XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Detected virtualization 'lxc'.
Detected architecture 'x86-64'.

Welcome to Ubuntu Vivid Vervet (development branch)!

Set hostname to <ubuntu>.
/etc/mtab is not a symlink or not pointing to /proc/self/mounts. This
is not supported anymore. Please make sure to replace this file by a
symlink to avoid incorrect or misleading mount(8) output.
Failed to install release agent, ignoring: No such file or directory

That's the same behaviour as with debian jessie and systemd. At this
point even an lxc-attach does not work...

I have no idea, what is going on, maybe there is something too old on
debian wheezy as a host?

Best regards

Dirk

[Serge Hallyn]

Post by Dirk Geschke
Hi Serge,

Post by Serge Hallyn

Post by Dirk Geschke
I just to follow
https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/
once more to install a new container and it fails. First of all it
was a problem with the access to the directory
~/.local/share/lxc/jessie1
The owner changed to a mapped one -> 100000 and then there was no
access for the lxcuser, which has uid 1001. I fixed this via setting
write access for the users group.
lxc-create -t download -n jessie1 -- -d debian -r jessie -a amd64
which worked without problems (except warnings regarding reopen tty).
~$ lxc-start -n jessie1
lxc_container: Permission denied - Unable to create /dev/.lxc for autodev
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
Here it ends, nothing more happens and only a kill -9 works...
/sbin/init -> /lib/systemd/systemd
I suspect, this does not work at all without cgroup namespace support
in the kernel? Or am I missing something else?

There's something else you're missing, but I'm not sure what. What is
your environment (os/release and any custom installs)? Try 1.1.0, and
make sure to re-create the container as the new config file should be
more correct for systemd backed containers.

the host is Debian wheezy, kernel 3.18.4 and a backported shadow
package for newuidmap & Co.
For LXC I have now lxc-1.1, cgmanager-0.35 and lxcfs-0.5. This works
fine with an debian-container for wheezy and jessie. But if I switch
jessie to systemd, it fails. I just tried an upgrade to experimental,
but this fails, too.
Now I started with a fresh template download of ubuntu vivid. This
works fine, too, but with upstart. If I install systemd an put a
$ lxc-start -n ubuntu -F -l DEBUG
WARN: could not reopen tty: Permission denied
systemd 218 running in system mode. (+PAM +AUDIT +SELINUX +IMA
+APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL
+XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Detected virtualization 'lxc'.
Detected architecture 'x86-64'.
Welcome to Ubuntu Vivid Vervet (development branch)!
Set hostname to <ubuntu>.
/etc/mtab is not a symlink or not pointing to /proc/self/mounts. This
is not supported anymore. Please make sure to replace this file by a
symlink to avoid incorrect or misleading mount(8) output.
Failed to install release agent, ignoring: No such file or directory
That's the same behaviour as with debian jessie and systemd. At this
point even an lxc-attach does not work...
I have no idea, what is going on, maybe there is something too old on
debian wheezy as a host?

Hm, seems unlikely. From lxc's pov things seem ok since init is starting.
Could you show the container configuration file as well as any files
which are lxc.include'd from it? I'd start systemd with its debug options
to get more info about where it's going wrong.

When you say even lxc-attach doesn't work - what happens when you run it?
That's definately weird. lxc-attach requires nothing from the container
itself other than a /bin/sh in rootfs and a init task that exists.

[Dirk Geschke]

Hi Serge,

Post by Serge Hallyn
Hm, seems unlikely. From lxc's pov things seem ok since init is starting.
Could you show the container configuration file as well as any files
which are lxc.include'd from it? I'd start systemd with its debug options
to get more info about where it's going wrong.
When you say even lxc-attach doesn't work - what happens when you run it?
That's definately weird. lxc-attach requires nothing from the container
itself other than a /bin/sh in rootfs and a init task that exists.

I'm a step further, I guess...

lxc-attach simply does nothing, it is blocked without any action but
I can terminate it with CTRL-C.

I get further, if I kill lxcfs, in this case I get these messages:

Failed to open pin file: Transport endpoint is not connected
Failed to allocate manager object: Transport endpoint is not
connected
[!!!!!!] Failed to allocate manager object, freezing.

But then I can attach to the container. As expected, a df shows:

df: '/proc/cpuinfo': Transport endpoint is not connected
df: '/proc/meminfo': Transport endpoint is not connected
df: '/proc/stat': Transport endpoint is not connected
df: '/proc/uptime': Transport endpoint is not connected
df: '/sys/fs/cgroup/blkio': Transport endpoint is not connected
df: '/sys/fs/cgroup/cpu': Transport endpoint is not connected
...

So there seems to be a problem in the communication between
systemd and lxcfs. A ps does not work, mentions /proc should
be mounted, but it is there:

lrwxrwxrwx 1 root root 0 Feb 9 18:37 /proc/1/exe -> /lib/systemd/systemd

I guess, this is due to the now missing lxcfs...

The installed systemd is 218-7ubuntu1.

Attached is the debug output and the config file. Except the entry
lxc.init_cmd to start systemd, there are no changes to the default.

lxcfs is started this way:

~geschke/lxcfs/lxcfs -d -s -f -o allow_other /usr/local/var/lib/lxcfs

This one is linked against the newest libfuse, but this doesn't
change anything...

Best regards

Dirk

[Dirk Geschke]

Hi Serge,

now I started lxcfs within gdb:

(gdb) where
#0 0x00007ffff706b610 in __recvmsg_nocancel ()
at ../sysdeps/unix/syscall-template.S:82
#1 0x0000000000403cdd in recv_creds (sock=8, cred=0x7fffffffe5b0,
v=0x7fffffffe5af "1\377\377\377\377\377\377\377\377\377\377\377\377\001")
at lxcfs.c:799
#2 0x00000000004046f2 in do_write_pids (tpid=27028,
contrl=0x60a4c0 "name=systemd", cg=0x60c250 "wheezy/ubuntu-12",
file=0x60bad5 "/cgroup.procs", buf=0x62e920 "1\n") at lxcfs.c:1109
#3 0x0000000000404a16 in cg_write (
path=0x60bab0 "/cgroup/name=systemd/wheezy/ubuntu-12/cgroup.procs",
buf=0x7ffff7fd3060 "1\n", size=2, offset=0, fi=0x7fffffffe850)
at lxcfs.c:1187
#4 0x00000000004069b9 in lxcfs_write (
path=0x60bab0 "/cgroup/name=systemd/wheezy/ubuntu-12/cgroup.procs",
buf=0x7ffff7fd3060 "1\n", size=2, offset=0, fi=0x7fffffffe850)
at lxcfs.c:2016
#5 0x00007ffff7bb1513 in fuse_fs_write_buf (fs=0x60c040,
path=0x60bab0 "/cgroup/name=systemd/wheezy/ubuntu-12/cgroup.procs",
buf=***@entry=0x7fffffffe900, off=***@entry=0, fi=***@entry=0x7fffffffe850)
at fuse.c:1878
#6 0x00007ffff7bb1686 in fuse_lib_write_buf (req=0x60aa20, ino=23,
buf=0x7fffffffe900, off=0, fi=0x7fffffffe850) at fuse.c:3278
#7 0x00007ffff7bb9209 in do_write_buf (ibuf=0x7fffffffe990,
inarg=<optimized out>, nodeid=<optimized out>, req=<optimized out>)
at fuse_lowlevel.c:1300
#8 fuse_ll_process_buf (data=<optimized out>, buf=0x7fffffffe990,
ch=<optimized out>) at fuse_lowlevel.c:2437
#9 0x00007ffff7bb588f in fuse_session_loop (se=0x60b720) at fuse_loop.c:40
#10 0x00007ffff7bade48 in fuse_loop (f=***@entry=0x60bd30) at fuse.c:4313
#11 0x00007ffff7bbd95f in fuse_main_common (argc=<optimized out>,
argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
user_data=<optimized out>, compat=<optimized out>) at helper.c:357
#12 0x0000000000406ccd in main (argc=7, argv=0x7fffffffebe8) at lxcfs.c:2158

Do you have any idea what is connected to sock=8? lsof does not
work as long as the fuse mount exist (even so df). So I see the
lxcfs socket:

# netstat -pan |grep 228936
unix 3 [ ] DGRAM 228936 26992/lxcfs @00073

But what should be on the other side of the socket, hence, what is
not answering here?

Best regards

Dirk

[Serge Hallyn]

Post by Dirk Geschke
Hi Serge,
(gdb) where
#0 0x00007ffff706b610 in __recvmsg_nocancel ()
at ../sysdeps/unix/syscall-template.S:82
#1 0x0000000000403cdd in recv_creds (sock=8, cred=0x7fffffffe5b0,
v=0x7fffffffe5af "1\377\377\377\377\377\377\377\377\377\377\377\377\001")
at lxcfs.c:799
#2 0x00000000004046f2 in do_write_pids (tpid=27028,
contrl=0x60a4c0 "name=systemd", cg=0x60c250 "wheezy/ubuntu-12",
file=0x60bad5 "/cgroup.procs", buf=0x62e920 "1\n") at lxcfs.c:1109

Hm. What happens here is lxcfs forks a child which jumps
into the container's pidns and listens over a socket for
pids to send back as a struct cred so that the kernel will
xlate into the container's pidns.

It looks like the parent is waiting for the client to send a
cred back. This really shouldn't happen. Does 'ps -ef | grep lxcfs'
show that there is only one lxcfs task?

I'd expect this sort of thing before commit 01e718521ab0639ffec091645c0c6eea52ca4f8e
but definately not in 0.5.

Post by Dirk Geschke
#3 0x0000000000404a16 in cg_write (
path=0x60bab0 "/cgroup/name=systemd/wheezy/ubuntu-12/cgroup.procs",
buf=0x7ffff7fd3060 "1\n", size=2, offset=0, fi=0x7fffffffe850)
at lxcfs.c:1187
#4 0x00000000004069b9 in lxcfs_write (
path=0x60bab0 "/cgroup/name=systemd/wheezy/ubuntu-12/cgroup.procs",
buf=0x7ffff7fd3060 "1\n", size=2, offset=0, fi=0x7fffffffe850)
at lxcfs.c:2016
#5 0x00007ffff7bb1513 in fuse_fs_write_buf (fs=0x60c040,
path=0x60bab0 "/cgroup/name=systemd/wheezy/ubuntu-12/cgroup.procs",
at fuse.c:1878
#6 0x00007ffff7bb1686 in fuse_lib_write_buf (req=0x60aa20, ino=23,
buf=0x7fffffffe900, off=0, fi=0x7fffffffe850) at fuse.c:3278
#7 0x00007ffff7bb9209 in do_write_buf (ibuf=0x7fffffffe990,
inarg=<optimized out>, nodeid=<optimized out>, req=<optimized out>)
at fuse_lowlevel.c:1300
#8 fuse_ll_process_buf (data=<optimized out>, buf=0x7fffffffe990,
ch=<optimized out>) at fuse_lowlevel.c:2437
#9 0x00007ffff7bb588f in fuse_session_loop (se=0x60b720) at fuse_loop.c:40
#11 0x00007ffff7bbd95f in fuse_main_common (argc=<optimized out>,
argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
user_data=<optimized out>, compat=<optimized out>) at helper.c:357
#12 0x0000000000406ccd in main (argc=7, argv=0x7fffffffebe8) at lxcfs.c:2158
Do you have any idea what is connected to sock=8? lsof does not
work as long as the fuse mount exist (even so df). So I see the

yeah the sock=8 presumably is the socketpair created in do_write_pids
to talk to the child we fork.

Post by Dirk Geschke
# netstat -pan |grep 228936
But what should be on the other side of the socket, hence, what is
not answering here?
Best regards
Dirk
--
+----------------------------------------------------------------------+
| Dr. Dirk Geschke / Plankensteinweg 61 / 85435 Erding |
| Telefon: [phone] / Mobil: 0176-96906350 / Fax: [phone] |
+----------------------------------------------------------------------+

[Dirk Geschke]

Hi Serge,

Post by Dirk Geschke
(gdb) where
#0 0x00007ffff706b610 in __recvmsg_nocancel ()
at ../sysdeps/unix/syscall-template.S:82
#1 0x0000000000403cdd in recv_creds (sock=8, cred=0x7fffffffe5b0,
v=0x7fffffffe5af "1\377\377\377\377\377\377\377\377\377\377\377\377\001")
at lxcfs.c:799

hmm, maybe this is the reason? There is a Zombie in the process
list:

root 27218 0.0 0.0 0 0 pts/1 Z 20:51 0:00 [lxcfs]
<defunct>

So probably lxcfs forks off a second copy of itself and communicates
via a socketpair with each other. But if the one fails, it won't
answer?

This one is a child of the first one:

$ ps -lp 27218
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
5 Z 0 27218 26992 0 80 0 - 0 ? pts/1 00:00:00 lxcf <defunct>

Maybe it is an lxcfs problem at all?

Best regards

Dirk

[Serge Hallyn]

Post by Dirk Geschke
Hi Serge,

Post by Dirk Geschke
(gdb) where
#0 0x00007ffff706b610 in __recvmsg_nocancel ()
at ../sysdeps/unix/syscall-template.S:82
#1 0x0000000000403cdd in recv_creds (sock=8, cred=0x7fffffffe5b0,
v=0x7fffffffe5af "1\377\377\377\377\377\377\377\377\377\377\377\377\001")
at lxcfs.c:799

hmm, maybe this is the reason? There is a Zombie in the process
root 27218 0.0 0.0 0 0 pts/1 Z 20:51 0:00 [lxcfs]
<defunct>
So probably lxcfs forks off a second copy of itself and communicates
via a socketpair with each other. But if the one fails, it won't
answer?
$ ps -lp 27218
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
5 Z 0 27218 26992 0 80 0 - 0 ? pts/1 00:00:00 lxcf <defunct>
Maybe it is an lxcfs problem at all?

How have you been installing lxcfs? Is it possible that you have
an old copy sitting around?

[Dirk Geschke]

Hi Serge,

Post by Serge Hallyn

Post by Dirk Geschke
Maybe it is an lxcfs problem at all?

How have you been installing lxcfs? Is it possible that you have
an old copy sitting around?

Argh, that's the problem:

1061 if (setns(newnsfd, 0) < 0)

There is no setns in glibc of debian wheezy, therefore I copied
this from lxc:

/* Define setns() if missing from the C library */
#ifndef HAVE_SETNS
static inline int setns(int fd, int nstype)
{
#ifdef __NR_setns
return syscall(__NR_setns, fd, nstype);
#elif defined(__NR_set_ns)
return syscall(__NR_set_ns, fd, nstype);
#else
errno = ENOSYS;
return -1;
#endif
}
#endif

There is a __NR_setns:

/usr/include/x86_64-linux-gnu/asm/unistd_64.h:#define __NR_setns 308

but somehow this is not included during compile time, therefore
setns is ...hmm... empty?

It results in an

errno = -38 ;
return -1;

Call me simply stupid, I had forgotten this adjustment.

I simply added

#define __NR_setns 308

to lxcfs.c and now it works, it boots ubuntu!

Ok, I still these errors on login:

Failed to set cpu.shares on /wheezy/ubuntu-18: Permission denied
Failed to set cpu.cfs_period_us on /wheezy/ubuntu-18: Permission denied
Failed to set cpu.cfs_quota_us on /wheezy/ubuntu-18: Permission denied
Failed to set blkio.weight on /wheezy/ubuntu-18: Permission denied
Failed to set memory.limit_in_bytes on /wheezy/ubuntu-18: Permission
denied
Failed to reset devices.list on /wheezy/ubuntu-18: Permission denied
Failed to reset devices.list on /wheezy/ubuntu-18/user.slice: Permission
denied
Failed to reset devices.list on
/wheezy/ubuntu-18/user.slice/user-0.slice: Permission denied
Failed to reset devices.list on
/wheezy/ubuntu-18/user.slice/user-0.slice/***@0.service: Permission
denied
Failed to reset devices.list on /wheezy/ubuntu-18/system.slice:
Permission denied
Cannot determine UID from slice user-0.slice
Failed to reset devices.list on
/wheezy/ubuntu-18/user.slice/user-0.slice/session-2.scope: Permission
denied

But it works!

Please accept my apologies for being a moron...

Best regards and many thanks for your help and patience!

Dirk

[Serge Hallyn]

Post by Dirk Geschke
Hi Serge,

Post by Serge Hallyn

Post by Dirk Geschke
Maybe it is an lxcfs problem at all?

How have you been installing lxcfs? Is it possible that you have
an old copy sitting around?

1061 if (setns(newnsfd, 0) < 0)
There is no setns in glibc of debian wheezy, therefore I copied
/* Define setns() if missing from the C library */
#ifndef HAVE_SETNS
static inline int setns(int fd, int nstype)
{
#ifdef __NR_setns
return syscall(__NR_setns, fd, nstype);
#elif defined(__NR_set_ns)
return syscall(__NR_set_ns, fd, nstype);
#else
errno = ENOSYS;
return -1;
#endif
}
#endif
/usr/include/x86_64-linux-gnu/asm/unistd_64.h:#define __NR_setns 308
but somehow this is not included during compile time, therefore
setns is ...hmm... empty?
It results in an
errno = -38 ;
return -1;
Call me simply stupid, I had forgotten this adjustment.
I simply added
#define __NR_setns 308
to lxcfs.c and now it works, it boots ubuntu!
Failed to set cpu.shares on /wheezy/ubuntu-18: Permission denied
Failed to set cpu.cfs_period_us on /wheezy/ubuntu-18: Permission denied
Failed to set cpu.cfs_quota_us on /wheezy/ubuntu-18: Permission denied
Failed to set blkio.weight on /wheezy/ubuntu-18: Permission denied
Failed to set memory.limit_in_bytes on /wheezy/ubuntu-18: Permission
denied
Failed to reset devices.list on /wheezy/ubuntu-18: Permission denied
Failed to reset devices.list on /wheezy/ubuntu-18/user.slice: Permission
denied
Failed to reset devices.list on
/wheezy/ubuntu-18/user.slice/user-0.slice: Permission denied
Failed to reset devices.list on
denied
Permission denied
Cannot determine UID from slice user-0.slice
Failed to reset devices.list on
/wheezy/ubuntu-18/user.slice/user-0.slice/session-2.scope: Permission
denied
But it works!
Please accept my apologies for being a moron...
Best regards and many thanks for your help and patience!

NP :)

-serge