Discussion:
sshd-keygen fails during container boot
(too old to reply)
Peter Steele
2015-12-04 16:33:43 UTC
Permalink
I'm seeing these messages on some of my containers during their initial
start-up:

systemd: Failed at step CGROUP spawning /usr/sbin/sshd-keygen: No such
file or directory
systemd: sshd-keygen.service: main process exited, code=exited,
status=219/CGROUP
systemd: Failed to start OpenSSH Server Key Generation.
systemd: Unit sshd-keygen.service entered failed state.

The net effect of this of course is that I cannot ssh into the
containers that encounter this problem. The odd thing is that
/usr/sbin/sshd-keygen *does* exist, and restarting the containers
corrects the problem (the keygen works on the reboot). It's not clear to
me why the containers think this file is missing and why some containers
running pretty much the identical image do not hit this problem. Has
anyone seen this?
Serge Hallyn
2015-12-04 21:38:08 UTC
Permalink
Post by Peter Steele
I'm seeing these messages on some of my containers during their
systemd: Failed at step CGROUP spawning /usr/sbin/sshd-keygen: No
such file or directory
systemd: sshd-keygen.service: main process exited, code=exited,
status=219/CGROUP
systemd: Failed to start OpenSSH Server Key Generation.
systemd: Unit sshd-keygen.service entered failed state.
The net effect of this of course is that I cannot ssh into the
containers that encounter this problem. The odd thing is that
/usr/sbin/sshd-keygen *does* exist, and restarting the containers
corrects the problem (the keygen works on the reboot). It's not
clear to me why the containers think this file is missing and why
some containers running pretty much the identical image do not hit
this problem. Has anyone seen this?
My guess is that the no such file or directory is talking about a
cgroup dir. what does /proc/1/cgroup in the container show? Make sure
to run the latest lxcfs on the host, as that's needed because
systemd moves itself to name=systemd:/init.scope cgroup.

When systemd is having trouble, you can usually find out more info by running

lxc-start -n containername -F -o /dev/stdout -- /sbin/init log_target=console log_level=debug
Peter Steele
2015-12-04 21:59:55 UTC
Permalink
Post by Serge Hallyn
My guess is that the no such file or directory is talking about a
cgroup dir. what does /proc/1/cgroup in the container show? Make sure
to run the latest lxcfs on the host, as that's needed because
systemd moves itself to name=systemd:/init.scope cgroup.
When systemd is having trouble, you can usually find out more info by running
lxc-start -n containername -F -o /dev/stdout -- /sbin/init log_target=console log_level=debug
I'm actually not (yet) running lxcfs. My understanding was that it isn't
absolutely required but it does offer several benefits. I'd planned to
tackle lxcfs after getting things running without it first, and my
containers appear to be running reasonably okay (although I am seeing
some odd behavior). If it's *is* needed, then I guess it's time to
tackle lxcfs.
Serge Hallyn
2015-12-07 15:49:56 UTC
Permalink
Post by Peter Steele
Post by Serge Hallyn
My guess is that the no such file or directory is talking about a
cgroup dir. what does /proc/1/cgroup in the container show? Make sure
to run the latest lxcfs on the host, as that's needed because
systemd moves itself to name=systemd:/init.scope cgroup.
When systemd is having trouble, you can usually find out more info by running
lxc-start -n containername -F -o /dev/stdout -- /sbin/init log_target=console log_level=debug
I'm actually not (yet) running lxcfs. My understanding was that it
isn't absolutely required but it does offer several benefits. I'd
planned to tackle lxcfs after getting things running without it
first, and my containers appear to be running reasonably okay
(although I am seeing some odd behavior). If it's *is* needed, then
I guess it's time to tackle lxcfs.
Oh right, since you're running your containers unconfined, you should
be fine without it.

Still would be useful to see systemd's output :)
Peter Steele
2015-12-07 22:23:12 UTC
Permalink
Post by Serge Hallyn
Post by Peter Steele
I'm actually not (yet) running lxcfs. My understanding was that it
isn't absolutely required but it does offer several benefits. I'd
planned to tackle lxcfs after getting things running without it
first, and my containers appear to be running reasonably okay
(although I am seeing some odd behavior). If it's *is* needed, then
I guess it's time to tackle lxcfs.
Oh right, since you're running your containers unconfined, you should
be fine without it.
Still would be useful to see systemd's output :)
I'm having some trouble reproducing this behavior when I launch a
container with

lxc-start -n containername -F -o /dev/stdout -- /sbin/init log_target=console log_level=debug


I'm seeing these errors when the containers are launched as part of my
automated installation framework. The installs that are done in this
manner are hands-free and the containers, once installed, are launched
with a simple

lxc-start -n containername

Multiple containers are created and launched in parallel as part of this
installation system. This is all controlled through automation (Python),
and the framework expects the launched container processes to become
daemons. I don't get the detailed startup output that happens when
lxc-start is run in the foreground. If I create a container
interactively and launch it with -F option manually, there are no errors
in /var/log/messages.

The errors I'm seeing are not limited to the sshd-keygen errors I
originally reported. In further tests since then I see similar errors
being reported for a variety of things. For example:

Dec 7 13:52:00 pws-vm-00 systemd: Failed at step CGROUP spawning
/usr/bin/kmod: No such file or directory
Dec 7 13:52:00 pws-vm-00 systemd: Mounted Huge Pages File System.
Dec 7 13:52:00 pws-vm-00 systemd: kmod-static-nodes.service: main
process exited, code=exited, status=219/CGROUP
Dec 7 13:52:00 pws-vm-00 systemd: Failed to start Create list of
required static device nodes for the current kernel.
Dec 7 13:52:00 pws-vm-00 systemd: Unit kmod-static-nodes.service
entered failed state.

Dec 7 13:52:01 pws-vm-00 systemd: Failed at step CGROUP spawning
/etc/rc.d/init.d/jexec: No such file or directory
Dec 7 13:52:01 pws-vm-00 systemd: jexec.service: control process
exited, code=exited status=219
Dec 7 13:52:01 pws-vm-00 systemd: Failed to start LSB: Supports the
direct execution of binary formats..
Dec 7 13:52:01 pws-vm-00 systemd: Unit jexec.service entered failed state.

I've also seen errors reported for network.service and others. In every
case, if I reboot my server forcing the hosted containers to be
restarted, everything comes up fine. If I run the same installation
framework using the same container images but instead use libvirt-lxc
create/start commands, I do not see these systemd errors. To the best of
my knowledge, the LXC config I'm using more or less matches the config
I'm using with my libvirt containers, for example

# cat /var/lib/lxc/vm-00/config
lxc.utsname = pws-vm-00
lxc.include = /var/lib/hf/lxc.conf
lxc.network.veth.pair = vm-00
lxc.network.hwaddr = fe:d6:e8:96:7e:2d
lxc.rootfs = /hf/cs/vm-00/rootfs
lxc.cgroup.memory.limit_in_bytes = 1073741824
lxc.cgroup.memory.memsw.limit_in_bytes = 2147483648
lxc.hook.autodev = /var/lib/lxc/vm-00/autodev
lxc.cgroup.devices.allow = b 8:3 rwm

# cat /var/lib/hf/lxc.conf
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64

# Networking defaults
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0

Am I missing anything in my config? I am running LXC version 1.1.5 under
a CentOS host. All containers are also CentOS and are privileged.

Peter
Serge Hallyn
2015-12-08 16:00:23 UTC
Permalink
Post by Peter Steele
Post by Serge Hallyn
Post by Peter Steele
I'm actually not (yet) running lxcfs. My understanding was that it
isn't absolutely required but it does offer several benefits. I'd
planned to tackle lxcfs after getting things running without it
first, and my containers appear to be running reasonably okay
(although I am seeing some odd behavior). If it's *is* needed, then
I guess it's time to tackle lxcfs.
Oh right, since you're running your containers unconfined, you should
be fine without it.
Still would be useful to see systemd's output :)
I'm having some trouble reproducing this behavior when I launch a
container with
lxc-start -n containername -F -o /dev/stdout -- /sbin/init log_target=console log_level=debug
I'm seeing these errors when the containers are launched as part of
my automated installation framework. The installs that are done in
this manner are hands-free and the containers, once installed, are
launched with a simple
Ok, can you change the launch command in the scripts to

lxc-start -n $containername -L /tmp/$containername.cout -l trace -o /tmp/$containername.dout -- /sbin/init log_target=console log_level=debug

The console output will go into the .cout file and lxc debug output into .dout.
Peter Steele
2015-12-08 19:10:10 UTC
Permalink
Post by Serge Hallyn
Ok, can you change the launch command in the scripts to
lxc-start -n $containername -L /tmp/$containername.cout -l trace -o /tmp/$containername.dout -- /sbin/init log_target=console log_level=debug
The console output will go into the .cout file and lxc debug output into .dout.
I've actually made some progress in reproducing this outside of my
framework. I originally thought the problem only occurred during the
first boot of the containers. I've discovered that it can happen any
time the server is rebooted and the containers are started when the
server comes up. I've only seen this problem when multiple containers
are starting at the same time.

I incorporated your modified start command into a test as follows:

# for vm in `lxc-ls`; do lxc-start -n $vm -L /tmp/$vm.cout -l trace -o
/tmp/$vm.dout -- /sbin/init log_target=console log_level=debug; done

This starts all of my previously created containers at roughly the same
time, and when I do this some of the containers encounter the systemd
errors I've been seeing. Which containers hit these errors vary from
test to test. In looking at the .dout logs, I noticed the following:

lxc-start 1449591253.647 DEBUG lxc_conf -
conf.c:setup_rootfs:1295 - mounted '/hf/cs/vm-00/rootfs' on
'/usr/lib64/lxc/rootfs'
lxc-start 1449591253.647 INFO lxc_conf -
conf.c:setup_utsname:928 - 'pws-vm-00' hostname has been setup
lxc-start 1449591253.660 DEBUG lxc_conf -
conf.c:setup_hw_addr:2368 - mac address 'fe:d6:e8:96:7e:2d' on 'eth0'
has been setup
lxc-start 1449591253.660 DEBUG lxc_conf -
conf.c:setup_netdev:2595 - 'eth0' has been setup
lxc-start 1449591253.660 INFO lxc_conf -
conf.c:setup_network:2616 - network has been setup
lxc-start 1449591253.660 INFO lxc_conf -
conf.c:mount_autodev:1157 - Mounting container /dev
lxc-start 1449591253.661 INFO lxc_conf -
conf.c:mount_autodev:1179 - Mounted tmpfs onto /usr/lib64/lxc/rootfs/dev
lxc-start 1449591253.661 INFO lxc_conf -
conf.c:mount_autodev:1197 - Mounted container /dev
lxc-start 1449591253.661 ERROR lxc_utils -
utils.c:open_without_symlink:1626 - No such file or directory - Error
examining fuse in /usr/lib64/lxc/rootfs/sys/fs/fuse/connections
lxc-start 1449591253.661 INFO lxc_conf -
conf.c:mount_entry:1727 - failed to mount '/sys/fs/fuse/connections' on
'/usr/lib64/lxc/rootfs/sys/fs/fuse/connections' (optional): No such file
or directory

All of the containers report this error, but what caught my eye is the
mount point referenced in this error. Is this same mount point used for
all containers that are being started? I assume this error is misleading
but I tried changing my for loop to add a 1 second delay in starting
each container, and after doing this there were no systemd errors.
Unfortunately, adding a delay in my install framework had no effect, so
I suspect the apparent positive results in adding a delay to the for
loop was just luck.

This same container reported the following errors in its /var/log/messages:

Dec 8 08:06:39 pws-vm-00 systemd: Starting Dump dmesg to /var/log/dmesg...
Dec 8 08:06:39 pws-vm-00 systemd: Failed at step CGROUP spawning
/etc/rc.d/init.d/jexec: No such file or directory
Dec 8 08:06:39 pws-vm-00 systemd: Starting Permit User Sessions...
Dec 8 08:06:39 pws-vm-00 systemd: Starting LSB: Bring up/down networking...
Dec 8 08:06:39 pws-vm-00 systemd: jexec.service: control process
exited, code=exited status=219
Dec 8 08:06:39 pws-vm-00 systemd: Failed to start LSB: Supports the
direct execution of binary formats..
Dec 8 08:06:39 pws-vm-00 systemd: Unit jexec.service entered failed state.

The .cout file for this same container didn't really have any thing of
note, other than it also reported some of these errors:

Starting LSB: Bring up/down networking...
Starting D-Bus System Message Bus...
OK Started D-Bus System Message Bus.
FAILED Failed to start LSB: Supports the direct execution of binary
formats..
See 'systemctl status jexec.service' for details.

There was nothing related to this error in the .dout file. The start of
this file does have some warnings:

lxc-start 1449590798.820 INFO lxc_start_ui -
lxc_start.c:main:264 - using rcfile /var/lib/lxc/vm-00/config
lxc-start 1449590798.822 WARN lxc_confile -
confile.c:config_pivotdir:1801 - lxc.pivotdir is ignored. It will soon
become an error.
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
cpuset unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpu
unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
blkio unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
memory unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
devices unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
freezer unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
net_cls unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
perf_event unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 WARN lxc_cgfs -
cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup
hugetlb unknown to /var/lib/lxc vm-00
lxc-start 1449590798.823 INFO lxc_start -
start.c:lxc_check_inherited:226 - closed inherited fd 4
lxc-start 1449590798.825 INFO lxc_container -
lxccontainer.c:do_lxcapi_start:712 - Attempting to set proc title to
[lxc monitor] /var/lib/lxc vm-00
lxc-start 1449590798.825 ERROR lxc_utils -
utils.c:setproctitle:1461 - Invalid argument - setting cmdline failed

and there were some later on related to secomp:

lxc-start 1449590798.825 INFO lxc_seccomp -
seccomp.c:parse_config_v2:426 - Adding native rule for finit_module
action 327681
lxc-start 1449590798.825 WARN lxc_seccomp -
seccomp.c:do_resolve_add_rule:233 - Seccomp: got negative # for syscall:
finit_module
lxc-start 1449590798.825 WARN lxc_seccomp -
seccomp.c:do_resolve_add_rule:234 - This syscall will NOT be blacklisted
lxc-start 1449590798.825 INFO lxc_seccomp -
seccomp.c:parse_config_v2:429 - Adding compat rule for finit_module
action 327681
lxc-start 1449590798.825 WARN lxc_seccomp -
seccomp.c:do_resolve_add_rule:233 - Seccomp: got negative # for syscall:
finit_module
lxc-start 1449590798.825 WARN lxc_seccomp -
seccomp.c:do_resolve_add_rule:234 - This syscall will NOT be blacklisted
lxc-start 1449590798.825 INFO lxc_seccomp -
seccomp.c:parse_config_v2:324 - processing: .delete_module errno 1.

along with a few other unrelated warnings. There are no errors in any of
the logs that point to an obvious cause for these system errors, at
least not to my eyes. Plus, I get a completely different set of failed
VMs each time I run through an install, with different services being
impacted each time. Is there anything in particular I should look for in
the .cout and .dout logs that might help explain what's going on?
Peter Steele
2015-12-08 22:21:34 UTC
Permalink
Post by Serge Hallyn
Ok, can you change the launch command in the scripts to
lxc-start -n $containername -L /tmp/$containername.cout -l trace -o
/tmp/$containername.dout -- /sbin/init log_target=console
log_level=debug
The console output will go into the .cout file and lxc debug output into .dout.
I did another test and hit the original sshd-keygen error on one of my
containers. The /var/log/messages file for that container reported the
following:

Dec 8 11:58:13 pws-vm-04 systemd: Starting OpenSSH Server Key Generation...
Dec 8 11:58:13 pws-vm-04 systemd: Failed at step CGROUP spawning
/usr/sbin/sshd-keygen: No such file or directory
Dec 8 11:58:13 pws-vm-04 systemd: sshd-keygen.service: main process
exited, code=exited, status=219/CGROUP
Dec 8 11:58:13 pws-vm-04 systemd: Failed to start OpenSSH Server Key
Generation.
Dec 8 11:58:13 pws-vm-04 systemd: Unit sshd-keygen.service entered
failed state.
Dec 8 11:58:16 pws-vm-04 systemd: Starting OpenSSH server daemon...
Dec 8 11:58:16 pws-vm-04 systemd: Started OpenSSH server daemon.
Dec 8 11:58:16 pws-vm-04 sshd: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:58:16 pws-vm-04 sshd: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:58:16 pws-vm-04 sshd: Could not load host key:
/etc/ssh/ssh_host_ed25519_key
Dec 8 11:58:58 pws-vm-04 sshd[722]: error: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:58:58 pws-vm-04 sshd[722]: error: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:58:58 pws-vm-04 sshd[722]: error: Could not load host key:
/etc/ssh/ssh_host_ed25519_key
Dec 8 11:59:00 pws-vm-04 sshd[724]: error: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:59:00 pws-vm-04 sshd[724]: error: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:59:00 pws-vm-04 sshd[724]: error: Could not load host key:
/etc/ssh/ssh_host_ed25519_key
Dec 8 11:59:00 pws-vm-04 sshd[726]: error: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:59:00 pws-vm-04 sshd[726]: error: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:59:00 pws-vm-04 sshd[726]: error: Could not load host key:
/etc/ssh/ssh_host_ed25519_key
Dec 8 11:59:00 pws-vm-04 sshd[728]: error: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:59:00 pws-vm-04 sshd[728]: error: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:59:00 pws-vm-04 sshd[728]: error: Could not load host key:
/etc/ssh/ssh_host_ed25519_key
Dec 8 11:59:00 pws-vm-04 sshd[730]: error: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:59:00 pws-vm-04 sshd[730]: error: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:59:00 pws-vm-04 sshd[730]: error: Could not load host key:
/etc/ssh/ssh_host_ed25519_key
Dec 8 11:59:02 pws-vm-04 sshd[831]: error: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:59:02 pws-vm-04 sshd[831]: error: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:59:02 pws-vm-04 sshd[831]: error: Could not load host key:
/etc/ssh/ssh_host_ed25519_key
Dec 8 11:59:02 pws-vm-04 sshd[833]: error: Could not load host key:
/etc/ssh/ssh_host_rsa_key
Dec 8 11:59:02 pws-vm-04 sshd[833]: error: Could not load host key:
/etc/ssh/ssh_host_ecdsa_key
Dec 8 11:59:02 pws-vm-04 sshd[833]: error: Could not load host key:
/etc/ssh/ssh_host_ed25519_key

The .cout file for this container looked normal except, of course, for
some ssh related messages:

Starting OpenSSH Server Key Generation...
FAILED Failed to start OpenSSH Server Key Generation.
See 'systemctl status sshd-keygen.service' for details.
Starting OpenSSH server daemon...

The .dout file had no errors related to sshd in this particular test.

In another test, I tried creating several containers with the command

lxc-create -n testN -t download -- -d centos -r 7 -a amd64

and then started them all in the same manner as my other tests using:

for vm in `lxc-ls`; do lxc-start -n $vm -L /tmp/$vm.cout -l trace -o
/tmp/$vm.dout -- /sbin/init log_target=console log_level=debug; done

In this case of course the containers are using the stock downloaded
CentOS 7 image instead of my custom image. I was unable to reproduce the
systemd error through multiple start/stop tests of my containers. They
always started up without any complaints. Granted, these stock images
are simpler than my custom images, but this seems to point to something
in my image that's causing this issue. At least that gives me a bit to
go on, although it's hard to understand what additional rpm modules
would cause systemd to behave this way during bootup.

Peter
Peter Steele
2015-12-08 23:06:05 UTC
Permalink
Post by Peter Steele
In this case of course the containers are using the stock downloaded
CentOS 7 image instead of my custom image. I was unable to reproduce
the systemd error through multiple start/stop tests of my
containers. They always started up without any complaints. Granted,
these stock images are simpler than my custom images, but this seems
to point to something in my image that's causing this issue. At
least that gives me a bit to go on, although it's hard to understand
what additional rpm modules would cause systemd to behave this way
during bootup.
Are there specific lxc related configuration files needed in a CentOS
template that my custom image might not have? I'm asking because in
looking at /usr/share/lxc/templates/lxc-centos I see there are several
configuration steps that are performed against the installed CentOS
rootfs, such as disabling selinix, creating a minimal fstab, etc. Most
of these are already done as part of my image creation process, but I
see there are other steps I do not include, such as creating
/etc/init/lxc-sysinit.conf. This is done as part of a specific check for
CentOS release 6, so I wonder if there is something equivalent needed
for CentOS 7?
Serge Hallyn
2015-12-09 04:36:40 UTC
Permalink
Post by Peter Steele
Post by Serge Hallyn
Ok, can you change the launch command in the scripts to
lxc-start -n $containername -L /tmp/$containername.cout -l trace -o /tmp/$containername.dout -- /sbin/init log_target=console log_level=debug
The console output will go into the .cout file and lxc debug output into .dout.
I've actually made some progress in reproducing this outside of my
framework. I originally thought the problem only occurred during the
first boot of the containers. I've discovered that it can happen any
time the server is rebooted and the containers are started when the
server comes up.
What do you mean by "when the server comes up"? If you bring up the
server, let it set for 5 mins, then start them, they still fail?
Post by Peter Steele
I've only seen this problem when multiple
containers are starting at the same time.
What lxc version are you using again?
Post by Peter Steele
# for vm in `lxc-ls`; do lxc-start -n $vm -L /tmp/$vm.cout -l trace
-o /tmp/$vm.dout -- /sbin/init log_target=console log_level=debug;
done
This starts all of my previously created containers at roughly the
same time, and when I do this some of the containers encounter the
systemd errors I've been seeing. Which containers hit these errors
vary from test to test. In looking at the .dout logs, I noticed the
lxc-start 1449591253.647 DEBUG lxc_conf -
conf.c:setup_rootfs:1295 - mounted '/hf/cs/vm-00/rootfs' on
'/usr/lib64/lxc/rootfs'
lxc-start 1449591253.647 INFO lxc_conf -
conf.c:setup_utsname:928 - 'pws-vm-00' hostname has been setup
lxc-start 1449591253.660 DEBUG lxc_conf -
conf.c:setup_hw_addr:2368 - mac address 'fe:d6:e8:96:7e:2d' on
'eth0' has been setup
lxc-start 1449591253.660 DEBUG lxc_conf -
conf.c:setup_netdev:2595 - 'eth0' has been setup
lxc-start 1449591253.660 INFO lxc_conf -
conf.c:setup_network:2616 - network has been setup
lxc-start 1449591253.660 INFO lxc_conf -
conf.c:mount_autodev:1157 - Mounting container /dev
lxc-start 1449591253.661 INFO lxc_conf -
conf.c:mount_autodev:1179 - Mounted tmpfs onto
/usr/lib64/lxc/rootfs/dev
lxc-start 1449591253.661 INFO lxc_conf -
conf.c:mount_autodev:1197 - Mounted container /dev
lxc-start 1449591253.661 ERROR lxc_utils -
utils.c:open_without_symlink:1626 - No such file or directory -
Error examining fuse in
/usr/lib64/lxc/rootfs/sys/fs/fuse/connections
Ok, so this shows that in the container 'sys/fs' existed,
but fuse did not. This suggests that the fuse kernel module
was not done loading yet.

Could you add to /lib/systemd/system/lxc.service the line

ExecStartPre=modprobe fuse

and see if that helps? (I'm not sure if you'd also need to sleep
a short time to give syfs time to catch up, or if the modprobe
would wait... you could just use a script that waits until
/sys/fs/fuse exists on the host)
Peter Steele
2015-12-09 17:28:33 UTC
Permalink
Post by Serge Hallyn
What do you mean by "when the server comes up"? If you bring up the
server, let it set for 5 mins, then start them, they still fail?
What I meant here was that when my server boots, it launches our
management software, which in turns launches the containers that are
defined on that server. The systemd errors occur as the containers are
started. Delaying when the containers are started doesn't have any
effect. I've found though that if I put a five second delay between
starting each container, the systemd errors don't occur (at least not in
the tests I've run so far). I haven't had this issue with libvirt-lxc,
and I hope there is a better solution than this arbitrary five second delay.
Post by Serge Hallyn
What lxc version are you using again?
1.1.5.
Post by Serge Hallyn
Post by Serge Hallyn
Ok, so this shows that in the container 'sys/fs' existed,
but fuse did not. This suggests that the fuse kernel module
was not done loading yet.
Could you add to /lib/systemd/system/lxc.service the line
ExecStartPre=modprobe fuse
and see if that helps? (I'm not sure if you'd also need to sleep
a short time to give syfs time to catch up, or if the modprobe
would wait... you could just use a script that waits until
/sys/fs/fuse exists on the host)
I added this line to lxc.service and that cleared up the fuse issues.
This did not have any effect on the systemd errors though.

Peter
Serge Hallyn
2015-12-09 17:43:48 UTC
Permalink
Post by Peter Steele
Post by Serge Hallyn
What do you mean by "when the server comes up"? If you bring up the
server, let it set for 5 mins, then start them, they still fail?
What I meant here was that when my server boots, it launches our
management software, which in turns launches the containers that are
defined on that server. The systemd errors occur as the containers
are started. Delaying when the containers are started doesn't have
any effect. I've found though that if I put a five second delay
between starting each container, the systemd errors don't occur (at
least not in the tests I've run so far). I haven't had this issue
with libvirt-lxc, and I hope there is a better solution than this
arbitrary five second delay.
Post by Serge Hallyn
What lxc version are you using again?
1.1.5.
Post by Serge Hallyn
Post by Serge Hallyn
Ok, so this shows that in the container 'sys/fs' existed,
but fuse did not. This suggests that the fuse kernel module
was not done loading yet.
Could you add to /lib/systemd/system/lxc.service the line
ExecStartPre=modprobe fuse
and see if that helps? (I'm not sure if you'd also need to sleep
a short time to give syfs time to catch up, or if the modprobe
would wait... you could just use a script that waits until
/sys/fs/fuse exists on the host)
I added this line to lxc.service and that cleared up the fuse
issues. This did not have any effect on the systemd errors though.
And "the systemd errors" is the ssh-keygen ones only? Or is there
more?

And you do, or do not, also get these with containers created
through the download template?
Peter Steele
2015-12-09 18:07:53 UTC
Permalink
Post by Serge Hallyn
And "the systemd errors" is the ssh-keygen ones only? Or is there
more?
Various services are being impacted, for example, I saw these errors in
a run yesterday:

Dec 7 13:52:00 pws-vm-00 systemd: Failed at step CGROUP spawning
/usr/bin/kmod: No such file or directory
Dec 7 13:52:00 pws-vm-00 systemd: Mounted Huge Pages File System.
Dec 7 13:52:00 pws-vm-00 systemd: kmod-static-nodes.service: main
process exited, code=exited, status=219/CGROUP
Dec 7 13:52:00 pws-vm-00 systemd: Failed to start Create list of
required static device nodes for the current kernel.
Dec 7 13:52:00 pws-vm-00 systemd: Unit kmod-static-nodes.service
entered failed state.

Dec 7 13:52:01 pws-vm-00 systemd: Failed at step CGROUP spawning
/etc/rc.d/init.d/jexec: No such file or directory
Dec 7 13:52:01 pws-vm-00 systemd: jexec.service: control process
exited, code=exited status=219
Dec 7 13:52:01 pws-vm-00 systemd: Failed to start LSB: Supports the
direct execution of binary formats..
Dec 7 13:52:01 pws-vm-00 systemd: Unit jexec.service entered failed state.

At least a half dozen different services have failed in the various
tests I've done, and the set is always different from run to run.
Post by Serge Hallyn
And you do, or do not, also get these with containers created
through the download template?
Most of my tests have been with my custom containers of course since we
need the additional tools and files that make up our management
software. I did a test though where I blew away the containers that were
created by my install framework and replaced them all with the generic
CentOS download template. I was unable to reproduce the systemd errors
with this simple container. I then installed the additional OS modules
and other third party packages that we use in our software on top of
this basic container and the systemd errors returned. I'm going to break
this process down a bit more to see if I can identify what additions to
the base container cause systemd to fail.

Peter
Serge Hallyn
2015-12-09 18:18:52 UTC
Permalink
Post by Peter Steele
Post by Serge Hallyn
And "the systemd errors" is the ssh-keygen ones only? Or is there
more?
Various services are being impacted, for example, I saw these errors
Dec 7 13:52:00 pws-vm-00 systemd: Failed at step CGROUP spawning
/usr/bin/kmod: No such file or directory
Dec 7 13:52:00 pws-vm-00 systemd: Mounted Huge Pages File System.
Dec 7 13:52:00 pws-vm-00 systemd: kmod-static-nodes.service: main
process exited, code=exited, status=219/CGROUP
Dec 7 13:52:00 pws-vm-00 systemd: Failed to start Create list of
required static device nodes for the current kernel.
Dec 7 13:52:00 pws-vm-00 systemd: Unit kmod-static-nodes.service
entered failed state.
This is the kind of thing I'd expect when using cgmanager or lxcfs,
but not with straight lxc+cgfs.

Can you show what /sys/fs/cgroup tree and /proc/1/cgroup looks like in a
working container?
Post by Peter Steele
Dec 7 13:52:01 pws-vm-00 systemd: Failed at step CGROUP spawning
/etc/rc.d/init.d/jexec: No such file or directory
Dec 7 13:52:01 pws-vm-00 systemd: jexec.service: control process
exited, code=exited status=219
Dec 7 13:52:01 pws-vm-00 systemd: Failed to start LSB: Supports the
direct execution of binary formats..
Dec 7 13:52:01 pws-vm-00 systemd: Unit jexec.service entered failed state.
At least a half dozen different services have failed in the various
tests I've done, and the set is always different from run to run.
Post by Serge Hallyn
And you do, or do not, also get these with containers created
through the download template?
Most of my tests have been with my custom containers of course since
we need the additional tools and files that make up our management
software. I did a test though where I blew away the containers that
were created by my install framework and replaced them all with the
generic CentOS download template. I was unable to reproduce the
systemd errors with this simple container. I then installed the
additional OS modules and other third party packages that we use in
our software on top of this basic container and the systemd errors
returned. I'm going to break this process down a bit more to see if
I can identify what additions to the base container cause systemd to
fail.
Interesting.

I suppose just looking at the 'capsh --print' output difference for the
bounding set between the custom containers spawned by lxc and libvirt-lxc could
be enlightening.
Peter Steele
2015-12-09 19:46:51 UTC
Permalink
Post by Serge Hallyn
This is the kind of thing I'd expect when using cgmanager or lxcfs,
but not with straight lxc+cgfs. Can you show what /sys/fs/cgroup tree
and /proc/1/cgroup looks like in a working container?
As requested:

# ll /sys/fs/cgroup(top level only)
total 0
drwxr-xr-x 3 root root 60 Dec 9 10:12 blkio
lrwxrwxrwx 1 root root 11 Dec 9 10:12 cpu -> cpu,cpuacct
drwxr-xr-x 3 root root 60 Dec 9 10:12 cpu,cpuacct
lrwxrwxrwx 1 root root 11 Dec 9 10:12 cpuacct -> cpu,cpuacct
drwxr-xr-x 3 root root 60 Dec 9 10:12 cpuset
drwxr-xr-x 3 root root 60 Dec 9 10:12 devices
drwxr-xr-x 3 root root 60 Dec 9 10:12 freezer
drwxr-xr-x 3 root root 60 Dec 9 10:12 hugetlb
drwxr-xr-x 3 root root 60 Dec 9 10:12 memory
lrwxrwxrwx 1 root root 16 Dec 9 10:12 net_cls -> net_cls,net_prio
drwxr-xr-x 3 root root 60 Dec 9 10:12 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Dec 9 10:12 net_prio -> net_cls,net_prio
drwxr-xr-x 3 root root 60 Dec 9 10:12 perf_event
dr-xr-xr-x 4 root root 0 Dec 9 10:28 systemd

# cat /proc/1/cgroup
10:hugetlb:/lxc/vm-00
9:perf_event:/lxc/vm-00
8:net_cls,net_prio:/lxc/vm-00
7:freezer:/lxc/vm-00
6:devices:/lxc/vm-00
5:memory:/lxc/vm-00
4:blkio:/lxc/vm-00
3:cpu,cpuacct:/lxc/vm-00
2:cpuset:/lxc/vm-00
1:name=systemd:/system.slice/supervisord.service

And for a bonus:

# mount
/dev/md1 on / type ext4 (rw,relatime,stripe=256,data=ordered)
none on /dev type tmpfs (rw,relatime,size=100k,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
sysfs on /sys/devices/virtual/net type sysfs
(rw,nosuid,nodev,noexec,relatime)
sysfs on /sys/fs/fuse/connections type sysfs
(rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs
(rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
cgroup_root on /sys/fs/cgroup/hugetlb type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/hugetlb/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup_root on /sys/fs/cgroup/perf_event type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/perf_event/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup_root on /sys/fs/cgroup/net_cls,net_prio type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/net_cls,net_prio/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup_root on /sys/fs/cgroup/freezer type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/freezer/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)
cgroup_root on /sys/fs/cgroup/devices type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/devices/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
cgroup_root on /sys/fs/cgroup/memory type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/memory/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,memory)
cgroup_root on /sys/fs/cgroup/blkio type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/blkio/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)
cgroup_root on /sys/fs/cgroup/cpu,cpuacct type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/cpu,cpuacct/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup_root on /sys/fs/cgroup/cpuset type tmpfs
(ro,relatime,size=10240k,mode=755)
cgroup on /sys/fs/cgroup/cpuset/lxc/vm-00 type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
devpts on /dev/lxc/console type devpts
(rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty1 type devpts
(rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty2 type devpts
(rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty3 type devpts
(rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/lxc/tty4 type devpts
(rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
Post by Serge Hallyn
Interesting.
I suppose just looking at the 'capsh --print' output difference for the
bounding set between the custom containers spawned by lxc and libvirt-lxc could
be enlightening.
Here's the diff:

# sdiff lxc libvirt
cap_chown cap_chown
cap_dac_override cap_dac_override
cap_dac_read_search cap_dac_read_search
cap_fowner cap_fowner
cap_fsetid cap_fsetid
cap_kill cap_kill
cap_setgid cap_setgid
cap_setuid cap_setuid
cap_setpcap cap_setpcap
cap_linux_immutable cap_linux_immutable
cap_net_bind_service cap_net_bind_service
cap_net_broadcast cap_net_broadcast
cap_net_admin cap_net_admin
cap_net_raw cap_net_raw
cap_ipc_lock cap_ipc_lock
cap_ipc_owner cap_ipc_owner
cap_sys_rawio
cap_sys_chroot cap_sys_chroot
cap_sys_ptrace cap_sys_ptrace
cap_sys_pacct
cap_sys_admin cap_sys_admin
cap_sys_boot cap_sys_boot
cap_sys_nice
cap_sys_resource cap_sys_resource
cap_sys_tty_config cap_sys_tty_config
cap_mknod <
cap_lease cap_lease
cap_audit_write cap_audit_write
cap_audit_control | cap_setfcap
cap_setfcap,cap_syslog |
cap_mac_override
Post by Serge Hallyn
cap_syslog
I've tried another config as well that is more similar, but the systemd
errors still occur:

# sdiff lxc libvirt
cap_chown cap_chown
cap_dac_override cap_dac_override
cap_dac_read_search cap_dac_read_search
cap_fowner cap_fowner
cap_fsetid cap_fsetid
cap_kill cap_kill
cap_setgid cap_setgid
cap_setuid cap_setuid
cap_setpcap cap_setpcap
cap_linux_immutable cap_linux_immutable
cap_net_bind_service cap_net_bind_service
cap_net_broadcast cap_net_broadcast
cap_net_admin cap_net_admin
cap_net_raw cap_net_raw
cap_ipc_lock cap_ipc_lock
cap_ipc_owner cap_ipc_owner
cap_sys_rawio cap_sys_rawio
cap_sys_chroot cap_sys_chroot
cap_sys_ptrace cap_sys_ptrace
cap_sys_pacct cap_sys_pacct
cap_sys_admin cap_sys_admin
cap_sys_boot cap_sys_boot
cap_sys_nice cap_sys_nice
cap_sys_resource cap_sys_resource
cap_sys_tty_config cap_sys_tty_config
cap_mknod <
cap_lease cap_lease
cap_audit_write cap_audit_write
cap_audit_control <
cap_setfcap cap_setfcap
Post by Serge Hallyn
cap_mac_override
cap_syslog cap_syslog
Peter Steele
2015-12-09 21:56:36 UTC
Permalink
Post by Peter Steele
Post by Serge Hallyn
I suppose just looking at the 'capsh --print' output difference for the
bounding set between the custom containers spawned by lxc and
libvirt-lxc could
be enlightening.
# sdiff lxc libvirt
My apologies here. The output I had pasted in was nicely column aligned,
with spaces. Something got lost along the way...

Peter
Peter Steele
2015-12-09 22:31:21 UTC
Permalink
Post by Peter Steele
Post by Peter Steele
Post by Serge Hallyn
I suppose just looking at the 'capsh --print' output difference for the
bounding set between the custom containers spawned by lxc and libvirt-lxc could
be enlightening.
# sdiff lxc libvirt
My apologies here. The output I had pasted in was nicely column
aligned, with spaces. Something got lost along the way...
Peter
Actually, some tabs got mixed in. Hopefully this will look better:

cap_chown cap_chown
cap_dac_override cap_dac_override
cap_dac_read_search cap_dac_read_search
cap_fowner cap_fowner
cap_fsetid cap_fsetid
cap_kill cap_kill
cap_setgid cap_setgid
cap_setuid cap_setuid
cap_setpcap cap_setpcap
cap_linux_immutable cap_linux_immutable
cap_net_bind_service cap_net_bind_service
cap_net_broadcast cap_net_broadcast
cap_net_admin cap_net_admin
cap_net_raw cap_net_raw
cap_ipc_lock cap_ipc_lock
cap_ipc_owner cap_ipc_owner
Post by Peter Steele
cap_sys_rawio
cap_sys_chroot cap_sys_chroot
cap_sys_ptrace cap_sys_ptrace
Post by Peter Steele
cap_sys_pacct
cap_sys_admin cap_sys_admin
cap_sys_boot cap_sys_boot
Post by Peter Steele
cap_sys_nice
cap_sys_resource cap_sys_resource
cap_sys_tty_config cap_sys_tty_config
cap_mknod <
cap_lease cap_lease
cap_audit_write cap_audit_write
cap_audit_control | cap_setfcap
cap_setfcap,cap_syslog | cap_mac_override
Post by Peter Steele
cap_syslog
Serge Hallyn
2015-12-10 02:43:15 UTC
Permalink
Post by Peter Steele
Post by Peter Steele
Post by Peter Steele
Post by Serge Hallyn
I suppose just looking at the 'capsh --print' output difference for the
bounding set between the custom containers spawned by lxc and libvirt-lxc could
be enlightening.
# sdiff lxc libvirt
My apologies here. The output I had pasted in was nicely column
aligned, with spaces. Something got lost along the way...
Peter
cap_chown cap_chown
cap_dac_override cap_dac_override
cap_dac_read_search cap_dac_read_search
cap_fowner cap_fowner
cap_fsetid cap_fsetid
cap_kill cap_kill
cap_setgid cap_setgid
cap_setuid cap_setuid
cap_setpcap cap_setpcap
cap_linux_immutable cap_linux_immutable
cap_net_bind_service cap_net_bind_service
cap_net_broadcast cap_net_broadcast
cap_net_admin cap_net_admin
cap_net_raw cap_net_raw
cap_ipc_lock cap_ipc_lock
cap_ipc_owner cap_ipc_owner
Post by Peter Steele
cap_sys_rawio
Looking through the systemd source, the only obvious thing is that
systmed won't mount configfs or debugfs without rawio. That
doesn't sound relevant here though.
Post by Peter Steele
cap_sys_chroot cap_sys_chroot
cap_sys_ptrace cap_sys_ptrace
Post by Peter Steele
cap_sys_pacct
cap_sys_admin cap_sys_admin
cap_sys_boot cap_sys_boot
Post by Peter Steele
cap_sys_nice
cap_sys_resource cap_sys_resource
cap_sys_tty_config cap_sys_tty_config
cap_mknod <
Ok, systemd does behave differently if it shouldn't be able
to create devices. If you add
lxc.cap.drop = mknod sys_rawio
to your configs does that help?
Post by Peter Steele
cap_lease cap_lease
cap_audit_write cap_audit_write
cap_audit_control | cap_setfcap
cap_setfcap,cap_syslog | cap_mac_override
Post by Peter Steele
cap_syslog
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Peter Steele
2015-12-10 14:13:00 UTC
Permalink
Post by Serge Hallyn
Ok, systemd does behave differently if it shouldn't be able
to create devices. If you add
lxc.cap.drop = mknod sys_rawio
to your configs does that help?
This did not help. I took it a step further and did an install with the
lxc capabilities configured to be as similar as possible to my libvirt
containers and even with this I saw the systemd errors. The only
difference between the cap sets of the two was cap_audit_control; the
lxc containers would not start without this capability but libvirt
containers didn't seem to need it.
Peter Steele
2015-12-11 15:38:37 UTC
Permalink
Post by Peter Steele
Post by Serge Hallyn
Ok, systemd does behave differently if it shouldn't be able
to create devices. If you add
lxc.cap.drop = mknod sys_rawio
to your configs does that help?
This did not help. I took it a step further and did an install with the
lxc capabilities configured to be as similar as possible to my libvirt
containers and even with this I saw the systemd errors. The only
difference between the cap sets of the two was cap_audit_control; the
lxc containers would not start without this capability but libvirt
containers didn't seem to need it.
I don't know if this is relevant, but we are running the 4.0.5 release
of the kernel-ml package set from elrepo. The stock CentOS 7.1 kernel
(3.10) has a bug that impacts bond modes 5 and 6 in containers, so we
had to find an alternative kernel. Other than a problem with RAID 1
mdadm volumes, the 4.0.5 kernel has been solid for us with libvirt based
containers.

I did another test this morning, installing six containers based on the
downloaded CentOS template. When these containers are started
simultaneously there are no errors reported with systemd. I then went
into each container and updated the set of CentOS packages making up the
template to include the additional rpms that we use in our containers.
The default template has something like 157 rpms. After installing the
additional rpms, the containers had 354 installed packages. I then did
another test of shutting down all the containers and restarting them
simultaneously using

for vm in `lxc-ls`; do lxc-start -n $vm; done

I hit the systemd errors on the very first try. This would seem to imply
the problem may be related to one of the additional CentOS rpms that we
use, although it certainly isn't clear which one (ones?) this might be.
I'm going to iteratively reduce the set of packages we use to try to
narrow down the cultprit.

Peter

Loading...