Discussion:
unable to start containers ("Permission denied - Failed to mount")
(too old to reply)
Tomasz Chmielewski
2018-09-24 13:32:02 UTC
Permalink
I'm not able to start any container today.

# lxc start preprod-app
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app
/var/snap/lxd/common/lxd/containers
/var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf:
Try `lxc info --show-log preprod-app` for more info


# lxc info --show-log preprod-app
Name: preprod-app
Remote: unix://
Architecture: x86_64
Created: 2018/09/05 15:01 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc preprod-app 20180924132438.883 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132438.883 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc preprod-app 20180924132438.887 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132438.887 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc preprod-app 20180924132438.917 ERROR dir -
storage/dir.c:dir_mount:195 - Permission denied - Failed to mount
"/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on
"/var/snap/lxd/common/lxc/"
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs
"/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto
"/var/snap/lxd/common/lxc/" with options "(null)"
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for
lxc preprod-app 20180924132438.917 ERROR conf - conf.c:lxc_setup:3510
- Failed to setup rootfs
lxc preprod-app 20180924132438.917 ERROR start -
start.c:do_start:1234 - Failed to setup container "preprod-app"
lxc preprod-app 20180924132438.918 ERROR sync - sync.c:__sync_wait:59
- An error occurred in another process (expected sequence number 5)
lxc preprod-app 20180924132439.235 ERROR start -
start.c:__lxc_start:1910 - Failed to spawn container "preprod-app"
lxc preprod-app 20180924132439.235 ERROR lxccontainer -
lxccontainer.c:wait_on_daemonized_start:840 - Received container state
"ABORTING" instead of "RUNNING"
lxc preprod-app 20180924132439.963 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132439.101 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc 20180924132439.380 WARN commands -
commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to
receive response for command "get_state"


# snap list
Name Version Rev Tracking Publisher Notes
core 16-2.35 5328 stable canonical✓ core
lxd 3.5 8774 stable canonical✓ -


This is on Ubuntu 18.04.


Tomasz Chmielewski
https://lxadm.com
Tomasz Chmielewski
2018-09-24 13:40:57 UTC
Permalink
Turns out something changed the permissions on "containers" directory:


# lxc storage list
+---------+-------------+--------+-----------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+---------+-------------+--------+-----------+---------+
| default | | btrfs | /data/lxd | 12 |
+---------+-------------+--------+-----------+---------+


# ls -l /data/lxd
total 0
drwxr-xr-x 1 root root 90 Sep 24 13:05 archives
drwx------ 1 root root 518 Sep 24 13:12 containers <--------- here
drwx--x--x 1 root root 0 Mar 28 16:14 custom
drwx------ 1 root root 0 Sep 21 06:05 images
drwx------ 1 root root 0 Sep 24 05:48 snapshots


This fixed it:

chmod 711 /data/lxd/containers/


I'm 99% sure we did not change the permissions on that directory...


Tomasz
Post by Tomasz Chmielewski
I'm not able to start any container today.
# lxc start preprod-app
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app
/var/snap/lxd/common/lxd/containers
Try `lxc info --show-log preprod-app` for more info
# lxc info --show-log preprod-app
Name: preprod-app
Remote: unix://
Architecture: x86_64
Created: 2018/09/05 15:01 UTC
Status: Stopped
Type: persistent
Profiles: default
lxc preprod-app 20180924132438.883 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132438.883 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc preprod-app 20180924132438.887 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132438.887 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc preprod-app 20180924132438.917 ERROR dir -
storage/dir.c:dir_mount:195 - Permission denied - Failed to mount
"/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on
"/var/snap/lxd/common/lxc/"
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs
"/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto
"/var/snap/lxd/common/lxc/" with options "(null)"
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_setup:3510 - Failed to setup rootfs
lxc preprod-app 20180924132438.917 ERROR start -
start.c:do_start:1234 - Failed to setup container "preprod-app"
lxc preprod-app 20180924132438.918 ERROR sync -
sync.c:__sync_wait:59 - An error occurred in another process (expected
sequence number 5)
lxc preprod-app 20180924132439.235 ERROR start -
start.c:__lxc_start:1910 - Failed to spawn container "preprod-app"
lxc preprod-app 20180924132439.235 ERROR lxccontainer -
lxccontainer.c:wait_on_daemonized_start:840 - Received container state
"ABORTING" instead of "RUNNING"
lxc preprod-app 20180924132439.963 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132439.101 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc 20180924132439.380 WARN commands -
commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to
receive response for command "get_state"
# snap list
Name Version Rev Tracking Publisher Notes
core 16-2.35 5328 stable canonical✓ core
lxd 3.5 8774 stable canonical✓ -
This is on Ubuntu 18.04.
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Christian Brauner
2018-09-24 13:43:20 UTC
Permalink
Odd, the new storage snapshot api performs an on-disk upgrade but it
shouldn't touch the containers directory...
//cc Stéphane

Christian
Post by Tomasz Chmielewski
# lxc storage list
+---------+-------------+--------+-----------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+---------+-------------+--------+-----------+---------+
| default | | btrfs | /data/lxd | 12 |
+---------+-------------+--------+-----------+---------+
# ls -l /data/lxd
total 0
drwxr-xr-x 1 root root 90 Sep 24 13:05 archives
drwx------ 1 root root 518 Sep 24 13:12 containers <--------- here
drwx--x--x 1 root root 0 Mar 28 16:14 custom
drwx------ 1 root root 0 Sep 21 06:05 images
drwx------ 1 root root 0 Sep 24 05:48 snapshots
chmod 711 /data/lxd/containers/
I'm 99% sure we did not change the permissions on that directory...
Tomasz
Post by Tomasz Chmielewski
I'm not able to start any container today.
# lxc start preprod-app
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app
/var/snap/lxd/common/lxd/containers
Try `lxc info --show-log preprod-app` for more info
# lxc info --show-log preprod-app
Name: preprod-app
Remote: unix://
Architecture: x86_64
Created: 2018/09/05 15:01 UTC
Status: Stopped
Type: persistent
Profiles: default
lxc preprod-app 20180924132438.883 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132438.883 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc preprod-app 20180924132438.887 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132438.887 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc preprod-app 20180924132438.917 ERROR dir -
storage/dir.c:dir_mount:195 - Permission denied - Failed to mount
"/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on
"/var/snap/lxd/common/lxc/"
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs
"/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto
"/var/snap/lxd/common/lxc/" with options "(null)"
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for
lxc preprod-app 20180924132438.917 ERROR conf -
conf.c:lxc_setup:3510 - Failed to setup rootfs
lxc preprod-app 20180924132438.917 ERROR start -
start.c:do_start:1234 - Failed to setup container "preprod-app"
lxc preprod-app 20180924132438.918 ERROR sync -
sync.c:__sync_wait:59 - An error occurred in another process (expected
sequence number 5)
lxc preprod-app 20180924132439.235 ERROR start -
start.c:__lxc_start:1910 - Failed to spawn container "preprod-app"
lxc preprod-app 20180924132439.235 ERROR lxccontainer -
lxccontainer.c:wait_on_daemonized_start:840 - Received container state
"ABORTING" instead of "RUNNING"
lxc preprod-app 20180924132439.963 WARN conf -
conf.c:lxc_map_ids:2917 - newuidmap binary is missing
lxc preprod-app 20180924132439.101 WARN conf -
conf.c:lxc_map_ids:2923 - newgidmap binary is missing
lxc 20180924132439.380 WARN commands -
commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to
receive response for command "get_state"
# snap list
Name Version Rev Tracking Publisher Notes
core 16-2.35 5328 stable canonical✓ core
lxd 3.5 8774 stable canonical✓ -
This is on Ubuntu 18.04.
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Continue reading on narkive:
Loading...