[lxc-users] Error: /proc must be mounted

[Itamar Gal]

Dear lxc-users,

First I will give an extremely brief description of my problem. I recently
upgrade some packages on my server. Containers which have been created since
the upgrade are having some issues. Specifically, if I run the ps command I get
the following error:

$ ps

Error: /proc must be mounted
To mount /proc at boot you need an /etc/fstab line like:
proc /proc proc defaults
In the meantime, run "mount proc /proc -t proc"

Now I will give more background and describe some of my attempts to further
investigate and resolve the issue.


I have a server running Ubuntu 12.04.5 LTS (Precise Pangolin). I recently
performed an upgrade of all system packages (i.e. apt-get upgrade) after not
having done so for some time. Here are the contents of my apt sources file:

$ cat /etc/apt/sources.list

# Standard packages
deb http://us.archive.ubuntu.com/ubuntu/ precise main restricted
universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main
restricted universe multiverse
deb http://security.ubuntu.com/ubuntu precise-security main
restricted universe multiverse

# PPA for virtual box
deb http://download.virtualbox.org/virtualbox/debian precise contrib

In particular, this operation resulted in an upgrade of the kernel and of some
LXC-related packages. Here is some version information related to LXC:

$ uname -a

Linux myhostname 3.2.0-86-generic #124-Ubuntu SMP
Wed Jun 17 21:40:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ apt-show-versions | grep lxc

liblxc1/precise uptodate 2.0.5+master~20161122-0513-0ubuntu1~precise
lxc/precise uptodate 2.0.5+master~20161122-0019-0ubuntu1~precise
lxc-common/precise uptodate 2.0.5+master~20161122-0513-0ubuntu1~precise
lxc-templates/precise uptodate 2.0.5+master~20161122-0513-0ubuntu1~precise
lxc1/precise uptodate 2.0.5+master~20161122-0513-0ubuntu1~precise
lxcfs/precise uptodate 2.0.4-0ubuntu1~ubuntu12.04.1~ppa1
lxctl/precise uptodate 0.3.1+debian-1
python3-lxc/precise uptodate 2.0.5+master~20161122-0513-0ubuntu1~precise


Here is an example of how I might go about reproducing my problem. First I
create a container, start it and log in via the console:

sudo lxc-create -n "temp-test" -t "ubuntu" -- --release "trusty"
sudo lxc-start -n "temp-test"
sudo lxc-console -n "temp-test"


Once inside the container, I try to run the ps command and follow some
instructions:

$ ps

Error: /proc must be mounted
To mount /proc at boot you need an /etc/fstab line like:
proc /proc proc defaults
In the meantime, run "mount proc /proc -t proc"


$ sudo mount proc /proc -t proc

mount: proc already mounted


$ sudo ps

Error: /proc must be mounted
To mount /proc at boot you need an /etc/fstab line like:
proc /proc proc defaults
In the meantime, run "mount proc /proc -t proc"


Then I check my fstab (it's empty):

$ cat /etc/fstab

# UNCONFIGURED FSTAB FOR BASE SYSTEM


Then I try an strace:

$ strace ps aux 2>&1 | tail

ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS, 0x7ffc59676200) = -1 EINVAL (Invalid argument)
geteuid() = 1000
open("/proc/uptime", O_RDONLY) = -1 ENOTCONN (Transport
endpoint is not connected)
write(2, "Error: /proc must be mounted\n T"..., 176Error: /proc
must be mounted
To mount /proc at boot you need an /etc/fstab line like:
proc /proc proc defaults
In the meantime, run "mount proc /proc -t proc"
) = 176
exit_group(102) = ?


I try to google for the error:

Transport endpoint is not connected

Unfortunately I was unable to find a solution that way. I also looked at the
lxcfs upstart log (on the host):

$ sudo cat /var/log/upstart/lxcfs.log

bindings.c: 4484: collect_and_mount_subsystems: Failed to preserve
initial mount namespace.

This lead to more fruitless google searches. I think that covers most of it.
I'm not sure how to proceed. Any help or advice would be greatly appreciated.

Cheers,
Itamar

[Fajar A. Nugraha]

Post by Itamar Gal
I have a server running Ubuntu 12.04.5 LTS (Precise Pangolin). I recently

ouch

I HIGHLY recommend you upgrade your OS to xenial. Or even install xenial to
a new disk using zfs root, if you have it (that way you can optionally run
you original precise root as a container later)

Post by Itamar Gal
$ apt-show-versions | grep lxc
liblxc1/precise uptodate 2.0.5+master~20161122-0513-0ubuntu1~precise

at least that's good. and you also have lxcfs

Post by Itamar Gal
Once inside the container, I try to run the ps command and follow some
$ ps
Error: /proc must be mounted
proc /proc proc defaults
In the meantime, run "mount proc /proc -t proc"
$ sudo mount proc /proc -t proc
mount: proc already mounted

depending on your versions, lxcfs might get killed/restarted during
upgrade. Try:
- check if lxcfs is running (in the host, "ps -ef | grep lxcfs")
- if lxcfs is running, create a new container, using the DOWNLOAD template
(NOT ubuntu template) and start it. Does ps work there?
- if it does, restart your problematic container. Does it work now?
--> if YES, problem solved. It was lxcfs restarted problem
--> if NO, check the config difference between your old new newly-created
container.

[Itamar Gal]

Hey Fajar,

Thank you for your help. Following your advice, I checked to make sure
that lxcfs is running:

$ pgrep lxcfs | xargs ps -f -p

UID PID PPID C STIME TTY TIME CMD
root 29988 1 0 13:17 ? 00:00:00 /usr/bin/lxcfs
/var/lib/lxcfs


It appears that is running. I then tried creating a new container as
you suggested:

sudo lxc-create --template=download --name=lxc-debug


I chose the following options from the interactive prompt:

Distribution: ubuntu
Release: trusty
Architecture: amd64


Then I started the container and checked to make sure it was running:

$ sudo lxc-start -n lxc-debug
$ sudo lxc-ls --fancy | grep debug

lxc-debug RUNNING 0 - 10.0.3.142 -


According to the printed instructions no user accounts are created for
downloaded containers and you have to use lxc-attach to create the
initial account. Just as a sanity check I tried logging in using
lxc-console and the ubuntu user, but sure enough the usual password
from the ubuntu template didn't work (i.e. I was unable to
authenticate). So I tried using lxc-attach instead. Unfortunately
lxc-attach was unsuccessful:

$ sudo lxc-attach -n lxc-debug -- "echo testing"

lxc-attach: lxc-debug: attach.c:
lxc_attach_to_ns: 252 No such file or directory - failed to open
namespace: 'mnt'.
lxc-attach: lxc-debug: attach.c: lxc_attach: 981 failed to enter
the namespace


In fact, it turns out that I get the same error message when
attempting to use lxc-attach with my older (working) containers.
Moving on, I used chroot to set the password for the ubuntu user:

$ sudo chroot /var/lib/lxc/lxc-debug/rootfs passwd ubuntu


After setting the password I can use lxc-console to enter the
container, but the ps command still doesn't work - it generates the
same error message.

Any further thoughts? Thanks again for your help.

Cheers,
Itamar

Post by Fajar A. Nugraha

Post by Itamar Gal
I have a server running Ubuntu 12.04.5 LTS (Precise Pangolin). I recently

ouch
I HIGHLY recommend you upgrade your OS to xenial. Or even install xenial to
a new disk using zfs root, if you have it (that way you can optionally run
you original precise root as a container later)

Post by Itamar Gal
$ apt-show-versions | grep lxc
liblxc1/precise uptodate 2.0.5+master~20161122-0513-0ubuntu1~precise

at least that's good. and you also have lxcfs

Post by Itamar Gal
Once inside the container, I try to run the ps command and follow some
$ ps
Error: /proc must be mounted
proc /proc proc defaults
In the meantime, run "mount proc /proc -t proc"
$ sudo mount proc /proc -t proc
mount: proc already mounted

depending on your versions, lxcfs might get killed/restarted during upgrade.
- check if lxcfs is running (in the host, "ps -ef | grep lxcfs")
- if lxcfs is running, create a new container, using the DOWNLOAD template
(NOT ubuntu template) and start it. Does ps work there?
- if it does, restart your problematic container. Does it work now?
--> if YES, problem solved. It was lxcfs restarted problem
--> if NO, check the config difference between your old new newly-created
container.
--
Fajar
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users

...

[Fajar A. Nugraha]

Post by Itamar Gal
Hey Fajar,
Thank you for your help. Following your advice, I checked to make sure
$ pgrep lxcfs | xargs ps -f -p
UID PID PPID C STIME TTY TIME CMD
root 29988 1 0 13:17 ? 00:00:00 /usr/bin/lxcfs
/var/lib/lxcfs

You should also be able to look at /var/lib/lxcfs and see what's in there.
There should be cgroup and proc mounts with the corrent content.

Post by Itamar Gal
It appears that is running. I then tried creating a new container as
sudo lxc-create --template=download --name=lxc-debug
Distribution: ubuntu
Release: trusty
Architecture: amd64
$ sudo lxc-start -n lxc-debug
$ sudo lxc-ls --fancy | grep debug
lxc-debug RUNNING 0 - 10.0.3.142 -
According to the printed instructions no user accounts are created for
downloaded containers and you have to use lxc-attach to create the
initial account. Just as a sanity check I tried logging in using
lxc-console and the ubuntu user, but sure enough the usual password
from the ubuntu template didn't work (i.e. I was unable to
authenticate). So I tried using lxc-attach instead. Unfortunately
$ sudo lxc-attach -n lxc-debug -- "echo testing"
lxc_attach_to_ns: 252 No such file or directory - failed to open
namespace: 'mnt'.
lxc-attach: lxc-debug: attach.c: lxc_attach: 981 failed to enter
the namespace

...

Hmmm ... that seems like a much bigger problem.

Are you sure you have correct version of everything? e.g. lxc userland and
lxcfs?
Are there perhaps stray versions somewhere (e.g. have you compiled lxc in
the past, and install it in the /usr/local)

"which lxc-attach" and "lxc-attach --version" should give more insight.

Post by Itamar Gal
In fact, it turns out that I get the same error message when
attempting to use lxc-attach with my older (working) containers.
$ sudo chroot /var/lib/lxc/lxc-debug/rootfs passwd ubuntu
After setting the password I can use lxc-console to enter the
container, but the ps command still doesn't work - it generates the
same error message.
Any further thoughts? Thanks again for your help.

You can try this command in the container. Here's from mine:

# grep /proc /proc/mounts
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
...
lxcfs /proc/cpuinfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

If you're missing the first entry, or the lxcfs entries, or don't even have
/proc/mounts altogether, then I'm not sure what else to check.

You could PROBABLY try to upgrade the kernel (
http://packages.ubuntu.com/precise-updates/linux-generic-lts-trusty), to
see if for some reason the problem is caused by missing namespace support
in your kernel. If that doesn't work, or you can't afford downtime on this
server, then my best advice at this point would be to "setup a new trusty
server as lxc host"

[Itamar Gal]

Post by Fajar A. Nugraha
You should also be able to look at /var/lib/lxcfs and see what's in there.
There should be cgroup and proc mounts with the corrent content.

Yup - looks like it's all there.

$ ls /var/lib/lxcfs

cgroup
proc

$ mountpoint /var/lib/lxcfs

/var/lib/lxcfs is a mountpoint

I did experience a couple of anomalies related to this mountpoint though.
First, the "tree" command produces "error opening dir" messages:

$ sudo tree /var/lib/lxcfs

/var/lib/lxcfs
├── cgroup
│ ├── blkio [error opening dir]
│ ├── cpu [error opening dir]
│ ├── cpuacct [error opening dir]
│ ├── cpuset [error opening dir]
│ ├── devices [error opening dir]
│ ├── freezer [error opening dir]
│ ├── memory [error opening dir]
│ └── perf_event [error opening dir]
└── proc [error opening dir]

10 directories, 0 files


Also, when I tried using the "find" command I got the following error message:

"Transport endpoint is not connected"

I then started getting the same error with the "ls" command. So I checked to
make sure lxcfs was still running and (to my surprise) found that it wasn't.
So apparently lxcfs crashed while I was inspecting the /var/lic/lxcfs mount?
I'm not sure why that would happen. Anyway, I restarted the service:

sudo service lxcfs start

After restarting "ls" and "find" worked again, although "tree" still does not.
Here is what I get from the "find" command:

$ find /var/lib/lxcfs/

/var/lib/lxcfs/
/var/lib/lxcfs/proc
/var/lib/lxcfs/proc/cpuinfo
/var/lib/lxcfs/proc/meminfo
/var/lib/lxcfs/proc/stat
/var/lib/lxcfs/proc/uptime
/var/lib/lxcfs/proc/diskstats
/var/lib/lxcfs/proc/swaps
/var/lib/lxcfs/cgroup
/var/lib/lxcfs/cgroup/perf_event
find: `/var/lib/lxcfs/cgroup/perf_event': Software caused connection abort
/var/lib/lxcfs/cgroup/blkio
/var/lib/lxcfs/cgroup/freezer
/var/lib/lxcfs/cgroup/devices
/var/lib/lxcfs/cgroup/memory
/var/lib/lxcfs/cgroup/cpuacct
/var/lib/lxcfs/cgroup/cpu
/var/lib/lxcfs/cgroup/cpuset

Post by Fajar A. Nugraha
Hmmm ... that seems like a much bigger problem.
Are you sure you have correct version of everything? e.g. lxc userland and
lxcfs?
Are there perhaps stray versions somewhere (e.g. have you compiled lxc in
the past, and install it in the /usr/local)
"which lxc-attach" and "lxc-attach --version" should give more insight.

I'm not sure that I know exactly what you mean by "correct version" in this
context, but I'm fairly certain that the only lxc software on this system has
been installed by the package manager (i.e. apt-get). Here's what I did to
double-check the state of lxc on the system. First, I followed your suggestion
and checked to see which lxc-attach binary was in my path (and check its
version number as well):

$ which -a lxc-attach
/usr/bin/lxc-attach

$ type lxc-attach
lxc-attach is hashed (/usr/bin/lxc-attach)

$ lxc-attach --version
2.0.0


I also checked the file-system for any other instances of lxc-attach:

$ sudo updatedb
$ locate lxc-attach

/usr/bin/lxc-attach
/usr/share/bash-completion/completions/lxc-attach
/usr/share/man/ja/man1/lxc-attach.1.gz
/usr/share/man/ko/man1/lxc-attach.1.gz
/usr/share/man/man1/lxc-attach.1.gz


Just to be really sure, I also checked the entire file-system for any instances
of other lxc tools:

$ sudo updatedb
$ for c in /usr/bin/lxc-*; do locate $(basename $c) | grep "${c}$"; done

/usr/bin/lxc-attach
/usr/bin/lxc-autostart
/usr/bin/lxc-cgroup
/usr/bin/lxc-checkconfig
/usr/bin/lxc-checkpoint
/usr/bin/lxc-clone
/usr/bin/lxc-config
/usr/bin/lxc-console
/usr/bin/lxc-copy
/usr/bin/lxc-create
/usr/bin/lxc-destroy
/usr/bin/lxc-device
/usr/bin/lxc-execute
/usr/bin/lxc-freeze
/usr/bin/lxc-info
/usr/bin/lxc-ls
/usr/bin/lxc-monitor
/usr/bin/lxc-snapshot
/usr/bin/lxc-start
/usr/bin/lxc-start-ephemeral
/usr/bin/lxc-stop
/usr/bin/lxc-top
/usr/bin/lxc-unfreeze
/usr/bin/lxc-unshare
/usr/bin/lxc-usernsexec
/usr/bin/lxc-wait

So I think it's probably safe to say that there's only the one set of lxc
tools installed. Here's what dpkg has to say about the version information:

$ dpkg-query -W "*lxc*"

liblxc0 1.0.0~alpha1-0ubuntu14~ubuntu12.04.1
liblxc1 2.0.5+master~20161122-1752-0ubuntu1~precise
lxc 2.0.5+master~20161122-1752-0ubuntu1~precise
lxc-common 2.0.5+master~20161122-1752-0ubuntu1~precise
lxc-dbg
lxc-dev
lxc-templates 2.0.5+master~20161122-1752-0ubuntu1~precise
lxc1 2.0.5+master~20161122-1752-0ubuntu1~precise
lxcfs 2.0.4-0ubuntu1~ubuntu12.04.1~ppa1
lxcguest
lxctl 0.3.1+debian-1
python3-lxc 2.0.5+master~20161122-1752-0ubuntu1~precise

Are those not the correct versions?

Post by Fajar A. Nugraha
# grep /proc /proc/mounts
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
...
lxcfs /proc/cpuinfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
If you're missing the first entry, or the lxcfs entries, or don't even have
/proc/mounts altogether, then I'm not sure what else to check.

...

I tried grepping in /proc/mounts as you suggest. My output isn't identical,
but I also don't see anything there that's jumping out as a problem:

$ grep /proc /proc/mounts

proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys/net proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
lxcfs /proc/cpuinfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

I should also mention that the "pgrep" command appears to be functioning
normally and I also appear to be able to access the proc file-system (in the
containers) manually without any issues. For example, this won't work:

$ pgrep ssh | xargs ps -f -p

Error: /proc must be mounted
To mount /proc at boot you need an /etc/fstab line like:
proc /proc proc defaults
In the meantime, run "mount proc /proc -t proc"

But this will:

$ for pid in $(pgrep ssh); do cat /proc/${pid}/cmdline; echo; done

/usr/sbin/sshd-D
sshd: user1 [priv]
sshd: ***@pts/5
sshd: user2 [priv]
sshd: ***@pts/6
sshd: user2 [priv]
sshd: ***@pts/7

[...]

Post by Fajar A. Nugraha
You could PROBABLY try to upgrade the kernel
(http://packages.ubuntu.com/precise-updates/linux-generic-lts-trusty), to
see if for some reason the problem is caused by missing namespace support in
your kernel. If that doesn't work, or you can't afford downtime on this
server, then my best advice at this point would be to "setup a new trusty
server as lxc host"

Yeah, that sounds about right. My plan is to setup a temporary lxc host as a
backup and then upgrade the main server to Ubuntu 16.04 as you suggested
previously. Thanks again for your help!

[Fajar A. Nugraha]

Post by Itamar Gal
$ sudo tree /var/lib/lxcfs
/var/lib/lxcfs
├── cgroup
│ ├── blkio [error opening dir]
sudo service lxcfs start
After restarting "ls" and "find" worked again, although "tree" still does not.
$ find /var/lib/lxcfs/
/var/lib/lxcfs/
/var/lib/lxcfs/proc
/var/lib/lxcfs/proc/cpuinfo

Did you restart your container after you restart lxcfs? Sounds like your
lxcfs is somehow messed up (probably an old version running, crashed or not
yet restarted during last update), and restarting lxcfs fixed that. However
restarting lxcfs will broke running containers, so you should stop-start
the containers as well.

If that works, then perhaps there's still hope for your aging precise
server after all :)

[Itamar Gal]

Post by Fajar A. Nugraha
Did you restart your container after you restart lxcfs? Sounds like your
lxcfs is somehow messed up (probably an old version running, crashed or not
yet restarted during last update), and restarting lxcfs fixed that. However
restarting lxcfs will broke running containers, so you should stop-start the
containers as well.
If that works, then perhaps there's still hope for your aging precise server
after all :)

Unfortunately restarting the container after restarting lxcfs did not
seem to have any effect. I think I'm going to have to let this one go.
I have an opportunity to schedule some downtime next month, so I'll
just upgrade the server then. Thanks again for your help!