Discussion:
[lxc-users] Does cpu cgroup has been enabled in lxc/lxd
kemi
2018-11-01 01:55:16 UTC
Permalink
Hi, Everyone
I am new comer of LXC/LXD community, and want to run a container on a limited cpu set.

The followings are my steps:
a) lxd init
b) lxc launch Ubuntu:18.04 first
c) lxc stop first
d) lxc config set first limits.cpu 0 // set container running on CPU 0
e) lxc start first
f) lxc exec first -- bash
g) nproc // the expected result would be 1, however, it still equals to cpu number of host
h) ls /sys/devices/system/cpu // the expected result should only include cpu0 directory, however, it's not

So, it seems that CPU cgroup has not been enabled in LXC/LXD, right? The version of lxc is 2.0.11 on Ubuntu 16.04.
Anyone can help me on that, thx very much.
Fajar A. Nugraha
2018-11-01 05:26:38 UTC
Permalink
Post by kemi
Hi, Everyone
I am new comer of LXC/LXD community, and want to run a container on a limited cpu set.
a) lxd init
b) lxc launch Ubuntu:18.04 first
c) lxc stop first
d) lxc config set first limits.cpu 0 // set container running on CPU 0
I'm not sure, but I believe "0" here means all cpu, and not pin to cpu 0?

Try changing this to "1", "0-0", and "1-2". Observe the difference.
Post by kemi
e) lxc start first
f) lxc exec first -- bash
g) nproc // the expected result would be 1, however, it still equals
to cpu number of host
h) ls /sys/devices/system/cpu // the expected result should only
include cpu0 directory, however, it's not
g) and h) read files from /proc, not cgroup. You need lxcfs. You should
already have that on ubuntu though.
Post by kemi
So, it seems that CPU cgroup has not been enabled in LXC/LXD, right? The
version of lxc is 2.0.11 on Ubuntu 16.04.
Anyone can help me on that, thx very much.
If this is a new install, I highly suggest you just switch to ubuntu 18.04
+ lxd-3 host.
If you simply want to have "correct" /proc entries, make sure lxcfs is
installed, and then restart lxd (if needed).
--
Fajar
kemi
2018-11-01 05:35:04 UTC
Permalink
Hi, Fajar
thx for your reply.
Post by Fajar A. Nugraha
Post by kemi
Hi, Everyone
I am new comer of LXC/LXD community, and want to run a container on a limited cpu set.
a) lxd init
b) lxc launch Ubuntu:18.04 first
c) lxc stop first
d) lxc config set first limits.cpu 0 // set container running on CPU 0
I'm not sure, but I believe "0" here means all cpu, and not pin to cpu 0?
Hmm, I expected to pin on CPU 0. Seems I misunderstood the *limit* configuration here:)
I will try use another number as you suggested.
Post by Fajar A. Nugraha
Try changing this to "1", "0-0", and "1-2". Observe the difference.
Post by kemi
e) lxc start first
f) lxc exec first -- bash
g) nproc // the expected result would be 1, however, it still equals
to cpu number of host
h) ls /sys/devices/system/cpu // the expected result should only
include cpu0 directory, however, it's not
g) and h) read files from /proc, not cgroup. You need lxcfs. You should
already have that on ubuntu though.
Hmm, I will take a look at it. thx for suggestion
Post by Fajar A. Nugraha
Post by kemi
So, it seems that CPU cgroup has not been enabled in LXC/LXD, right? The
version of lxc is 2.0.11 on Ubuntu 16.04.
Anyone can help me on that, thx very much.
If this is a new install, I highly suggest you just switch to ubuntu 18.04
+ lxd-3 host.
Lxd has a simple online testing using lxd-3.6. May I use it for testing purpose?
https://linuxcontainers.org/lxd/try-it/
Post by Fajar A. Nugraha
If you simply want to have "correct" /proc entries, make sure lxcfs is
installed, and then restart lxd (if needed).
Not only I want to get correct number of cpus in container, but still hope to have independent
procfs and sysfs virtualized in container.
Post by Fajar A. Nugraha
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
kemi
2018-11-01 06:38:38 UTC
Permalink
Post by kemi
Hi, Fajar
thx for your reply.
Post by Fajar A. Nugraha
Post by kemi
Hi, Everyone
I am new comer of LXC/LXD community, and want to run a container on a
limited cpu set.
a) lxd init
b) lxc launch Ubuntu:18.04 first
c) lxc stop first
d) lxc config set first limits.cpu 0 // set container running on CPU 0
I'm not sure, but I believe "0" here means all cpu, and not pin to cpu 0?
Hmm, I expected to pin on CPU 0. Seems I misunderstood the *limit* configuration here:)
I will try use another number as you suggested.
Post by Fajar A. Nugraha
Try changing this to "1", "0-0", and "1-2". Observe the difference.
have tried. `nproc` works well.
Post by kemi
Post by Fajar A. Nugraha
Post by kemi
e) lxc start first
f) lxc exec first -- bash
g) nproc // the expected result would be 1, however, it still equals
to cpu number of host
h) ls /sys/devices/system/cpu // the expected result should only
include cpu0 directory, however, it's not
g) and h) read files from /proc, not cgroup. You need lxcfs. You should
already have that on ubuntu though.
/proc/cpuinfo also matches the expected result.
However, it seems that sysfs in container still shares with host /sys file system.
Right?
Fajar A. Nugraha
2018-11-01 06:53:31 UTC
Permalink
Post by kemi
Post by Fajar A. Nugraha
g) and h) read files from /proc, not cgroup. You need lxcfs. You should
already have that on ubuntu though.
/proc/cpuinfo also matches the expected result.
However, it seems that sysfs in container still shares with host /sys file system.
Right?
Correct. See https://linuxcontainers.org/lxcfs/introduction/
--
Fajar
kemi
2018-11-01 07:16:17 UTC
Permalink
Post by Fajar A. Nugraha
Post by kemi
Post by Fajar A. Nugraha
g) and h) read files from /proc, not cgroup. You need lxcfs. You should
already have that on ubuntu though.
/proc/cpuinfo also matches the expected result.
However, it seems that sysfs in container still shares with host /sys file system.
Right?
Correct. See https://linuxcontainers.org/lxcfs/introduction/
OK, then I have a question on scalability and security issues on running multiple containers.

Background: Our customers hope to run hundreds or even thousands of containers in their production environment.

Sharing sysfs of containers with host sysfs in lxc/lxd may have:
a) security issue.
If a malicious program in a container changes a sensitive file in /sys,
e.g. reduce CPU frequency, does it really works? Does it affect other running containers?

b) Scalability issue.
E.g. During launching a ubuntu OS(not kernel) or Android OS in a container,it usually use udev/ueventd
to manage their device. This device manager daemon will read or write uevent file in /sys, the kernel
then broadcast a uevent to all the listeners(udev daemon) via netlink, if there are already hundreds
of containers in the system, all of udev daemons need to deal with it, it would lead to a long boot
latency which we have observed in docker.

Anyway to fix that?
Fajar A. Nugraha
2018-11-01 07:30:54 UTC
Permalink
Post by kemi
Post by Fajar A. Nugraha
Post by kemi
Post by Fajar A. Nugraha
g) and h) read files from /proc, not cgroup. You need lxcfs. You
should
Post by Fajar A. Nugraha
Post by kemi
Post by Fajar A. Nugraha
already have that on ubuntu though.
/proc/cpuinfo also matches the expected result.
However, it seems that sysfs in container still shares with host /sys file system.
Right?
Correct. See https://linuxcontainers.org/lxcfs/introduction/
OK, then I have a question on scalability and security issues on running
multiple containers.
Background: Our customers hope to run hundreds or even thousands of
containers in their production environment.
a) security issue.
If a malicious program in a container changes a sensitive file in /sys,
e.g. reduce CPU frequency, does it really works? Does it affect other running containers?
Why don't you try it and see :)

Even privileged container should get something like this

# echo 1000000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_min_freq
-su: /sys/devices/system/cpu/cpufreq/policy1/scaling_min_freq: Read-only
file system


There were some known security issues with /sys in the past (not cpufreq
though), but even back then it should be non issue for the default lxd
containers, which are unprivileged.



b) Scalability issue.
Post by kemi
E.g. During launching a ubuntu OS(not kernel) or Android OS in a
container,it usually use udev/ueventd
to manage their device. This device manager daemon will read or write
uevent file in /sys, the kernel
then broadcast a uevent to all the listeners(udev daemon) via netlink, if
there are already hundreds
of containers in the system, all of udev daemons need to deal with it, it
would lead to a long boot
latency which we have observed in docker.
LXD containers don't use udev.
Post by kemi
Anyway to fix that?
Try it, and if you find anything wrong, ask.
--
Fajar
kemi
2018-11-01 08:04:00 UTC
Permalink
Post by Fajar A. Nugraha
Post by kemi
Post by Fajar A. Nugraha
Post by kemi
Post by Fajar A. Nugraha
g) and h) read files from /proc, not cgroup. You need lxcfs. You
should
Post by Fajar A. Nugraha
Post by kemi
Post by Fajar A. Nugraha
already have that on ubuntu though.
/proc/cpuinfo also matches the expected result.
However, it seems that sysfs in container still shares with host /sys file system.
Right?
Correct. See https://linuxcontainers.org/lxcfs/introduction/
OK, then I have a question on scalability and security issues on running
multiple containers.
Background: Our customers hope to run hundreds or even thousands of
containers in their production environment.
a) security issue.
If a malicious program in a container changes a sensitive file in /sys,
e.g. reduce CPU frequency, does it really works? Does it affect other running containers?
Why don't you try it and see :)
this is just an example I used to help describe my question clearly.
I believe there are some other security issues in lxc/ldx due to shared sysfs.
Post by Fajar A. Nugraha
Even privileged container should get something like this
# echo 1000000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_min_freq
-su: /sys/devices/system/cpu/cpufreq/policy1/scaling_min_freq: Read-only
file system
There were some known security issues with /sys in the past (not cpufreq
though), but even back then it should be non issue for the default lxd
containers, which are unprivileged.
Yes. Using unprivileged container is a workaround, not very good though.
Post by Fajar A. Nugraha
b) Scalability issue.
Post by kemi
E.g. During launching a ubuntu OS(not kernel) or Android OS in a
container,it usually use udev/ueventd
to manage their device. This device manager daemon will read or write
uevent file in /sys, the kernel
then broadcast a uevent to all the listeners(udev daemon) via netlink, if
there are already hundreds
of containers in the system, all of udev daemons need to deal with it, it
would lead to a long boot
latency which we have observed in docker.
LXD containers don't use udev.
I have no idea on how LXD container works now.
Maybe udev is disabled or some other mechanism may be used to manage device.
Post by Fajar A. Nugraha
Post by kemi
Anyway to fix that?
Try it, and if you find anything wrong, ask.
The reason why I have not tried it is there is no available android image provided on existed
images server for LXD container. Do you know something about that?
So, I have to make a Android image first. Anyway I will try it next.
Thx for your answers again.
Fajar A. Nugraha
2018-11-01 08:30:33 UTC
Permalink
Post by kemi
The reason why I have not tried it is there is no available android image
provided on existed
images server for LXD container. Do you know something about that?
I don't believe anybody has succesfully run android in lxd yet (sucess as
in "you can use vnc or similar to view the screen"). Perhaps porting a
working docker setup is a good start (I assume this is not you or your
team, apologies if you're already familiar with it) :
https://github.com/butomo1989/docker-android

There were some hints on debugging android on lxd in the list archive:
https://discuss.linuxcontainers.org/t/how-to-debug-container-boot-on-lxd/849
, which might be relevant to what you want.
--
Fajar
Jäkel, Guido
2018-11-01 10:00:17 UTC
Permalink
Post by kemi
I have no idea on how LXD container works now.
Maybe udev is disabled or some other mechanism may be used to manage device.
LXC/LXD is not limited to that, but the probably most used model for Containers it that it should form a key-ready environment for a Linux userland to run applications, where all resourced