Discussion:
LXC a feature complete replacement of OpenVZ?
(too old to reply)
Christian Haintz
2010-05-13 16:17:54 UTC
Permalink
Hi,

At first LXC seams to be a great work from what we have read already.

There are still a few open questions for us (we are currently running
dozens of OpenVZ Hardwarenodes).

1) OpenVZ in the long-term seams to be a dead end. Will LXC be a
feature complete replacement for OpenVZ in the 1.0 Version?

As of the current version
2) is there IPTable support, any sort of control like the OpenVZ
IPTable config.
3) Is there support for tun/tap device
4) is there support for correct memory info and disk space info (are
df and top are showing the container ressources or the resources of
the hardwarenode)
5) is there something compared to the fine grained controll about
memory resources like vmguarpages/privmpages/oomguarpages in LXC?
6) is LXC production ready?

Thanks in Advance, and we are looking forward to switch to Linux
Containers when all Questions are answered with yes :-)

Regards,
Christian

--
Christian Haintz
Student of Software Development and Business Management
Graz, University of Technology
Gordon Henderson
2010-05-13 17:17:22 UTC
Permalink
Post by Christian Haintz
Hi,
At first LXC seams to be a great work from what we have read already.
There are still a few open questions for us (we are currently running
dozens of OpenVZ Hardwarenodes).
I can't answer for the developers, but here's my answers/observations
based on what I've seen and used ...
Post by Christian Haintz
1) OpenVZ in the long-term seams to be a dead end. Will LXC be a
feature complete replacement for OpenVZ in the 1.0 Version?
I looked at OpenVZ and while it looked promising, didn't seem to be going
anywhere. I also struggled to get their patches into a recent kernel and
it looked like there was no Debian support for it. LXC was in the kernel
as standard - I doubt it'll come out now... (and there is a back-ported
lxc debian package that works fine under Lenny)
Post by Christian Haintz
As of the current version
2) is there IPTable support, any sort of control like the OpenVZ
IPTable config.
I run iptables - and in some cases different iptable setups in each
container on a host (which also has it's own iptables).

Seems to "just work". Each container has an eth0 and the host has a br0
(as well as an eth0).

Logging is at the kernel level though, so goes into the log-files on the
server host rather than in the container - it may be possible to isolate
that, but it's not something I'm too bothered with.

My iptables are just shell-scripts that get called as part of the boot
sequence - I really don't know what sort of control OpenVZ gives you.
Post by Christian Haintz
3) Is there support for tun/tap device
Doesn't look like it yet...

http://www.mail-archive.com/lxc-users at lists.sourceforge.net/msg00239.html
Post by Christian Haintz
4) is there support for correct memory info and disk space info (are
df and top are showing the container ressources or the resources of
the hardwarenode)
Something I'm looking at myself - top gives your own processes, but cpu
usage is for the whole machine. 'df' I can get by manipulating /etc/mtab -
then I get the size of the entire partition my host is running under. I'm
not doing anything 'clever' like creating a file and loopback mounting it
- all my containers in a host are currently on the same partition. I'm not
looking to give fixed-size disks to each container though. YMMV.

However gathering cpu stats for each container is something I am
interested in - and was about to post to the list about it - I think there
are files (on the host) under /cgroup/container-name/cpuacct.stat and a
few others which might help me though, but I'm going to have to look them
up...
Post by Christian Haintz
5) is there something compared to the fine grained controll about
memory resources like vmguarpages/privmpages/oomguarpages in LXC?
Pass..
Post by Christian Haintz
6) is LXC production ready?
Not sure who could make that definitive decision ;-)

It sounds like the lack of tun/tap might be a show-stopper for you though.
(come back next week ;-)

However, I'm using it in production - got a dozen LAMPy type boxes running
it so-far with several containers inside, and a small number of asterisk
hosts. (I'm not mixing the LAMP and asterisk hosts though) My clients
haven't noticed any changes which makes me happy. I don't think what I'm
doing is very stressful to the systems though, but so-far I'm very happy
with it.

I did test it to my own satisfaction before I committed myself to it on
servers 300 miles away though. One test was to create 20 containers on an
old 1.8GHz celeron box, each running asterisk with one connected to the
next and so on - then place a call into the first. It manged 3 loops
playing media before it had any problems - and those were due to kernel
context/network switching rather than anything to do with the LXC setup.
(I suspect there is more network overhead though due to the bridge and
vlan nature of the underlying plumbing)

So right now, I'm happy with LXC - I've no need for other virtualisation
as I'm purely running Linux, so don't need to host Win, different kernels,
etc. And for me, it's a management tool - I can now take a container and
move it to different hardware (not yet a proper "live migration", but the
final rsync is currently only a few minutes and I can live with that) I
have also saved myself a headache or two by moving old servers with OS's I
couldn't upgrade into new hardware - so I have one server running Debian
Lenny, kernel 2.6.33.1 hosting an old Debian Woody server inside a
container running the customers custom application which they developed 6
years ago... They're happy as they got new hardware and I'm happy as I
didn't have to worry about migrating their code to a new version of Debian
on new hardware... And I can also take that entire image now and move it
to another server if I needed to load-balance, upgrade, cater for h/w
failure, etc.

I'm using kernel 2.6.33.x (which I custom compile for the server hardware)
and Debian Lenny FWIW.

I'm trying to not sound like a complete fanboi, but until the start of
this year, I had no interest in virtualisation at all, but once I got into
it and saw it as a management tool, I was sold - and LXC is the solution
that seemed to work the best for me. (and more-so as a lot of the servers I
have don't have those magic instructions to make XEN or KVM go faster)

Hope this helps,

Gordon
Daniel Lezcano
2010-05-13 21:22:50 UTC
Permalink
Post by Christian Haintz
Hi,
At first LXC seams to be a great work from what we have read already.
There are still a few open questions for us (we are currently running
dozens of OpenVZ Hardwarenodes).
1) OpenVZ in the long-term seams to be a dead end. Will LXC be a
feature complete replacement for OpenVZ in the 1.0 Version?
Theorically speaking, LXC is not planned to be a replacement to OpenVZ.
When a specific functionality is missing, it is added. Sometimes that
needs a kernel development implying an attempt to mainline inclusion.

When the users of LXC want a new functionality, they send a patchset or
ask if it possible to implement it. Often, the modifications need a
kernel modification at that takes sometime to reach the upstream kernel
(eg. sysfs per namespace).

Practically speaking, LXC evolves following the needs (eg. entering a
container) of the users and that may lead to a replacement of OpenVZ.

The version 1.0 is planned to be a stable version, with documentation
and frozen API.
Post by Christian Haintz
As of the current version
2) is there IPTable support, any sort of control like the OpenVZ
IPTable config.
The iptables support in the container is depending on the kernel version
you are using. AFAICS, iptables per namespace is implemented now.
Post by Christian Haintz
3) Is there support for tun/tap device
The drivers are ready to be used in the container but not sysfs and that
unfortunately prevent to create a tun/tap in a container.

sysfs per namespace is on the way to be merged upstream.
Post by Christian Haintz
4) is there support for correct memory info and disk space info (are
df and top are showing the container ressources or the resources of
the hardwarenode)
No and that will not be supported by the kernel but it is possible to do
that with fuse. I did a prototype here:

http://lxc.sourceforge.net/download/procfs/procfs.tar.gz

But I gave up with it because I have too much things to do with lxc and
not enough free time. Anyone is welcome to improve it ;)
Post by Christian Haintz
5) is there something compared to the fine grained controll about
memory resources like vmguarpages/privmpages/oomguarpages in LXC?
I don't know these controls you are talking about but LXC is plugged
with the cgroups. One of the subsystem of the cgroup is the memory
controller allowing to assign an amount of physical memory and swap
space to the container. There are some mechanism for notification as
well. There are some other resource controller like io (new), freezer,
cpuset, net_cls and device whitelist (googling one of these name + lwn
may help).
Post by Christian Haintz
6) is LXC production ready?
yes and no :)

If you plan to run several webserver (not a full system) or non-root
applications, then yes IMHO it is ready for production.

If you plan to run a full system and you have very aggressive users
inside with root privilege then it may not be ready yet. If you setup a
full system and you plan to have only the administrator of the host to
be the administrator of the containers, and the users inside the
container are never root, then IMHO it ready if you accept for example
to have the iptables logs to go to the host system.

Really, it depends of what you want to do ...

I don't know OpenVZ very well, but AFAIK it is focused on system
container while LXC can setup different level of isolation allowing to
run an application sharing a filesystem or a network for example, as
well as running a full system. But this flexibility is a drawback too
because the administrator of the container needs a bit of knowledge on
the system administration and the container technology.
Post by Christian Haintz
Thanks in Advance, and we are looking forward to switch to Linux
Containers when all Questions are answered with yes :-)
Hope that helped.

Thanks
-- Daniel
Ahrenholz, Jeffrey M
2010-05-14 21:28:08 UTC
Permalink
Post by Daniel Lezcano
Post by Christian Haintz
3) Is there support for tun/tap device
The drivers are ready to be used in the container but not
sysfs and that
unfortunately prevent to create a tun/tap in a container.
sysfs per namespace is on the way to be merged upstream.
The workaround here is to create a TUN/TAP on the host and install it into the container using something like 'ip link set tap0 netns nnnn'. The app running in the container may not complain (from my experience) and can open and use the tap0.

Another option (which cannot be done with OpenVZ) is creating a TUN/TAP on the host, opening the socket end from a host process, then pushing it into the namespace container. Once in the namespace you can rename it to anything you'd like. Processes in the namespace may read/write as a normal device, and you get all of the packets in the host system. This is useful for connecting together containers with simulators such as ns-3.

-Jeff
Christian Haintz
2010-06-01 21:26:52 UTC
Permalink
Hi,

At first, thanks for all the great feedback and the quickly ongoing
development to lxc.
Post by Daniel Lezcano
Post by Christian Haintz
6) is LXC production ready?
yes and no :)
If you plan to run several webserver (not a full system) or non-root
applications, then yes IMHO it is ready for production.
If you plan to run a full system and you have very aggressive users
inside with root privilege then it may not be ready yet. If you
setup a full system and you plan to have only the administrator of
the host to be the administrator of the containers, and the users
inside the container are never root, then IMHO it ready if you
accept for example to have the iptables logs to go to the host system.
In my opinion there is not a big different if i run some software
which might have a security bug which people could exploit or if i
have a root user who trys to escape the container. In both ways i need
isolation which i can trust.
For me this is the main reason doing things in isolation like lxc or
openvz, because i don't need the overhead of kvm or xen but i still
need isolation which jail a software or a system - root users inside
or not.

It looks to me like you already know a way how to escape from a
container, don't you? And if so, is that a desired behavior or just a
bug?
The point i'd like to come: Is one goal of lxc to make it a container
where nothing/nobody can escape or is this feature just "nice-to-have"
but not a "must have" on the roadmap?
Post by Daniel Lezcano
Really, it depends of what you want to do ...
I don't know OpenVZ very well, but AFAIK it is focused on system
container while LXC can setup different level of isolation allowing
to run an application sharing a filesystem or a network for example,
as well as running a full system. But this flexibility is a drawback
too because the administrator of the container needs a bit of
knowledge on the system administration and the container technology.
For me, all aspects of lxc are interesting, I am not only focused to
full system virtualization. I am also thinking of jailing just some
apps with some libs in containers (e.g. python). But in the end, for
me it is about encapsulation with no escape :-)

Regards,
Christian

--
Christian Haintz
Student of Software Development and Business Management
Graz, University of Technology

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20100601/44e08552/attachment.html>
Daniel Lezcano
2010-06-02 09:25:10 UTC
Permalink
Post by Christian Haintz
Hi,
At first, thanks for all the great feedback and the quickly ongoing
development to lxc.
Post by Daniel Lezcano
Post by Christian Haintz
6) is LXC production ready?
yes and no :)
If you plan to run several webserver (not a full system) or non-root
applications, then yes IMHO it is ready for production.
If you plan to run a full system and you have very aggressive users
inside with root privilege then it may not be ready yet. If you setup
a full system and you plan to have only the administrator of the host
to be the administrator of the containers, and the users inside the
container are never root, then IMHO it ready if you accept for
example to have the iptables logs to go to the host system.
In my opinion there is not a big different if i run some software
which might have a security bug which people could exploit or if i
have a root user who trys to escape the container. In both ways i need
isolation which i can trust.
For me this is the main reason doing things in isolation like lxc or
openvz, because i don't need the overhead of kvm or xen but i still
need isolation which jail a software or a system - root users inside
or not.
It looks to me like you already know a way how to escape from a
container, don't you?
No, you can't escape the container. I meant a root user in a container
has a nuisance power for the host system, eg. send falsified packets to
the network.
Post by Christian Haintz
And if so, is that a desired behavior or just a bug?
The point i'd like to come: Is one goal of lxc to make it a container
where nothing/nobody can escape or is this feature just "nice-to-have"
but not a "must have" on the roadmap?
Post by Daniel Lezcano
Really, it depends of what you want to do ...
I don't know OpenVZ very well, but AFAIK it is focused on system
container while LXC can setup different level of isolation allowing
to run an application sharing a filesystem or a network for example,
as well as running a full system. But this flexibility is a drawback
too because the administrator of the container needs a bit of
knowledge on the system administration and the container technology.
For me, all aspects of lxc are interesting, I am not only focused to
full system virtualization. I am also thinking of jailing just some
apps with some libs in containers (e.g. python). But in the end, for
me it is about encapsulation with no escape :-)
From a design POV with the namespaces, an application can't escape.

Thanks
-- Daniel

Continue reading on narkive:
Loading...