Discussion:
Converting existing CentOS 6.x to container within Ubuntu 12.04 - can that be simple?
(too old to reply)
Whit Blauvelt
2012-10-27 18:36:23 UTC
Permalink
Hi,

I'd welcome advice on whether there's a sane, relatively simple way to take
a backup copy of a CentOS 6 system, which happens to be sitting on a
partition on a Ubuntu 12.04 VMware VM, and set it up to run in a container
there. It's been a year since I've done anything with LXC. I see that it's
been nicely streamlined in Ubuntu 12.04. Googling for pointers to answer my
current question, I'm mostly seeing older discussion of older Ubuntu and
CentOS versions, which I expect partially doesn't apply now.

I have this notion that it might be simple indeed to set this up. But that
notion is admittedly foggy. If it is simple, is there a guide to this sort
of thing somewhere? I see a template for a fresh CentOS 6 guest here on the
list I could try, but if there's a way to more directly just use the already
configured backup rather than build a fresh instance that would be even
better.

Thanks,
Whit
Fajar A. Nugraha
2012-10-28 09:50:07 UTC
Permalink
Post by Whit Blauvelt
I have this notion that it might be simple indeed to set this up.
Sure it is. Well, kindof :)
Post by Whit Blauvelt
But that
notion is admittedly foggy. If it is simple, is there a guide to this sort
of thing somewhere? I see a template for a fresh CentOS 6 guest here on the
list I could try, but if there's a way to more directly just use the already
configured backup rather than build a fresh instance that would be even
better.
Did your search brought you to
http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6 ? :D

If yes, that guide assumes you have a "working" centos 6 setup
already, in the form of one created using "yum install --installroot".
You could change that to "a filesystem-level backup of a working
centos installation", and pretty much do the same modifications. In
particular, lxc-sysinit.conf and fstab.

There might be other modifcations required (I forgot which ones, try
looking at http://wiki.1tux.org/wiki/Centos6/Installation/Minimal_installation_using_yum#Post-install_configuration
and see which ones is relevant), just try it and see how it goes.

As usual, create backups before you modify anything. Just in case.
--
Fajar
Whit Blauvelt
2012-11-01 13:47:25 UTC
Permalink
Post by Fajar A. Nugraha
Did your search brought you to
http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6 ? :D
Did not, and that's a very nice recipe.

My current question is if there's an available bridging scheme that will
work in my context. The host is an ESXi VMware VM (currently CentOS 6, but
could be Ubuntu 12.04 if helpful). The CentOS 6 guest on that host needs to
end up with a unique IP on the VMware LAN. VMware does not work unless it
can assign the host's IP by dhcp, and at least so far in my experiments will
not do that if I set the host to use bro0 rather than eth0.

Once the host is up, I can add additional LAN IPs to eth0 without problem.
What's not clear is which, if any, of the bridging schemes for the LXC guest
on the host can enable that guest to take its own IP on the LAN.

Why am I trying such a silly trick? Because I have some perfectly good KVM
VMs, but no tool that can convert them to VDOs to put on VMware - the
existing tools satisfy the common demand, which sanely is to get VMs off of
VMware and onto KVM, not the other way around. But my client is committed to
a cloud provider with an ancient, creaky VMware beneath a crippled user
interface.

So _if_ I can take the LXC guest creation recipe above - which is even
easier to follow that it looks at first glance - and then manage the right
bridging trick with it, this will be far more efficient than configuring
VMware VMs from scratch to duplicate the existing, highly-configured KVM
VMs. It could even enable combining some of the less stressed KVM VMs onto
single LXC hosts on VMware, to cut back a bit on the cloud fees.

But ... can it be done? Looking at this page,
http://wiki.1tux.org/wiki/Ubuntu/Bridge, it's not clear if it can be. In
KVM, I always just set up a real bridge on the host - the thing it seems I
can't do in this VMware setting. In all cases, the LXC guests need to end up
with a LAN IP on which they can be addressed from other systems, but not
necessarily the LXC host.

Thanks,
Whit
Fajar A. Nugraha
2012-11-01 13:58:13 UTC
Permalink
Post by Whit Blauvelt
The host is an ESXi VMware VM (currently CentOS 6, but
could be Ubuntu 12.04 if helpful).
Ubuntu will definitely be easier. It has new-enough lxc version, plus
you won't have selinux fiasco (search the list archive for details).
Post by Whit Blauvelt
The CentOS 6 guest on that host needs to
end up with a unique IP on the VMware LAN.
Sure. Easy.
Post by Whit Blauvelt
But ... can it be done? Looking at this page,
http://wiki.1tux.org/wiki/Ubuntu/Bridge, it's not clear if it can be.
Sure you can.

http://wiki.1tux.org/wiki/Ubuntu/Bridge#Bridging_a_real_network_interface

It that setup, both the host (the example assumes ubuntu host) and the
containers which uses that bridge will be on the same LAN, getting IP
address from the same DHCP server (whatever that is) on that network.
--
Fajar
Whit Blauvelt
2012-11-01 14:37:40 UTC
Permalink
Fajar,

Thanks for the quick response. I've gotten a bit farther with VMware. It
will allow br0 to be the interface on its guest - it just can't assign that
by dhcp. But when I get the invocation right for a static assignment, it
takes. It had been seeming that br0 for the host interface just wasn't an
option under VMware.

Is a real bridge the only way to get an unique IP on the same LAN for the
LXC guest, or would some combination of assigning a second IP to the LXC
host, and then DNAT'ing ans SNAT'ing to it, work reasonably well? I just ask
in case at some point the cloud provider sees that we're violating their TOS
with the static IP assignment. Most probably they won't.

Whit
Fajar A. Nugraha
2012-11-01 14:44:40 UTC
Permalink
Post by Whit Blauvelt
Fajar,
Thanks for the quick response. I've gotten a bit farther with VMware. It
will allow br0 to be the interface on its guest - it just can't assign that
by dhcp. But when I get the invocation right for a static assignment, it
takes. It had been seeming that br0 for the host interface just wasn't an
option under VMware.
It shouldn't matter.

Vmware shouldn't care what the interface name is under the guest OS.
Or what you do with it (bridge or not).

If you have an Ubuntu host, with a wired ethernet interface (e.g.
eth0), then you can create a bridge on top of that. It doesn't matter
whether the ubuntu host is actually native (on real server), a KVM
guest, or a vmware guest.

... of course there are probably some settings (even on real server +
switch) that limits only ONE MAC address on a single port. In this
case you can't use bridge. Again, it doesn't matter whether it's
vmware or not. It's a matter whether the switch (real or virtual,
doesn't matter) allows more than one MAC on a single port.
Post by Whit Blauvelt
Is a real bridge the only way to get an unique IP on the same LAN for the
LXC guest,
That's the easy way.
Post by Whit Blauvelt
or would some combination of assigning a second IP to the LXC
host, and then DNAT'ing ans SNAT'ing to it, work reasonably well?
That's the hard way. If you use this one you'd probably want to use a
host-only bridge (e.g.
http://wiki.1tux.org/wiki/Ubuntu/Bridge#Bridge_with_IP_address) and
setup NAT rules manually.
--
Fajar
Whit Blauvelt
2012-11-01 20:21:44 UTC
Permalink
I must be real close, thanks to your help. Have lxc on Ubuntu 12.04, with a
bridge set up, and an existing CentOS 6.3 file system copied to the system
and modified per the tux1 directions.

But ...

# lxc-start --name obscured
lxc-start: Invalid argument - failed to umount 'dev/pts'
lxc-start: failed to setup the new pts instance
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'obscured'

Emptying the dev/pts directory fixed that, but then:

obscured login: root
Password: init: tty (/dev/tty2) main process (131) terminated with status 1
init: tty (/dev/tty2) main process ended, respawning
init: tty (/dev/tty3) main process (133) terminated with status 1
init: tty (/dev/tty3) main process ended, respawning
init: tty (/dev/tty4) main process (135) terminated with status 1
init: tty (/dev/tty4) main process ended, respawning
init: tty (/dev/tty5) main process (137) terminated with status 1
init: tty (/dev/tty5) main process ended, respawning
init: tty (/dev/tty6) main process (139) terminated with status 1
init: tty (/dev/tty6) main process ended, respawning

Should I have just emptied the whole /dev directory and recreated only the
specifically given entries given in the tux1 wiki? I'd run each command
given there, but not on an empty /dev directory.

Thanks again,
Whit
Fajar A. Nugraha
2012-11-01 21:46:19 UTC
Permalink
Post by Whit Blauvelt
obscured login: root
Password: init: tty (/dev/tty2) main process (131) terminated with status 1
init: tty (/dev/tty2) main process ended, respawning
init: tty (/dev/tty3) main process (133) terminated with status 1
init: tty (/dev/tty3) main process ended, respawning
init: tty (/dev/tty4) main process (135) terminated with status 1
init: tty (/dev/tty4) main process ended, respawning
init: tty (/dev/tty5) main process (137) terminated with status 1
init: tty (/dev/tty5) main process ended, respawning
init: tty (/dev/tty6) main process (139) terminated with status 1
init: tty (/dev/tty6) main process ended, respawning
I use this: lxc.tty = 1. which means, only tty1 is active :)

You could either:
- delete /dev/tty[2-6], or
- use lxc.tty = 6. Haven't test this though.

I prefer the first one. The tty's are used only when use
"lxc-console", and by default it connects to tty1, so IMHO there's no
point in having tty >= 2.
--
Fajar
Whit Blauvelt
2012-11-01 22:30:31 UTC
Permalink
Post by Fajar A. Nugraha
I use this: lxc.tty = 1. which means, only tty1 is active :)
- delete /dev/tty[2-6], or
- use lxc.tty = 6. Haven't test this though.
I prefer the first one. The tty's are used only when use
"lxc-console", and by default it connects to tty1, so IMHO there's no
point in having tty >= 2.
Went the route of deleting all of dev, then recreating the specific parts in
the document. At this point it lxc-starts, and due to my having cribbed this
line from a doc elsewhere for the config file

lxc.network.ipv4 = 10.196.58.117

it comes up with that IP. Still needs work on the routing and netmask but I
can reconfigure that by hand for the moment.

What it does do is connect nicely to the host, on 10.196.58.116, and vice
versa. What it doesn't do, unlike the host, is connect to anywhere else,
despite being set up with identical routes.

Now, on other VMs here I've tested if VMware will allow multiple IPs per
machine. It does, when they're both just assigned to the base machine, and
so would have the same MAC. On the host ip_forward = 1 so that's not it. I
wonder if the trick would be to have the guest have the _same_ MAC as the
host rather than an arbitrary one? So ...

Ah if I do that, then VMware manages to force both the IP belonging to the
host and the appropriate routing tables onto it. That's funny. But it can't
be recovered from. Changing the IP doesn't result in it being able to
connect even to the host.

I'm thinking the VMware restriction - apparently that it come from a known
MAC - may be incompatible with using br0. So I guess I have to work out the
DNAT/SNAT formula, and assign the 2nd IP for the guest on the host level and
then forward it through.

I really do appreciate the advice and encouragement, Fajar.

Whit
Fajar A. Nugraha
2012-11-01 22:35:54 UTC
Permalink
Post by Whit Blauvelt
I'm thinking the VMware restriction - apparently that it come from a known
MAC - may be incompatible with using br0. So I guess I have to work out the
DNAT/SNAT formula, and assign the 2nd IP for the guest on the host level and
then forward it through.
Think of this from another perspective: other linux virtualizations
(e.g. KVM and xen) also use the same network configuration (bridge to
a physical network device) by default. So while xen (when using PV)
can run under vmware, your particular setup would prevent it from
getting network connectivity too. Just saying that it's not lxc's
fault :)

Having said that, I recall some VPS providers enforcing the same
limitation, so your condition is quite common. Please share whatever
ended up working for you so others can benefit from it as well.
--
Fajar
Whit Blauvelt
2012-11-08 17:32:57 UTC
Permalink
Post by Fajar A. Nugraha
Having said that, I recall some VPS providers enforcing the same
limitation, so your condition is quite common. Please share whatever
ended up working for you so others can benefit from it as well.
I've almost got it fully working. Taking some ideas from here:

http://www.activestate.com/blog/2011/10/virtualization-ec2-cloud-using-lxc

I've created a virtual bridge on the host, put a subnet on that bridge, and
connected the guest to the host through the subnet. I've also added a second
IP to the host's LAN interface, and used DNAT and SNAT in iptables to
connect that to the guest.

The private bridge is on 192.168.3.0/24, and the LAN on 10.196.58.0/24, with
the second LAN IP on the host 10.196.58.117, and the IP of the guest on the
private bridge 192.168.3.134.

So my two iptables rules on the lxc host are simply:

iptables -t nat -A PREROUTING -d 10.196.58.117 -j DNAT --to-destination 192.168.3.134
iptables -t nat -A POSTROUTING -s 192.168.3.134 -j SNAT --to 10.196.58.117

which results in these rules in the nat table:

Chain PREROUTING (policy ACCEPT 2676 packets, 427K bytes)
num pkts bytes target prot opt in out source destination
1 73 3672 DNAT all -- * * 0.0.0.0/0 10.196.58.117 to:192.168.3.134

Chain POSTROUTING (policy ACCEPT 37927 packets, 2358K bytes)
num pkts bytes target prot opt in out source destination
1 1621 121K SNAT all -- * * 192.168.3.134 0.0.0.0/0 to:10.196.58.117

That's working to a large extent, but not completely. From outside I can SSH
into the guest at the host's 10.196.58.117 address. And from the guest, I
can ping out to anywhere. But from the guest, I am not able to SSH to
anywhere. I can't mount filesystems from other systems either. In some cases
SSH attempts will simply give a "Host key verification failed" message -
this to hosts which the lxc host can SSH to with no problem. In others SSH
just hangs. File system mounts fail with "mount error(13): Permission
denied" - this with mount commands which work perfectly on the lxc host.

So the solution so far is fine for an lxc container that's simply going to
be a server, without needing to mount any filesystems hosted on other
systems or SSH to other systems. It's working within the restrictions of the
VMware host that's underneath this, which allows multiple IPs on its guests
(such as the lxc host here) but not multiple MAC addresses. The MAC address
restriction prevents the lxc guest from sharing the host's bridge set up on
the host's VMware-LAN-facing interface.

My real-world use for this setup requires mounting filesystems which are
outside the guest. If there's a way to mount them to the lxc host and make
them available to the guest I haven't found it. SSH'ing out from the lxc
guest isn't a requirement. But I suspect that and the filesystem mounts are
failing for a common reason.

Ideas?

Thanks,
Whit
Fajar A. Nugraha
2012-11-09 00:35:22 UTC
Permalink
Post by Whit Blauvelt
Post by Fajar A. Nugraha
Having said that, I recall some VPS providers enforcing the same
limitation, so your condition is quite common. Please share whatever
ended up working for you so others can benefit from it as well.
That's working to a large extent, but not completely. From outside I can SSH
into the guest at the host's 10.196.58.117 address. And from the guest, I
can ping out to anywhere. But from the guest, I am not able to SSH to
anywhere. I can't mount filesystems from other systems either. In some cases
SSH attempts will simply give a "Host key verification failed" message -
this to hosts which the lxc host can SSH to with no problem. In others SSH
just hangs. File system mounts fail with "mount error(13): Permission
denied" - this with mount commands which work perfectly on the lxc host.
Ideas?
(1) I'm not sure you can do nfs-mount inside an lxc container
(2) For connectivity issues, try with something simple first, like
telnet on wget

In any case, I suggest you start a new thread (or two, one for each
issue) to get attention from others to that specific issue.
--
Fajar
Dan Kegel
2012-11-09 01:02:47 UTC
Permalink
Post by Fajar A. Nugraha
(1) I'm not sure you can do nfs-mount inside an lxc container
Agreed. I had to rip out all use of nfs inside my lxc containers;
app-level things like scp and wget work fine.
Jäkel, Guido
2012-11-09 08:43:48 UTC
Permalink
Post by Fajar A. Nugraha
(1) I'm not sure you can do nfs-mount inside an lxc container
Yes, you can for the simplest solution.

But also, you can mount it on the host and propagate it (or any subtree, e.g. for a concrete container) via an bind-mount to the container. If you have a lot of containers, this will reduce the number of NFS-mounts to one per host. And if the containers will use the same set of files, there will use local locking and share the same fs-cache.

Also, as the network traffic caused by NFS operations will be handled by the host and there is no "traffic" caused by file access inside the containers, the container don't need to have network access to the NFS server used. With other words, the NFS server don't need to be exposed to the network domain of the containers but just to that of the host.


A entry in an lxc fstab file (referred by lxc.mount=) like

/mnt/ext_nfs/container_foo mnt/my_nfs_part none bind 0 0

will propagate (a former at host at /mnt/ext_nfs mounted) external NFS source (with a tree container_foo) to the mount point /mnt/my_nfs_part of the container foo.


This paradigm will also serve the principle "Separation of Concerns", because the container don't have to know about the source of the shared file space. It might be shifted, splitted or reconfigured otherwise in case of external needs and it even don't need to be served by NFS.


Guido
Whit Blauvelt
2012-11-09 15:19:41 UTC
Permalink
Post by Jäkel, Guido
Post by Fajar A. Nugraha
(1) I'm not sure you can do nfs-mount inside an lxc container
Yes, you can for the simplest solution.
Still doesn't work for me, when the container is on a private bridge from
the host and DNATed and SNATed to a second IP on the host's WAN-facing
interface. Either it won't work in this particular configuration, or I have
to add something more for it to.

As for other connections initiated from the container, wget to outside sites
works fine. On the other hand ftp makes the initial connection and sends the
user name from the prompt, but fails to progress to the password prompt,
instead failing the ftp connection with "authentication failed" despite no
chance to enter the password. Trying "yum update" fails with a flurry of
"Not found" messages for packages required by dependencies. But ping works
to any and all addresses.

And as I mentioned before SSH works into the guest, but SSH out from the
guest consistently fails with a "Host key verification failed" message. This
breakage may well follow from the way I copied in a full existing CentOS 6
filesystem and then modified it to create the guest. Wish I knew how to
diagnose what's wrong. I'm sure packets captured in the NAT stages would
show something, but what to expect as normal there is outside my range of
knowledge.
Post by Jäkel, Guido
But also, you can mount it on the host and propagate it (or any subtree,
e.g. for a concrete container) via an bind-mount to the container. If you
have a lot of containers, this will reduce the number of NFS-mounts to one
per host. And if the containers will use the same set of files, there will
use local locking and share the same fs-cache.
...
That's a powerful argument for this as the right way. Thanks. Now to
understand the scheme.
Post by Jäkel, Guido
A entry in an lxc fstab file (referred by lxc.mount=) like
/mnt/ext_nfs/container_foo mnt/my_nfs_part none bind 0 0
Is there a way to do mounts by hand, or is does this only work if done in
the container's fstab file?

Thanks,
Whit
Whit Blauvelt
2012-11-09 21:59:58 UTC
Permalink
Post by Whit Blauvelt
Is there a way to do mounts by hand, or is does this only work if done in
the container's fstab file?
Yes you can use "mount --bind old new" to test by hand.
Curiously, that even works when I have first a remote mount (in this case
CIFS) on the host mounted into the lxc guest's space, for instance to

/var/lib/lxc/guest/rootfs/mnt/xyz

which initially leaves the mount visible from the host but not the guest.
But then

mount -o bind /var/lib/lxc/guest/rootfs/mnt/xyz /var/lib/lxc/guest/rootfs/mnt/xyz

brings the mount into visibility on the guest.

Now if I can just figure out why yum and ftp don't want to work from inside
the guest. It seems a side effect of having the guest on a private bridge of
the host - or perhaps something odd the VMware LAN underneath the host is
imposing.

Whit
Dan Kegel
2012-11-09 22:33:31 UTC
Permalink
SSH out from the guest consistently fails with a "Host key verification failed"
That might just mean .ssh/known_hosts doesn't have the host in it
inside the container.

Maybe you want -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ?
mount -o bind /var/lib/lxc/guest/rootfs/mnt/xyz /var/lib/lxc/guest/rootfs/mnt/xyz
brings the mount into visibility on the guest.
IIRC, that failed with NFS for me. You may be lucky you're using CIFS.
Whit Blauvelt
2012-11-09 23:03:41 UTC
Permalink
Post by Dan Kegel
SSH out from the guest consistently fails with a "Host key verification failed"
That might just mean .ssh/known_hosts doesn't have the host in it
inside the container.
Maybe you want -o StrictHostKeyChecking=no ...
Oddly,

# ssh 10.196.58.16 -o StrictHostKeyChecking=no
Warning: Permanently added '10.196.58.16' (RSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Now, that never got to the point of offering me a password prompt. Those
lines are without any action on my part. In that way it's acting just like
ftp - where ftp offers me a userid prompt but then doesn't offer me a
password prompt either, instead seemingling getting bad password attempts
automatically somehow.
Post by Dan Kegel
mount -o bind /var/lib/lxc/guest/rootfs/mnt/xyz /var/lib/lxc/guest/rootfs/mnt/xyz
brings the mount into visibility on the guest.
IIRC, that failed with NFS for me. You may be lucky you're using CIFS.
You mounted NFS on the host, and then tried to bind it to the guest? Didn't
work?

Whit
Dan Kegel
2012-11-09 23:13:14 UTC
Permalink
Post by Whit Blauvelt
Post by Dan Kegel
Post by Whit Blauvelt
mount -o bind /var/lib/lxc/guest/rootfs/mnt/xyz /var/lib/lxc/guest/rootfs/mnt/xyz
brings the mount into visibility on the guest.
IIRC, that failed with NFS for me. You may be lucky you're using CIFS.
You mounted NFS on the host, and then tried to bind it to the guest? Didn't
work?
Right. As I recall, the system exploded. Or at least did not work
properly after that. Hung on file access, maybe. It was painful
enough that I've purged the incident from memory, and just
avoid nfs inside lxc.
Whit Blauvelt
2012-11-10 01:07:26 UTC
Permalink
Post by Dan Kegel
Post by Whit Blauvelt
You mounted NFS on the host, and then tried to bind it to the guest? Didn't
work?
Right. As I recall, the system exploded. Or at least did not work
properly after that. Hung on file access, maybe. It was painful
enough that I've purged the incident from memory, and just
avoid nfs inside lxc.
Hmm. Looking here:

https://help.ubuntu.com/12.04/serverguide/lxc.html

when using lxc in Ubuntu, it looks like Apparmor steps all over it,
particularly when it comes to mounting. Ubuntu even has Apparmor as an lxc
dependency, so if you remove Apparmor it takes lxc with it.

I'm going to have to follow the instructions there to make lxc "unconfined."
Having a feature designed to break things, on the assumption that
right-thinking people just shouldn't want to do those things anyway, is bad
design. Especially when it's linked to an experimental feature like lxc,
which people should want to explore to discover its unconfined best uses
before deciding which aspects of it to lock down.

Whit
Whit Blauvelt
2012-11-10 01:25:00 UTC
Permalink
Post by Whit Blauvelt
https://help.ubuntu.com/12.04/serverguide/lxc.html
when using lxc in Ubuntu, it looks like Apparmor steps all over it,
particularly when it comes to mounting. Ubuntu even has Apparmor as an lxc
dependency, so if you remove Apparmor it takes lxc with it.
I'm going to have to follow the instructions there to make lxc "unconfined."
Having a feature designed to break things, on the assumption that
right-thinking people just shouldn't want to do those things anyway, is bad
design. Especially when it's linked to an experimental feature like lxc,
which people should want to explore to discover its unconfined best uses
before deciding which aspects of it to lock down.
Okay, it was Apparmor that was breaking the mounting capability within the
container - although learning that it's far better to do that from the host
was worth the discussion here. I can understand Apparmor being set by
default to block mounting other parts of the host filesystem. Having it
blocked from mounting filesystems elsewhere is absurd. It's the
responsibilty of the systems offering mounts to define who can use them.
There's no good reason to duplicate that responsibilty on the lxc host.

Unfortunately getting Apparmor out of the way doesn't resolve the bizarre
inability to complete an ssh or ftp login dialog - the lxc guest still fails
to show the password prompts on the console, while evidently feeding the
remote end something that looks like line feeds so as to fail the login.

Whit
Dan Kegel
2012-11-10 01:47:33 UTC
Permalink
Post by Whit Blauvelt
Okay, it was Apparmor that was breaking the mounting capability within the
container - although learning that it's far better to do that from the host
was worth the discussion here.
Even doing the nfs on the host failed for me. I had to utterly
purge all NFS from both host and guest for things to work.
Rob Landley
2012-11-11 12:09:28 UTC
Permalink
On Fri, Nov 9, 2012 at 3:03 PM, Whit Blauvelt <whit at transpect.com>
Post by Whit Blauvelt
Post by Dan Kegel
Post by Whit Blauvelt
mount -o bind /var/lib/lxc/guest/rootfs/mnt/xyz
/var/lib/lxc/guest/rootfs/mnt/xyz
Post by Whit Blauvelt
Post by Dan Kegel
Post by Whit Blauvelt
brings the mount into visibility on the guest.
IIRC, that failed with NFS for me. You may be lucky you're using
CIFS.
Post by Whit Blauvelt
You mounted NFS on the host, and then tried to bind it to the
guest? Didn't
Post by Whit Blauvelt
work?
Right. As I recall, the system exploded. Or at least did not work
properly after that. Hung on file access, maybe. It was painful
enough that I've purged the incident from memory, and just
avoid nfs inside lxc.
I fixed cifs to be container aware back in 2010. I spent months
examining NFS trying to do similar fixes, but NFS is a crawling horror
full of bad ideas like superblock merging, and the NFS v2, v3, v4, and
v4.1 (pnfs) protocols have an awful lot of divergent spaghetti
codepaths.

Making CIFS container-aware was a couple lines. Doing the same for NFS
is a thesis project. (It's not that NFS and containers and NFS don't
mix, it's that NFS and _anything_ doesn't mix. Only Sun could design a
"stateless filesystem server" when the point of a filesystem is to
maintain state. The whole design is a giant contradiction in terms with
decades of crap built on top to try to bury the bad ideas under so much
complexity the problems are less obvious. I'll stop now.)

Really: go look at the Plan 9 filesystem (the "9p" driver, the
userspace TCP/IP server is called "diod", and kvm has a built-in 9p
server called "virtfs" using a virtio transport. It's been in the
kernel for years and uses the same "one pipe per mount" connection type
of CIFS (so you can use it over a serial port if you really need to),
but unlike that windows-derived monstrosity the protocol going over
said pipe is not crazy.

I posted a patch to containerize 9p over a year ago, never checked if
it got merged:

http://marc.info/?l=v9fs-developer&m=130597202601764

Rob

Loading...