[Lxc-users] lxcbr0 versus virbr0 (Ubuntu)

[Ulli Horlacher]

Prologue: I run LXC successful for nearly 2 years on Ubuntu 10.04, using
veth / br0. Every container has its own IP address, no NAT. I run
production services like http://fex.rus.uni-stuttgart.de/ on it, rocksolid.

I have now set up second server with Ubuntu 12.04 and there have changed a
lot of things, starting with networking.

Reading https://help.ubuntu.com/12.04/serverguide/lxc.html it says
one can use lxcbr0 or virbr0 for bridging, but without further explanation.

What is "better"? Or is lxcbr0 only for NAT?
Is virbr0 the successor of br0?
Probably I am missing some basic documentation...

The new server has six GbE interfaces and I have set up "ethernet bonding":
three real interfaces build one virtual interface.

I have successfully assigned a single test-IP to bond1:

root at vms3:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 129.69.202.142 0.0.0.0 UG 100 0 0 bond0
10.0.3.0 0.0.0.0 255.255.255.0 U 0 0 0 lxcbr0
129.69.1.0 0.0.0.0 255.255.255.0 U 0 0 0 bond1
129.69.202.128 0.0.0.0 255.255.255.240 U 0 0 0 bond0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 bond0

(lxcbr0 was automaticaly started when I installed lxc)

How shall I continue?
Binding which bridge type how to bond1?


Below is my current network setup:

root at vms3:~# cat /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
bond-master bond0

auto eth1
iface eth1 inet manual
bond-master bond0

auto eth2
iface eth2 inet manual
bond-master bond0

auto eth3
iface eth3 inet manual
bond-master bond1

auto eth4
iface eth4 inet manual
bond-master bond1

auto eth5
iface eth5 inet manual
bond-master bond1

auto bond0
iface bond0 inet static
address 129.69.202.131
netmask 255.255.255.240
network 129.69.202.128
broadcast 129.69.202.143
gateway 129.69.202.142
bond-mode balance-rr
bond-miimon 100
bond-slaves none

auto bond1
iface bond1 inet static
#up ifconfig bond1 up
address 129.69.1.42
netmask 255.255.255.0
network 129.69.1.0
broadcast 129.69.1.255
bond-mode balance-rr
bond-miimon 100
bond-slaves none

[Christoph Mitasch]

Hello,

I started to run LXC in production with Ubuntu 12.04 a few months ago. Without any problems so far too.

Post by Ulli Horlacher
Reading https://help.ubuntu.com/12.04/serverguide/lxc.html it says
one can use lxcbr0 or virbr0 for bridging, but without further
explanation.
What is "better"? Or is lxcbr0 only for NAT?
Is virbr0 the successor of br0?
Probably I am missing some basic documentation...

See /etc/default/lxc for a short explanation of lxcbr0.
My experience was that this ensures that a newly created container without any network configuration has outgoing access to the network/Internet.

See /etc/init/lxc-net.conf for details. Actually a minimal DHCP + MASQUERADING environment is set up in the 10.0.3.0/24 network using dnsmasq and iptables.

I'm not using the lxcbr0 bridge, it's just nice for fresh installations where you have Internet access automatically after installing.

If you set your own network bridge (lxc.network.type=veth,lxc.network.link=br0,lxc.network.flags=up) for a container everything should work for you as in the past.

Post by Ulli Horlacher
The new server has six GbE interfaces and I have set up "ethernet
three real interfaces build one virtual interface.
root at vms3:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref
Use Iface
0.0.0.0 129.69.202.142 0.0.0.0 UG 100 0
0 bond0
10.0.3.0 0.0.0.0 255.255.255.0 U 0 0
0 lxcbr0
129.69.1.0 0.0.0.0 255.255.255.0 U 0 0
0 bond1
129.69.202.128 0.0.0.0 255.255.255.240 U 0 0
0 bond0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0
0 bond0
(lxcbr0 was automaticaly started when I installed lxc)

I've setup my bridge using /etc/network/interfaces. You can disable lxcbr0 if you don't like it in /etc/default/lxc.
# Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your
# containers. Set to "false" if you'll use virbr0 or another existing
# bridge, or mavlan to your host's NIC.
USE_LXC_BRIDGE="true"

I would also like to point out the new apparmor profile that is automatically assigend to containers since Ubuntu 12.04. That enhances the security of the containers.
https://help.ubuntu.com/12.04/serverguide/lxc.html#lxc-apparmor

Regards,
Christoph

PS: I just came back from a great LinuxCon in Barcelona. I did a presentation about LXC there. If you're interested:
http://www.thomas-krenn.com/en/wiki/Event-News:_LinuxCon_Europe_2012

[Serge Hallyn]

Post by Ulli Horlacher
Prologue: I run LXC successful for nearly 2 years on Ubuntu 10.04, using
veth / br0. Every container has its own IP address, no NAT. I run
production services like http://fex.rus.uni-stuttgart.de/ on it, rocksolid.
I have now set up second server with Ubuntu 12.04 and there have changed a
lot of things, starting with networking.
Reading https://help.ubuntu.com/12.04/serverguide/lxc.html it says
one can use lxcbr0 or virbr0 for bridging, but without further explanation.
What is "better"? Or is lxcbr0 only for NAT?

virbr0 is the libvirt bridge, it's essentially identical to lxcbr0. It's
just that older lxc didn't provide lxcbr0.

Since you have a real bridge, it is better to keep using br0. In fact,
edit /etc/default/lxc to set USE_LXC_BRIDGE="false" to avoid creating
lxcbr0 at all.

-serge

[Ulli Horlacher]

Post by Serge Hallyn
Since you have a real bridge, it is better to keep using br0.

I have just discovered, that br0 is still available!
I was in mistake to think only lxcbr0 and virbr0 are choosable.

Post by Serge Hallyn
In fact, edit /etc/default/lxc to set USE_LXC_BRIDGE="false" to avoid
creating lxcbr0 at all.

This is good documented. I found it quick :-)

Is there a comprehensive documentation about Linux bridging in general or
LXC networking in special?

[Serge Hallyn]

Post by Ulli Horlacher

Post by Serge Hallyn
Since you have a real bridge, it is better to keep using br0.

I have just discovered, that br0 is still available!
I was in mistake to think only lxcbr0 and virbr0 are choosable.

Post by Serge Hallyn
In fact, edit /etc/default/lxc to set USE_LXC_BRIDGE="false" to avoid
creating lxcbr0 at all.

This is good documented. I found it quick :-)
Is there a comprehensive documentation about Linux bridging in general or
LXC networking in special?

The ubuntu server guide does have a section on bridging

https://help.ubuntu.com/12.04/serverguide/network-configuration.html
https://help.ubuntu.com/12.10/serverguide/network-configuration.html

as does the community guide

https://help.ubuntu.com/community/NetworkConnectionBridge

lxcbr0 is described briefly in the server guide section on lxc:

https://help.ubuntu.com/12.04/serverguide/lxc.html
https://help.ubuntu.com/12.10/serverguide/lxc.html