Discussion:
LAN for LXD containers (with multiple LXD servers)?
(too old to reply)
Tomasz Chmielewski
2016-09-18 09:16:11 UTC
Permalink
It's easy to create a "LAN" for LXD containers on a single LXD server -
just attach them to the same bridge, use the same subnet (i.e.
10.10.10.0/24) - done. Containers can communicate with each other using
their private IP address.

However, with more then one LXD server *not* in the same LAN (i.e. two
LXD servers in different datacentres), the things get tricky.


Is anyone using such setups, with multiple LXD servers and containers
being able to communicate with each other?


LXD1: IP 1.2.3.4, Europe LXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.10.20
container2, 10.10.10.11 container5, 10.10.10.21
container3, 10.10.10.12 container6, 10.10.10.22


LXD3: IP 3.4.5.6, US
container7, 10.10.10.30
container8, 10.10.10.31
container8, 10.10.10.32


While I can imagine setting up many OpenVPN tunnels between all LXD
servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the
routes as containers are stopped/started/migrated, it's a bit of a
management nightmare. And even more so if the number of LXD servers
grows.

Hints, discussion?


Tomasz Chmielewski
https://lxadm.com
Micky Del Favero
2016-09-18 11:32:11 UTC
Permalink
Post by Tomasz Chmielewski
While I can imagine setting up many OpenVPN tunnels between all LXD
servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the
routes as containers are stopped/started/migrated, it's a bit of a
management nightmare. And even more so if the number of LXD servers
grows.
Hints, discussion?
If you use the same subnet for the container as in your example
10.10.10.0/24 there's no routing nightmare, you have only to setup
openvpn in bridge mode and so you'll need only 2 tunnels: LXD1-LXD2 and
LXD2-LXD3, obviously if LXD2 will go offline you'll have a problem.

If you'll have many containers using the same subnet for all will be
impossible so you'll need to managed routing between hosts to be able to
reach every container.

I'm about to design and deploy similar setup: many LXD hosts running
containers that can be started on a hosts and eventually migrated to
another host in the same or in a different datacenter.

I'm thinking not to use the same subnet for all containers but to use a
different subnet for every host, so following your example it will be
like the following:

LXD1: IP 1.2.3.4, Europe LXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.20.10
container2, 10.10.10.11 container5, 10.10.20.11
container3, 10.10.10.12 container6, 10.10.20.12

LXD3: IP 3.4.5.6, US
container7, 10.10.30.10
container8, 10.10.30.11
container9, 10.10.30.12

on every hosts a dhcp server is use to dynamically configure network on
starting container and, based on container's hostname, to dynamically
update dns servers so containers will be always reachable via hostname.

Routing problem remains, I'll probably choose to deploy some tunnels
between hosts to connect each other, then running an OSPF daemon for
dynamic routing management.

Ciao, Micky
--
The sysadmin has all the answers, expecially "No"
Sergiusz Pawlowicz
2016-09-18 12:05:13 UTC
Permalink
While I can imagine setting up many OpenVPN tunnels between all LXD servers
I cannot imagine that :-) :-)

Use tinc, mate. Your life begins :-)

https://www.tinc-vpn.org/

cheers,
Serge
Tomasz Chmielewski
2016-09-18 12:11:26 UTC
Permalink
Post by Sergiusz Pawlowicz
While I can imagine setting up many OpenVPN tunnels between all LXD servers
I cannot imagine that :-) :-)
Use tinc, mate. Your life begins :-)
https://www.tinc-vpn.org/
I did some reading about tinc before, and according to documentation and
mailing lists:

- performance may not be so great

- it gets problematic as the number of tinc instances grows (few will be
OK, dozens will work, but beyond that, the things might get slowish)

- if I'm not mistaken, you need to run a tinc instance per LXD client,
not per LXD server, so that's extra management and performance overhead
(i.e. if two tinc clients are running on the same server, they would
still encrypt the traffic to each other)


Tomasz Chmielewski
https://lxadm.com
Sergiusz Pawlowicz
2016-09-18 16:12:05 UTC
Permalink
Post by Tomasz Chmielewski
Post by Sergiusz Pawlowicz
While I can imagine setting up many OpenVPN tunnels between all LXD servers
I cannot imagine that :-) :-)
Use tinc, mate. Your life begins :-)
https://www.tinc-vpn.org/
I did some reading about tinc before, and according to documentation and
- performance may not be so great
- it gets problematic as the number of tinc instances grows (few will be OK,
dozens will work, but beyond that, the things might get slowish)
- if I'm not mistaken, you need to run a tinc instance per LXD client, not
per LXD server, so that's extra management and performance overhead (i.e. if
two tinc clients are running on the same server, they would still encrypt
the traffic to each other)
I think you must read more, because everything you wrote is simply false.

Or, maybe, try it? :-) :-) :-)

cheers!
S.
Ron Kelley
2016-09-18 13:14:41 UTC
Permalink
(Long reply follows…)

Personally, I think you need to look at the big picture for such deployments. From what I read below, you are asking, “how do I extend my layer-2 subnets between data centers such that container1 in Europe can talk with container6 in Asia, etc”. If this is true, I think you need to look at deploying data center hardware (servers with multiple NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC, etc). And, you must look at how your failover services will work in this design. It’s easy to get a couple of servers working with a simple design, but those simple designs tend to go to production very fast without proper testing and design.


All that said, here is one way I would tackle this type of request:

* Get servers with at least 3 NICs (preferably 5)
* One iLO/DRAC/IPMI interface for out-of-band management
* One for Container server management (ie: LXD1 IP 1.2.3.4) - use a second NIC for redundancy in a bonded configuration
* One for Container hosting network (ie container1, container2, etc) - use a second NIC for redundancy and VLANs to separate traffic

* Get firewalls in each location with L2TP/IPSEC support (pfSense works great)
* Extend your L2 networks between your sites with L2TP
* Secure the connection with IPSSEC

* On your LXD servers, create 2 bonded NICs
* One for container management (eth0, eth1)
* One for hosting network (eth2, eth3)
* Use VLANs on hosting network to separate traffic
* Configure your containers with the appropriate VLAN tag (ie: 501)

Once the above is done, your containers can talk w/each other in different locations. You can use firewall rules to allow/deny IP connections from your container VMs. You can extend both your container management and hosting networks across the L2 tunnel allowing you to move VMs at will.


General Notes:
-----------------------
* For server bonded connections, I use linux mode type 6; works well, provides great throughput, requires no special configuration on directly-connected switches.
* On the LXD side, create multiple profiles with VLAN configurations. Personally, I have 2 profiles: one for VLAN 501 and one for VLAN 502. Local firewall provides security between container networks.
* Be mindful of the services you share across the tunnels. Things like iSCSI, NFS, etc will kill your network performance because of the chatty type of traffic.


Some good references:
-----------------------
https://doc.pfsense.org/index.php/L2TP/IPsec
http://archive.openflow.org/wk/index.php/Tunneling_-_GRE/L2TP
http://www.networkworld.com/article/2163334/tech-primers/what-can-l2tp-do-for-your-network-.html

Caution: L2 networks have a lot of broadcast traffic. If your site-to-site connections are slow, your entire extended L2 network will suffer. Must find a way to suppress L2 broadcast/multicast between sites.


Hope this helps. Happy to share my LXD configurations with anyone...

-Ron






On Sep 18, 2016, at 5:16 AM, Tomasz Chmielewski <***@wpkg.org> wrote:

It's easy to create a "LAN" for LXD containers on a single LXD server - just attach them to the same bridge, use the same subnet (i.e. 10.10.10.0/24) - done. Containers can communicate with each other using their private IP address.

However, with more then one LXD server *not* in the same LAN (i.e. two LXD servers in different datacentres), the things get tricky.


Is anyone using such setups, with multiple LXD servers and containers being able to communicate with each other?


LXD1: IP 1.2.3.4, Europe LXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.10.20
container2, 10.10.10.11 container5, 10.10.10.21
container3, 10.10.10.12 container6, 10.10.10.22


LXD3: IP 3.4.5.6, US
container7, 10.10.10.30
container8, 10.10.10.31
container8, 10.10.10.32


While I can imagine setting up many OpenVPN tunnels between all LXD servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the routes as containers are stopped/started/migrated, it's a bit of a management nightmare. And even more so if the number of LXD servers grows.

Hints, discussion?


Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
lxc-***@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
Tomasz Chmielewski
2016-09-18 13:41:45 UTC
Permalink
Post by Ron Kelley
(Long reply follows…)
Personally, I think you need to look at the big picture for such
deployments. From what I read below, you are asking, “how do I extend
my layer-2 subnets between data centers such that container1 in Europe
can talk with container6 in Asia, etc”. If this is true, I think you
need to look at deploying data center hardware (servers with multiple
NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC,
etc). And, you must look at how your failover services will work in
this design. It’s easy to get a couple of servers working with a
simple design, but those simple designs tend to go to production very
fast without proper testing and design.
Well, it's not only about deploying on "different continents".

It can be also in the same datacentre, where the hosting doesn't give
you a LAN option.

For example - Amazon AWS, same region, same availability zone.

The servers will have "private" addresses like 10.x.x.x, traffic there
will be private to your servers, but there will be no LAN. You can't
assign your own LAN addresses (10.x.x.x).

This means, while you can launch several LXD containers on every of
these servers - but their "LAN" will be limited per each LXD server
(unless we do some special tricks).

Some other hostings offer a public IP, or several public IPs per
servers, in the same datacentre, but again, no LAN.


Tomasz Chmielewski
https://lxadm.com
Ron Kelley
2016-09-18 15:41:25 UTC
Permalink
So, just for clarity, you are saying each LXD server will have no separate network connection for the containers. Thus, all containers are private to the LXD server, and any outbound traffic must traverse the container server interface. Is this correct? If so, sorry, I must have missed this requirement in your initial email.
Post by Ron Kelley
(Long reply follows…)
Personally, I think you need to look at the big picture for such
deployments. From what I read below, you are asking, “how do I extend
my layer-2 subnets between data centers such that container1 in Europe
can talk with container6 in Asia, etc”. If this is true, I think you
need to look at deploying data center hardware (servers with multiple
NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC,
etc). And, you must look at how your failover services will work in
this design. It’s easy to get a couple of servers working with a
simple design, but those simple designs tend to go to production very
fast without proper testing and design.
Well, it's not only about deploying on "different continents".

It can be also in the same datacentre, where the hosting doesn't give you a LAN option.

For example - Amazon AWS, same region, same availability zone.

The servers will have "private" addresses like 10.x.x.x, traffic there will be private to your servers, but there will be no LAN. You can't assign your own LAN addresses (10.x.x.x).

This means, while you can launch several LXD containers on every of these servers - but their "LAN" will be limited per each LXD server (unless we do some special tricks).

Some other hostings offer a public IP, or several public IPs per servers, in the same datacentre, but again, no LAN.


Tomasz Chmielewski
https://lxadm.com
Tilak Waelde
2016-09-18 20:12:06 UTC
Permalink
Post by Ron Kelley
Hope this helps. Happy to share my LXD configurations with anyone...
-Ron
Please do! I'd really love to see a description of a production lxd /
lxc setup with proper networking and multiple hosts!

I haven't played around with it yet, but is it possible to include some
sort of VRF-lite[0] into such a setup for multi tenancy purposes? Other
than by using VLANs one could use the same IP ranges multiple times from
what I've come to understand?
I'm not sure how a user could put the containers interfaces into a
different network namespace..

cheers,
Tilak

[0] : https://www.kernel.org/doc/Documentation/networking/vrf.txt
Tomasz Chmielewski
2016-09-19 17:46:04 UTC
Permalink
Post by Tilak Waelde
Post by Ron Kelley
Hope this helps. Happy to share my LXD configurations with anyone...
Please do! I'd really love to see a description of a production lxd /
lxc setup with proper networking and multiple hosts!
I haven't played around with it yet, but is it possible to include
some sort of VRF-lite[0] into such a setup for multi tenancy purposes?
Other than by using VLANs one could use the same IP ranges multiple
times from what I've come to understand?
I'm not sure how a user could put the containers interfaces into a
different network namespace..
Hi,

after some experimenting with VXLAN, I've summed up a working "LAN for
multiple LXC servers" here:

https://lxadm.com/Unicast_VXLAN:_overlay_network_for_multiple_servers_with_dozens_of_containers


It is using in kernel VXLAN, and thus performs very well (almost wire
speed, and much much better than any userspace programs).

On the other hand, it provides no encryption between LXD servers (or, in
fact, any other virtualization), so may depend on your exact
requirements.


Tomasz Chmielewski
brian mullan
2016-09-19 13:48:35 UTC
Permalink
Tomasz

After experimenting with TINC, OpenVPN & others I found and have used
PeerVPN to do this:

https://peervpn.net/

PeerVPN is a full-mesh VPN and is auto-learning..

- Ethernet tunneling support using TAP devices.
- IPv6 support.
- Full mesh network topology.
- Automatically builds tunnels through firewalls and NATs without any
further setup (for example, port forwarding).
- Shared key encryption and authentication support.

At the time *I found PeerVPN much simpler to configure that TINC !*

A PeerVPN mesh is extremely easy to configure as you only have to install
PeerVPN on each host server and edit its config file on that server.

The brief tutorial page (https://peervpn.net/tutorial/) shows that config
for a basic PeerVPN only requires about 7 lines on each of your servers.

In Apr 2015 I documented a proof-of-concept test setup
<https://bmullan.wordpress.com/2015/05/12/proof-of-concept-using-mesh-vpn-to-interconnect-lxc-containers-on-multiple-hosts-on-multiple-clouds/>
I'd done to interconnect LXC containers on AWS, DIgital Ocean and here
locally (so much like you want to do I think).

*NOTE: * that documentation *was before* my use of LXD !

*The reason I mention this is that with the advent of LXD the entire
container IP addressing setup using PeerVPN becomes even easier ! * This
is because when I did those tests I never took the time to create a single
DHCP source for all lxc containers on all hosts. In my 2015
proof-of-concept test of PeerVPN I just a quick & dirty config using the
assumption that no 2 containers (for my test purposes) would get assigned
the same 10.0.3.x IP address by their local Server/Host.
Today, LXD *during installation* allows you much more control over each
Server/Host's LXDBR0 bridge IP and the 10.x.x.x IP address range that is
used for that Host's lxc containers.

*So if you were to use PeerVPN, during LXD installation on each server
just reserve a portion of the 10.x.x.x IP address range for each
Server/Host to avoid any possibility of any 2 containers on any 2 Hosts
getting assigned the same IP.*

The last config step on each Server/Host is to connect the PeerVPN mesh
Tunnel End Point (TEP to the local LXDBR0 bridge to enable all the
containers attached to that bridge to pass data over the VPN tunnel to the
other servers since my documentation set up a Layer 2 VPN (note PeerVPN
supports a Layer 3 tunnel as well I believe).

Since the PeerVPN TEP "interface" ... called "peervpn0" in my tutorial
example is just like any other Linux ethernet interface you only have to
use the "ip link" command to connect "peervpn0" to the LXD "lxdbr0" bridge.

*$ sudo ip link set dev peervpn0 master lxdbr0*

After you've done that on all 3 hosts your containers on all the hosts
should be able to reach each other since they would all be on what
logically is the same "ethernet" via the VPN & the fact that you setup all
of the LXD installs with the same same 10.x.x.x network.

Also, the containers on each Server/Host still get external Internet access
(if you permit it) via their local Host.

Anyway, I thought I'd send this info in case it is helpful as this should
work with servers locally or remotely using LXD.

Brian
Sergiusz Pawlowicz
2016-09-19 14:20:08 UTC
Permalink
Automatically builds tunnels through firewalls and NATs without any further
setup (for example, port forwarding).
I would not appreciate something which "automatically" (whatever it
means) traverse my firewalls, to be honest. We should treat our data
seriously, Brian.

Serge
brian mullan
2016-09-20 13:10:57 UTC
Permalink
Serge,
*> Automatically builds tunnels through firewalls and NATs without any
further> setup (for example, port forwarding).I would not appreciate
something which "automatically" (whatever itmeans) traverse my firewalls,
to be honest. We should treat our dataseriously, Brian.*
First, a sysadmin person has to install/setup/configure PeerVPN on each
server so I guess like installing/configuring TINC or any other VPN
solution there is some assumption of some sort of "trust" in that person's
work.

Second, in PeerVPN's configuration file
<https://github.com/peervpn/peervpn/blob/master/peervpn.conf> on each
server you (re the sysadmin) have to
specify 2 security related items:

PORT xxxxx # the Port to be opened/used by PeerVPN

But you point out a good question -
regarding that bullet by the author on the PeerVPN web page. Tobias Volk
may be referring to something else as *it CLEARLY states in the short
PeerVPN tutorial <https://peervpn.net/tutorial/> you MUST port-forward the
"port" configured for PeerVPN to use if Nodes are behind a NAT. *

*I know PeerVPN doesn't work if you have not done that from my own use.:*


*Configuration of node A*
port 7000
networkname ExampleNet
psk mysecretpassword
enabletunneling yes
interface peervpn0
ifconfig4 10.8.0.1/24
This will open UDP port 7000 and create a virtual ethernet interface with
the name peervpn0 and the IP address 10.8.0.1.
Please note that Node A needs to be directly reachable from Node B.
*If Node A is behind a NAT device, you will have to forward port 7000. *
*Configuration of node B*
port 7000
networkname ExampleNet
psk mysecretpassword
enabletunneling yes
interface peervpn0
ifconfig4 10.8.0.2/24
initpeers node-a.example.com 7000
Replace node-a.example.com with the real address of Node A.

Further there is the shared PSK crypto key generation that also limits
connections to "peers" sharing the "same" PSK "seed" in the configuration
file.

In a PeerVPN mesh different server/hosts can have multiple PSK "seed"
configured to allow any 1 host to "peer" with different specific systems in
the "mesh" who have a matching PSK "seed" configured.
I can email Tobias and ask for clarification as to what "bullet" means.

Brian
brian mullan
2016-09-20 21:47:29 UTC
Permalink
I got a response back from Tobias Volk (the PeerVPN author)

He changed the bullet on the main web page to help clarify that there is no
automatic tunnel through firewalls..

text was...
*Automatically builds tunnels through firewalls and NATs without any
further setup (for example, port forwarding).*

text changed to...
*No NAT reconfiguration necessary, only one node with a public address is
required*
Post by brian mullan
Serge,
*> Automatically builds tunnels through firewalls and NATs without any
further> setup (for example, port forwarding).I would not appreciate
something which "automatically" (whatever itmeans) traverse my firewalls,
to be honest. We should treat our dataseriously, Brian.*
First, a sysadmin person has to install/setup/configure PeerVPN on each
server so I guess like installing/configuring TINC or any other VPN
solution there is some assumption of some sort of "trust" in that person's
work.
Second, in PeerVPN's configuration file
<https://github.com/peervpn/peervpn/blob/master/peervpn.conf> on each
server you (re the sysadmin) have to
PORT xxxxx # the Port to be opened/used by PeerVPN
But you point out a good question -
regarding that bullet by the author on the PeerVPN web page. Tobias Volk
may be referring to something else as *it CLEARLY states in the short
PeerVPN tutorial <https://peervpn.net/tutorial/> you MUST port-forward the
"port" configured for PeerVPN to use if Nodes are behind a NAT. *
*I know PeerVPN doesn't work if you have not done that from my own use.:*
*Configuration of node A*
port 7000
networkname ExampleNet
psk mysecretpassword
enabletunneling yes
interface peervpn0
ifconfig4 10.8.0.1/24
This will open UDP port 7000 and create a virtual ethernet interface with
the name peervpn0 and the IP address 10.8.0.1.
Please note that Node A needs to be directly reachable from Node B.
*If Node A is behind a NAT device, you will have to forward port 7000. *
*Configuration of node B*
port 7000
networkname ExampleNet
psk mysecretpassword
enabletunneling yes
interface peervpn0
ifconfig4 10.8.0.2/24
initpeers node-a.example.com 7000
Replace node-a.example.com with the real address of Node A.
Further there is the shared PSK crypto key generation that also limits
connections to "peers" sharing the "same" PSK "seed" in the configuration
file.
In a PeerVPN mesh different server/hosts can have multiple PSK "seed"
configured to allow any 1 host to "peer" with different specific systems in
the "mesh" who have a matching PSK "seed" configured.
I can email Tobias and ask for clarification as to what "bullet" means.
Brian
Raymond Ferguson
2016-09-20 13:40:45 UTC
Permalink
Maybe a VPN container that peers with the other VPN containers and runs a
dynamic routing protocol? Then just setup a route on the lxd host to the
VPN guest. Make it config driven so you just push a new peer list to
update. Or not and just push the updated container?
Post by Tomasz Chmielewski
It's easy to create a "LAN" for LXD containers on a single LXD server -
just attach them to the same bridge, use the same subnet (i.e.
10.10.10.0/24) - done. Containers can communicate with each other using
their private IP address.
However, with more then one LXD server *not* in the same LAN (i.e. two LXD
servers in different datacentres), the things get tricky.
Is anyone using such setups, with multiple LXD servers and containers
being able to communicate with each other?
LXD1: IP 1.2.3.4, Europe LXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.10.20
container2, 10.10.10.11 container5, 10.10.10.21
container3, 10.10.10.12 container6, 10.10.10.22
LXD3: IP 3.4.5.6, US
container7, 10.10.10.30
container8, 10.10.10.31
container8, 10.10.10.32
While I can imagine setting up many OpenVPN tunnels between all LXD
servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the
routes as containers are stopped/started/migrated, it's a bit of a
management nightmare. And even more so if the number of LXD servers grows.
Hints, discussion?
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Ruzsinszky Attila
2016-09-21 20:30:33 UTC
Permalink
Hi,

Why don't you make a test with OpenVSwitch?
You can setup an SDN with it.

TIA,
Ruzsi

Continue reading on narkive:
Loading...