[lxc-users] Networking issues with LXC containers in EC2

[Peter Steele]

I first brought this issue up several weeks ago and have just got back
to the work where I originally ran into this problem. The scenario is
simple enough:

- Create two EC2 instances running CentOS 7.1
- Configure these instances to used bridged networking
- Create a LXC container running under each instance, using the command

lxc-create -t download -n test1 --dir=/hf/test1/rootfs -- -d centos
-r 7 -a amd64

Each container ends up with a config that looks something like this:

lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.rootfs = /hf/test1/rootfs
lxc.utsname = test1
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

- Assign each EC2 instance and container a static IP address in the same
subnet. In my test I assigned the two EC2 instances 10.0.0.102 and
10.0.0.108 and the two containers 10.0.0.103 and 10.0.0.109.

In this scenario, each EC2 host can ping their own LXC container, and
they can ping each other. Likewise, the containers can ping their host.
However, Instance 102 cannot ping the container hosted on instance 108,
and similarly instance 108 cannot ping the container hosted on instance
102. If I configure this exact same scenario on real hardware or on KVM
based virtual machines, this "crosstalk" problem does not
occur--instance 102 can ping container 109 for example, even though it
is hosted on a difference instance.

From what I've read, I understand that Amazon has implemented some
special/restricted behavior for the networking stack of EC2 instances.
The question I have is whether I can accomplish what I've attempted
here, specifically, can I access a LXC container hosted on one EC2
instance directly from another EC2 instance or from another LXC
container hosted on another EC2 instance?


Peter

[Fajar A. Nugraha]

Post by Peter Steele
From what I've read, I understand that Amazon has implemented some
special/restricted behavior for the networking stack of EC2 instances. The
question I have is whether I can accomplish what I've attempted here,
specifically, can I access a LXC container hosted on one EC2 instance
directly from another EC2 instance or from another LXC container hosted on
another EC2 instance?

You might want to ask them first. Looks like it's only available for
VPC setup: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

If they do allow multiple IP address, then the next step is to check
whether they allow multiple MACs (which is what you get when you use
bridge). There's a workaround for this if the ONLY limitation is the
MAC, using proxyarp.

[Peter Steele]

I should have added that I have no issue running our software on a
single EC2 instance with containers running on that instance. We can
assign multiple IPs to the instance itself, as well as to the containers
running under the instance, and the containers can all communicate with
each other as well as with the host. The problem occurs when we have
more than one EC2 instance and need to have the containers in separate
instances to communicate with each other. You're right though: If no one
on this list has actually dealt with this issue themselves, the quickest
answer is probably to talk to AWS directly.

Thanks.

Peter

Post by Fajar A. Nugraha

Post by Peter Steele
From what I've read, I understand that Amazon has implemented some
special/restricted behavior for the networking stack of EC2 instances. The
question I have is whether I can accomplish what I've attempted here,
specifically, can I access a LXC container hosted on one EC2 instance
directly from another EC2 instance or from another LXC container hosted on
another EC2 instance?

You might want to ask them first. Looks like it's only available for
VPC setup: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
If they do allow multiple IP address, then the next step is to check
whether they allow multiple MACs (which is what you get when you use
bridge). There's a workaround for this if the ONLY limitation is the
MAC, using proxyarp.

[Fajar A. Nugraha]

I should have added that I have no issue running our software on a single
EC2 instance with containers running on that instance. We can assign
multiple IPs to the instance itself, as well as to the containers running
under the instance, and the containers can all communicate with each other
as well as with the host.

can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?

if "no", then you might hit the multiple mac problem

[Peter Steele]

Post by Fajar A. Nugraha

I should have added that I have no issue running our software on a single
EC2 instance with containers running on that instance. We can assign
multiple IPs to the instance itself, as well as to the containers running
under the instance, and the containers can all communicate with each other
as well as with the host.

can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?
if "no", then you might hit the multiple mac problem

Sadly the answer is no. They cannot even ping another host in the same
VPC...

Peter

[Fajar A. Nugraha]

Post by Peter Steele

Post by Fajar A. Nugraha

I should have added that I have no issue running our software on a single
EC2 instance with containers running on that instance. We can assign
multiple IPs to the instance itself, as well as to the containers running
under the instance, and the containers can all communicate with each other
as well as with the host.

can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?
if "no", then you might hit the multiple mac problem

Sadly the answer is no. They cannot even ping another host in the same
VPC...

Looks like multiple mac problem. As in, EC2 only allows one mac from
your interface.

Proxyarp should work:

(1) Make SURE your EC2 instances (I'd call them "host" from now on)
supports multiple IPs (private or elastic/public IPs, depending on
your needs). The easiest way is to add those IPs to your host
interface, make sure that that new IP can be accessed (e.g. ping that
IP from another host), and then remove it.

(2) Enable proxy arp on the host

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

of course, adjust to your environment (e.g. change interface name if
needed). You can also add entries in /etc/sysctl.conf or /etc/sysctl.d
so that this setting will persist on reboot.

(3) See https://www.mail-archive.com/lxc-***@lists.linuxcontainers.org/msg02380.html

This should make all outgoing packets use eth0's MAC, and the host
will effectively function as a router.

[Fajar A. Nugraha]

Post by Fajar A. Nugraha

Post by Peter Steele

Post by Fajar A. Nugraha

I should have added that I have no issue running our software on a single
EC2 instance with containers running on that instance. We can assign
multiple IPs to the instance itself, as well as to the containers running
under the instance, and the containers can all communicate with each other
as well as with the host.

can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?
if "no", then you might hit the multiple mac problem

Sadly the answer is no. They cannot even ping another host in the same
VPC...

Looks like multiple mac problem. As in, EC2 only allows one mac from
your interface.

Actually my reply on your past thread should be simpler:
https://lists.linuxcontainers.org/pipermail/lxc-users/2015-September/010069.html

[Peter Steele]

Post by Fajar A. Nugraha

Post by Peter Steele

Post by Fajar A. Nugraha

I should have added that I have no issue running our software on a single
EC2 instance with containers running on that instance. We can assign
multiple IPs to the instance itself, as well as to the containers running
under the instance, and the containers can all communicate with each other
as well as with the host.

can the containers in that setup communicate with systems outside the
host (e.g. access the internet)?
if "no", then you might hit the multiple mac problem

Sadly the answer is no. They cannot even ping another host in the same
VPC...

Looks like multiple mac problem. As in, EC2 only allows one mac from
your interface.
(1) Make SURE your EC2 instances (I'd call them "host" from now on)
supports multiple IPs (private or elastic/public IPs, depending on
your needs). The easiest way is to add those IPs to your host
interface, make sure that that new IP can be accessed (e.g. ping that
IP from another host), and then remove it.
(2) Enable proxy arp on the host
echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

It turned out that proxy arp was indeed the solution here, but a few
other parameters had to be set as well. I just need to run the following
commands on each EC2 instance:

echo 1 > /proc/sys/net/ipv4/conf/br0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp_pvlan
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/br0/send_redirects

With these settings, my containers and hosts can all talk to each other
just like they were all residing on the same subnet. An easy solution in
the end.

Peter