Discussion:
Macvlan setup issues
(too old to reply)
Mark Haney
2016-04-28 17:02:37 UTC
Permalink
Hi all. I'm really new to LXC/LXD as well as with macvlan setup, so bear
with me. I'm having all kinds of trouble getting the macvlan setup to work
on my Ubuntu 16.04 VM. The documentation isn't terribly clear, or at least
it seems to be missing something that is keeping my containers from being
accessible inside my office network directly. I'm hoping it's something
simple, but as I've run out of good Google searches, I'm not sure what else
to try. Here's my problem (and setup):

I've been able to get containers of various distros running file with
bridging, but I'd like to be able to access them as part of my LAN
directly.


The IP of the 16.04 host: 10.42.204.50 (gw 10.42.204.1)

Now, I went through the docs to setup macvlans as this appeared to be the
simplest (best?) method of setting up the containers to access my LAN. So,
I ran dpkg-reconfigure lxd and removed the bridging. Then set the macvlan
according to the docs
<https://www.stgraber.org/2016/03/19/lxd-2-0-your-first-lxd-container-312/>
:

lxc profile device set default eth0 nictype macvlan
lxc profile device set default eth0 parent ens160

The second command might not be correct, as the documentation doesn't
specify precisely what the 'parent' is, though I believe it's the host
interface name (in my case ens160). Regardless, this seems to have worked,
at least partly because, when I launched an image of Ubuntu 15.10, the eth0
interface was correctly given a DHCP address from my LAN (10.42.204.153)
and a default gateway (10.42.204.1).

However, I cannot ping anything on my network from the container with a
'Destination Host Unreachable' message. IP forwarding is enabled, but I
don't think I need that for this setup. The problem is, none of the docs
say I need to do anything else to get macvlans working, which is why this
is so frustrating. Is this just a case of I'm missing something the docs
assume I have knowledge of? Or an actual problem.

I'm no Linux slouch, I've been using them since the RH3 days, but macvlans
are new to me, so is LXD.

Any ideas?
--
Mark Haney ::: Senior Systems Engineer
*VIF* *International Education*
P.O. Box 3566 ::: Chapel Hill, N.C. 27515 ::: USA
919-265-5006 office

Global learning for all.
www.viflearn.com
Find VIF on Facebook <http://facebook.com/VIFInternationalEducation> |
Twitter <https://twitter.com/vifglobaled> | LinkedIn
<http://www.linkedin.com/company/vif-international-education>

Recognized as a ‘Best for the World’
<http://bestfortheworld.bcorporation.net/> B Corp!
Fajar A. Nugraha
2016-04-29 06:22:09 UTC
Permalink
Post by Mark Haney
Hi all. I'm really new to LXC/LXD as well as with macvlan setup, so bear
with me. I'm having all kinds of trouble getting the macvlan setup to work
on my Ubuntu 16.04 VM. The documentation isn't terribly clear, or at least
it seems to be missing something that is keeping my containers from being
accessible inside my office network directly. I'm hoping it's something
simple, but as I've run out of good Google searches, I'm not sure what else
I've been able to get containers of various distros running file with
bridging, but I'd like to be able to access them as part of my LAN directly.
The IP of the 16.04 host: 10.42.204.50 (gw 10.42.204.1)
Now, I went through the docs to setup macvlans as this appeared to be the
simplest (best?) method of setting up the containers to access my LAN. So,
I ran dpkg-reconfigure lxd and removed the bridging. Then set the macvlan
lxc profile device set default eth0 nictype macvlan
lxc profile device set default eth0 parent ens160
The second command might not be correct, as the documentation doesn't
specify precisely what the 'parent' is, though I believe it's the host
interface name (in my case ens160). Regardless, this seems to have worked,
at least partly because, when I launched an image of Ubuntu 15.10, the eth0
interface was correctly given a DHCP address from my LAN (10.42.204.153) and
a default gateway (10.42.204.1).
However, I cannot ping anything on my network from the container with a
'Destination Host Unreachable' message. IP forwarding is enabled, but I
don't think I need that for this setup. The problem is, none of the docs
say I need to do anything else to get macvlans working, which is why this is
so frustrating. Is this just a case of I'm missing something the docs
assume I have knowledge of? Or an actual problem.
I'm no Linux slouch, I've been using them since the RH3 days, but macvlans
are new to me, so is LXD.
The usual questions:
- do you get the SAME result when you use a MANUALLY created bridge
with eth0 as one of the slaves?
- does your network (e.g. switch) allows multiple macs on a single port?

The usual culprit (e.g. with EC2) is that the network only allows one mac.
--
Fajar
Mark Haney
2016-04-29 12:39:58 UTC
Permalink
Well, as this is a VM running on an ESXi 6 cluster, I'm having a hard time
believing that would be the case. However, just to make certain, I've
created the bridge manually and still have the same problem. If our
switches (Cisco 3750s) had issues with running multiple MACs on the IBM
Blade Server NICs, I'm pretty sure we'd have seen that long ago.

Any other ideas? Or should I just scrap this entire setup since it appears
that only LXD experts can manage to get this particular setup working
properly?
Post by Mark Haney
Post by Mark Haney
Hi all. I'm really new to LXC/LXD as well as with macvlan setup, so bear
with me. I'm having all kinds of trouble getting the macvlan setup to
work
Post by Mark Haney
on my Ubuntu 16.04 VM. The documentation isn't terribly clear, or at
least
Post by Mark Haney
it seems to be missing something that is keeping my containers from being
accessible inside my office network directly. I'm hoping it's something
simple, but as I've run out of good Google searches, I'm not sure what
else
Post by Mark Haney
I've been able to get containers of various distros running file with
bridging, but I'd like to be able to access them as part of my LAN
directly.
Post by Mark Haney
The IP of the 16.04 host: 10.42.204.50 (gw 10.42.204.1)
Now, I went through the docs to setup macvlans as this appeared to be the
simplest (best?) method of setting up the containers to access my LAN.
So,
Post by Mark Haney
I ran dpkg-reconfigure lxd and removed the bridging. Then set the
macvlan
Post by Mark Haney
lxc profile device set default eth0 nictype macvlan
lxc profile device set default eth0 parent ens160
The second command might not be correct, as the documentation doesn't
specify precisely what the 'parent' is, though I believe it's the host
interface name (in my case ens160). Regardless, this seems to have
worked,
Post by Mark Haney
at least partly because, when I launched an image of Ubuntu 15.10, the
eth0
Post by Mark Haney
interface was correctly given a DHCP address from my LAN (10.42.204.153)
and
Post by Mark Haney
a default gateway (10.42.204.1).
However, I cannot ping anything on my network from the container with a
'Destination Host Unreachable' message. IP forwarding is enabled, but I
don't think I need that for this setup. The problem is, none of the docs
say I need to do anything else to get macvlans working, which is why
this is
Post by Mark Haney
so frustrating. Is this just a case of I'm missing something the docs
assume I have knowledge of? Or an actual problem.
I'm no Linux slouch, I've been using them since the RH3 days, but
macvlans
Post by Mark Haney
are new to me, so is LXD.
- do you get the SAME result when you use a MANUALLY created bridge
with eth0 as one of the slaves?
- does your network (e.g. switch) allows multiple macs on a single port?
The usual culprit (e.g. with EC2) is that the network only allows one mac.
--
Fajar
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
--
Mark Haney ::: Senior Systems Engineer
*VIF* *International Education*
P.O. Box 3566 ::: Chapel Hill, N.C. 27515 ::: USA
919-265-5006 office

Global learning for all.
www.viflearn.com
Find VIF on Facebook <http://facebook.com/VIFInternationalEducation> |
Twitter <https://twitter.com/vifglobaled> | LinkedIn
<http://www.linkedin.com/company/vif-international-education>

Recognized as a ‘Best for the World’
<http://bestfortheworld.bcorporation.net/> B Corp!
Fajar A. Nugraha
2016-04-29 18:39:20 UTC
Permalink
Post by Mark Haney
Well, as this is a VM running on an ESXi 6 cluster, I'm having a hard time
believing that would be the case. However, just to make certain, I've
created the bridge manually and still have the same problem. If our
switches (Cisco 3750s) had issues with running multiple MACs on the IBM
Blade Server NICs, I'm pretty sure we'd have seen that long ago.
Your switch can handle multiple macs, but does vmware allows it as
well (this IS, on a vmware VM, right)? AFAIK some configurations only
allows one MAC per VM "port".
Post by Mark Haney
Any other ideas? Or should I just scrap this entire setup since it appears
that only LXD experts can manage to get this particular setup working
properly?
Start with a simpler configuration first. e.g. use a physical server
(or your laptop) that is connected to a switch.
--
Fajar
Steve Adams
2016-04-29 19:22:27 UTC
Permalink
Sent from my Windows Phone
________________________________
From: Steve Adams<mailto:***@outlook.com>
Sent: ‎4/‎29/‎2016 12:21 PM
To: Mark Haney<mailto:***@vifprogram.com>
Subject: RE:Macvlan setup issues

The simplest way that I have found to accomplish this is with a simple bridge. I am assuming that you are using Ubuntu and have an Ethernet connection on the host. The bridge can easily be setup in /etc/network/ interfaces (there is tons of documentation). The resulting br0, or whatever you chose to call it, would then be used as the bridge device in containers config. You can than set up in the containers /etc/network/interfaces to have IP assigned by the router DHCP or assign your own static IP in the same subnet.
Keep in mind that I have only done this with nested containers as I use a laptop primarily and wifi adds a level of complexity that I have not felt the need to deal with. But it did work perfectly in assigning IPs to the nested containers in the same subnet as the host container.

Sent from my Windows Phone

-----Original Message-----
From: "Mark Haney" <***@vifprogram.com>
Sent: ‎4/‎28/‎2016 10:02 AM
To: "lxc-***@lists.linuxcontainers.org" <lxc-***@lists.linuxcontainers.org>
Subject: [lxc-users] Macvlan setup issues

Hi all. I'm really new to LXC/LXD as well as with macvlan setup, so bear
with me. I'm having all kinds of trouble getting the macvlan setup to work
on my Ubuntu 16.04 VM. The documentation isn't terribly clear, or at least
it seems to be missing something that is keeping my containers from being
accessible inside my office network directly. I'm hoping it's something
simple, but as I've run out of good Google searches, I'm not sure what else
to try. Here's my problem (and setup):

I've been able to get containers of various distros running file with
bridging, but I'd like to be able to access them as part of my LAN
directly.


The IP of the 16.04 host: 10.42.204.50 (gw 10.42.204.1)

Now, I went through the docs to setup macvlans as this appeared to be the
simplest (best?) method of setting up the containers to access my LAN. So,
I ran dpkg-reconfigure lxd and removed the bridging. Then set the macvlan
according to the docs
<https://www.stgraber.org/2016/03/19/lxd-2-0-your-first-lxd-container-312/>
:

lxc profile device set default eth0 nictype macvlan
lxc profile device set default eth0 parent ens160

The second command might not be correct, as the documentation doesn't
specify precisely what the 'parent' is, though I believe it's the host
interface name (in my case ens160). Regardless, this seems to have worked,
at least partly because, when I launched an image of Ubuntu 15.10, the eth0
interface was correctly given a DHCP address from my LAN (10.42.204.153)
and a default gateway (10.42.204.1).

However, I cannot ping anything on my network from the container with a
'Destination Host Unreachable' message. IP forwarding is enabled, but I
don't think I need that for this setup. The problem is, none of the docs
say I need to do anything else to get macvlans working, which is why this
is so frustrating. Is this just a case of I'm missing something the docs
assume I have knowledge of? Or an actual problem.

I'm no Linux slouch, I've been using them since the RH3 days, but macvlans
are new to me, so is LXD.

Any ideas?


--

Mark Haney ::: Senior Systems Engineer
*VIF* *International Education*
P.O. Box 3566 ::: Chapel Hill, N.C. 27515 ::: USA
919-265-5006 office

Global learning for all.
www.viflearn.com
Find VIF on Facebook <http://facebook.com/VIFInternationalEducation> |
Twitter <https://twitter.com/vifglobaled> | LinkedIn
<http://www.linkedin.com/company/vif-international-education>

Recognized as a ‘Best for the World’
<http://bestfortheworld.bcorporation.net/> B Corp!
Dan Kegel
2016-04-29 21:45:19 UTC
Permalink
I'm playing with macvlan, too, which is a challenge as I am quite
ignorant about it, bridges, and lxc.

Novice gotchas so far:
- I have to run lxc under sudo (I think?), and after I do that,
.config/lxc is owned by root, and I can't even do lxc --help :-(
- I keep typing lxc when I mean lxd and vice versa, it'd be cool if
lxc and lxd detected that and said "Did you mean..."
- if you have btrfs, the right answer to lxd init's question about zfs
is "dir", maybe "btrfs" should be a synonym for "dir"
- if you don't care, the right answer to lxc init's question about
listening address is 0.0.0.0, that should be in the prompt, maybe
- After letting "lxd init" create a bridge, uninstalling doesn't get
rid of the bridge; use "ip link del lxdbr0"
(plus possibly "nmcli connection delete lxdbr0" if network manager has
gotten a whiff of it?)
- lxc init will abort if there are any cached images, have to do "lxc
image list" then "lxc image delete foo" (need 'lxc image delete all'?)
- there's no way to assign an alias when doing 'lxc launch", so it's
hard to write a demo script that cleans up after itself?
- lxc image list pretty-prints, not very scriptable

About macvlan:
I think the way to find the parent interface is to list 'em with
either ifconfig or "ip link".
Since "ip link" says I have two interfaces, lo and enp6s0, I gather
the commands to use macvlan are
lxc profile device set default eth0 nictype macvlan
lxc profile device set default eth0 parent enp6s0
Unfortunately, this fails for me with
error: Error calling 'lxd forkstart demo /var/lib/lxd/containers
/var/log/lxd/demo/lxc.conf': err='exit status 1'
sudo lxc info --show-log demo says
lxc 20160429143729.079 ERROR lxc_conf -
conf.c:instantiate_macvlan:2668 - failed to create macvlan interface
'mcFODJ4S' on 'eth0' : Invalid argument

And even trying lxd without macvlan now fails with that error message,
even after doing
sudo apt purge lxd lxd-client :-( :-(

sudo lxc profile device get default eth0 nictype still shows macvlan.
Where the heck is that stored? Or is it the default?

Here's the script I was using to try to demo this:

-- snip ---

#!/bin/sh
set -ex
# Very destructive demo of lxd networking. Read before running.

cleanup() {
IMAGES=`sudo lxc image list | grep UTC | awk '{print $3}'`
if test "$IMAGES"
then
sudo lxc image delete $IMAGES
fi
sudo lxc delete demo || true
sudo apt-get purge lxd lxd-client || true
sudo rm -rf $HOME/.config/lxc
}

cleanup

# OK, now install lxc and configure it
sudo apt-get install lxd
sudo lxd init \
--auto \
--network-address 0.0.0.0 \
--network-port 8443 \
--storage-backend dir \
--trust-password demosecret \
#

# Verify network works inside container
sudo lxc launch ubuntu: demo
sudo lxc exec demo ping -c 2 8.8.8.8
sudo lxc stop demo
sudo lxc delete demo

# Switch to macvlan
ip link
REAL_ETHERNET=enp6s0 # assume the non-lo interface ip link spits out is enp6s0
sudo lxc profile device set myvlan eth0 nictype macvlan
sudo lxc profile device set myvlan eth0 parent $REAL_ETHERNET

# Verify network works inside container with our special tweaks
sudo lxc launch -p default -p myvlan ubuntu: demo
sudo lxc exec demo ping -c 2 8.8.8.8
sudo lxc stop demo
sudo lxc delete demo

# Undo everything
cleanup

-- snip ---
Dan Kegel
2016-04-29 22:35:42 UTC
Permalink
I clearly need to take a step back and make sure my user's in the lxd
group and the "my first container" demo
https://www.stgraber.org/2016/03/19/lxd-2-0-your-first-lxd-container-312/
works. It doesn't here anymore. Having polluted two ubuntu 16.04
systems here,
I'm going to head back home where it was working last night.

Probably all that sudo was the result of me fat-fingering lxc vs lxd
once, and from then on,
~/.config/lxc was owned by root, and nothing would run without sudo anymore.
Continue reading on narkive:
Loading...