[lxc-users] /etc/resolv.conf occasionally does not get written in LXC container with static conf

[Marat Khalili]

Occasionally after reboot of the host the /etc/resolv.conf file in some
container comes up containing only two comment lines (taken from
/etc/resolvconf/resolv.conf.d/head). It should be filled in accordance
with dns-nameservers line in /etc/network/interfaces (network
configuration is all static), but it doesn't, though IP address is
assigned correctly. Any of the following fixes the problem:
* /etc/init.d/resolvconf reload
* ifdown/ifup
* lxc-stop/lxc-start
* lxc-attach shutdown -r now

What's bizzare is there're many containers with similar configuration on
this host (actually, most created with the same script), but they
usually come up ok, and there's nothing particularly different in
successful and unsuccessful syslogs until some service inside (e.g.
Apache) realizes it's got no DNS.

Since it is hard to reproduce the problem without rebooting production
server, I don't even know where to dig. Probably someone has seen this
kind of behaviour before?

It's Ubuntu 16.04 on both host and container:
Linux host 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux

--

With Best Regards,
Marat Khalili

[David Favor]

Post by Marat Khalili
Occasionally after reboot of the host the /etc/resolv.conf file in some
container comes up containing only two comment lines (taken from
/etc/resolvconf/resolv.conf.d/head). It should be filled in accordance
with dns-nameservers line in /etc/network/interfaces (network
configuration is all static), but it doesn't, though IP address is
* /etc/init.d/resolvconf reload
* ifdown/ifup
* lxc-stop/lxc-start
* lxc-attach shutdown -r now
What's bizzare is there're many containers with similar configuration on
this host (actually, most created with the same script), but they
usually come up ok, and there's nothing particularly different in
successful and unsuccessful syslogs until some service inside (e.g.
Apache) realizes it's got no DNS.
Since it is hard to reproduce the problem without rebooting production
server, I don't even know where to dig. Probably someone has seen this
kind of behaviour before?
Linux host 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux

There seems to be some ugliness with regeneration of /etc/resolv.conf at boot time.

I've seen some glitches around this process also.

In my case, I run a local copy of dnsmasq + completely disable all the
systemd-resolved lunacy, so here's how I deal with this...

Which might be more brute force than what your after + works well.

1) Completely destroy systemd-resolved

systemctl stop systemd-resolved
systemctl disable systemd-resolved
apt-get purge libnss-resolve

2) Create /etc/dnsmasq-caching.conf

port=53
listen-address=127.0.0.1
listen-address=127.0.0.53

interface=lo
bind-interfaces

no-resolv
no-hosts

# Google
server=8.8.8.8
server=8.8.4.4

3) Add to /etc/rc.local

echo "starting" | logger --tag "rc-local: dnsmasq"
dnsmasq -C /etc/dnsmasq-caching.conf

4) Create /etc/resolv.conf.override

nameserver 127.0.0.1

5) apt-get install incron

6) echo root > /etc/incron.allow

7) incrontab -e

/run/resolvconf/resolv.conf IN_CLOSE_WRITE,IN_NO_LOOP /bin/cp -f /etc/resolv.conf.override /run/resolvconf/resolv.conf
/etc/resolv.conf.override IN_CLOSE_WRITE,IN_NO_LOOP /bin/cp -f /etc/resolv.conf.override /run/resolvconf/resolv.conf

At this point the highly broken systemd-resolved is completely off.

Anytime /run/resolvconf/resolv.conf (resolvconf -u for example) or /etc/resolv.conf.override changes
/run/resolvconf/resolv.conf is overwritten using your /etc/resolv.conf.override file, so you always
know exactly what's in your resolv.conf file.

Like I said, might be overkill + ensures you know exactly how your resolution is occurring.

[Marat Khalili]

Thank you for sharing, I was suspecting systemd too. It is very sad that such drastic measures as incrontab are needed :(

[David Favor]

Post by Marat Khalili
Thank you for sharing, I was suspecting systemd too. It is very sad that
such drastic measures as incrontab are needed :(

Yes.

Most of the Ubuntu bugs I've opened recently have to do with systemd oddities.

Very ugly.