Discussion:
live migration using lxd 0.4
(too old to reply)
KATOH Yasufumi
2015-03-24 07:35:42 UTC
Permalink
Hi,

I try to live migration using LXD, but not working.

* OS: Ubuntu Trusty x 2
* source host: lxd01
* destination host: lxd02
* LXD: Install from daily PPA
$ dpkg-query --show lxd
lxd 0.4-0ubuntu1~ppa1~trusty

I ran as follows. Is there something wrong or something mistake?

1. Change /etc/init/lxd.conf on both hosts
exec /usr/bin/lxd --group lxd --tcp (address of each host):8443 --debug >> /var/log/lxd.log 2>&1
2. Get and import image on lxd01
lxd-images import lxc ubuntu trusty amd64 --alias ubuntu
3. set password on both hosts
lxc config set password hogehoge
4. add remotes on lxd01
lxc remote add lxd01 lxd01:8443
lxc remote add lxd02 lxd02:8443
5. edit profile on both hosts
lxc config profile edit default
name: default
config:
raw.lxc: |-
lxc.tty=0
lxc.console=none
lxc.cgroup.devices.deny=c 5:1 rwm
devices:
eth0:
nictype: bridged
parent: lxcbr0
type: nic
6. Launch container on lxd01
lxc launch ubuntu ubuntu01
lxc list
+----------+---------+-----------+------+
| NAME | STATE | IPV4 | IPV6 |
+----------+---------+-----------+------+
| ubuntu01 | RUNNING | 10.0.3.68 | |
+----------+---------+-----------+------+
7. move the container, then error
lxc move lxd01:ubuntu01 lxd02:ubuntu02
error: Get https://10.200.200.212:8443/1.0/operations/21334af8-9b9b-40f3-bd87-e53309cee7da/wait: EOF

lxd on destination host (lxd02) output the log:
https://gist.github.com/tenforward/3ebccf5c18cf8f2486ea

Thanks,
KATOH Yasufumi
Tycho Andersen
2015-03-24 14:34:36 UTC
Permalink
Hi,
Post by KATOH Yasufumi
Hi,
I try to live migration using LXD, but not working.
* OS: Ubuntu Trusty x 2
* source host: lxd01
* destination host: lxd02
* LXD: Install from daily PPA
$ dpkg-query --show lxd
lxd 0.4-0ubuntu1~ppa1~trusty
I ran as follows. Is there something wrong or something mistake?
1. Change /etc/init/lxd.conf on both hosts
exec /usr/bin/lxd --group lxd --tcp (address of each host):8443 --debug >> /var/log/lxd.log 2>&1
2. Get and import image on lxd01
lxd-images import lxc ubuntu trusty amd64 --alias ubuntu
3. set password on both hosts
lxc config set password hogehoge
4. add remotes on lxd01
lxc remote add lxd01 lxd01:8443
lxc remote add lxd02 lxd02:8443
5. edit profile on both hosts
lxc config profile edit default
name: default
raw.lxc: |-
lxc.tty=0
lxc.console=none
lxc.cgroup.devices.deny=c 5:1 rwm
nictype: bridged
parent: lxcbr0
type: nic
6. Launch container on lxd01
lxc launch ubuntu ubuntu01
lxc list
+----------+---------+-----------+------+
| NAME | STATE | IPV4 | IPV6 |
+----------+---------+-----------+------+
| ubuntu01 | RUNNING | 10.0.3.68 | |
+----------+---------+-----------+------+
7. move the container, then error
lxc move lxd01:ubuntu01 lxd02:ubuntu02
error: Get https://10.200.200.212:8443/1.0/operations/21334af8-9b9b-40f3-bd87-e53309cee7da/wait: EOF
https://gist.github.com/tenforward/3ebccf5c18cf8f2486ea
Sorry about that. This was a bug that I inadvertently introduce just
before 0.4 was released. It is fixed by
141734d852b9ffb34eca10366e96d57c93ff23e8, which went in shortly after
0.4 was tagged, and will be in 0.5 due to be released today.

You'll also find that you need some other stuff to do live migration
(not documented anywhere, unfortunately):

1. you can't use lxcfs (or any other fuse fs)
2. only non-uidmapped containers work (you can do this in lxd by just
setting the uidmap for lxd to be 0-65536, or by setting the uidmap
in the container's raw.lxc config)
3. you can't be using systemd on the host
4. probably some others that I've forgotten about at the moment :)

Thanks for trying the code though! I'll hopefully be sending out fixes
for some of this stuff in the coming weeks. Let me know if you have
any problems and I can try and give you workarounds.

Tycho
Post by KATOH Yasufumi
Thanks,
KATOH Yasufumi
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Bostjan Skufca
2015-03-24 15:36:53 UTC
Permalink
Post by Tycho Andersen
Hi,
You'll also find that you need some other stuff to do live migration
...
2. only non-uidmapped containers work (you can do this in lxd by just
setting the uidmap for lxd to be 0-65536, or by setting the uidmap
in the container's raw.lxc config)
What is the reason for this? Missing implementation or some technical
obstacle? Just curious.



b.
Tycho Andersen
2015-03-24 16:01:34 UTC
Permalink
Post by Bostjan Skufca
Post by Tycho Andersen
Hi,
You'll also find that you need some other stuff to do live migration
...
2. only non-uidmapped containers work (you can do this in lxd by just
setting the uidmap for lxd to be 0-65536, or by setting the uidmap
in the container's raw.lxc config)
What is the reason for this? Missing implementation or some technical
obstacle? Just curious.
I think just missing implementation at this point (both in kernel and
userland). The problem is that once you enter the user namespace, you
can't do a lot of stuff that the restore code needs to do. The CRIU
folks introduced usernsd a couple of months ago to deal with this
(i.e. a daemon that performs the privileged requests on behalf of the
process), but it is still very young and the last I played with it it
didn't quite work.

You can see some of their discussion: http://criu.org/UserNamespace

Tycho
Post by Bostjan Skufca
b.
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
KATOH Yasufumi
2015-03-26 07:52:17 UTC
Permalink
Post by Tycho Andersen
On Tue, 24 Mar 2015 08:34:36 -0600
in message "Re: [lxc-users] live migration using lxd 0.4"
Post by Tycho Andersen
Sorry about that. This was a bug that I inadvertently introduce just
before 0.4 was released. It is fixed by
141734d852b9ffb34eca10366e96d57c93ff23e8, which went in shortly after
0.4 was tagged, and will be in 0.5 due to be released today.
Thank you for your reply! I'm trying to do live migration using 0.5
now :-) But I have not succeeded yet. (^^;)

| 2015/03/26 16:41:59 operation %!s(func()
| shared.OperationResult=0x4c5d20) finished: { checkpoint failed}
Post by Tycho Andersen
You'll also find that you need some other stuff to do live migration
1. you can't use lxcfs (or any other fuse fs)
I stopped lxcfs.
Post by Tycho Andersen
2. only non-uidmapped containers work (you can do this in lxd by just
setting the uidmap for lxd to be 0-65536, or by setting the uidmap
in the container's raw.lxc config)
I have a question. What does non-uidmapped container means? Should I
configure "security.privileged true"? Would you please tell me the way
to configure it specifically?
Post by Tycho Andersen
3. you can't be using systemd on the host
I use busybox and ubuntu container. :)
Post by Tycho Andersen
4. probably some others that I've forgotten about at the moment :)
Thanks for trying the code though! I'll hopefully be sending out fixes
for some of this stuff in the coming weeks. Let me know if you have
any problems and I can try and give you workarounds.
Tycho
Thanks,
KATOH Yasufumi
Tycho Andersen
2015-03-26 17:39:01 UTC
Permalink
Hi KATOH,
Post by KATOH Yasufumi
Post by Tycho Andersen
On Tue, 24 Mar 2015 08:34:36 -0600
in message "Re: [lxc-users] live migration using lxd 0.4"
Post by Tycho Andersen
Sorry about that. This was a bug that I inadvertently introduce just
before 0.4 was released. It is fixed by
141734d852b9ffb34eca10366e96d57c93ff23e8, which went in shortly after
0.4 was tagged, and will be in 0.5 due to be released today.
Thank you for your reply! I'm trying to do live migration using 0.5
now :-) But I have not succeeded yet. (^^;)
I'm working on getting everything working again with privileged LXC
(several recent changes have broken things), once that happens the LXD
bits are in place to make it will be easy to do migration in LXD.
Post by KATOH Yasufumi
| 2015/03/26 16:41:59 operation %!s(func()
| shared.OperationResult=0x4c5d20) finished: { checkpoint failed}
Post by Tycho Andersen
You'll also find that you need some other stuff to do live migration
1. you can't use lxcfs (or any other fuse fs)
I stopped lxcfs.
Post by Tycho Andersen
2. only non-uidmapped containers work (you can do this in lxd by just
setting the uidmap for lxd to be 0-65536, or by setting the uidmap
in the container's raw.lxc config)
I have a question. What does non-uidmapped container means? Should I
configure "security.privileged true"? Would you please tell me the way
to configure it specifically?
I just tried to do this, and it appears to be broken. I sent a PR to
fix it,

https://github.com/lxc/lxd/pull/438

So you'll need that as well as an LXC built with my two recent c/r
patches to the lxc-devel list (which haven't been merged yet). If you
have all that, I think you can do something like

lxc profile create migratable
lxc profile edit migratable

# paste the content below; these are a bunch of fixes for current criu
# limitations.

name: migratable
config:
raw.lxc: |
lxc.console = none
lxc.cgroup.devices.deny = c 5:1 rwm
lxc.start.auto =
lxc.start.auto = proc:mixed sys:mixed
security.privileged: "true"
devices: {}

Then do:

# assuming lxd2 is your target remote
lxc profile copy migratable lxd2:

lxc init ubuntu migratee
lxc config profile apply migratee migratable

lxc start migratee

# assuming lxd2 is the http:// url for your default lxd
lxc move lxd:migratee lxd2:migratee

And you should see it try to migrate. Currently it will still fail because of
external bind mounts like binfmt and some others. I'm working on a patch for
this now, but you could disable the additional bind mounts if you're not using
them and it should work.

Tycho
Post by KATOH Yasufumi
Post by Tycho Andersen
3. you can't be using systemd on the host
I use busybox and ubuntu container. :)
Post by Tycho Andersen
4. probably some others that I've forgotten about at the moment :)
Thanks for trying the code though! I'll hopefully be sending out fixes
for some of this stuff in the coming weeks. Let me know if you have
any problems and I can try and give you workarounds.
Tycho
Thanks,
KATOH Yasufumi
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
KATOH Yasufumi
2015-03-27 09:36:22 UTC
Permalink
Hi,

Thanks for your reply and detailed description!! I will try it next week.
Post by Tycho Andersen
Post by KATOH Yasufumi
On Thu, 26 Mar 2015 11:39:01 -0600
in message "Re: [lxc-users] live migration using lxd 0.4"
Post by Tycho Andersen
Hi KATOH,
Post by KATOH Yasufumi
On Tue, 24 Mar 2015 08:34:36 -0600
in message "Re: [lxc-users] live migration using lxd 0.4"
Sorry about that. This was a bug that I inadvertently introduce just
before 0.4 was released. It is fixed by
141734d852b9ffb34eca10366e96d57c93ff23e8, which went in shortly after
0.4 was tagged, and will be in 0.5 due to be released today.
Thank you for your reply! I'm trying to do live migration using 0.5
now :-) But I have not succeeded yet. (^^;)
I'm working on getting everything working again with privileged LXC
(several recent changes have broken things), once that happens the LXD
bits are in place to make it will be easy to do migration in LXD.
Post by KATOH Yasufumi
| 2015/03/26 16:41:59 operation %!s(func()
| shared.OperationResult=0x4c5d20) finished: { checkpoint failed}
You'll also find that you need some other stuff to do live migration
1. you can't use lxcfs (or any other fuse fs)
I stopped lxcfs.
2. only non-uidmapped containers work (you can do this in lxd by just
setting the uidmap for lxd to be 0-65536, or by setting the uidmap
in the container's raw.lxc config)
I have a question. What does non-uidmapped container means? Should I
configure "security.privileged true"? Would you please tell me the way
to configure it specifically?
I just tried to do this, and it appears to be broken. I sent a PR to
fix it,
https://github.com/lxc/lxd/pull/438
So you'll need that as well as an LXC built with my two recent c/r
patches to the lxc-devel list (which haven't been merged yet). If you
have all that, I think you can do something like
lxc profile create migratable
lxc profile edit migratable
# paste the content below; these are a bunch of fixes for current criu
# limitations.
name: migratable
raw.lxc: |
lxc.console = none
lxc.cgroup.devices.deny = c 5:1 rwm
lxc.start.auto =
lxc.start.auto = proc:mixed sys:mixed
security.privileged: "true"
devices: {}
# assuming lxd2 is your target remote
lxc init ubuntu migratee
lxc config profile apply migratee migratable
lxc start migratee
# assuming lxd2 is the http:// url for your default lxd
lxc move lxd:migratee lxd2:migratee
And you should see it try to migrate. Currently it will still fail because of
external bind mounts like binfmt and some others. I'm working on a patch for
this now, but you could disable the additional bind mounts if you're not using
them and it should work.
Tycho
KATOH Yasufumi
2015-04-08 12:06:48 UTC
Permalink
Hi,
Post by KATOH Yasufumi
On Fri, 27 Mar 2015 18:36:22 +0900
in message "Re: [lxc-users] live migration using lxd 0.4"
Post by KATOH Yasufumi
Thanks for your reply and detailed description!! I will try it next week.
I tried live migration using lxd 0.6, and work fine!! :-D

* OS: Ubuntu Trusty x 2
* source host: lxd01
* destination host: lxd02
* LXD: Install from lxd-git-master PPA
$ dpkg-query --show lxd
lxd 0.6-0ubuntu1~ubuntu14.04.1~ppa1

1. remove lxcfs
apt-get remove lxcfs
2. set password (both hosts)
lxc config set password hogehoge
3. get image on lxd01
lxd-images import lxc ubuntu trusty amd64 --alias ubuntu
4. add remotes on lxd01
lxc remote add lxd01 lxd01:8443
lxc remote add lxd02 lxd02:8443
5. copy & edit profile
lxc config profile copy default criu
lxc config profile edit criu
name: criu
config:
raw.lxc: |
lxc.tty=0
lxc.console=none
lxc.cgroup.devices.deny=c 5:1 rwm
security.privileged: "true"
devices: {}
6. copy profile
lxc config profile copy lxd01:criu lxd02:criu
7. create container on lxd01
lxc init ubuntu ct01
8. apply profile on lxd01
lxc config profile apply ct01 criu
9. start container on lxd01
lxc start ct01
lxc list
+------+---------+------+------+
| NAME | STATE | IPV4 | IPV6 |
+------+---------+------+------+
| ct01 | RUNNING | | |
+------+---------+------+------+
10. migrate container
lxc move lxd01:ct01 lxd02:ct01
$ lxc list lxd02:
+------+---------+------+------+
| NAME | STATE | IPV4 | IPV6 |
+------+---------+------+------+
| ct01 | RUNNING | | |
+------+---------+------+------+

Thanks!!
Tycho Andersen
2015-04-08 14:44:18 UTC
Permalink
Hi,
Post by KATOH Yasufumi
Hi,
Post by KATOH Yasufumi
On Fri, 27 Mar 2015 18:36:22 +0900
in message "Re: [lxc-users] live migration using lxd 0.4"
Post by KATOH Yasufumi
Thanks for your reply and detailed description!! I will try it next week.
I tried live migration using lxd 0.6, and work fine!! :-D
Glad to hear it. Thanks for fiddling with it :)

Tycho
Post by KATOH Yasufumi
* OS: Ubuntu Trusty x 2
* source host: lxd01
* destination host: lxd02
* LXD: Install from lxd-git-master PPA
$ dpkg-query --show lxd
lxd 0.6-0ubuntu1~ubuntu14.04.1~ppa1
1. remove lxcfs
apt-get remove lxcfs
2. set password (both hosts)
lxc config set password hogehoge
3. get image on lxd01
lxd-images import lxc ubuntu trusty amd64 --alias ubuntu
4. add remotes on lxd01
lxc remote add lxd01 lxd01:8443
lxc remote add lxd02 lxd02:8443
5. copy & edit profile
lxc config profile copy default criu
lxc config profile edit criu
name: criu
raw.lxc: |
lxc.tty=0
lxc.console=none
lxc.cgroup.devices.deny=c 5:1 rwm
security.privileged: "true"
devices: {}
6. copy profile
lxc config profile copy lxd01:criu lxd02:criu
7. create container on lxd01
lxc init ubuntu ct01
8. apply profile on lxd01
lxc config profile apply ct01 criu
9. start container on lxd01
lxc start ct01
lxc list
+------+---------+------+------+
| NAME | STATE | IPV4 | IPV6 |
+------+---------+------+------+
| ct01 | RUNNING | | |
+------+---------+------+------+
10. migrate container
lxc move lxd01:ct01 lxd02:ct01
+------+---------+------+------+
| NAME | STATE | IPV4 | IPV6 |
+------+---------+------+------+
| ct01 | RUNNING | | |
+------+---------+------+------+
Thanks!!
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Continue reading on narkive:
Loading...