Discussion:
lxd hang - "panic: runtime error: slice bounds out of range"
(too old to reply)
Tomasz Chmielewski
2017-01-31 07:17:58 UTC
Permalink
lxd process on one of my servers started to hang a few days ago.

In syslog, I can see the following being repeated (log below).

I see a similar issue here:

https://github.com/lxc/lxd/issues/2089

but it's closed.

Running:

ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - client

On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.


It seems to recover if I kill this process:

root 26853 0.0 0.0 222164 12228 ? Ssl 07:13 0:00
/usr/bin/lxd waitready --timeout=600


Sometimes need to kill it a few times until it recovers.


Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after
600s timeout.
Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds out
of range
Jan 31 06:26:06 lxd01 lxd[4402]: goroutine 27 [running]:
Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Jan 31 06:26:06 lxd01 lxd[4402]:
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
Jan 31 06:26:06 lxd01 lxd[4402]:
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
Jan 31 06:26:06 lxd01 lxd[4402]:
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0,
0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
Jan 31 06:26:06 lxd01 lxd[4402]:
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
Jan 31 06:26:06 lxd01 lxd[4402]:
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
Jan 31 06:26:06 lxd01 lxd[4402]:
main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
Jan 31 06:26:06 lxd01 lxd[4402]: main.snapshotDelete.func1(0xc8200c8000,
0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
Jan 31 06:26:06 lxd01 lxd[4402]:
main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
Jan 31 06:26:06 lxd01 lxd[4402]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after 600s
timeout.
Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds out
of range
Jan 31 06:36:06 lxd01 lxd[21952]: goroutine 16 [running]:
Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Jan 31 06:36:06 lxd01 lxd[21952]:
main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
Jan 31 06:36:06 lxd01 lxd[21952]:
main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
Jan 31 06:36:06 lxd01 lxd[21952]:
main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938,
0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
Jan 31 06:36:06 lxd01 lxd[21952]:
main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
Jan 31 06:36:06 lxd01 lxd[21952]:
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
Jan 31 06:36:06 lxd01 lxd[21952]:
main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
Jan 31 06:36:06 lxd01 lxd[21952]:
main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
Jan 31 06:36:06 lxd01 lxd[21952]:
main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
Jan 31 06:36:06 lxd01 lxd[21952]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after
600s timeout.




Tomasz Chmielewski
https://lxadm.com
Tomasz Chmielewski
2017-01-31 07:29:06 UTC
Permalink
I think it may be related to
https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/

I have a docker container, with several dockers inside, and with lxd
snapshots.

Doing this:

# lxc delete docker
error: Get
http://unix.socket/1.0/operations/7d30bf41-3af6-4b48-b42c-06fdd2bba48b/wait:
EOF

Results in:

Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out
of range
Jan 31 07:28:36 lxd01 lxd[8363]: goroutine 867 [running]:
Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900,
0xc8204a0480, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*containerLXC).Delete(0xc8204a0480, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
Jan 31 07:28:36 lxd01 lxd[8363]:
main.containerDeleteSnapshots(0xc820090a00, 0xc82017c017, 0x6, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/containers.go:208
+0x4ac
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*containerLXC).Delete(0xc8204a00c0, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2371
+0x696
Jan 31 07:28:36 lxd01 lxd[8363]:
main.containerDelete.func1(0xc8200c8840, 0x0, 0x0)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_delete.go:22
+0x3e
Jan 31 07:28:36 lxd01 lxd[8363]:
main.(*operation).Run.func1(0xc8200c8840, 0xc82011f320)
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 07:28:36 lxd01 lxd[8363]: created by main.(*operation).Run
Jan 31 07:28:36 lxd01 lxd[8363]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Unit entered failed
state.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Service hold-off time
over, scheduling restart.
Jan 31 07:28:36 lxd01 systemd[1]: Stopped LXD - main daemon.
Jan 31 07:28:36 lxd01 systemd[1]: Starting LXD - main daemon...
Jan 31 07:28:36 lxd01 lxd[11969]: t=2017-01-31T07:28:36+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 07:28:37 lxd01 systemd[1]: Started LXD - main daemon.




Tomasz
Post by Tomasz Chmielewski
lxd process on one of my servers started to hang a few days ago.
In syslog, I can see the following being repeated (log below).
https://github.com/lxc/lxd/issues/2089
but it's closed.
ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - client
On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.
root 26853 0.0 0.0 222164 12228 ? Ssl 07:13 0:00
/usr/bin/lxd waitready --timeout=600
Sometimes need to kill it a few times until it recovers.
Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after
600s timeout.
Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds out
of range
Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0,
0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200c8000, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after
600s timeout.
Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds
out of range
Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938,
0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after
600s timeout.
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Tomasz Chmielewski
2017-02-01 06:19:44 UTC
Permalink
Unfortunately it's still crashing, around 1 day after removing the
docker container:

Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s
timeout.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited,
code=exited status=1
Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed
state.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time
over, scheduling restart.
Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon...
Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out
of range
Feb 1 06:16:20 lxd01 lxd[29235]: goroutine 16 [running]:
Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Feb 1 06:16:20 lxd01 lxd[29235]:
main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
Feb 1 06:16:20 lxd01 lxd[29235]:
main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a,
0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
Feb 1 06:16:20 lxd01 lxd[29235]:
main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0,
0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
Feb 1 06:16:20 lxd01 lxd[29235]:
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240,
0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
Feb 1 06:16:20 lxd01 lxd[29235]:
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
Feb 1 06:16:20 lxd01 lxd[29235]:
main.(*containerLXC).Delete(0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
Feb 1 06:16:20 lxd01 lxd[29235]:
main.snapshotDelete.func1(0xc8205400b0, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
Feb 1 06:16:20 lxd01 lxd[29235]:
main.(*operation).Run.func1(0xc8205400b0, 0xc820258f60)
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run
Feb 1 06:16:20 lxd01 lxd[29235]:
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT


Any clues what's causing this and how to fix?


Tomasz
Post by Tomasz Chmielewski
I think it may be related to
https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
I have a docker container, with several dockers inside, and with lxd
snapshots.
# lxc delete docker
error: Get
EOF
Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out
of range
Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900,
0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.containerDeleteSnapshots(0xc820090a00, 0xc82017c017, 0x6, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/containers.go:208
+0x4ac
main.(*containerLXC).Delete(0xc8204a00c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2371
+0x696
main.containerDelete.func1(0xc8200c8840, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_delete.go:22
+0x3e
main.(*operation).Run.func1(0xc8200c8840, 0xc82011f320)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 07:28:36 lxd01 lxd[8363]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Unit entered failed
state.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Service hold-off time
over, scheduling restart.
Jan 31 07:28:36 lxd01 systemd[1]: Stopped LXD - main daemon.
Jan 31 07:28:36 lxd01 systemd[1]: Starting LXD - main daemon...
Jan 31 07:28:36 lxd01 lxd[11969]: t=2017-01-31T07:28:36+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 07:28:37 lxd01 systemd[1]: Started LXD - main daemon.
Tomasz
Post by Tomasz Chmielewski
lxd process on one of my servers started to hang a few days ago.
In syslog, I can see the following being repeated (log below).
https://github.com/lxc/lxd/issues/2089
but it's closed.
ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - client
On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.
root 26853 0.0 0.0 222164 12228 ? Ssl 07:13 0:00
/usr/bin/lxd waitready --timeout=600
Sometimes need to kill it a few times until it recovers.
Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after
600s timeout.
Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds
out of range
Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0,
0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200c8000, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after
600s timeout.
Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds
out of range
Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938,
0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after
600s timeout.
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Stéphane Graber
2017-02-01 10:02:24 UTC
Permalink
Hey there,

I wrote a fix for that function just yesterday which I think should fix
your issue. It's been merged in git but isn't in any released version of
LXD yet.

Since you're using LXD 2.0.8, I made a build of 2.0.8 with that one fix
applied at: https://dl.stgraber.org/lxd-2.0.8-btrfs
SHA256: 4d9a7ef7c4635d7dd6c3e41f0eb1a3db12d38a8148b3940aa801c7355510e815

Stéphane
Unfortunately it's still crashing, around 1 day after removing the docker
Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s
timeout.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited,
code=exited status=1
Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed state.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time over,
scheduling restart.
Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon...
Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out of
range
Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, 0x0,
0x0, 0x0, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, 0x0,
0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0,
0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240, 0x7feeacffd0c0,
0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
Feb 1 06:16:20 lxd01 lxd[29235]: main.(*containerLXC).Delete(0xc8204fa480,
0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
Feb 1 06:16:20 lxd01 lxd[29235]: main.snapshotDelete.func1(0xc8205400b0,
0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
Feb 1 06:16:20 lxd01 lxd[29235]: main.(*operation).Run.func1(0xc8205400b0,
0xc820258f60)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Any clues what's causing this and how to fix?
Tomasz
Post by Tomasz Chmielewski
I think it may be related to
https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
I have a docker container, with several dockers inside, and with lxd
snapshots.
# lxc delete docker
error: Get
EOF
Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out
of range
Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900,
0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.containerDeleteSnapshots(0xc820090a00, 0xc82017c017, 0x6, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/containers.go:208
+0x4ac
main.(*containerLXC).Delete(0xc8204a00c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2371
+0x696
main.containerDelete.func1(0xc8200c8840, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_delete.go:22
+0x3e
main.(*operation).Run.func1(0xc8200c8840, 0xc82011f320)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 07:28:36 lxd01 lxd[8363]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Unit entered failed
state.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Service hold-off time
over, scheduling restart.
Jan 31 07:28:36 lxd01 systemd[1]: Stopped LXD - main daemon.
Jan 31 07:28:36 lxd01 systemd[1]: Starting LXD - main daemon...
Jan 31 07:28:36 lxd01 lxd[11969]: t=2017-01-31T07:28:36+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 07:28:37 lxd01 systemd[1]: Started LXD - main daemon.
Tomasz
Post by Tomasz Chmielewski
lxd process on one of my servers started to hang a few days ago.
In syslog, I can see the following being repeated (log below).
https://github.com/lxc/lxd/issues/2089
but it's closed.
ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - client
On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.
root 26853 0.0 0.0 222164 12228 ? Ssl 07:13 0:00
/usr/bin/lxd waitready --timeout=600
Sometimes need to kill it a few times until it recovers.
Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after
600s timeout.
Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds
out of range
Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0,
0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200c8000, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after
600s timeout.
Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds
out of range
Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938,
0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after
600s timeout.
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
--
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
Tomasz Chmielewski
2017-02-01 10:13:25 UTC
Permalink
FYI it was crashing in the following conditions:

- privileged container (i.e. the one for docker)
- on btrfs filesystem
- btrfs subvolume created inside the container (docker would create such
subvolumes)
- snapshot of the container made


Will your fix eventually go to i.e. 2.0.9?


Tomasz
Post by Stéphane Graber
Hey there,
I wrote a fix for that function just yesterday which I think should fix
your issue. It's been merged in git but isn't in any released version of
LXD yet.
Since you're using LXD 2.0.8, I made a build of 2.0.8 with that one fix
applied at: https://dl.stgraber.org/lxd-2.0.8-btrfs
4d9a7ef7c4635d7dd6c3e41f0eb1a3db12d38a8148b3940aa801c7355510e815
Stéphane
Unfortunately it's still crashing, around 1 day after removing the docker
Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s
timeout.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited,
code=exited status=1
Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed state.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time over,
scheduling restart.
Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon...
Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out of
range
Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, 0x0,
0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, 0x0,
0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0,
0xc8204fa480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240,
0x7feeacffd0c0,
0xc8204fa480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204fa480,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8205400b0,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8205400b0,
0xc820258f60)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Any clues what's causing this and how to fix?
Tomasz
Post by Tomasz Chmielewski
I think it may be related to
https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
I have a docker container, with several dockers inside, and with lxd
snapshots.
# lxc delete docker
error: Get
EOF
Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out
of range
Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900,
0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.containerDeleteSnapshots(0xc820090a00, 0xc82017c017, 0x6, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/containers.go:208
+0x4ac
main.(*containerLXC).Delete(0xc8204a00c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2371
+0x696
main.containerDelete.func1(0xc8200c8840, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_delete.go:22
+0x3e
main.(*operation).Run.func1(0xc8200c8840, 0xc82011f320)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 07:28:36 lxd01 lxd[8363]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Unit entered failed
state.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Service hold-off time
over, scheduling restart.
Jan 31 07:28:36 lxd01 systemd[1]: Stopped LXD - main daemon.
Jan 31 07:28:36 lxd01 systemd[1]: Starting LXD - main daemon...
Jan 31 07:28:36 lxd01 lxd[11969]: t=2017-01-31T07:28:36+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 07:28:37 lxd01 systemd[1]: Started LXD - main daemon.
Tomasz
Post by Tomasz Chmielewski
lxd process on one of my servers started to hang a few days ago.
In syslog, I can see the following being repeated (log below).
https://github.com/lxc/lxd/issues/2089
but it's closed.
ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - client
On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.
root 26853 0.0 0.0 222164 12228 ? Ssl 07:13 0:00
/usr/bin/lxd waitready --timeout=600
Sometimes need to kill it a few times until it recovers.
Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after
600s timeout.
Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds
out of range
Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0,
0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200c8000, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after
600s timeout.
Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds
out of range
Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938,
0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after
600s timeout.
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
Stéphane Graber
2017-02-01 10:25:10 UTC
Permalink
Post by Tomasz Chmielewski
- privileged container (i.e. the one for docker)
- on btrfs filesystem
- btrfs subvolume created inside the container (docker would create such
subvolumes)
- snapshot of the container made
Yeah, shouldn't really matter since the whole code was removed and
redone not to rely on parsing "btrfs subvolume list" :)
Post by Tomasz Chmielewski
Will your fix eventually go to i.e. 2.0.9?
2.0.9 was released before I wrote that fix (should hit Ubuntu users in
the next couple of weeks). The fix will be in 2.0.10 (I just pushed it
to the stable-2.0 branch).
Post by Tomasz Chmielewski
Tomasz
Post by Stéphane Graber
Hey there,
I wrote a fix for that function just yesterday which I think should fix
your issue. It's been merged in git but isn't in any released version of
LXD yet.
Since you're using LXD 2.0.8, I made a build of 2.0.8 with that one fix
applied at: https://dl.stgraber.org/lxd-2.0.8-btrfs
SHA256: 4d9a7ef7c4635d7dd6c3e41f0eb1a3db12d38a8148b3940aa801c7355510e815
Stéphane
Unfortunately it's still crashing, around 1 day after removing the docker
Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s
timeout.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited,
code=exited status=1
Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed state.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time over,
scheduling restart.
Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon.
Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon...
Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out of
range
Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, 0x0,
0x0, 0x0, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, 0x0,
0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0,
0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240,
0x7feeacffd0c0,
0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204fa480,
0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8205400b0,
0x0, 0x0)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8205400b0,
0xc820258f60)
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run
Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Any clues what's causing this and how to fix?
Tomasz
Post by Tomasz Chmielewski
I think it may be related to
https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
I have a docker container, with several dockers inside, and with lxd
snapshots.
# lxc delete docker
error: Get
EOF
Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out
of range
Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900,
0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204a0480, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.containerDeleteSnapshots(0xc820090a00, 0xc82017c017, 0x6, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/containers.go:208
+0x4ac
main.(*containerLXC).Delete(0xc8204a00c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2371
+0x696
main.containerDelete.func1(0xc8200c8840, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_delete.go:22
+0x3e
main.(*operation).Run.func1(0xc8200c8840, 0xc82011f320)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 07:28:36 lxd01 lxd[8363]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Main process exited,
code=exited, status=2/INVALIDARGUMENT
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Unit entered failed
state.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Failed with result
'exit-code'.
Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Service hold-off time
over, scheduling restart.
Jan 31 07:28:36 lxd01 systemd[1]: Stopped LXD - main daemon.
Jan 31 07:28:36 lxd01 systemd[1]: Starting LXD - main daemon...
Jan 31 07:28:36 lxd01 lxd[11969]: t=2017-01-31T07:28:36+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 07:28:37 lxd01 systemd[1]: Started LXD - main daemon.
Tomasz
Post by Tomasz Chmielewski
lxd process on one of my servers started to hang a few days ago.
In syslog, I can see the following being repeated (log below).
https://github.com/lxc/lxd/issues/2089
but it's closed.
ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2
amd64 Container hypervisor based on LXC - client
On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.
root 26853 0.0 0.0 222164 12228 ? Ssl 07:13 0:00
/usr/bin/lxd waitready --timeout=600
Sometimes need to kill it a few times until it recovers.
Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after
600s timeout.
Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds
out of range
Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0,
0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200c8000, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after
600s timeout.
Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn
msg="CGroup memory swap accounting is disabled, swap limits will be
ignored."
Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds
out of range
Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0, 0x0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
+0x104f
main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a,
0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
+0x72
main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938,
0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
+0xb0
main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
+0x5c
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60,
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
+0x22b
main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
+0x30e
main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
+0x3e
main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
+0x3a
Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
+0x127
Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after
600s timeout.
Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
_______________________________________________
lxc-users mailing list
http://lists.linuxcontainers.org/listinfo/lxc-users
--
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
Continue reading on narkive:
Loading...