Running GlusterFS inside docker container

As a part of GlusterFS 3.5 testing and hackathon, I decided to put GlusterFS inside a docker container.So I installed docker on my Fedora20 desktop

[sourcecode language=”text” gutter=”false”]
$ yum install docker-io -y
$ systemctl enable docker.service
ln -s ‘/usr/lib/systemd/system/docker.service’ ‘/etc/systemd/system/multi-user.target.wants/docker.service’
$ systemctl start docker.service
$ docker version
Client version: 0.7.6
..
Server version: 0.7.6
[/sourcecode]

and then started a Fedora container

[sourcecode language=”text” gutter=”false”]
$ docker run -i -t mattdm/fedora /bin/bash
[/sourcecode]

Once I am inside the container I installed GlusterFS packages

[sourcecode language=”text” gutter=”false”]
bash-4.2# yum install glusterfs glusterfs-server -y
[/sourcecode]

And then tried to create volume

[sourcecode language=”text” gutter=”false”]
bash-4.2# /usr/sbin/glusterd
bash-4.2# gluster volume create vol 172.17.0.3:/mnt/brick/ force
[/sourcecode]

but I got following error:-

[sourcecode language=”text” gutter=”false”]
volume create: vol: failed: Glusterfs is not supported on brick: 172.17.0.3:/mnt/brick.
Setting extended attributes failed, reason: Operation not permitted.
[/sourcecode]

From above error it looked as setting up extended attributes are not supported, which is a basic need to use GlusteFS. So I tried to test them manually. I was able to set extended attributes in user namespace but not in trusted namespace.

[sourcecode language=”text” gutter=”false”]
bash-4.2# yum install attr -y
bash-4.2# setfattr -n user.foo1 -v "bar" a
bash-4.2# touch a; setfattr -n trusted.foo1 -v "bar" a
setfattr: a: Operation not permitted
[/sourcecode]

With some internet search I figured out that CAP_SYS_ADMIN is needed for setting up extended attributes in trusted namespace and to get that inside docker we need to run an image with –privileged=true option like

[sourcecode language=”text” gutter=”false”]
$ docker run –privileged=true -i -t mattdm/fedora /bin/bash
[/sourcecode]

With that I was able to create the volume and start it

[sourcecode language=”text” gutter=”false”]
bash-4.2# gluster volume create vol 172.17.0.3:/mnt/brick/ force
bash-4.2# gluster volume start vol
[/sourcecode]

But when I tried to mount the volume I got following error:-

[sourcecode language=”text” gutter=”false”]
E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory)
[/sourcecode]

this turned out to be image specific problem, which I am using (mattdm/fedora). I had to mknod for /dev/fuse

[sourcecode language=”text” gutter=”false”]
bash-4.2# mknod /dev/fuse c 10 229
[/sourcecode]

and after that I was able to mount volume.

On Fedroa 20 for docker version 0.7.6 the default storage driver for docker is device-mapper on which extended attributes are supported. AUFS storage driver does not support extended attributes as of now.I have tried with btrfs storage driver with docker 0.8 as well and was able to use GlusterFS. To use btrfs storage driver, we need to start docker daemon with following command :-

[sourcecode language=”text” gutter=”false”]
$ docker -d -s btrfs
[/sourcecode]

Above will only work if Docker is running on a btrfs partition already prepared by the host system.