GlusterFS HowTo Tutorial For Distributed Storage in Docker, Kubernetes, LXC, KVM, Proxmox
This can be used on almost anything, since Gluster is a userspace tool, based on FUSE. This means that all Gluster appears as to any application is just a directory.
Applications don't need specific support for Gluster, so long as you can tell the application to use a certain directory for storage.
One application can be for redundant and scaled storage, including for within Docker and Kubernetes, LXC, Proxmox, OpenStack, etc or just your image/web/video files or even database.
In this example, we assume that each node needs a full copy of the data and has a full storage brick in each node. In practice, when you scale to very large amounts of storage nodes, you would not likely want each node to have a full copy of the data.
However, in our case, in a smaller cluster, it would be too risky not to have at least 2-3 bricks or full replicas in the cluster.
One final production consideration, is that gluster has no inherent security to prevent clients from mounting your volumes aside from being IP based (but of course an attacker could get an IP from the correct subnet or even physically or remotely gain control of an allowed IP). This is both convenient and a huge security hole on the part of GlusterFS. The ideal security situation is that gluster nodes and clients communicate across a separate VLAN and an encrypted and secure VPN tunnel.
In this example I am using 3 nodes which are named gluster1, 2 and 3.
Step 1 - Install Gluster on All Nodes:
apt install glusterfs-server
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [147 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8182 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2596 B]
Fetched 8532 kB in 2s (4525 kB/s)
Reading package lists...
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
attr bzip2 ca-certificates file fuse glusterfs-client glusterfs-common ibverbs-providers keyutils libacl1-dev libaio1 libattr1-dev libc-dev-bin libc6-dev libevent-2.1-6 libfuse2
libgfapi0 libgfchangelog0 libgfdb0 libgfrpc0 libgfxdr0 libglusterfs-dev libglusterfs0 libibverbs1 libicu63 libldap-2.4-2 libldap-common libmagic-mgc libmagic1 libmpdec2 libnfsidmap2
libnl-3-200 libnl-route-3-200 libpython-stdlib libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.7 libpython3.7-minimal libpython3.7-stdlib
librdmacm1 libreadline5 libreadline7 libsasl2-2 libsasl2-modules libsasl2-modules-db libsqlite3-0 libtirpc-common libtirpc3 liburcu6 libwrap0 libxml2 linux-libc-dev manpages manpages-dev
mime-support nfs-common openssl python python-minimal python2 python2-minimal python2.7 python2.7-minimal python3 python3-asn1crypto python3-certifi python3-cffi-backend python3-chardet
python3-cryptography python3-idna python3-jwt python3-minimal python3-pkg-resources python3-prettytable python3-requests python3-six python3-urllib3 python3.7 python3.7-minimal
readline-common rpcbind sensible-utils ucf xfsprogs xz-utils
Suggested packages:
bzip2-doc glibc-doc libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql man-browser open-iscsi watchdog
python-doc python-tk python2-doc python2.7-doc binutils binfmt-support python3-doc python3-tk python3-venv python-cryptography-doc python3-cryptography-vectors python3-crypto
python3-setuptools python3-openssl python3-socks python3.7-venv python3.7-doc readline-doc xfsdump acl quota
The following NEW packages will be installed:
attr bzip2 ca-certificates file fuse glusterfs-client glusterfs-common glusterfs-server ibverbs-providers keyutils libacl1-dev libaio1 libattr1-dev libc-dev-bin libc6-dev libevent-2.1-6
libfuse2 libgfapi0 libgfchangelog0 libgfdb0 libgfrpc0 libgfxdr0 libglusterfs-dev libglusterfs0 libibverbs1 libicu63 libldap-2.4-2 libldap-common libmagic-mgc libmagic1 libmpdec2
libnfsidmap2 libnl-3-200 libnl-route-3-200 libpython-stdlib libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.7 libpython3.7-minimal
libpython3.7-stdlib librdmacm1 libreadline5 libreadline7 libsasl2-2 libsasl2-modules libsasl2-modules-db libsqlite3-0 libtirpc-common libtirpc3 liburcu6 libwrap0 libxml2 linux-libc-dev
manpages manpages-dev mime-support nfs-common openssl python python-minimal python2 python2-minimal python2.7 python2.7-minimal python3 python3-asn1crypto python3-certifi
python3-cffi-backend python3-chardet python3-cryptography python3-idna python3-jwt python3-minimal python3-pkg-resources python3-prettytable python3-requests python3-six python3-urllib3
python3.7 python3.7-minimal readline-common rpcbind sensible-utils ucf xfsprogs xz-utils
0 upgraded, 88 newly installed, 0 to remove and 0 not upgraded.
Need to get 62.5 MB of archives.
After this operation, 178 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://deb.debian.org/debian buster/main amd64 libpython2.7-minimal amd64 2.7.16-2+deb10u1 [395 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 python2.7-minimal amd64 2.7.16-2+deb10u1 [1369 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 python2-minimal amd64 2.7.16-1 [41.4 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 python-minimal amd64 2.7.16-1 [21.0 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 mime-support all 3.62 [37.2 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 readline-common all 7.0-5 [70.6 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libreadline7 amd64 7.0-5 [151 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 libsqlite3-0 amd64 3.27.2-3+deb10u1 [641 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libpython2.7-stdlib amd64 2.7.16-2+deb10u1 [1912 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 python2.7 amd64 2.7.16-2+deb10u1 [305 kB]
Get:11 http://deb.debian.org/debian buster/main amd64 libpython2-stdlib amd64 2.7.16-1 [20.8 kB]
Get:12 http://deb.debian.org/debian buster/main amd64 libpython-stdlib amd64 2.7.16-1 [20.8 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 python2 amd64 2.7.16-1 [41.6 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 python amd64 2.7.16-1 [22.8 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libpython3.7-minimal amd64 3.7.3-2+deb10u3 [589 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 python3.7-minimal amd64 3.7.3-2+deb10u3 [1737 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 python3-minimal amd64 3.7.3-1 [36.6 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libmpdec2 amd64 2.4.2-2 [87.2 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libpython3.7-stdlib amd64 3.7.3-2+deb10u3 [1734 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 python3.7 amd64 3.7.3-2+deb10u3 [330 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 libpython3-stdlib amd64 3.7.3-1 [20.0 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 python3 amd64 3.7.3-1 [61.5 kB]
Get:23 http://deb.debian.org/debian buster/main amd64 sensible-utils all 0.0.12 [15.8 kB]
Get:24 http://deb.debian.org/debian buster/main amd64 bzip2 amd64 1.0.6-9.2~deb10u1 [48.4 kB]
Get:25 http://deb.debian.org/debian buster/main amd64 libmagic-mgc amd64 1:5.35-4+deb10u2 [242 kB]
Get:26 http://deb.debian.org/debian buster/main amd64 libmagic1 amd64 1:5.35-4+deb10u2 [118 kB]
Get:27 http://deb.debian.org/debian buster/main amd64 file amd64 1:5.35-4+deb10u2 [66.4 kB]
Get:28 http://deb.debian.org/debian buster/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-1+deb10u2 [69.2 kB]
Get:29 http://deb.debian.org/debian buster/main amd64 libsasl2-2 amd64 2.1.27+dfsg-1+deb10u2 [106 kB]
Get:30 http://deb.debian.org/debian-security buster/updates/main amd64 libldap-common all 2.4.47+dfsg-3+deb10u7 [90.1 kB]
Get:31 http://deb.debian.org/debian-security buster/updates/main amd64 libldap-2.4-2 amd64 2.4.47+dfsg-3+deb10u7 [224 kB]
Get:32 http://deb.debian.org/debian buster/main amd64 manpages all 4.16-2 [1295 kB]
Get:33 http://deb.debian.org/debian buster/main amd64 ucf all 3.0038+nmu1 [69.0 kB]
Get:34 http://deb.debian.org/debian-security buster/updates/main amd64 xz-utils amd64 5.2.4-1+deb10u1 [183 kB]
Get:35 http://deb.debian.org/debian buster/main amd64 attr amd64 1:2.4.48-4 [41.4 kB]
Get:36 http://deb.debian.org/debian-security buster/updates/main amd64 openssl amd64 1.1.1n-0+deb10u2 [855 kB]
Get:37 http://deb.debian.org/debian buster/main amd64 ca-certificates all 20200601~deb10u2 [166 kB]
Get:38 http://deb.debian.org/debian buster/main amd64 libfuse2 amd64 2.9.9-1+deb10u1 [128 kB]
Get:39 http://deb.debian.org/debian buster/main amd64 fuse amd64 2.9.9-1+deb10u1 [72.3 kB]
Get:40 http://deb.debian.org/debian buster/main amd64 libaio1 amd64 0.3.112-3 [11.2 kB]
Get:41 http://deb.debian.org/debian buster/main amd64 libtirpc-common all 1.1.4-0.4 [16.7 kB]
Get:42 http://deb.debian.org/debian buster/main amd64 libtirpc3 amd64 1.1.4-0.4 [93.5 kB]
Get:43 http://deb.debian.org/debian buster/main amd64 libglusterfs0 amd64 5.5-3 [2740 kB]
Get:44 http://deb.debian.org/debian buster/main amd64 libgfxdr0 amd64 5.5-3 [2493 kB]
Get:45 http://deb.debian.org/debian buster/main amd64 libgfrpc0 amd64 5.5-3 [2512 kB]
Get:46 http://deb.debian.org/debian buster/main amd64 libgfapi0 amd64 5.5-3 [2535 kB]
Get:47 http://deb.debian.org/debian buster/main amd64 libgfchangelog0 amd64 5.5-3 [2493 kB]
Get:48 http://deb.debian.org/debian buster/main amd64 libgfdb0 amd64 5.5-3 [2491 kB]
Get:49 http://deb.debian.org/debian buster/main amd64 libnl-3-200 amd64 3.4.0-1 [63.0 kB]
Get:50 http://deb.debian.org/debian buster/main amd64 libnl-route-3-200 amd64 3.4.0-1 [162 kB]
Get:51 http://deb.debian.org/debian buster/main amd64 libibverbs1 amd64 22.1-1 [51.2 kB]
Get:52 http://deb.debian.org/debian buster/main amd64 libpython3.7 amd64 3.7.3-2+deb10u3 [1498 kB]
Get:53 http://deb.debian.org/debian buster/main amd64 librdmacm1 amd64 22.1-1 [65.3 kB]
Get:54 http://deb.debian.org/debian buster/main amd64 liburcu6 amd64 0.10.2-1 [66.4 kB]
Get:55 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u3 [8293 kB]
Get:56 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u3 [689 kB]
Get:57 http://deb.debian.org/debian buster/main amd64 libc-dev-bin amd64 2.28-10+deb10u1 [276 kB]
Get:58 http://deb.debian.org/debian buster/main amd64 linux-libc-dev amd64 4.19.235-1 [1510 kB]
Get:59 http://deb.debian.org/debian buster/main amd64 libc6-dev amd64 2.28-10+deb10u1 [2692 kB]
Get:60 http://deb.debian.org/debian buster/main amd64 libattr1-dev amd64 1:2.4.48-4 [34.9 kB]
Get:61 http://deb.debian.org/debian buster/main amd64 libacl1-dev amd64 2.2.53-4 [91.7 kB]
Get:62 http://deb.debian.org/debian buster/main amd64 libglusterfs-dev amd64 5.5-3 [2608 kB]
Get:63 http://deb.debian.org/debian buster/main amd64 python3-prettytable all 0.7.2-4 [22.8 kB]
Get:64 http://deb.debian.org/debian buster/main amd64 python3-certifi all 2018.8.24-1 [140 kB]
Get:65 http://deb.debian.org/debian buster/main amd64 python3-pkg-resources all 40.8.0-1 [153 kB]
Get:66 http://deb.debian.org/debian buster/main amd64 python3-chardet all 3.0.4-3 [80.5 kB]
Get:67 http://deb.debian.org/debian buster/main amd64 python3-idna all 2.6-1 [34.3 kB]
Get:68 http://deb.debian.org/debian buster/main amd64 python3-six all 1.12.0-1 [15.7 kB]
Get:69 http://deb.debian.org/debian buster/main amd64 python3-urllib3 all 1.24.1-1 [97.1 kB]
Get:70 http://deb.debian.org/debian buster/main amd64 python3-requests all 2.21.0-1 [66.9 kB]
Get:71 http://deb.debian.org/debian buster/main amd64 python3-jwt all 1.7.0-2 [20.5 kB]
Get:72 http://deb.debian.org/debian buster/main amd64 libreadline5 amd64 5.2+dfsg-3+b13 [120 kB]
Get:73 http://deb.debian.org/debian buster/main amd64 xfsprogs amd64 4.20.0-1 [909 kB]
Get:74 http://deb.debian.org/debian buster/main amd64 glusterfs-common amd64 5.5-3 [5271 kB]
Get:75 http://deb.debian.org/debian buster/main amd64 glusterfs-client amd64 5.5-3 [2493 kB]
Get:76 http://deb.debian.org/debian buster/main amd64 glusterfs-server amd64 5.5-3 [2665 kB]
Get:77 http://deb.debian.org/debian buster/main amd64 ibverbs-providers amd64 22.1-1 [187 kB]
Get:78 http://deb.debian.org/debian buster/main amd64 keyutils amd64 1.6-6 [51.7 kB]
Get:79 http://deb.debian.org/debian buster/main amd64 libevent-2.1-6 amd64 2.1.8-stable-4 [177 kB]
Get:80 http://deb.debian.org/debian buster/main amd64 libnfsidmap2 amd64 0.25-5.1 [32.0 kB]
Get:81 http://deb.debian.org/debian buster/main amd64 libsasl2-modules amd64 2.1.27+dfsg-1+deb10u2 [104 kB]
Get:82 http://deb.debian.org/debian buster/main amd64 libwrap0 amd64 7.6.q-28 [58.7 kB]
Get:83 http://deb.debian.org/debian buster/main amd64 manpages-dev all 4.16-2 [2232 kB]
Get:84 http://deb.debian.org/debian buster/main amd64 rpcbind amd64 1.2.5-0.3+deb10u1 [47.1 kB]
Get:85 http://deb.debian.org/debian buster/main amd64 nfs-common amd64 1:1.3.4-2.5+deb10u1 [231 kB]
Get:86 http://deb.debian.org/debian buster/main amd64 python3-asn1crypto all 0.24.0-1 [78.2 kB]
Get:87 http://deb.debian.org/debian buster/main amd64 python3-cffi-backend amd64 1.12.2-1 [79.7 kB]
Get:88 http://deb.debian.org/debian buster/main amd64 python3-cryptography amd64 2.6.1-3+deb10u2 [219 kB]
Fetched 62.5 MB in 3s (23.2 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libpython2.7-minimal:amd64.
(Reading database ... 11168 files and directories currently installed.)
Preparing to unpack .../00-libpython2.7-minimal_2.7.16-2+deb10u1_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../01-python2.7-minimal_2.7.16-2+deb10u1_amd64.deb ...
Unpacking python2.7-minimal (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2-minimal.
Preparing to unpack .../02-python2-minimal_2.7.16-1_amd64.deb ...
Unpacking python2-minimal (2.7.16-1) ...
Selecting previously unselected package python-minimal.
Preparing to unpack .../03-python-minimal_2.7.16-1_amd64.deb ...
Unpacking python-minimal (2.7.16-1) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../04-mime-support_3.62_all.deb ...
Unpacking mime-support (3.62) ...
Selecting previously unselected package readline-common.
Preparing to unpack .../05-readline-common_7.0-5_all.deb ...
Unpacking readline-common (7.0-5) ...
Selecting previously unselected package libreadline7:amd64.
Preparing to unpack .../06-libreadline7_7.0-5_amd64.deb ...
Unpacking libreadline7:amd64 (7.0-5) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../07-libsqlite3-0_3.27.2-3+deb10u1_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.27.2-3+deb10u1) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../08-libpython2.7-stdlib_2.7.16-2+deb10u1_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../09-python2.7_2.7.16-2+deb10u1_amd64.deb ...
Unpacking python2.7 (2.7.16-2+deb10u1) ...
Selecting previously unselected package libpython2-stdlib:amd64.
Preparing to unpack .../10-libpython2-stdlib_2.7.16-1_amd64.deb ...
Unpacking libpython2-stdlib:amd64 (2.7.16-1) ...
Selecting previously unselected package libpython-stdlib:amd64.
Preparing to unpack .../11-libpython-stdlib_2.7.16-1_amd64.deb ...
Unpacking libpython-stdlib:amd64 (2.7.16-1) ...
Setting up libpython2.7-minimal:amd64 (2.7.16-2+deb10u1) ...
Setting up python2.7-minimal (2.7.16-2+deb10u1) ...
Linking and byte-compiling packages for runtime python2.7...
Setting up python2-minimal (2.7.16-1) ...
Selecting previously unselected package python2.
(Reading database ... 11984 files and directories currently installed.)
Preparing to unpack .../python2_2.7.16-1_amd64.deb ...
Unpacking python2 (2.7.16-1) ...
Setting up python-minimal (2.7.16-1) ...
Selecting previously unselected package python.
(Reading database ... 12017 files and directories currently installed.)
Preparing to unpack .../python_2.7.16-1_amd64.deb ...
Unpacking python (2.7.16-1) ...
Selecting previously unselected package libpython3.7-minimal:amd64.
Preparing to unpack .../libpython3.7-minimal_3.7.3-2+deb10u3_amd64.deb ...
Unpacking libpython3.7-minimal:amd64 (3.7.3-2+deb10u3) ...
Selecting previously unselected package python3.7-minimal.
Preparing to unpack .../python3.7-minimal_3.7.3-2+deb10u3_amd64.deb ...
Unpacking python3.7-minimal (3.7.3-2+deb10u3) ...
Setting up libpython3.7-minimal:amd64 (3.7.3-2+deb10u3) ...
Setting up python3.7-minimal (3.7.3-2+deb10u3) ...
Selecting previously unselected package python3-minimal.
(Reading database ... 12271 files and directories currently installed.)
Preparing to unpack .../python3-minimal_3.7.3-1_amd64.deb ...
Unpacking python3-minimal (3.7.3-1) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../libmpdec2_2.4.2-2_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-2) ...
Selecting previously unselected package libpython3.7-stdlib:amd64.
Preparing to unpack .../libpython3.7-stdlib_3.7.3-2+deb10u3_amd64.deb ...
Unpacking libpython3.7-stdlib:amd64 (3.7.3-2+deb10u3) ...
Selecting previously unselected package python3.7.
Preparing to unpack .../python3.7_3.7.3-2+deb10u3_amd64.deb ...
Unpacking python3.7 (3.7.3-2+deb10u3) ...
Selecting previously unselected package libpython3-stdlib:amd64.
Preparing to unpack .../libpython3-stdlib_3.7.3-1_amd64.deb ...
Unpacking libpython3-stdlib:amd64 (3.7.3-1) ...
Setting up python3-minimal (3.7.3-1) ...
Selecting previously unselected package python3.
(Reading database ... 12683 files and directories currently installed.)
Preparing to unpack .../00-python3_3.7.3-1_amd64.deb ...
Unpacking python3 (3.7.3-1) ...
Selecting previously unselected package sensible-utils.
Preparing to unpack .../01-sensible-utils_0.0.12_all.deb ...
Unpacking sensible-utils (0.0.12) ...
Selecting previously unselected package bzip2.
Preparing to unpack .../02-bzip2_1.0.6-9.2~deb10u1_amd64.deb ...
Unpacking bzip2 (1.0.6-9.2~deb10u1) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../03-libmagic-mgc_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking libmagic-mgc (1:5.35-4+deb10u2) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../04-libmagic1_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.35-4+deb10u2) ...
Selecting previously unselected package file.
Preparing to unpack .../05-file_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking file (1:5.35-4+deb10u2) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../06-libsasl2-modules-db_2.1.27+dfsg-1+deb10u2_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u2) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../07-libsasl2-2_2.1.27+dfsg-1+deb10u2_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u2) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../08-libldap-common_2.4.47+dfsg-3+deb10u7_all.deb ...
Unpacking libldap-common (2.4.47+dfsg-3+deb10u7) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../09-libldap-2.4-2_2.4.47+dfsg-3+deb10u7_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u7) ...
Selecting previously unselected package manpages.
Preparing to unpack .../10-manpages_4.16-2_all.deb ...
Unpacking manpages (4.16-2) ...
Selecting previously unselected package ucf.
Preparing to unpack .../11-ucf_3.0038+nmu1_all.deb ...
Moving old data out of the way
Unpacking ucf (3.0038+nmu1) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../12-xz-utils_5.2.4-1+deb10u1_amd64.deb ...
Unpacking xz-utils (5.2.4-1+deb10u1) ...
Selecting previously unselected package attr.
Preparing to unpack .../13-attr_1%3a2.4.48-4_amd64.deb ...
Unpacking attr (1:2.4.48-4) ...
Selecting previously unselected package openssl.
Preparing to unpack .../14-openssl_1.1.1n-0+deb10u2_amd64.deb ...
Unpacking openssl (1.1.1n-0+deb10u2) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../15-ca-certificates_20200601~deb10u2_all.deb ...
Unpacking ca-certificates (20200601~deb10u2) ...
Selecting previously unselected package libfuse2:amd64.
Preparing to unpack .../16-libfuse2_2.9.9-1+deb10u1_amd64.deb ...
Unpacking libfuse2:amd64 (2.9.9-1+deb10u1) ...
Selecting previously unselected package fuse.
Preparing to unpack .../17-fuse_2.9.9-1+deb10u1_amd64.deb ...
Unpacking fuse (2.9.9-1+deb10u1) ...
Selecting previously unselected package libaio1:amd64.
Preparing to unpack .../18-libaio1_0.3.112-3_amd64.deb ...
Unpacking libaio1:amd64 (0.3.112-3) ...
Selecting previously unselected package libtirpc-common.
Preparing to unpack .../19-libtirpc-common_1.1.4-0.4_all.deb ...
Unpacking libtirpc-common (1.1.4-0.4) ...
Selecting previously unselected package libtirpc3:amd64.
Preparing to unpack .../20-libtirpc3_1.1.4-0.4_amd64.deb ...
Unpacking libtirpc3:amd64 (1.1.4-0.4) ...
Selecting previously unselected package libglusterfs0:amd64.
Preparing to unpack .../21-libglusterfs0_5.5-3_amd64.deb ...
Unpacking libglusterfs0:amd64 (5.5-3) ...
Selecting previously unselected package libgfxdr0:amd64.
Preparing to unpack .../22-libgfxdr0_5.5-3_amd64.deb ...
Unpacking libgfxdr0:amd64 (5.5-3) ...
Selecting previously unselected package libgfrpc0:amd64.
Preparing to unpack .../23-libgfrpc0_5.5-3_amd64.deb ...
Unpacking libgfrpc0:amd64 (5.5-3) ...
Selecting previously unselected package libgfapi0:amd64.
Preparing to unpack .../24-libgfapi0_5.5-3_amd64.deb ...
Unpacking libgfapi0:amd64 (5.5-3) ...
Selecting previously unselected package libgfchangelog0:amd64.
Preparing to unpack .../25-libgfchangelog0_5.5-3_amd64.deb ...
Unpacking libgfchangelog0:amd64 (5.5-3) ...
Selecting previously unselected package libgfdb0:amd64.
Preparing to unpack .../26-libgfdb0_5.5-3_amd64.deb ...
Unpacking libgfdb0:amd64 (5.5-3) ...
Selecting previously unselected package libnl-3-200:amd64.
Preparing to unpack .../27-libnl-3-200_3.4.0-1_amd64.deb ...
Unpacking libnl-3-200:amd64 (3.4.0-1) ...
Selecting previously unselected package libnl-route-3-200:amd64.
Preparing to unpack .../28-libnl-route-3-200_3.4.0-1_amd64.deb ...
Unpacking libnl-route-3-200:amd64 (3.4.0-1) ...
Selecting previously unselected package libibverbs1:amd64.
Preparing to unpack .../29-libibverbs1_22.1-1_amd64.deb ...
Unpacking libibverbs1:amd64 (22.1-1) ...
Selecting previously unselected package libpython3.7:amd64.
Preparing to unpack .../30-libpython3.7_3.7.3-2+deb10u3_amd64.deb ...
Unpacking libpython3.7:amd64 (3.7.3-2+deb10u3) ...
Selecting previously unselected package librdmacm1:amd64.
Preparing to unpack .../31-librdmacm1_22.1-1_amd64.deb ...
Unpacking librdmacm1:amd64 (22.1-1) ...
Selecting previously unselected package liburcu6:amd64.
Preparing to unpack .../32-liburcu6_0.10.2-1_amd64.deb ...
Unpacking liburcu6:amd64 (0.10.2-1) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../33-libicu63_63.1-6+deb10u3_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u3) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../34-libxml2_2.9.4+dfsg1-7+deb10u3_amd64.deb ...
Unpacking libxml2:amd64 (2.9.4+dfsg1-7+deb10u3) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../35-libc-dev-bin_2.28-10+deb10u1_amd64.deb ...
Unpacking libc-dev-bin (2.28-10+deb10u1) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../36-linux-libc-dev_4.19.235-1_amd64.deb ...
Unpacking linux-libc-dev:amd64 (4.19.235-1) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../37-libc6-dev_2.28-10+deb10u1_amd64.deb ...
Unpacking libc6-dev:amd64 (2.28-10+deb10u1) ...
Selecting previously unselected package libattr1-dev:amd64.
Preparing to unpack .../38-libattr1-dev_1%3a2.4.48-4_amd64.deb ...
Unpacking libattr1-dev:amd64 (1:2.4.48-4) ...
Selecting previously unselected package libacl1-dev:amd64.
Preparing to unpack .../39-libacl1-dev_2.2.53-4_amd64.deb ...
Unpacking libacl1-dev:amd64 (2.2.53-4) ...
Selecting previously unselected package libglusterfs-dev.
Preparing to unpack .../40-libglusterfs-dev_5.5-3_amd64.deb ...
Unpacking libglusterfs-dev (5.5-3) ...
Selecting previously unselected package python3-prettytable.
Preparing to unpack .../41-python3-prettytable_0.7.2-4_all.deb ...
Unpacking python3-prettytable (0.7.2-4) ...
Selecting previously unselected package python3-certifi.
Preparing to unpack .../42-python3-certifi_2018.8.24-1_all.deb ...
Unpacking python3-certifi (2018.8.24-1) ...
Selecting previously unselected package python3-pkg-resources.
Preparing to unpack .../43-python3-pkg-resources_40.8.0-1_all.deb ...
Unpacking python3-pkg-resources (40.8.0-1) ...
Selecting previously unselected package python3-chardet.
Preparing to unpack .../44-python3-chardet_3.0.4-3_all.deb ...
Unpacking python3-chardet (3.0.4-3) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../45-python3-idna_2.6-1_all.deb ...
Unpacking python3-idna (2.6-1) ...
Selecting previously unselected package python3-six.
Preparing to unpack .../46-python3-six_1.12.0-1_all.deb ...
Unpacking python3-six (1.12.0-1) ...
Selecting previously unselected package python3-urllib3.
Preparing to unpack .../47-python3-urllib3_1.24.1-1_all.deb ...
Unpacking python3-urllib3 (1.24.1-1) ...
Selecting previously unselected package python3-requests.
Preparing to unpack .../48-python3-requests_2.21.0-1_all.deb ...
Unpacking python3-requests (2.21.0-1) ...
Selecting previously unselected package python3-jwt.
Preparing to unpack .../49-python3-jwt_1.7.0-2_all.deb ...
Unpacking python3-jwt (1.7.0-2) ...
Selecting previously unselected package libreadline5:amd64.
Preparing to unpack .../50-libreadline5_5.2+dfsg-3+b13_amd64.deb ...
Unpacking libreadline5:amd64 (5.2+dfsg-3+b13) ...
Selecting previously unselected package xfsprogs.
Preparing to unpack .../51-xfsprogs_4.20.0-1_amd64.deb ...
Unpacking xfsprogs (4.20.0-1) ...
Selecting previously unselected package glusterfs-common.
Preparing to unpack .../52-glusterfs-common_5.5-3_amd64.deb ...
Unpacking glusterfs-common (5.5-3) ...
Selecting previously unselected package glusterfs-client.
Preparing to unpack .../53-glusterfs-client_5.5-3_amd64.deb ...
Unpacking glusterfs-client (5.5-3) ...
Selecting previously unselected package glusterfs-server.
Preparing to unpack .../54-glusterfs-server_5.5-3_amd64.deb ...
Unpacking glusterfs-server (5.5-3) ...
Selecting previously unselected package ibverbs-providers:amd64.
Preparing to unpack .../55-ibverbs-providers_22.1-1_amd64.deb ...
Unpacking ibverbs-providers:amd64 (22.1-1) ...
Selecting previously unselected package keyutils.
Preparing to unpack .../56-keyutils_1.6-6_amd64.deb ...
Unpacking keyutils (1.6-6) ...
Selecting previously unselected package libevent-2.1-6:amd64.
Preparing to unpack .../57-libevent-2.1-6_2.1.8-stable-4_amd64.deb ...
Unpacking libevent-2.1-6:amd64 (2.1.8-stable-4) ...
Selecting previously unselected package libnfsidmap2:amd64.
Preparing to unpack .../58-libnfsidmap2_0.25-5.1_amd64.deb ...
Unpacking libnfsidmap2:amd64 (0.25-5.1) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../59-libsasl2-modules_2.1.27+dfsg-1+deb10u2_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27+dfsg-1+deb10u2) ...
Selecting previously unselected package libwrap0:amd64.
Preparing to unpack .../60-libwrap0_7.6.q-28_amd64.deb ...
Unpacking libwrap0:amd64 (7.6.q-28) ...
Selecting previously unselected package manpages-dev.
Preparing to unpack .../61-manpages-dev_4.16-2_all.deb ...
Unpacking manpages-dev (4.16-2) ...
Selecting previously unselected package rpcbind.
Preparing to unpack .../62-rpcbind_1.2.5-0.3+deb10u1_amd64.deb ...
Unpacking rpcbind (1.2.5-0.3+deb10u1) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../63-nfs-common_1%3a1.3.4-2.5+deb10u1_amd64.deb ...
Unpacking nfs-common (1:1.3.4-2.5+deb10u1) ...
Selecting previously unselected package python3-asn1crypto.
Preparing to unpack .../64-python3-asn1crypto_0.24.0-1_all.deb ...
Unpacking python3-asn1crypto (0.24.0-1) ...
Selecting previously unselected package python3-cffi-backend.
Preparing to unpack .../65-python3-cffi-backend_1.12.2-1_amd64.deb ...
Unpacking python3-cffi-backend (1.12.2-1) ...
Selecting previously unselected package python3-cryptography.
Preparing to unpack .../66-python3-cryptography_2.6.1-3+deb10u2_amd64.deb ...
Unpacking python3-cryptography (2.6.1-3+deb10u2) ...
Setting up mime-support (3.62) ...
Setting up libmagic-mgc (1:5.35-4+deb10u2) ...
Setting up attr (1:2.4.48-4) ...
Setting up manpages (4.16-2) ...
Setting up libtirpc-common (1.1.4-0.4) ...
Setting up libsqlite3-0:amd64 (3.27.2-3+deb10u1) ...
Setting up libsasl2-modules:amd64 (2.1.27+dfsg-1+deb10u2) ...
Setting up libmagic1:amd64 (1:5.35-4+deb10u2) ...
Setting up linux-libc-dev:amd64 (4.19.235-1) ...
Setting up file (1:5.35-4+deb10u2) ...
Setting up libfuse2:amd64 (2.9.9-1+deb10u1) ...
Setting up bzip2 (1.0.6-9.2~deb10u1) ...
Setting up libldap-common (2.4.47+dfsg-3+deb10u7) ...
Setting up libicu63:amd64 (63.1-6+deb10u3) ...
Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u2) ...
Setting up libwrap0:amd64 (7.6.q-28) ...
Setting up xz-utils (5.2.4-1+deb10u1) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
Setting up libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u2) ...
Setting up libevent-2.1-6:amd64 (2.1.8-stable-4) ...
Setting up keyutils (1.6-6) ...
Setting up sensible-utils (0.0.12) ...
Setting up liburcu6:amd64 (0.10.2-1) ...
Setting up libnl-3-200:amd64 (3.4.0-1) ...
Setting up libmpdec2:amd64 (2.4.2-2) ...
Setting up libaio1:amd64 (0.3.112-3) ...
Setting up libc-dev-bin (2.28-10+deb10u1) ...
Setting up openssl (1.1.1n-0+deb10u2) ...
Setting up readline-common (7.0-5) ...
Setting up libxml2:amd64 (2.9.4+dfsg1-7+deb10u3) ...
Setting up libreadline7:amd64 (7.0-5) ...
Setting up libtirpc3:amd64 (1.1.4-0.4) ...
Setting up fuse (2.9.9-1+deb10u1) ...
Setting up manpages-dev (4.16-2) ...
Setting up libpython3.7-stdlib:amd64 (3.7.3-2+deb10u3) ...
Setting up libreadline5:amd64 (5.2+dfsg-3+b13) ...
Setting up libpython3.7:amd64 (3.7.3-2+deb10u3) ...
Setting up libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u7) ...
Setting up rpcbind (1.2.5-0.3+deb10u1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rpcbind.service → /lib/systemd/system/rpcbind.service.
Created symlink /etc/systemd/system/sockets.target.wants/rpcbind.socket → /lib/systemd/system/rpcbind.socket.
Setting up libnl-route-3-200:amd64 (3.4.0-1) ...
Setting up libpython2.7-stdlib:amd64 (2.7.16-2+deb10u1) ...
Setting up libglusterfs0:amd64 (5.5-3) ...
Setting up ca-certificates (20200601~deb10u2) ...
Updating certificates in /etc/ssl/certs...
137 added, 0 removed; done.
Setting up ucf (3.0038+nmu1) ...
Setting up libc6-dev:amd64 (2.28-10+deb10u1) ...
Setting up libnfsidmap2:amd64 (0.25-5.1) ...
Setting up libpython3-stdlib:amd64 (3.7.3-1) ...
Setting up libgfxdr0:amd64 (5.5-3) ...
Setting up python3.7 (3.7.3-2+deb10u3) ...
Setting up libibverbs1:amd64 (22.1-1) ...
Setting up libattr1-dev:amd64 (1:2.4.48-4) ...
Setting up python2.7 (2.7.16-2+deb10u1) ...
Setting up ibverbs-providers:amd64 (22.1-1) ...
Setting up libpython2-stdlib:amd64 (2.7.16-1) ...
Setting up libgfdb0:amd64 (5.5-3) ...
Setting up python3 (3.7.3-1) ...
running python rtupdate hooks for python3.7...
running python post-rtupdate hooks for python3.7...
Setting up python2 (2.7.16-1) ...
Setting up nfs-common (1:1.3.4-2.5+deb10u1) ...
Creating config file /etc/idmapd.conf with new version
Adding system user `statd' (UID 106) ...
Adding new user `statd' (UID 106) with group `nogroup' ...
Not creating home directory `/var/lib/nfs'.
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-client.target → /lib/systemd/system/nfs-client.target.
Created symlink /etc/systemd/system/remote-fs.target.wants/nfs-client.target → /lib/systemd/system/nfs-client.target.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up python3-six (1.12.0-1) ...
Setting up libpython-stdlib:amd64 (2.7.16-1) ...
Setting up python3-certifi (2018.8.24-1) ...
Setting up python3-idna (2.6-1) ...
Setting up xfsprogs (4.20.0-1) ...
Setting up python3-urllib3 (1.24.1-1) ...
Setting up python3-prettytable (0.7.2-4) ...
Setting up python (2.7.16-1) ...
Setting up python3-asn1crypto (0.24.0-1) ...
Setting up libgfrpc0:amd64 (5.5-3) ...
Setting up python3-cffi-backend (1.12.2-1) ...
Setting up libacl1-dev:amd64 (2.2.53-4) ...
Setting up python3-pkg-resources (40.8.0-1) ...
Setting up librdmacm1:amd64 (22.1-1) ...
Setting up python3-jwt (1.7.0-2) ...
Setting up libgfchangelog0:amd64 (5.5-3) ...
Setting up python3-chardet (3.0.4-3) ...
Setting up python3-cryptography (2.6.1-3+deb10u2) ...
Setting up python3-requests (2.21.0-1) ...
Setting up libgfapi0:amd64 (5.5-3) ...
Setting up libglusterfs-dev (5.5-3) ...
Setting up glusterfs-common (5.5-3) ...
Adding group `gluster' (GID 109) ...
Done.
Setting up glusterfs-client (5.5-3) ...
Setting up glusterfs-server (5.5-3) ...
glusterd.service is a disabled or a static unit, not starting it.
glustereventsd.service is a disabled or a static unit, not starting it.
Processing triggers for systemd (241-7~deb10u8) ...
Processing triggers for libc-bin (2.28-10+deb10u1) ...
Processing triggers for ca-certificates (20200601~deb10u2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Step 2 - Start Glusterd on All Nodes
systemctl start glusterd
#enable on boot too or you will find your volumes do not work by themselves
systemctl enable glusterd
Step 3 - Connect Gluster Nodes
gluster1 IP = 10.13.132.79
gluster2 IP = 10.13.132.68
gluster3 IP = 10.13.132.21
On gluster1:
#connect to server 2
gluster peer probe 10.13.132.68
#connect to server 3
gluster peer probe 10.13.132.21
You should see this after each peer probe:
peer probe: success.
On gluster2:
#connect to server 1
gluster peer probe 10.13.132.79
You should see this after each peer probe:
peer probe: success.
On gluster3:
#connect to server 1
gluster peer probe 10.13.132.79
You should see this after each peer probe:
peer probe: success.
Check the peer status from gluster1 to make sure all is well:
gluster peer status Number of Peers: 2
Hostname: 10.13.132.68
Uuid: 5b34c83d-489d-4981-9c59-ac991e1a014f
State: Peer in Cluster (Connected)
Hostname: 10.13.132.21
Uuid: 19e86290-3632-4e4f-9f74-4124bd61c6a0
State: Peer in Cluster (Connected)
Step 3 - Create your first Gluster Volume aka Make Bricks!
This can be done on any member of the Gluster cluster
mkdir -p /rttgluster/realtechtalkVolume/brick0
Format is like this:
gluster volume create VolumeName replica NumberOfServers IP:/VolumePath IP:/VolumePath IP:/VolumePath
Example based on the IPs in this blog and the /rttgluster directory as a volume location
gluster volume create realtechtalkVolume replica 3 10.13.132.79:/rttgluster/realtechtalkVolume/brick0 10.13.132.68:/rttgluster/realtechtalkVolume/brick0 10.13.132.21:/rttgluster/realtechtalkVolume/brick0
Step 4 - Start The Volume
gluster volume start realtechtalkVolume
You should see a success message and then do a "gluster volume info" and it should show the 3 nodes, bricks, and "Status: Started"
If the status is "Created" then you probably didn't do the gluster volume start like above.
Did you get an error when creating the volume?
If you are using a directory on the / root partition, it will complain as it's not recommended, but if you want to force it, then just add the "force" at the end and it will create:
Volume: failed: The brick 10.13.132.79:/rttgluster is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior.
gluster volume create realtechtalkVolume replica 3 10.13.132.79:/rttgluster/realtechtalkVolume/brick0 10.13.132.68:/rttgluster/realtechtalkVolume/brick0 10.13.132.21:/rttgluster/realtechtalkVolume/brick0 force
Step 5 - Let's use our gluster volume!
Let's move into our gluster volume in /rttgluster/realtechtalkVolume/brick0 and create a directory on any node (in our case gluster01).
Does it exist on the other nodes?
Oops, it looks like it doesn't quite work this way, files will appear in the brick0 dir but you cannot use it directly as you have to mount it using the client side "mount utility" like this:
Use the "volumename" of your volume if it is not called realtechtalkVolume
root@gluster02:~# mount -t glusterfs 10.13.132.79:realtechtalkVolume /gluster/
root@gluster02:~# cd /gluster/
root@gluster02:/gluster# ls
root@gluster02:/gluster# mkdir realtechtalkGlusterTest!
Now you'll see it does exist on gluster01's brick0 dir:
Making it permanent / automounting the Gluster Volume upon Boot
*Make sure that glusterd is enabled for bootup, otherwise this will fail until you manually start glusterd
systemctl enable glusterd
You will likely want the volumes to be mounted and survive a reboot, so you'll need to edit fstab on each host.
localhost:/realtechtalkVolume /gluster glusterfs defaults,_netdev 0 0
In our case, above, we are mounting with localhost (no need to specify an IP) since each server is part of the gluster volume.
The /realtechtalkVolume is the volume name and /gluster is the location we are going to mount to (I recommend keeping the mount location consistent across the nodes).
Tags:
glusterfs, howto, tutorial, distributed, docker, kubernetes, lxc, kvm, proxmoxthis, gluster, userspace, fuse, directory, applications, redundant, scaled, proxmox, openstack, etc, database, nodes, install, apt, server, http, deb, debian, org, bullseye, inrelease, kb, updates, amd, packages, fetched, lists, dependency, additional, installed, attr, bzip, ca, certificates, ibverbs, providers, keyutils, libacl, dev, libaio, libattr, libc, bin, libevent, libfuse, libgfapi, libgfchangelog, libgfdb, libgfrpc, libgfxdr, libglusterfs, libibverbs, libicu, libldap, libmagic, mgc, libmpdec, libnfsidmap, libnl, libpython, stdlib, minimal, librdmacm, libreadline, libsasl, modules, db, libsqlite, libtirpc, liburcu, libwrap, libxml, linux, manpages, mime, nfs, openssl, python, asn, crypto, certifi, cffi, backend, chardet, cryptography, idna, jwt, pkg, prettytable, requests, urllib, readline, rpcbind, sensible, utils, ucf, xfsprogs, xz, glibc, gssapi, mit, heimdal, ldap, otp, sql, browser, iscsi, watchdog, tk, binutils, binfmt, venv, vectors, setuptools, xfsdump, acl, quota, upgraded, newly, mb, archives, disk, dfsg, nmu, debconf, delaying, configuration, selecting, previously, unselected, directories, currently, preparing, unpack, minimal_, _amd, unpacking, support_, _all, common_, _, stdlib_, linking, byte, compiling, runtime, python_, utils_, mgc_, file_, db_, manpages_, ucf_, attr_, openssl_, certificates_, fuse_, bin_, dev_, prettytable_, certifi_, resources_, chardet_, idna_, six_, requests_, jwt_, xfsprogs_, client_, server_, providers_, keyutils_, modules_, rpcbind_, crypto_, backend_, cryptography_, update, alternatives, usr, lzma, auto, mode, symlink, systemd, multi, user, rarr, lib, sockets, socket, updating, ssl, certs, rtupdate, hooks, creating, config, idmapd, conf, adding, statd, uid, nogroup, var, fs, disabled, static, gid, glusterd, glustereventsd, processing, triggers, systemctl, ip, peer, probe, peers, hostname, uuid, ac, fstate, cluster, bd, volume, aka, bricks, mkdir, rttgluster, realtechtalkvolume, format, volumename, replica, numberofservers, volumepath, ips, blog, quot, info, didn, partition, recommended, override, node, doesn, dir, mount, utility, ls, realtechtalkglustertest, ll, automounting, volumes, mounted, reboot, edit, fstab, localhost, defaults, _netdev,