I was trying to upgrade the node according to this 'https://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/' but glusterd won't start, so there is no way to upgrade the volume :-/
I get the following error (/var/log/glusterfs/glusterd.log):
$this->bbcode_second_pass_code('', '
[2017-05-31 21:50:52.036026] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-31 21:50:52.319159] W [MSGID: 101095] [xlator.c:213:xlator_dynload] 0-xlator: libicui18n.so.58: cannot open shared object file: No such file or directory
[2017-05-31 21:50:52.319492] E [MSGID: 101002] [graph.y:213:volume_type] 0-parser: Volume 'management', line 2: type 'mgmt/glusterd' is not valid or not found on this machine
[2017-05-31 21:50:52.319915] E [MSGID: 101019] [graph.y:321:volume_end] 0-parser: "type" not specified for volume management
[2017-05-31 21:50:52.320180] E [MSGID: 100026] [glusterfsd.c:2308:glusterfs_process_volfp] 0-: failed to construct the graph
[2017-05-31 21:50:52.321072] E [MSGID: 101172] [graph.c:1024:glusterfs_graph_destroy] 0-graph: invalid argument: graph [Invalid argument]
[2017-05-31 21:50:52.321468] W [MSGID: 100032] [glusterfsd.c:1329:cleanup_and_exit] 0-: received signum (1), shutting down
')
I am sure it has something to do with this: "libicui18n.so.58" being missing as "icu 59.1-1" is current and apparently used by "3.9.1".
Just in case, here is the configuration and what things look like when 3.9.1 starts:
I haven't changed anything in '/etc/glusterfs' and 'glusterd.vol' isn't any different between 3.9 and 3.10:
$this->bbcode_second_pass_code('', '
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option ping-timeout 0
option event-threads 1
# option transport.address-family inet6
# option base-port 49152
end-volume
')
After downgrading back to 3.9.1 the server comes up without a hitch (well, with the usual moans)
$this->bbcode_second_pass_code('', '
[2017-05-31 22:07:02.202376] I [MSGID: 100030] [glusterfsd.c:2455:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.9.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-31 22:07:02.248316] I [MSGID: 106478] [glusterd.c:1379:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-05-31 22:07:02.248602] I [MSGID: 106479] [glusterd.c:1426:init] 0-management: Using /var/lib/glusterd as working directory
[2017-05-31 22:07:02.278633] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib/glusterfs/3.9.1/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-05-31 22:07:02.278826] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-05-31 22:07:02.278921] W [rpcsvc.c:1632:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-05-31 22:07:02.279018] E [MSGID: 106243] [glusterd.c:1650:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-05-31 22:07:25.209686] I [MSGID: 106513] [glusterd-store.c:2100:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30710
[2017-05-31 22:07:25.781340] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
[2017-05-31 22:07:26.063004] I [MSGID: 106498] [glusterd-handler.c:3662:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-31 22:07:26.102475] I [rpc-clnt.c:1045:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 22:07:26.144725] I [rpc-clnt.c:1045:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.socket.listen-backlog 128
8: option event-threads 1
9: option ping-timeout 0
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2017-05-31 22:07:26.102200] I [MSGID: 106498] [glusterd-handler.c:3662:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
...
')