glusterd fails to start after upgrade from 3.9.1 to 3.10.1

This forum is for discussion about general software issues.

glusterd fails to start after upgrade from 3.9.1 to 3.10.1

Postby redlichtie » Wed May 31, 2017 10:57 pm

I was trying to upgrade the node according to this 'https://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/' but glusterd won't start, so there is no way to upgrade the volume :-/

I get the following error (/var/log/glusterfs/glusterd.log):
$this->bbcode_second_pass_code('', '
[2017-05-31 21:50:52.036026] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-31 21:50:52.319159] W [MSGID: 101095] [xlator.c:213:xlator_dynload] 0-xlator: libicui18n.so.58: cannot open shared object file: No such file or directory
[2017-05-31 21:50:52.319492] E [MSGID: 101002] [graph.y:213:volume_type] 0-parser: Volume 'management', line 2: type 'mgmt/glusterd' is not valid or not found on this machine
[2017-05-31 21:50:52.319915] E [MSGID: 101019] [graph.y:321:volume_end] 0-parser: "type" not specified for volume management
[2017-05-31 21:50:52.320180] E [MSGID: 100026] [glusterfsd.c:2308:glusterfs_process_volfp] 0-: failed to construct the graph
[2017-05-31 21:50:52.321072] E [MSGID: 101172] [graph.c:1024:glusterfs_graph_destroy] 0-graph: invalid argument: graph [Invalid argument]
[2017-05-31 21:50:52.321468] W [MSGID: 100032] [glusterfsd.c:1329:cleanup_and_exit] 0-: received signum (1), shutting down
')

I am sure it has something to do with this: "libicui18n.so.58" being missing as "icu 59.1-1" is current and apparently used by "3.9.1".

Just in case, here is the configuration and what things look like when 3.9.1 starts:

I haven't changed anything in '/etc/glusterfs' and 'glusterd.vol' isn't any different between 3.9 and 3.10:
$this->bbcode_second_pass_code('', '
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option ping-timeout 0
option event-threads 1
# option transport.address-family inet6
# option base-port 49152
end-volume
')

After downgrading back to 3.9.1 the server comes up without a hitch (well, with the usual moans)
$this->bbcode_second_pass_code('', '
[2017-05-31 22:07:02.202376] I [MSGID: 100030] [glusterfsd.c:2455:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.9.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-05-31 22:07:02.248316] I [MSGID: 106478] [glusterd.c:1379:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-05-31 22:07:02.248602] I [MSGID: 106479] [glusterd.c:1426:init] 0-management: Using /var/lib/glusterd as working directory
[2017-05-31 22:07:02.278633] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib/glusterfs/3.9.1/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-05-31 22:07:02.278826] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-05-31 22:07:02.278921] W [rpcsvc.c:1632:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-05-31 22:07:02.279018] E [MSGID: 106243] [glusterd.c:1650:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-05-31 22:07:25.209686] I [MSGID: 106513] [glusterd-store.c:2100:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30710
[2017-05-31 22:07:25.781340] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
[2017-05-31 22:07:26.063004] I [MSGID: 106498] [glusterd-handler.c:3662:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-05-31 22:07:26.102475] I [rpc-clnt.c:1045:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-05-31 22:07:26.144725] I [rpc-clnt.c:1045:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.socket.listen-backlog 128
8: option event-threads 1
9: option ping-timeout 0
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2017-05-31 22:07:26.102200] I [MSGID: 106498] [glusterd-handler.c:3662:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
...
')
redlichtie
 
Posts: 3
Joined: Sat Sep 03, 2016 5:28 am

Re: glusterd fails to start after upgrade from 3.9.1 to 3.10

Postby WarheadsSE » Thu Jun 01, 2017 3:03 pm

$this->bbcode_second_pass_code('', '[2017-05-31 21:50:52.319492] E [MSGID: 101002] [graph.y:213:volume_type] 0-parser: Volume 'management', line 2: type 'mgmt/glusterd' is not valid or not found on this machine')

Sure ICU is an issue, but this error screams more important to me. It might be that ICU is preventing the proper recognition, but I can't count on that fact.

Seeing as ICU was last built on Apr 29, and glusterfs was May 31, I wonder what sub-dependency somehow is still linked to ICU 58
Core Developer
Remember: Arch Linux ARM is entirely community donation supported!
WarheadsSE
Developer
 
Posts: 6807
Joined: Mon Oct 18, 2010 2:12 pm

Re: glusterd fails to start after upgrade from 3.9.1 to 3.10

Postby redlichtie » Thu Jun 01, 2017 5:54 pm

I think that the configuration parser fails because the desired version of ICU isn't available.

The dependency seems a to be too strict as well. Can't a program just depend on ICU being present?
redlichtie
 
Posts: 3
Joined: Sat Sep 03, 2016 5:28 am


Return to General

Who is online

Users browsing this forum: No registered users and 9 guests