[pve-devel] [PATCH ha-manager 3/3] relocate policy: try to avoid already failed nodes

Thomas Lamprecht t.lamprecht at proxmox.com
Thu Jul 14 14:49:08 CEST 2016


If the failure policy triggers more often than 2 times we used an
already tried node again, even if there where other untried nodes,
we then cycled between those two nodes, if the active service count
did not change.

This does not make real sense as when it failed to start on a node
a short time ago it probably will also fail now (e.g. storage is
offline), whereas an untried node may have the chance to be fully
able to start the service, which is our goal.

Fix that by excluding those already tried nodes from the top
priority node list in 'select_service_node' if there are other
possible nodes to try. If there isn't any left we delete the
last tried one, so that the service tries another one even if it was
already tried (we want to fulfill the relocation policy after all).
This is not ideal but our default relocation try value is set to 1,
so that would only happen if a user set it explicitly to an high(er)
value.

select_service_node gets called in two places:
* next_state_started: there we want to use this behaviour
* recover_fenced service: there the try_next is always false
  as we just want to select a node to recover, if a relocation
  policy is then needed it is the duty for next_state_started
  to do so.
So we are safe to do this witouth changing behaviour execpt the one
described above.

If all tries fail we place the service in the error state, the
tried nodes entry gets cleanup after an user triggers an error
recovery by disabeling the service, so the information of the tried
nodes stays in the manager status until then.

Signed-off-by: Thomas Lamprecht <t.lamprecht at proxmox.com>
---
 src/PVE/HA/Manager.pm                              | 26 ++++++++-
 src/test/test-relocate-policy-default-group/README |  7 +++
 .../test-relocate-policy-default-group/cmdlist     |  4 ++
 .../hardware_status                                |  5 ++
 .../test-relocate-policy-default-group/log.expect  | 53 +++++++++++++++++
 .../manager_status                                 |  1 +
 .../service_config                                 |  3 +
 src/test/test-relocate-policy1/README              |  4 ++
 src/test/test-relocate-policy1/cmdlist             |  4 ++
 src/test/test-relocate-policy1/hardware_status     |  5 ++
 src/test/test-relocate-policy1/log.expect          | 68 ++++++++++++++++++++++
 src/test/test-relocate-policy1/manager_status      | 42 +++++++++++++
 src/test/test-relocate-policy1/service_config      |  9 +++
 src/test/test-resource-failure6/log.expect         | 59 +++++++++++++++++++
 14 files changed, 287 insertions(+), 3 deletions(-)
 create mode 100644 src/test/test-relocate-policy-default-group/README
 create mode 100644 src/test/test-relocate-policy-default-group/cmdlist
 create mode 100644 src/test/test-relocate-policy-default-group/hardware_status
 create mode 100644 src/test/test-relocate-policy-default-group/log.expect
 create mode 100644 src/test/test-relocate-policy-default-group/manager_status
 create mode 100644 src/test/test-relocate-policy-default-group/service_config
 create mode 100644 src/test/test-relocate-policy1/README
 create mode 100644 src/test/test-relocate-policy1/cmdlist
 create mode 100644 src/test/test-relocate-policy1/hardware_status
 create mode 100644 src/test/test-relocate-policy1/log.expect
 create mode 100644 src/test/test-relocate-policy1/manager_status
 create mode 100644 src/test/test-relocate-policy1/service_config
 create mode 100644 src/test/test-resource-failure6/log.expect

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 92cce9d..25de980 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -71,7 +71,7 @@ sub flush_master_status {
 } 
 
 sub select_service_node {
-    my ($groups, $online_node_usage, $service_conf, $current_node, $try_next) = @_;
+    my ($groups, $online_node_usage, $service_conf, $current_node, $try_next, $tried_nodes) = @_;
 
     my $group = {};
     # add all online nodes to default group to allow try_next when no group set
@@ -119,6 +119,26 @@ sub select_service_node {
 
     my $top_pri = $pri_list[0];
 
+    # try to avoid nodes where the service failed already if we want to relocate
+    if ($try_next) {
+	# first check if we have any untried node left
+	my $i = 0;
+	foreach my $node (@$tried_nodes) {
+	    $i++ if $pri_groups->{$top_pri}->{$node};
+	}
+
+	if ($i < scalar(keys %{$pri_groups->{$top_pri}})){
+	    # we have another one left so delete the tried ones
+	    foreach my $node (@$tried_nodes) {
+		delete $pri_groups->{$top_pri}->{$node};
+	    }
+	} else {
+	    # no untried (and active) node left, so delete the current node,
+	    # we want to try another one after all
+	    delete $pri_groups->{$top_pri}->{$current_node};
+	}
+    }
+
     my @nodes = sort { 
 	$online_node_usage->{$a} <=> $online_node_usage->{$b} || $a cmp $b
     } keys %{$pri_groups->{$top_pri}};
@@ -652,8 +672,8 @@ sub next_state_started {
 		}
 	    }
 
-	    my $node = select_service_node($self->{groups}, $self->{online_node_usage}, 
-					   $cd, $sd->{node}, $try_next);
+	    my $node = select_service_node($self->{groups}, $self->{online_node_usage},
+					   $cd, $sd->{node}, $try_next, $tried_nodes);
 
 	    if ($node && ($sd->{node} ne $node)) {
 		if ($cd->{type} eq 'vm') {
diff --git a/src/test/test-relocate-policy-default-group/README b/src/test/test-relocate-policy-default-group/README
new file mode 100644
index 0000000..18ee13a
--- /dev/null
+++ b/src/test/test-relocate-policy-default-group/README
@@ -0,0 +1,7 @@
+Test relocate policy on services with no group.
+Service 'fa:130' fails three times to restart and has a 'max_restart' policy
+of 0, thus will be relocated after each start try.
+As it has no group configured all available nodes should get chosen for
+when relocating.
+As we allow to relocate twice but the service fails three times we place
+it in the error state after all tries where used and all nodes where visited
diff --git a/src/test/test-relocate-policy-default-group/cmdlist b/src/test/test-relocate-policy-default-group/cmdlist
new file mode 100644
index 0000000..8f06508
--- /dev/null
+++ b/src/test/test-relocate-policy-default-group/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on"],
+    [ "service fa:130 enabled" ]
+]
diff --git a/src/test/test-relocate-policy-default-group/hardware_status b/src/test/test-relocate-policy-default-group/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-relocate-policy-default-group/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off" },
+  "node2": { "power": "off", "network": "off" },
+  "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-relocate-policy-default-group/log.expect b/src/test/test-relocate-policy-default-group/log.expect
new file mode 100644
index 0000000..a1e6795
--- /dev/null
+++ b/src/test/test-relocate-policy-default-group/log.expect
@@ -0,0 +1,53 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'fa:130' on node 'node2'
+info     20    node1/crm: service 'fa:130': state changed from 'started' to 'request_stop'
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     24    node3/crm: status change wait_for_quorum => slave
+info     40    node1/crm: service 'fa:130': state changed from 'request_stop' to 'stopped'
+info    120      cmdlist: execute service fa:130 enabled
+info    120    node1/crm: service 'fa:130': state changed from 'stopped' to 'started'  (node = node2)
+info    123    node2/lrm: starting service fa:130
+warn    123    node2/lrm: unable to start service fa:130
+err     123    node2/lrm: unable to start service fa:130 on local node after 0 retries
+warn    140    node1/crm: starting service fa:130 on node 'node2' failed, relocating service.
+info    140    node1/crm: relocate service 'fa:130' to node 'node1'
+info    140    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node2, target = node1)
+info    143    node2/lrm: service fa:130 - start relocate to node 'node1'
+info    143    node2/lrm: service fa:130 - end relocate to node 'node1'
+info    160    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node1)
+info    161    node1/lrm: got lock 'ha_agent_node1_lock'
+info    161    node1/lrm: status change wait_for_agent_lock => active
+info    161    node1/lrm: starting service fa:130
+warn    161    node1/lrm: unable to start service fa:130
+err     161    node1/lrm: unable to start service fa:130 on local node after 0 retries
+warn    180    node1/crm: starting service fa:130 on node 'node1' failed, relocating service.
+info    180    node1/crm: relocate service 'fa:130' to node 'node3'
+info    180    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node1, target = node3)
+info    181    node1/lrm: service fa:130 - start relocate to node 'node3'
+info    181    node1/lrm: service fa:130 - end relocate to node 'node3'
+info    200    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node3)
+info    205    node3/lrm: got lock 'ha_agent_node3_lock'
+info    205    node3/lrm: status change wait_for_agent_lock => active
+info    205    node3/lrm: starting service fa:130
+warn    205    node3/lrm: unable to start service fa:130
+err     205    node3/lrm: unable to start service fa:130 on local node after 0 retries
+err     220    node1/crm: recovery policy for service fa:130 failed, entering error state. Tried nodes: node2, node1, node3
+info    220    node1/crm: service 'fa:130': state changed from 'started' to 'error'
+err     225    node3/lrm: service fa:130 is in an error state and needs manual intervention. Look up 'ERROR RECOVERY' in the documentation.
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-relocate-policy-default-group/manager_status b/src/test/test-relocate-policy-default-group/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-relocate-policy-default-group/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-relocate-policy-default-group/service_config b/src/test/test-relocate-policy-default-group/service_config
new file mode 100644
index 0000000..c3cc873
--- /dev/null
+++ b/src/test/test-relocate-policy-default-group/service_config
@@ -0,0 +1,3 @@
+{
+    "fa:130": { "node": "node2", "max_restart": "0", "max_relocate": "2"  }
+}
diff --git a/src/test/test-relocate-policy1/README b/src/test/test-relocate-policy1/README
new file mode 100644
index 0000000..f0f12fd
--- /dev/null
+++ b/src/test/test-relocate-policy1/README
@@ -0,0 +1,4 @@
+Test if relocate policy selects the lowest populated node in addition to
+only those which weren't tried yet.
+As node 1 has the most services it should get selected as last even if its
+name sorts before the other ones.
diff --git a/src/test/test-relocate-policy1/cmdlist b/src/test/test-relocate-policy1/cmdlist
new file mode 100644
index 0000000..d253427
--- /dev/null
+++ b/src/test/test-relocate-policy1/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "service fa:130 enabled" ]
+]
diff --git a/src/test/test-relocate-policy1/hardware_status b/src/test/test-relocate-policy1/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-relocate-policy1/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off" },
+  "node2": { "power": "off", "network": "off" },
+  "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-relocate-policy1/log.expect b/src/test/test-relocate-policy1/log.expect
new file mode 100644
index 0000000..d53b4f4
--- /dev/null
+++ b/src/test/test-relocate-policy1/log.expect
@@ -0,0 +1,68 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: adding new service 'fa:130' on node 'node3'
+info     20    node1/crm: service 'fa:130': state changed from 'started' to 'request_stop'
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:100
+info     21    node1/lrm: service status vm:100 started
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     21    node1/lrm: starting service vm:102
+info     21    node1/lrm: service status vm:102 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:103
+info     23    node2/lrm: service status vm:103 started
+info     23    node2/lrm: starting service vm:104
+info     23    node2/lrm: service status vm:104 started
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service vm:105
+info     25    node3/lrm: service status vm:105 started
+info     40    node1/crm: service 'fa:130': state changed from 'request_stop' to 'stopped'
+info    120      cmdlist: execute service fa:130 enabled
+info    120    node1/crm: service 'fa:130': state changed from 'stopped' to 'started'  (node = node3)
+info    125    node3/lrm: starting service fa:130
+warn    125    node3/lrm: unable to start service fa:130
+err     125    node3/lrm: unable to start service fa:130 on local node after 0 retries
+warn    140    node1/crm: starting service fa:130 on node 'node3' failed, relocating service.
+info    140    node1/crm: relocate service 'fa:130' to node 'node2'
+info    140    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+info    145    node3/lrm: service fa:130 - start relocate to node 'node2'
+info    145    node3/lrm: service fa:130 - end relocate to node 'node2'
+info    160    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node2)
+info    163    node2/lrm: starting service fa:130
+warn    163    node2/lrm: unable to start service fa:130
+err     163    node2/lrm: unable to start service fa:130 on local node after 0 retries
+warn    180    node1/crm: starting service fa:130 on node 'node2' failed, relocating service.
+info    180    node1/crm: relocate service 'fa:130' to node 'node1'
+info    180    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node2, target = node1)
+info    183    node2/lrm: service fa:130 - start relocate to node 'node1'
+info    183    node2/lrm: service fa:130 - end relocate to node 'node1'
+info    200    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node1)
+info    201    node1/lrm: starting service fa:130
+warn    201    node1/lrm: unable to start service fa:130
+err     201    node1/lrm: unable to start service fa:130 on local node after 0 retries
+warn    220    node1/crm: starting service fa:130 on node 'node1' failed, relocating service.
+info    220    node1/crm: relocate service 'fa:130' to node 'node3'
+info    220    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node1, target = node3)
+info    221    node1/lrm: service fa:130 - start relocate to node 'node3'
+info    221    node1/lrm: service fa:130 - end relocate to node 'node3'
+info    240    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node3)
+info    245    node3/lrm: starting service fa:130
+info    245    node3/lrm: service status fa:130 started
+info    260    node1/crm: relocation policy successful for 'fa:130', tried nodes: node3, node2, node1, node3
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-relocate-policy1/manager_status b/src/test/test-relocate-policy1/manager_status
new file mode 100644
index 0000000..8cce913
--- /dev/null
+++ b/src/test/test-relocate-policy1/manager_status
@@ -0,0 +1,42 @@
+{
+    "master_node": "node1",
+    "node_status": {
+        "node1": "online",
+        "node2": "online",
+        "node3": "online"
+    },
+    "relocate_tried_nodes": {},
+    "service_status": {
+        "vm:100": {
+            "node": "node1",
+            "state": "started",
+            "uid": "hSIUPNL/lBjgyU4svobXlg"
+        },
+        "vm:101": {
+            "node": "node1",
+            "state": "started",
+            "uid": "vLuiMIZ5KBKzDZv2bkYLvA"
+        },
+        "vm:102": {
+            "node": "node1",
+            "state": "started",
+            "uid": "COPzO9cc+8Z3lUbWn8zCHA"
+        },
+        "vm:103": {
+            "node": "node2",
+            "state": "started",
+            "uid": "iktXhI6tCi8X6h8wQS9Uyw"
+        },
+        "vm:104": {
+            "node": "node2",
+            "state": "started",
+            "uid": "ySWup2on+tY88hdfzS1ymg"
+        },
+        "vm:105": {
+            "node": "node3",
+            "state": "started",
+            "uid": "RGRR9EOAzALG5cVMeWiKWA"
+        }
+    },
+    "timestamp": 10
+}
diff --git a/src/test/test-relocate-policy1/service_config b/src/test/test-relocate-policy1/service_config
new file mode 100644
index 0000000..d9f1823
--- /dev/null
+++ b/src/test/test-relocate-policy1/service_config
@@ -0,0 +1,9 @@
+{
+    "vm:100": { "node": "node1", "state": "enabled" },
+    "vm:101": { "node": "node1", "state": "enabled" },
+    "vm:102": { "node": "node1", "state": "enabled" },
+    "vm:103": { "node": "node2", "state": "enabled" },
+    "vm:104": { "node": "node2", "state": "enabled" },
+    "vm:105": { "node": "node3", "state": "enabled" },
+    "fa:130": { "node": "node3", "max_restart": "0", "max_relocate": "3"  }
+}
diff --git a/src/test/test-resource-failure6/log.expect b/src/test/test-resource-failure6/log.expect
new file mode 100644
index 0000000..8e06ec7
--- /dev/null
+++ b/src/test/test-resource-failure6/log.expect
@@ -0,0 +1,59 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'fa:130' on node 'node2'
+info     20    node1/crm: service 'fa:130': state changed from 'started' to 'request_stop'
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     24    node3/crm: status change wait_for_quorum => slave
+info     40    node1/crm: service 'fa:130': state changed from 'request_stop' to 'stopped'
+info    120      cmdlist: execute service fa:130 enabled
+info    120    node1/crm: service 'fa:130': state changed from 'stopped' to 'started'  (node = node2)
+info    123    node2/lrm: starting service fa:130
+warn    123    node2/lrm: unable to start service fa:130
+err     123    node2/lrm: unable to start service fa:130 on local node after 0 retries
+warn    140    node1/crm: starting service fa:130 on node 'node2' failed, relocating service.
+info    140    node1/crm: relocate service 'fa:130' to node 'node1'
+info    140    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node2, target = node1)
+info    143    node2/lrm: service fa:130 - start relocate to node 'node1'
+info    143    node2/lrm: service fa:130 - end relocate to node 'node1'
+info    160    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node1)
+info    161    node1/lrm: got lock 'ha_agent_node1_lock'
+info    161    node1/lrm: status change wait_for_agent_lock => active
+info    161    node1/lrm: starting service fa:130
+warn    161    node1/lrm: unable to start service fa:130
+err     161    node1/lrm: unable to start service fa:130 on local node after 0 retries
+warn    180    node1/crm: starting service fa:130 on node 'node1' failed, relocating service.
+info    180    node1/crm: relocate service 'fa:130' to node 'node3'
+info    180    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node1, target = node3)
+info    181    node1/lrm: service fa:130 - start relocate to node 'node3'
+info    181    node1/lrm: service fa:130 - end relocate to node 'node3'
+info    200    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node3)
+info    205    node3/lrm: got lock 'ha_agent_node3_lock'
+info    205    node3/lrm: status change wait_for_agent_lock => active
+info    205    node3/lrm: starting service fa:130
+warn    205    node3/lrm: unable to start service fa:130
+err     205    node3/lrm: unable to start service fa:130 on local node after 0 retries
+warn    220    node1/crm: starting service fa:130 on node 'node3' failed, relocating service.
+info    220    node1/crm: relocate service 'fa:130' to node 'node1'
+info    220    node1/crm: service 'fa:130': state changed from 'started' to 'relocate'  (node = node3, target = node1)
+info    225    node3/lrm: service fa:130 - start relocate to node 'node1'
+info    225    node3/lrm: service fa:130 - end relocate to node 'node1'
+info    240    node1/crm: service 'fa:130': state changed from 'relocate' to 'started'  (node = node1)
+info    241    node1/lrm: starting service fa:130
+info    241    node1/lrm: service status fa:130 started
+info    260    node1/crm: relocation policy successful for 'fa:130', tried nodes: node2, node1, node3, node1
+info    720     hardware: exit simulation - done
-- 
2.1.4





More information about the pve-devel mailing list