[pve-devel] applied: [PATCH ha-manager 1/3] add 'migrate' node shutdown policy

Thomas Lamprecht t.lamprecht at proxmox.com
Mon Nov 25 19:49:11 CET 2019


This adds handling for a new shutdown policy, namely "migrate".
If that is set then the LRM doesn't queues stop jobs, but transitions
to a new mode, namely 'maintenance'.

The LRM modes now get passed from the CRM in the NodeStatus update
method, this allows to detect such a mode and make node-status state
transitions. Effectively we only allow to transition if we're
currently online, else this is ignored. 'maintenance' does not
protects from fencing.

The moving then gets done by select service node. A node in
maintenance mode is not in "list_online_nodes" and so also not in
online_node_usage used to re-calculate if a service needs to be
moved. Only started services will get moved, this can be done almost
by leveraging exiting behavior, the next_state_started FSM state
transition method just needs to be thought to not early return for
nodes which are not online but in maintenance mode.

A few tests get adapted from the other policy tests is added to
showcase behavior with reboot, shutdown, and shutdown of the current
manager. It also shows the behavior when a service cannot be
migrated, albeit as our test system is limited to simulate maximal 9
migration failures, it "seems" to succeed after that. But note here
that the maximal retries would have been hit way more earlier, so
this is just artifact from our test system.

Besides some implementation details two question still are not solved
by this approach:
* what if a service cannot be moved away, either by errors or as no
  alternative node is found by select_service_node
  - retrying indefinitely, this happens currently. The user set this
    up like this in the first place. We will order SSH, pveproxy,
    after the LRM service to ensure that the're still the possibility
    for manual interventions
  - a idea would be to track the time and see if we're stuck (this is
    not to hard), in such a case we could stop the services after X
    minutes and continue.
* a full cluster shutdown, but that is even without this mode not to
  ideal, nodes will get fenced after no partition is quorate anymore,
  already. And as long as it's just a central setting in DC config,
  an admin has a single switch to flip to make it work, so not sure
  how much handling we want to do here, if we go over the point where
  we have no quorum we're dead anyhow, soo.. at least not really an
  issue of this series, orthogonal related yes, but not more.

For real world usability the datacenter.cfg schema needs to be
changed to allow the migrate shutdown policy, but that's trivial

Signed-off-by: Thomas Lamprecht <t.lamprecht at proxmox.com>
---
 src/PVE/HA/LRM.pm                             |  61 +++++++++-
 src/PVE/HA/Manager.pm                         |  19 ++--
 src/PVE/HA/NodeStatus.pm                      |  19 +++-
 .../cmdlist                                   |   4 +
 .../datacenter.cfg                            |   5 +
 .../hardware_status                           |   5 +
 .../log.expect                                | 106 ++++++++++++++++++
 .../manager_status                            |   1 +
 .../service_config                            |   4 +
 src/test/test-shutdown-policy3/cmdlist        |   4 +
 src/test/test-shutdown-policy3/datacenter.cfg |   5 +
 .../test-shutdown-policy3/hardware_status     |   5 +
 src/test/test-shutdown-policy3/log.expect     |  59 ++++++++++
 src/test/test-shutdown-policy3/manager_status |   1 +
 src/test/test-shutdown-policy3/service_config |   4 +
 src/test/test-shutdown-policy4/cmdlist        |   4 +
 src/test/test-shutdown-policy4/datacenter.cfg |   5 +
 .../test-shutdown-policy4/hardware_status     |   5 +
 src/test/test-shutdown-policy4/log.expect     |  55 +++++++++
 src/test/test-shutdown-policy4/manager_status |   1 +
 src/test/test-shutdown-policy4/service_config |   4 +
 src/test/test-shutdown-policy5/cmdlist        |   4 +
 src/test/test-shutdown-policy5/datacenter.cfg |   5 +
 .../test-shutdown-policy5/hardware_status     |   5 +
 src/test/test-shutdown-policy5/log.expect     |  58 ++++++++++
 src/test/test-shutdown-policy5/manager_status |   1 +
 src/test/test-shutdown-policy5/service_config |   4 +
 27 files changed, 440 insertions(+), 13 deletions(-)
 create mode 100644 src/test/test-shutdown-policy-migrate-fail1/cmdlist
 create mode 100644 src/test/test-shutdown-policy-migrate-fail1/datacenter.cfg
 create mode 100644 src/test/test-shutdown-policy-migrate-fail1/hardware_status
 create mode 100644 src/test/test-shutdown-policy-migrate-fail1/log.expect
 create mode 100644 src/test/test-shutdown-policy-migrate-fail1/manager_status
 create mode 100644 src/test/test-shutdown-policy-migrate-fail1/service_config
 create mode 100644 src/test/test-shutdown-policy3/cmdlist
 create mode 100644 src/test/test-shutdown-policy3/datacenter.cfg
 create mode 100644 src/test/test-shutdown-policy3/hardware_status
 create mode 100644 src/test/test-shutdown-policy3/log.expect
 create mode 100644 src/test/test-shutdown-policy3/manager_status
 create mode 100644 src/test/test-shutdown-policy3/service_config
 create mode 100644 src/test/test-shutdown-policy4/cmdlist
 create mode 100644 src/test/test-shutdown-policy4/datacenter.cfg
 create mode 100644 src/test/test-shutdown-policy4/hardware_status
 create mode 100644 src/test/test-shutdown-policy4/log.expect
 create mode 100644 src/test/test-shutdown-policy4/manager_status
 create mode 100644 src/test/test-shutdown-policy4/service_config
 create mode 100644 src/test/test-shutdown-policy5/cmdlist
 create mode 100644 src/test/test-shutdown-policy5/datacenter.cfg
 create mode 100644 src/test/test-shutdown-policy5/hardware_status
 create mode 100644 src/test/test-shutdown-policy5/log.expect
 create mode 100644 src/test/test-shutdown-policy5/manager_status
 create mode 100644 src/test/test-shutdown-policy5/service_config

diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm
index b5ef8b8..98466a2 100644
--- a/src/PVE/HA/LRM.pm
+++ b/src/PVE/HA/LRM.pm
@@ -16,6 +16,7 @@ use PVE::HA::Resources;
 my $valid_states = {
     wait_for_agent_lock => "waiting for agent lock",
     active => "got agent_lock",
+    maintenance => "going into maintenance",
     lost_agent_lock => "lost agent_lock",
 };
 
@@ -61,18 +62,27 @@ sub shutdown_request {
     }
 
     my $freeze_all;
+    my $maintenance;
     if ($shutdown_policy eq 'conditional') {
 	$freeze_all = $reboot;
     } elsif ($shutdown_policy eq 'freeze') {
 	$freeze_all = 1;
     } elsif ($shutdown_policy eq 'failover') {
 	$freeze_all = 0;
+    } elsif ($shutdown_policy eq 'migrate') {
+	$maintenance = 1;
     } else {
 	$haenv->log('err', "unknown shutdown policy '$shutdown_policy', fall back to conditional");
 	$freeze_all = $reboot;
     }
 
-    if ($shutdown) {
+    if ($maintenance) {
+	# we get marked as unaivalable by the manager, then all services will
+	# be migrated away, we'll still have the same "can we exit" clause than
+	# a normal shutdown -> no running service on this node
+	# FIXME: after X minutes, add shutdown command for remaining services,
+	# e.g., if they have no alternative node???
+    } elsif ($shutdown) {
 	# *always* queue stop jobs for all services if the node shuts down,
 	# independent if it's a reboot or a poweroff, else we may corrupt
 	# services or hinder node shutdown
@@ -89,7 +99,10 @@ sub shutdown_request {
 
     if ($shutdown) {
 	my $shutdown_type = $reboot ? 'reboot' : 'shutdown';
-	if ($freeze_all) {
+	if ($maintenance) {
+	    $haenv->log('info', "$shutdown_type LRM, doing maintenance, removing this node from active list");
+	    $self->{mode} = 'maintenance';
+	} elsif ($freeze_all) {
 	    $haenv->log('info', "$shutdown_type LRM, stop and freeze all services");
 	    $self->{mode} = 'restart';
 	} else {
@@ -101,7 +114,7 @@ sub shutdown_request {
 	$self->{mode} = 'restart';
     }
 
-    $self->{shutdown_request} = 1;
+    $self->{shutdown_request} = $haenv->get_time();
 
     eval { $self->update_lrm_status() or die "not quorate?\n"; };
     if (my $err = $@) {
@@ -300,6 +313,16 @@ sub work {
 	    $self->set_local_status({ state => 'lost_agent_lock'});
 	} elsif (!$self->get_protected_ha_agent_lock()) {
 	    $self->set_local_status({ state => 'lost_agent_lock'});
+	} elsif ($self->{mode} eq 'maintenance') {
+	    $self->set_local_status({ state => 'maintenance'});
+	}
+    } elsif ($state eq 'maintenance') {
+
+	if ($fence_request) {
+	    $haenv->log('err', "node need to be fenced during maintenance mode - releasing agent_lock\n");
+	    $self->set_local_status({ state => 'lost_agent_lock'});
+	} elsif (!$self->get_protected_ha_agent_lock()) {
+	    $self->set_local_status({ state => 'lost_agent_lock'});
 	}
     }
 
@@ -432,6 +455,38 @@ sub work {
 
 	$haenv->sleep(5);
 
+    } elsif ($state eq 'maintenance') {
+
+	my $startime = $haenv->get_time();
+	return if !$self->update_service_status();
+
+	# wait until all active services moved away
+	my $service_count = $self->active_service_count();
+
+	my $exit_lrm = 0;
+
+	if ($self->{shutdown_request}) {
+	    if ($service_count == 0 && $self->run_workers() == 0) {
+		if ($self->{ha_agent_wd}) {
+		    $haenv->watchdog_close($self->{ha_agent_wd});
+		    delete $self->{ha_agent_wd};
+		}
+
+		$exit_lrm = 1;
+
+		# restart with no or freezed services, release the lock
+		$haenv->release_ha_agent_lock();
+	    }
+	}
+
+	$self->manage_resources() if !$exit_lrm;
+
+	$self->update_lrm_status();
+
+	return 0 if $exit_lrm;
+
+	$haenv->sleep_until($startime + 5);
+
     } else {
 
 	die "got unexpected status '$state'\n";
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 3d09433..1f14754 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -369,15 +369,16 @@ sub manage {
 
     my ($haenv, $ms, $ns, $ss) = ($self->{haenv}, $self->{ms}, $self->{ns}, $self->{ss});
 
-    $ns->update($haenv->get_node_info());
-
-    if (!$ns->node_is_online($haenv->nodename())) {
-	$haenv->log('info', "master seems offline");
-	return;
-    }
-
+    my ($node_info) = $haenv->get_node_info();
     my ($lrm_results, $lrm_modes) = $self->read_lrm_status();
 
+    $ns->update($node_info, $lrm_modes);
+
+    if (!$ns->node_is_operational($haenv->nodename())) {
+	$haenv->log('info', "master seems offline");
+	return;
+    }
+
     my $sc = $haenv->read_service_config();
 
     $self->{groups} = $haenv->read_group_config(); # update
@@ -638,7 +639,9 @@ sub next_state_started {
 	if ($ns->node_is_offline_delayed($sd->{node})) {
 	    &$change_service_state($self, $sid, 'fence');
 	}
-	return;
+	if ($ns->get_node_state($sd->{node}) ne 'maintenance') {
+	    return;
+	}
     }
 
     if ($cd->{state} eq 'disabled' || $cd->{state} eq 'stopped') {
diff --git a/src/PVE/HA/NodeStatus.pm b/src/PVE/HA/NodeStatus.pm
index 8784110..9d58fa4 100644
--- a/src/PVE/HA/NodeStatus.pm
+++ b/src/PVE/HA/NodeStatus.pm
@@ -24,6 +24,7 @@ sub new {
 # possible node state:
 my $valid_node_states = {
     online => "node online and member of quorate partition",
+    maintenance => "node is a member of quorate partition but currently not able to do work",
     unknown => "not member of quorate partition, but possibly still running",
     fence => "node needs to be fenced",
     gone => "node vanished from cluster members list, possibly deleted"
@@ -38,6 +39,11 @@ sub get_node_state {
     return $self->{status}->{$node};
 }
 
+sub node_is_operational {
+    my ($self, $node) = @_;
+    return $self->node_is_online($node) || $self->get_node_state($node) eq 'maintenance';
+}
+
 sub node_is_online {
     my ($self, $node) = @_;
 
@@ -117,12 +123,13 @@ my $set_node_state = sub {
 };
 
 sub update {
-    my ($self, $node_info) = @_;
+    my ($self, $node_info, $lrm_modes) = @_;
 
     my $haenv = $self->{haenv};
 
     foreach my $node (sort keys %$node_info) {
 	my $d = $node_info->{$node};
+	my $lrm_mode = $lrm_modes->{$node} // 'unkown';
 	next if !$d->{online};
 
 	# record last time the node was online (required to implement fence delay)
@@ -131,11 +138,19 @@ sub update {
 	my $state = $self->get_node_state($node);
 
 	if ($state eq 'online') {
+	    if ($lrm_mode eq 'maintenance') {
+		#$haenv->log('info', "update node state maintance");
+		$set_node_state->($self, $node, 'maintenance');
+	    }
 	    # &$set_node_state($self, $node, 'online');
 	} elsif ($state eq 'unknown' || $state eq 'gone') {
 	    &$set_node_state($self, $node, 'online');
 	} elsif ($state eq 'fence') {
 	    # do nothing, wait until fenced
+	} elsif ($state eq 'maintenance') {
+	    if ($lrm_mode ne 'maintenance') {
+		$set_node_state->($self, $node, 'online');
+	    }
 	} else {
 	    die "detected unknown node state '$state";
 	}
@@ -149,7 +164,7 @@ sub update {
 
 	# node is not inside quorate partition, possibly not active
 
-	if ($state eq 'online') {
+	if ($state eq 'online' || $state eq 'maintenance') {
 	    &$set_node_state($self, $node, 'unknown');
 	} elsif ($state eq 'unknown') {
 
diff --git a/src/test/test-shutdown-policy-migrate-fail1/cmdlist b/src/test/test-shutdown-policy-migrate-fail1/cmdlist
new file mode 100644
index 0000000..8558351
--- /dev/null
+++ b/src/test/test-shutdown-policy-migrate-fail1/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on"],
+    [ "reboot node3" ]
+]
diff --git a/src/test/test-shutdown-policy-migrate-fail1/datacenter.cfg b/src/test/test-shutdown-policy-migrate-fail1/datacenter.cfg
new file mode 100644
index 0000000..de0bf81
--- /dev/null
+++ b/src/test/test-shutdown-policy-migrate-fail1/datacenter.cfg
@@ -0,0 +1,5 @@
+{
+    "ha": {
+        "shutdown_policy": "migrate"
+    }
+}
diff --git a/src/test/test-shutdown-policy-migrate-fail1/hardware_status b/src/test/test-shutdown-policy-migrate-fail1/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-shutdown-policy-migrate-fail1/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off" },
+  "node2": { "power": "off", "network": "off" },
+  "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-shutdown-policy-migrate-fail1/log.expect b/src/test/test-shutdown-policy-migrate-fail1/log.expect
new file mode 100644
index 0000000..79664c7
--- /dev/null
+++ b/src/test/test-shutdown-policy-migrate-fail1/log.expect
@@ -0,0 +1,106 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'fa:109' on node 'node3'
+info     20    node1/crm: adding new service 'vm:103' on node 'node3'
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service fa:109
+info     25    node3/lrm: service status fa:109 started
+info     25    node3/lrm: starting service vm:103
+info     25    node3/lrm: service status vm:103 started
+info    120      cmdlist: execute reboot node3
+info    120    node3/lrm: got shutdown request with shutdown policy 'migrate'
+info    120    node3/lrm: reboot LRM, doing maintenance, removing this node from active list
+info    120    node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info    120    node1/crm: relocate service 'fa:109' to node 'node1'
+info    120    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node1)
+info    120    node1/crm: migrate service 'vm:103' to node 'node1' (running)
+info    120    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node3, target = node1)
+info    125    node3/lrm: status change active => maintenance
+err     125    node3/lrm: service fa:109 not moved (migration error)
+info    125    node3/lrm: service vm:103 - start migrate to node 'node1'
+info    125    node3/lrm: service vm:103 - end migrate to node 'node1'
+err     140    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    140    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    140    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    140    node1/crm: relocate service 'fa:109' to node 'node2'
+info    140    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+info    141    node1/lrm: got lock 'ha_agent_node1_lock'
+info    141    node1/lrm: status change wait_for_agent_lock => active
+info    141    node1/lrm: starting service vm:103
+info    141    node1/lrm: service status vm:103 started
+err     145    node3/lrm: service fa:109 not moved (migration error)
+err     160    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    160    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    160    node1/crm: relocate service 'fa:109' to node 'node2'
+info    160    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+err     165    node3/lrm: service fa:109 not moved (migration error)
+err     180    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    180    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    180    node1/crm: relocate service 'fa:109' to node 'node2'
+info    180    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+err     185    node3/lrm: service fa:109 not moved (migration error)
+err     200    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    200    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    200    node1/crm: relocate service 'fa:109' to node 'node2'
+info    200    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+err     205    node3/lrm: service fa:109 not moved (migration error)
+err     220    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    220    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    220    node1/crm: relocate service 'fa:109' to node 'node2'
+info    220    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+err     225    node3/lrm: service fa:109 not moved (migration error)
+err     240    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    240    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    240    node1/crm: relocate service 'fa:109' to node 'node2'
+info    240    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+err     245    node3/lrm: service fa:109 not moved (migration error)
+err     260    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    260    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    260    node1/crm: relocate service 'fa:109' to node 'node2'
+info    260    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+err     265    node3/lrm: service fa:109 not moved (migration error)
+err     280    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    280    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    280    node1/crm: relocate service 'fa:109' to node 'node2'
+info    280    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+err     285    node3/lrm: service fa:109 not moved (migration error)
+err     300    node1/crm: service 'fa:109' - migration failed (exit code 1)
+info    300    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node3)
+info    300    node1/crm: relocate service 'fa:109' to node 'node2'
+info    300    node1/crm: service 'fa:109': state changed from 'started' to 'relocate'  (node = node3, target = node2)
+info    305    node3/lrm: service fa:109 - start relocate to node 'node2'
+info    305    node3/lrm: stopping service fa:109 (relocate)
+info    305    node3/lrm: service status fa:109 stopped
+info    305    node3/lrm: service fa:109 - end relocate to node 'node2'
+info    320    node1/crm: service 'fa:109': state changed from 'relocate' to 'started'  (node = node2)
+info    323    node2/lrm: got lock 'ha_agent_node2_lock'
+info    323    node2/lrm: status change wait_for_agent_lock => active
+info    323    node2/lrm: starting service fa:109
+info    323    node2/lrm: service status fa:109 started
+info    326    node3/lrm: exit (loop end)
+info    326       reboot: execute crm node3 stop
+info    325    node3/crm: server received shutdown request
+info    345    node3/crm: exit (loop end)
+info    345       reboot: execute power node3 off
+info    345       reboot: execute power node3 on
+info    345    node3/crm: status change startup => wait_for_quorum
+info    340    node3/lrm: status change startup => wait_for_agent_lock
+info    360    node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info    364    node3/crm: status change wait_for_quorum => slave
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-shutdown-policy-migrate-fail1/manager_status b/src/test/test-shutdown-policy-migrate-fail1/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-shutdown-policy-migrate-fail1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-shutdown-policy-migrate-fail1/service_config b/src/test/test-shutdown-policy-migrate-fail1/service_config
new file mode 100644
index 0000000..458bb5e
--- /dev/null
+++ b/src/test/test-shutdown-policy-migrate-fail1/service_config
@@ -0,0 +1,4 @@
+{
+    "vm:103": { "node": "node3", "state": "enabled" },
+    "fa:109": { "node": "node3", "state": "enabled" }
+}
diff --git a/src/test/test-shutdown-policy3/cmdlist b/src/test/test-shutdown-policy3/cmdlist
new file mode 100644
index 0000000..8558351
--- /dev/null
+++ b/src/test/test-shutdown-policy3/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on"],
+    [ "reboot node3" ]
+]
diff --git a/src/test/test-shutdown-policy3/datacenter.cfg b/src/test/test-shutdown-policy3/datacenter.cfg
new file mode 100644
index 0000000..de0bf81
--- /dev/null
+++ b/src/test/test-shutdown-policy3/datacenter.cfg
@@ -0,0 +1,5 @@
+{
+    "ha": {
+        "shutdown_policy": "migrate"
+    }
+}
diff --git a/src/test/test-shutdown-policy3/hardware_status b/src/test/test-shutdown-policy3/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-shutdown-policy3/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off" },
+  "node2": { "power": "off", "network": "off" },
+  "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-shutdown-policy3/log.expect b/src/test/test-shutdown-policy3/log.expect
new file mode 100644
index 0000000..6ecf211
--- /dev/null
+++ b/src/test/test-shutdown-policy3/log.expect
@@ -0,0 +1,59 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'ct:102' on node 'node3'
+info     20    node1/crm: adding new service 'vm:103' on node 'node3'
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service ct:102
+info     25    node3/lrm: service status ct:102 started
+info     25    node3/lrm: starting service vm:103
+info     25    node3/lrm: service status vm:103 started
+info    120      cmdlist: execute reboot node3
+info    120    node3/lrm: got shutdown request with shutdown policy 'migrate'
+info    120    node3/lrm: reboot LRM, doing maintenance, removing this node from active list
+info    120    node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info    120    node1/crm: relocate service 'ct:102' to node 'node1'
+info    120    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node3, target = node1)
+info    120    node1/crm: migrate service 'vm:103' to node 'node1' (running)
+info    120    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node3, target = node1)
+info    125    node3/lrm: status change active => maintenance
+info    125    node3/lrm: service ct:102 - start relocate to node 'node1'
+info    125    node3/lrm: stopping service ct:102 (relocate)
+info    125    node3/lrm: service status ct:102 stopped
+info    125    node3/lrm: service ct:102 - end relocate to node 'node1'
+info    125    node3/lrm: service vm:103 - start migrate to node 'node1'
+info    125    node3/lrm: service vm:103 - end migrate to node 'node1'
+info    140    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+info    140    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    141    node1/lrm: got lock 'ha_agent_node1_lock'
+info    141    node1/lrm: status change wait_for_agent_lock => active
+info    141    node1/lrm: starting service ct:102
+info    141    node1/lrm: service status ct:102 started
+info    141    node1/lrm: starting service vm:103
+info    141    node1/lrm: service status vm:103 started
+info    146    node3/lrm: exit (loop end)
+info    146       reboot: execute crm node3 stop
+info    145    node3/crm: server received shutdown request
+info    165    node3/crm: exit (loop end)
+info    165       reboot: execute power node3 off
+info    165       reboot: execute power node3 on
+info    165    node3/crm: status change startup => wait_for_quorum
+info    160    node3/lrm: status change startup => wait_for_agent_lock
+info    180    node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info    184    node3/crm: status change wait_for_quorum => slave
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-shutdown-policy3/manager_status b/src/test/test-shutdown-policy3/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-shutdown-policy3/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-shutdown-policy3/service_config b/src/test/test-shutdown-policy3/service_config
new file mode 100644
index 0000000..8ee94b5
--- /dev/null
+++ b/src/test/test-shutdown-policy3/service_config
@@ -0,0 +1,4 @@
+{
+    "vm:103": { "node": "node3", "state": "enabled" },
+    "ct:102": { "node": "node3", "state": "enabled" }
+}
diff --git a/src/test/test-shutdown-policy4/cmdlist b/src/test/test-shutdown-policy4/cmdlist
new file mode 100644
index 0000000..a86b9e2
--- /dev/null
+++ b/src/test/test-shutdown-policy4/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on"],
+    [ "shutdown node3" ]
+]
diff --git a/src/test/test-shutdown-policy4/datacenter.cfg b/src/test/test-shutdown-policy4/datacenter.cfg
new file mode 100644
index 0000000..de0bf81
--- /dev/null
+++ b/src/test/test-shutdown-policy4/datacenter.cfg
@@ -0,0 +1,5 @@
+{
+    "ha": {
+        "shutdown_policy": "migrate"
+    }
+}
diff --git a/src/test/test-shutdown-policy4/hardware_status b/src/test/test-shutdown-policy4/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-shutdown-policy4/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off" },
+  "node2": { "power": "off", "network": "off" },
+  "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-shutdown-policy4/log.expect b/src/test/test-shutdown-policy4/log.expect
new file mode 100644
index 0000000..2e31059
--- /dev/null
+++ b/src/test/test-shutdown-policy4/log.expect
@@ -0,0 +1,55 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'ct:102' on node 'node3'
+info     20    node1/crm: adding new service 'vm:103' on node 'node3'
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service ct:102
+info     25    node3/lrm: service status ct:102 started
+info     25    node3/lrm: starting service vm:103
+info     25    node3/lrm: service status vm:103 started
+info    120      cmdlist: execute shutdown node3
+info    120    node3/lrm: got shutdown request with shutdown policy 'migrate'
+info    120    node3/lrm: shutdown LRM, doing maintenance, removing this node from active list
+info    120    node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info    120    node1/crm: relocate service 'ct:102' to node 'node1'
+info    120    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node3, target = node1)
+info    120    node1/crm: migrate service 'vm:103' to node 'node1' (running)
+info    120    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node3, target = node1)
+info    125    node3/lrm: status change active => maintenance
+info    125    node3/lrm: service ct:102 - start relocate to node 'node1'
+info    125    node3/lrm: stopping service ct:102 (relocate)
+info    125    node3/lrm: service status ct:102 stopped
+info    125    node3/lrm: service ct:102 - end relocate to node 'node1'
+info    125    node3/lrm: service vm:103 - start migrate to node 'node1'
+info    125    node3/lrm: service vm:103 - end migrate to node 'node1'
+info    140    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+info    140    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    141    node1/lrm: got lock 'ha_agent_node1_lock'
+info    141    node1/lrm: status change wait_for_agent_lock => active
+info    141    node1/lrm: starting service ct:102
+info    141    node1/lrm: service status ct:102 started
+info    141    node1/lrm: starting service vm:103
+info    141    node1/lrm: service status vm:103 started
+info    146    node3/lrm: exit (loop end)
+info    146     shutdown: execute crm node3 stop
+info    145    node3/crm: server received shutdown request
+info    165    node3/crm: exit (loop end)
+info    165     shutdown: execute power node3 off
+info    180    node1/crm: node 'node3': state changed from 'maintenance' => 'unknown'
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-shutdown-policy4/manager_status b/src/test/test-shutdown-policy4/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-shutdown-policy4/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-shutdown-policy4/service_config b/src/test/test-shutdown-policy4/service_config
new file mode 100644
index 0000000..8ee94b5
--- /dev/null
+++ b/src/test/test-shutdown-policy4/service_config
@@ -0,0 +1,4 @@
+{
+    "vm:103": { "node": "node3", "state": "enabled" },
+    "ct:102": { "node": "node3", "state": "enabled" }
+}
diff --git a/src/test/test-shutdown-policy5/cmdlist b/src/test/test-shutdown-policy5/cmdlist
new file mode 100644
index 0000000..e84297f
--- /dev/null
+++ b/src/test/test-shutdown-policy5/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on"],
+    [ "shutdown node1" ]
+]
diff --git a/src/test/test-shutdown-policy5/datacenter.cfg b/src/test/test-shutdown-policy5/datacenter.cfg
new file mode 100644
index 0000000..de0bf81
--- /dev/null
+++ b/src/test/test-shutdown-policy5/datacenter.cfg
@@ -0,0 +1,5 @@
+{
+    "ha": {
+        "shutdown_policy": "migrate"
+    }
+}
diff --git a/src/test/test-shutdown-policy5/hardware_status b/src/test/test-shutdown-policy5/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-shutdown-policy5/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off" },
+  "node2": { "power": "off", "network": "off" },
+  "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-shutdown-policy5/log.expect b/src/test/test-shutdown-policy5/log.expect
new file mode 100644
index 0000000..15f67c2
--- /dev/null
+++ b/src/test/test-shutdown-policy5/log.expect
@@ -0,0 +1,58 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'ct:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node1'
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service ct:102
+info     21    node1/lrm: service status ct:102 started
+info     21    node1/lrm: starting service vm:103
+info     21    node1/lrm: service status vm:103 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info    120      cmdlist: execute shutdown node1
+info    120    node1/lrm: got shutdown request with shutdown policy 'migrate'
+info    120    node1/lrm: shutdown LRM, doing maintenance, removing this node from active list
+info    120    node1/crm: node 'node1': state changed from 'online' => 'maintenance'
+info    120    node1/crm: relocate service 'ct:102' to node 'node2'
+info    120    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    120    node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info    120    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info    121    node1/lrm: status change active => maintenance
+info    121    node1/lrm: service ct:102 - start relocate to node 'node2'
+info    121    node1/lrm: stopping service ct:102 (relocate)
+info    121    node1/lrm: service status ct:102 stopped
+info    121    node1/lrm: service ct:102 - end relocate to node 'node2'
+info    121    node1/lrm: service vm:103 - start migrate to node 'node2'
+info    121    node1/lrm: service vm:103 - end migrate to node 'node2'
+info    140    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node2)
+info    140    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node2)
+info    142    node1/lrm: exit (loop end)
+info    142     shutdown: execute crm node1 stop
+info    141    node1/crm: server received shutdown request
+info    143    node2/lrm: got lock 'ha_agent_node2_lock'
+info    143    node2/lrm: status change wait_for_agent_lock => active
+info    143    node2/lrm: starting service ct:102
+info    143    node2/lrm: service status ct:102 started
+info    143    node2/lrm: starting service vm:103
+info    143    node2/lrm: service status vm:103 started
+info    160    node1/crm: voluntary release CRM lock
+info    161    node1/crm: exit (loop end)
+info    161     shutdown: execute power node1 off
+info    161    node2/crm: got lock 'ha_manager_lock'
+info    161    node2/crm: status change slave => master
+info    161    node2/crm: node 'node1': state changed from 'maintenance' => 'unknown'
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-shutdown-policy5/manager_status b/src/test/test-shutdown-policy5/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-shutdown-policy5/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-shutdown-policy5/service_config b/src/test/test-shutdown-policy5/service_config
new file mode 100644
index 0000000..d37e5b3
--- /dev/null
+++ b/src/test/test-shutdown-policy5/service_config
@@ -0,0 +1,4 @@
+{
+    "vm:103": { "node": "node1", "state": "enabled" },
+    "ct:102": { "node": "node1", "state": "enabled" }
+}
-- 
2.20.1




More information about the pve-devel mailing list