[PVE-User] Config/Status commands stopped to respond
Dmytro O. Redchuk
dor at volz.ua
Tue Jan 28 12:31:35 CET 2020
У вт., 28-го січ. 2020, о 11:13 Gianni Milo wrote:
> What's the output of 'journalctl -u pvestatd.service" ? How about 'pvesm
> status' ?
Sorry, here is it:
root at nd2:~# journalctl -u pvestatd.service
-- Logs begin at Sun 2019-11-03 02:00:04 EET, end at Tue 2020-01-28 13:17:01 EET. --
Jan 28 12:09:03 nd2 systemd[1]: Stopping PVE Status Daemon...
Jan 28 12:10:33 nd2 systemd[1]: pvestatd.service: Stopping timed out. Terminating.
Jan 28 12:12:03 nd2 systemd[1]: pvestatd.service: State 'stop-sigterm' timed out. Killing.
Jan 28 12:12:03 nd2 systemd[1]: pvestatd.service: Killing process 2738 (pvestatd) with signal SIGKILL.
Jan 28 12:12:03 nd2 systemd[1]: pvestatd.service: Main process exited, code=killed, status=9/KILL
Jan 28 12:12:03 nd2 systemd[1]: Stopped PVE Status Daemon.
Jan 28 12:12:03 nd2 systemd[1]: pvestatd.service: Unit entered failed state.
Jan 28 12:12:03 nd2 systemd[1]: pvestatd.service: Failed with result 'timeout'.
Jan 28 12:12:03 nd2 systemd[1]: Starting PVE Status Daemon...
Jan 28 12:13:34 nd2 systemd[1]: pvestatd.service: Start operation timed out. Terminating.
Jan 28 12:13:34 nd2 systemd[1]: Failed to start PVE Status Daemon.
Jan 28 12:13:34 nd2 systemd[1]: pvestatd.service: Unit entered failed state.
Jan 28 12:13:34 nd2 systemd[1]: pvestatd.service: Failed with result 'timeout'.
Jan 28 12:14:34 nd2 systemd[1]: Starting PVE Status Daemon...
Jan 28 12:16:04 nd2 systemd[1]: pvestatd.service: Start operation timed out. Terminating.
Jan 28 12:16:04 nd2 systemd[1]: Failed to start PVE Status Daemon.
Jan 28 12:16:04 nd2 systemd[1]: pvestatd.service: Unit entered failed state.
Jan 28 12:16:04 nd2 systemd[1]: pvestatd.service: Failed with result 'timeout'.
I've tried to restart,
then restarted pvedaemon and pveproxy, and tried to start pvestatd again.
pvesm hungs too, "zfs list" works ok.
And all VMs seem to be working, no any issue with their disks.
At least it looks like this.
> On Tue, 28 Jan 2020 at 10:35, Dmytro O. Redchuk via pve-user <
> pve-user at pve.proxmox.com> wrote:
>
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: "Dmytro O. Redchuk" <dor at volz.ua>
> > To: PVE User List <pve-user at pve.proxmox.com>
> > Cc:
> > Bcc:
> > Date: Tue, 28 Jan 2020 12:35:08 +0200
> > Subject: Re: [PVE-User] Config/Status commands stopped to respond
> > У вт., 28-го січ. 2020, о 10:26 Gianni Milo wrote:
> > > First thing that comes to my mind when having only 2 nodes in the cluster
> > > is that perhaps the cluster is not quorate ? I would check that first and
> > First node's corosync reports ok for quorum:
> >
> > root at nd1:~# pvecm status
> > Quorum information
> > ------------------
> > Date: Tue Jan 28 12:29:52 2020
> > Quorum provider: corosync_votequorum
> > Nodes: 2
> > Node ID: 0x00000001
> > Ring ID: 1/36
> > Quorate: Yes
> >
> > Votequorum information
> > ----------------------
> > Expected votes: 2
> > Highest expected: 2
> > Total votes: 2
> > Quorum: 2
> > Flags: Quorate
> >
> > Membership information
> > ----------------------
> > Nodeid Votes Name
> > 0x00000001 1 10.24.0.1 (local)
> > 0x00000002 1 10.24.0.2
> >
> >
> > So, for the moment, I did the following on that "partially failed" node
> > (their logs has been empty for today):
> >
> > 1. systemctl restart pvedaemon.service -- ok, status is OK
> > 2. systemctl restart pveproxy.service -- ok, status is OK
> > 3. systemctl restart pvestatd.service --FAILED, timeout in the log:
> > Jan 28 12:16:04 nd2 systemd[1]: pvestatd.service: Start operation timed
> > out. Terminating.
> >
> >
> > Is it be because of some hunged node/container/process or dead lock file?
> >
> > What else that could be?
> >
> > Thank you!
> >
> >
> > > maybe restart the related services...
> > >
> > > G.
> > >
> > >
> > > On Tue, 28 Jan 2020 at 09:40, Dmytro O. Redchuk via pve-user <
> > > pve-user at pve.proxmox.com> wrote:
> > >
> > > >
> > > >
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: "Dmytro O. Redchuk" <dor at volz.ua>
> > > > To: pve-user at pve.proxmox.com
> > > > Cc:
> > > > Bcc:
> > > > Date: Tue, 28 Jan 2020 11:40:04 +0200
> > > > Subject: Config/Status commands stopped to respond
> > > > Hi masters,
> > > >
> > > > I am running two-nodes cluster (PM v.5.3),
> > > > and today I've found that one node stopped to respond to config/status
> > > > commands --- VMs in GUI are gray and marked with question mark,
> > > > commands like "pvecm status" or "qm list" hung (until ^C).
> > > >
> > > > So, I can login with ssh into that node,
> > > > and all VMs seem to be working fine.
> > > >
> > > > Please, is it possible to get it working without any VMs/node restart?
> > > >
> > > > What have I do?
> > > >
> > > > Could not find (or missed) anything useful in logfiles.
> > > >
> > > > Thank you!
> > > >
> > > > --
> > > > Dmytro O. Redchuk
> > > >
> > > >
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: "Dmytro O. Redchuk via pve-user" <pve-user at pve.proxmox.com>
> > > > To: pve-user at pve.proxmox.com
> > > > Cc: "Dmytro O. Redchuk" <dor at volz.ua>
> > > > Bcc:
> > > > Date: Tue, 28 Jan 2020 11:40:04 +0200
> > > > Subject: [PVE-User] Config/Status commands stopped to respond
> > > > _______________________________________________
> > > > pve-user mailing list
> > > > pve-user at pve.proxmox.com
> > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > >
> > > _______________________________________________
> > > pve-user mailing list
> > > pve-user at pve.proxmox.com
> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> > --
> > Dmytro O. Redchuk
> > (+380) 44 2474832
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: "Dmytro O. Redchuk via pve-user" <pve-user at pve.proxmox.com>
> > To: PVE User List <pve-user at pve.proxmox.com>
> > Cc: "Dmytro O. Redchuk" <dor at volz.ua>
> > Bcc:
> > Date: Tue, 28 Jan 2020 12:35:08 +0200
> > Subject: Re: [PVE-User] Config/Status commands stopped to respond
> > _______________________________________________
> > pve-user mailing list
> > pve-user at pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
--
Dmytro O. Redchuk
(+380) 44 2474832
More information about the pve-user
mailing list