From nick-liste at posteo.eu Wed Jan 2 10:57:39 2019 From: nick-liste at posteo.eu (Nicola Ferrari (#554252)) Date: Wed, 2 Jan 2019 10:57:39 +0100 Subject: [PVE-User] WebUI Asking login when changing node In-Reply-To: References: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> Message-ID: <943c127a-4c63-149f-596e-6bfbec99ea5b@posteo.eu> Il 29/12/2018 15:36, Alfio munoz ha scritto: > Hi, you can also check the date and the time of the three servers. > This did the trick! Time on two of three nodes was not synced with ntp: timesyncd unit of systemd was facing an error because the old "ntp" package was still installed and running. Now time is the same on every node and problem is fixed :) Here are the steps I followed on every node, for future reference: - removed the "old" debian ntp package apt-get purge ntp - Configure timezone with: dpkg-reconfigure tzdata timedatectl set-timezone Europe/Rome rm /etc/localtime ; ln -s /usr/share/zoneinfo/Europe/Rome /etc/localtime - Reconfigure /etc/systemd/timesyncd.conf with my internal ntp server [Time] Servers=192.168.x.y Start timesync: timedatectl set-ntp true Restart timesyncd unit systemctl restart systemd-timesyncd.service - check timesync status systemctl status systemd-timesyncd.service timedatectl status "NTP Sincronized" should answer "yes". If still no success, restart also systemd-networkd service and retry status commands systemctl restart systemd-networkd Cheers, Nick -- +---------------------+ | Linux User #554252 | +---------------------+ From sten.aus at eenet.ee Thu Jan 3 11:38:22 2019 From: sten.aus at eenet.ee (Sten Aus) Date: Thu, 3 Jan 2019 12:38:22 +0200 Subject: [PVE-User] PVE 4 -> 5, multipath differences? In-Reply-To: <20181228141739.GB4570@sv.lnf.it> References: <20181228141739.GB4570@sv.lnf.it> Message-ID: Hi I've encountered same "issue" or "feature", not sure. I've tried to edit/tweak multipath config, but "prio const" should say that every path is with same priority. As this is my third storage for not critical data, I don't worry much, but I haven't checked if two paths (prio=50) get more load than other two (prio=10). HTH Sten On 28/12/2018 16:17, Marco Gaiarin wrote: > I've just upgraded my cluster frm 4.4 to latest 5. > > Before that, i've also do a firmware upgrade of the SAN (HP MSA 1040), > but seems not related. > > In old PVE 4.4/jessie i got: > > dixie:~# multipath -ll > mpath0 (3600c0ff00026ed11a7cb565701000000) dm-1 HP,MSA 1040 SAN > size=1.4T features='1 queue_if_no_path' hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=1 status=active > |- 2:0:0:0 sda 8:0 active ready running > |- 3:0:0:0 sdb 8:16 active ready running > |- 4:0:0:0 sdc 8:32 active ready running > `- 5:0:0:0 sdd 8:48 active ready running > > in new PVE 5/stretch i got: > > root at ashpool:~# multipath -ll > mpath2 (3600c0ff00026ed11475d215a01000000) dm-0 HP,MSA 1040 SAN > size=1.8T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw > |-+- policy='round-robin 0' prio=50 status=active > | |- 12:0:0:4 sdg 8:96 active ready running > | `- 15:0:0:4 sdr 65:16 active ready running > `-+- policy='round-robin 0' prio=10 status=enabled > |- 13:0:0:4 sdj 8:144 active ready running > `- 14:0:0:4 sds 65:32 active ready running > > config file is the same. > > > Reading around seems that my SAN is a dual controller Active/Passive one, > so probably it is correct to have two path active and two ready, but... > i seek some feedback. ;-) > > Thanks. > From gaio at sv.lnf.it Mon Jan 7 10:37:52 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Mon, 7 Jan 2019 10:37:52 +0100 Subject: [PVE-User] PVE 4 -> 5, multipath differences? In-Reply-To: References: <20181228141739.GB4570@sv.lnf.it> Message-ID: <20190107093752.GA2772@sv.lnf.it> Mandi! Sten Aus In chel di` si favelave... > As this is my third storage for not critical data, I don't worry much, but I > haven't checked if two paths (prio=50) get more load than other two (prio=10). Apart from the prio, the differences between jessie multipath is that in jessie all path are active (and hwhandler is '0'), while in stretch hwhandler is 'ALUA' and only two path are active. Looking at SCSI path seems al correct: the path active are exactly the path where the volume in the SAN reside. Looking at code (git master branch) of multipath-tools seems that hardware get correctly identified. So, probably is really the right thing. But also, i've noted that for some strange mean the blacklist in multipath does not work as expected. I've: blacklist { wwid .* } but still i've log full of: Jan 7 09:55:20 ashpool multipathd[470]: sda: spurious uevent, path not found where SDA is: root at ashpool:~# ls -la /dev/disk/by-id/ | grep sda lrwxrwxrwx 1 root root 9 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000 -> ../../sda lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d -> ../../sda lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F -> ../../sda lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d -> ../../sda lrwxrwxrwx 1 root root 10 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d-part3 -> ../../sda3 the boot disk, in a hw raid controller. Boh... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From elacunza at binovo.es Mon Jan 7 11:02:00 2019 From: elacunza at binovo.es (Eneko Lacunza) Date: Mon, 7 Jan 2019 11:02:00 +0100 Subject: [PVE-User] PVE 4 -> 5, multipath differences? In-Reply-To: <20190107093752.GA2772@sv.lnf.it> References: <20181228141739.GB4570@sv.lnf.it> <20190107093752.GA2772@sv.lnf.it> Message-ID: <22ce2e4a-da19-213d-85ff-ca11f3a7c66f@binovo.es> Hi, I see the same behaviour with an EMC VNXe3200 (two priorities). I assume it is the right thing to do, host really only has 2x1Gbit channels to storage... :) El 7/1/19 a las 10:37, Marco Gaiarin escribi?: > Mandi! Sten Aus > In chel di` si favelave... > >> As this is my third storage for not critical data, I don't worry much, but I >> haven't checked if two paths (prio=50) get more load than other two (prio=10). > Apart from the prio, the differences between jessie multipath is that > in jessie all path are active (and hwhandler is '0'), while in stretch > hwhandler is 'ALUA' and only two path are active. > > Looking at SCSI path seems al correct: the path active are exactly the > path where the volume in the SAN reside. > Looking at code (git master branch) of multipath-tools seems that > hardware get correctly identified. > > So, probably is really the right thing. > > > But also, i've noted that for some strange mean the blacklist in multipath > does not work as expected. I've: > > blacklist { > wwid .* > } > > but still i've log full of: > > Jan 7 09:55:20 ashpool multipathd[470]: sda: spurious uevent, path not found > > where SDA is: > > root at ashpool:~# ls -la /dev/disk/by-id/ | grep sda > lrwxrwxrwx 1 root root 9 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000 -> ../../sda > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000-part1 -> ../../sda1 > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000-part2 -> ../../sda2 > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-0HP_LOGICAL_VOLUME_00000000-part3 -> ../../sda3 > lrwxrwxrwx 1 root root 9 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d -> ../../sda > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d-part1 -> ../../sda1 > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d-part2 -> ../../sda2 > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-3600508b1001c4972bd1d4906bb1dc75d-part3 -> ../../sda3 > lrwxrwxrwx 1 root root 9 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F -> ../../sda > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part1 -> ../../sda1 > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part2 -> ../../sda2 > lrwxrwxrwx 1 root root 10 gen 7 10:25 scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part3 -> ../../sda3 > lrwxrwxrwx 1 root root 9 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d -> ../../sda > lrwxrwxrwx 1 root root 10 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d-part1 -> ../../sda1 > lrwxrwxrwx 1 root root 10 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d-part2 -> ../../sda2 > lrwxrwxrwx 1 root root 10 gen 7 10:25 wwn-0x600508b1001c4972bd1d4906bb1dc75d-part3 -> ../../sda3 > > the boot disk, in a hw raid controller. > > > Boh... > -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From denis.morejon at etecsa.cu Mon Jan 7 18:11:35 2019 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Mon, 7 Jan 2019 12:11:35 -0500 Subject: [PVE-User] Is there a way to permit to start up VMs when no quorum? In-Reply-To: <22ce2e4a-da19-213d-85ff-ca11f3a7c66f@binovo.es> References: <20181228141739.GB4570@sv.lnf.it> <20190107093752.GA2772@sv.lnf.it> <22ce2e4a-da19-213d-85ff-ca11f3a7c66f@binovo.es> Message-ID: <6224747d-01f3-4429-5caa-86f31e7b1fba@etecsa.cu> Hi: (1) I don't know the idea behind keeping a VM from starting up when no quorum. It has been maybe, since my point of view, the worst of managing Proxmox cluster, because the stability of services (VM up and running) had to be first (before the sync of information, for instance). Is there a way to bypass this and permit to start up a VM even on no quorum? (2) I read that Proxmox supports just 30 (more less) nodes in a cluster. I supose that it's due to the multi-master scheme on ALL nodes. Other solutions like Open Nebula (For instance) has a central database and it facilitates to grow up. Even a multi-master scheme, in which not all nodes take plase, would be better to Proxmox so that It could be scalable to hundreds of nodes. Is there a vision on that? Thanks in advance! From dietmar at proxmox.com Mon Jan 7 19:23:53 2019 From: dietmar at proxmox.com (Dietmar Maurer) Date: Mon, 7 Jan 2019 19:23:53 +0100 (CET) Subject: [PVE-User] Is there a way to permit to start up VMs when no quorum? In-Reply-To: <6224747d-01f3-4429-5caa-86f31e7b1fba@etecsa.cu> References: <20181228141739.GB4570@sv.lnf.it> <20190107093752.GA2772@sv.lnf.it> <22ce2e4a-da19-213d-85ff-ca11f3a7c66f@binovo.es> <6224747d-01f3-4429-5caa-86f31e7b1fba@etecsa.cu> Message-ID: <579662038.88.1546885433890@webmail.proxmox.com> > I don't know the idea behind keeping a VM from starting up when no > quorum. It has been maybe, since my point of view, the worst of managing > Proxmox cluster, because the stability of services (VM up and running) > had to be first (before the sync of information, for instance). > > Is there a way to bypass this and permit to start up a VM even on no quorum? No. This is required to avoid split brain ... From denis.morejon at etecsa.cu Mon Jan 7 19:39:06 2019 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Mon, 7 Jan 2019 13:39:06 -0500 Subject: [PVE-User] Is there a way to permit to start up VMs when no quorum? In-Reply-To: <579662038.88.1546885433890@webmail.proxmox.com> References: <20181228141739.GB4570@sv.lnf.it> <20190107093752.GA2772@sv.lnf.it> <22ce2e4a-da19-213d-85ff-ca11f3a7c66f@binovo.es> <6224747d-01f3-4429-5caa-86f31e7b1fba@etecsa.cu> <579662038.88.1546885433890@webmail.proxmox.com> Message-ID: Could you give me an example please? In practice, I know a lot of people that are afraid of building a cluster because of the lost of quorum, an have a plain html web page with the url of each node instead. And this is sad. This is like assuming that the most important thing is to have the VMs UP! El 7/1/19 a las 13:23, Dietmar Maurer escribi?: >> I don't know the idea behind keeping a VM from starting up when no >> quorum. It has been maybe, since my point of view, the worst of managing >> Proxmox cluster, because the stability of services (VM up and running) >> had to be first (before the sync of information, for instance). >> >> Is there a way to bypass this and permit to start up a VM even on no quorum? > No. This is required to avoid split brain ... > > From hipantcii at gmail.com Tue Jan 8 00:49:49 2019 From: hipantcii at gmail.com (Ivan Dimitrov) Date: Tue, 8 Jan 2019 00:49:49 +0100 Subject: [PVE-User] Is there a way to permit to start up VMs when no quorum? In-Reply-To: <579662038.88.1546885433890@webmail.proxmox.com> References: <20181228141739.GB4570@sv.lnf.it> <20190107093752.GA2772@sv.lnf.it> <22ce2e4a-da19-213d-85ff-ca11f3a7c66f@binovo.es> <6224747d-01f3-4429-5caa-86f31e7b1fba@etecsa.cu> <579662038.88.1546885433890@webmail.proxmox.com> Message-ID: Hi, Actually there is a way to do this with a bit tinkering. The first way is described in proxmox documentation. Read this chapter for details: https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node In my home setup I have two node cluster and one of the nodes is my main server and the second one is for testing and very often not running. I do not sync machines machines between the nodes and I am careful with the VM IDs. In order to make it work I have raised the expected votes to 3 and given 2 votes to the main server. With such configuration the main node which is 24/7 will always start and run normally. root at hyper-convergence:~# pvecm status Quorum information ------------------ Date: Tue Jan 8 00:47:06 2019 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 0x00000001 Ring ID: 1/43248 Quorate: Yes Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 2 Quorum: 2 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 2 10.10.0.3 (local) P.S. Keep in mind this is homelab. I wouldn't recommend this for anything but testing and/or tinkering at home. With Best Regards, Ivan Dimitrov On Mon, 7 Jan 2019 at 19:23, Dietmar Maurer wrote: > > I don't know the idea behind keeping a VM from starting up when no > > quorum. It has been maybe, since my point of view, the worst of managing > > Proxmox cluster, because the stability of services (VM up and running) > > had to be first (before the sync of information, for instance). > > > > Is there a way to bypass this and permit to start up a VM even on no > quorum? > > No. This is required to avoid split brain ... > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From t.lamprecht at proxmox.com Tue Jan 8 08:59:08 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 8 Jan 2019 08:59:08 +0100 Subject: [PVE-User] Is there a way to permit to start up VMs when no quorum? In-Reply-To: References: <20181228141739.GB4570@sv.lnf.it> <20190107093752.GA2772@sv.lnf.it> <22ce2e4a-da19-213d-85ff-ca11f3a7c66f@binovo.es> <6224747d-01f3-4429-5caa-86f31e7b1fba@etecsa.cu> <579662038.88.1546885433890@webmail.proxmox.com> Message-ID: On 1/7/19 7:39 PM, Denis Morejon wrote:> Could you give me an example please? Dietmar did already, research split brain. >> In practice, I know a lot of people that are afraid of building a cluster because of the lost of quorum, an have a plain html web page with the url of each node instead. And this is sad. This is like assuming that the most important thing is to have the VMs UP! No, while important it really isn't the *most* important thing. The most important thing is no (data/shared resource) corruption of the VM/CT, and our quorum mechanisms are one part of the machinery to keep that this way. If your network is stable and either multicast works (on smaller systems or medium ones with _really fast_ switches, unicast works just as well) quorum really should not be a problem? And as really a lot of our users have clusters configured I do not think that "a lot of people are afraid of using them". > > El 7/1/19 a las 13:23, Dietmar Maurer escribi?: >>> I don't know the idea behind keeping a VM from starting up when no >>> quorum. It has been maybe, since my point of view, the worst of managing >>> Proxmox cluster, because the stability of services (VM up and running) >>> had to be first (before the sync of information, for instance). >>> >>> Is there a way to bypass this and permit to start up a VM even on no quorum? >> No. This is required to avoid split brain ... >> >> From marcomgabriel at gmail.com Fri Jan 11 11:18:09 2019 From: marcomgabriel at gmail.com (Marco M. Gabriel) Date: Fri, 11 Jan 2019 11:18:09 +0100 Subject: [PVE-User] lxc container can't start on a newly added cluster node Message-ID: Hi there, we recently added a 4th proxmox node to our 3-node proxmox cluster. When trying to migrate or start an lxc container to the new node, it doesn't start because it can't mount /dev/pts/x to /dev/console. I can move the lxc container to each of the 3 old nodes and start it there, but it refuses to start on the new node. Nodes 1 to 3 were installed with Version 4.4 and upgraded until the current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) Node 4 was installed a few days ago, but is on exactly the same version now. The lxc container is an ubuntu 16.04 based container from the proxmox repository and was installed about a month ago. Trace log of lxc-start (shortened): lxc-start 116 20190111094747.940 INFO conf - conf.c:lxc_fill_autodev:1220 - Populating "/dev" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" lxc-start 116 20190111094747.940 INFO conf - conf.c:lxc_fill_autodev:1297 - Populated "/dev" lxc-start 116 20190111094747.940 ERROR utils - utils.c:safe_mount:1791 - No such file or directory - Failed to mount /dev/pts/3 onto /dev/console lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup_dev_console:1765 - Failed to mount "/dev/pts/3" on "/dev/console" lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup:3656 - Failed to setup console lxc-start 116 20190111094747.940 ERROR start - start.c:do_start:1246 - Failed to setup container "116" Any hints how to fix this or what happens here? Best regards, Marco From gaio at sv.lnf.it Fri Jan 11 11:59:22 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Fri, 11 Jan 2019 11:59:22 +0100 Subject: [PVE-User] Filesystem corruption on a VM? In-Reply-To: <20181119140822.GG2916@sv.lnf.it> References: <20181114174054.GA18133@sv.lnf.it> <20181115105642.GD2709@sv.lnf.it> <20181115113550.GE2709@sv.lnf.it> <231749ab-a148-e10a-f6b6-6f9fc1c92e6d@firewall-services.com> <20181115114939.GG2709@sv.lnf.it> <20181115132420.GI2709@sv.lnf.it> <20181119140822.GG2916@sv.lnf.it> Message-ID: <20190111105922.GN2855@sv.lnf.it> I come back to this thread. > OK, with 16GB of ram 5 days passed without FS errors. I've upgraded SAN firmware to the latest available, then upgrade the cluster from PVE 4.4 to latest 5 version. Today is 2 weeks without filesystem corruption happens. Previous interval was lesser then 5 days, so i suppose that i've fixed that trouble. FYI. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From mcooper at coopfire.com Fri Jan 11 12:02:17 2019 From: mcooper at coopfire.com (Michael Cooper) Date: Fri, 11 Jan 2019 06:02:17 -0500 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: References: Message-ID: <5c3877bb.1c69fb81.1d89d.93fa@mx.google.com> I am not sure if this is a stupid question but did you make sure all storage is the same across the nodes? Just curious, Michael A Cooper Infrastruture Engineer http://www.coopfire.com From: Marco M. Gabriel Sent: Friday, January 11, 2019 5:18 AM To: PVE User List Subject: [PVE-User] lxc container can't start on a newly added cluster node Hi there, we recently added a 4th proxmox node to our 3-node proxmox cluster. When trying to migrate or start an lxc container to the new node, it doesn't start because it can't mount /dev/pts/x to /dev/console. I can move the lxc container to each of the 3 old nodes and start it there, but it refuses to start on the new node. Nodes 1 to 3 were installed with Version 4.4 and upgraded until the current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) Node 4 was installed a few days ago, but is on exactly the same version now. The lxc container is an ubuntu 16.04 based container from the proxmox repository and was installed about a month ago. Trace log of lxc-start (shortened): lxc-start 116 20190111094747.940 INFO conf - conf.c:lxc_fill_autodev:1220 - Populating "/dev" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" lxc-start 116 20190111094747.940 INFO conf - conf.c:lxc_fill_autodev:1297 - Populated "/dev" lxc-start 116 20190111094747.940 ERROR utils - utils.c:safe_mount:1791 - No such file or directory - Failed to mount /dev/pts/3 onto /dev/console lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup_dev_console:1765 - Failed to mount "/dev/pts/3" on "/dev/console" lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup:3656 - Failed to setup console lxc-start 116 20190111094747.940 ERROR start - start.c:do_start:1246 - Failed to setup container "116" Any hints how to fix this or what happens here? Best regards, Marco _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mcooper at coopfire.com Fri Jan 11 12:05:43 2019 From: mcooper at coopfire.com (Michael Cooper) Date: Fri, 11 Jan 2019 06:05:43 -0500 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: References: Message-ID: <5c387889.1c69fb81.1226d.8ea1@mx.google.com> The reason I asked if this is the mount point (/dev/pts/x) on the other 3 nodes, then it has to be the same on the 4th node for it to migrate properly. Let me know, Michael A Cooper Infrastruture Engineer http://www.coopfire.com From: Marco M. Gabriel Sent: Friday, January 11, 2019 5:18 AM To: PVE User List Subject: [PVE-User] lxc container can't start on a newly added cluster node Hi there, we recently added a 4th proxmox node to our 3-node proxmox cluster. When trying to migrate or start an lxc container to the new node, it doesn't start because it can't mount /dev/pts/x to /dev/console. I can move the lxc container to each of the 3 old nodes and start it there, but it refuses to start on the new node. Nodes 1 to 3 were installed with Version 4.4 and upgraded until the current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) Node 4 was installed a few days ago, but is on exactly the same version now. The lxc container is an ubuntu 16.04 based container from the proxmox repository and was installed about a month ago. Trace log of lxc-start (shortened): lxc-start 116 20190111094747.940 INFO conf - conf.c:lxc_fill_autodev:1220 - Populating "/dev" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" lxc-start 116 20190111094747.940 DEBUG conf - conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" lxc-start 116 20190111094747.940 INFO conf - conf.c:lxc_fill_autodev:1297 - Populated "/dev" lxc-start 116 20190111094747.940 ERROR utils - utils.c:safe_mount:1791 - No such file or directory - Failed to mount /dev/pts/3 onto /dev/console lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup_dev_console:1765 - Failed to mount "/dev/pts/3" on "/dev/console" lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup:3656 - Failed to setup console lxc-start 116 20190111094747.940 ERROR start - start.c:do_start:1246 - Failed to setup container "116" Any hints how to fix this or what happens here? Best regards, Marco _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From marcomgabriel at gmail.com Fri Jan 11 12:10:03 2019 From: marcomgabriel at gmail.com (Marco M. Gabriel) Date: Fri, 11 Jan 2019 12:10:03 +0100 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: <5c387889.1c69fb81.1226d.8ea1@mx.google.com> References: <5c387889.1c69fb81.1226d.8ea1@mx.google.com> Message-ID: Yes, all the mount points are the same on each node. As the container doesn't start, an existing /dev/pts/x on any other node wouldn't make a difference. I also can create other virtual terminals (screen, tmux, ssh, etc.) and I can see new /dev/pts/* devices being created, only lxc-start seems to have a problem creating/mounting a console there. Am Fr., 11. Jan. 2019 um 12:05 Uhr schrieb Michael Cooper < mcooper at coopfire.com>: > The reason I asked if this is the mount point (/dev/pts/x) on the other 3 > nodes, then it has to be the same on the 4th node for it to migrate > properly. > > > Let me know, > Michael A Cooper > Infrastruture Engineer > http://www.coopfire.com > > > From: Marco M. Gabriel > Sent: Friday, January 11, 2019 5:18 AM > To: PVE User List > Subject: [PVE-User] lxc container can't start on a newly added cluster node > > Hi there, > > we recently added a 4th proxmox node to our 3-node proxmox cluster. When > trying to migrate or start an lxc container to the new node, it doesn't > start because it can't mount /dev/pts/x to /dev/console. I can move the lxc > container to each of the 3 old nodes and start it there, but it refuses to > start on the new node. > > Nodes 1 to 3 were installed with Version 4.4 and upgraded until the > current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) > Node 4 was installed a few days ago, but is on exactly the same version > now. > > The lxc container is an ubuntu 16.04 based container from the proxmox > repository and was installed about a month ago. > > Trace log of lxc-start (shortened): > > lxc-start 116 20190111094747.940 INFO conf - > conf.c:lxc_fill_autodev:1220 - Populating "/dev" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" > lxc-start 116 20190111094747.940 INFO conf - > conf.c:lxc_fill_autodev:1297 - Populated "/dev" > lxc-start 116 20190111094747.940 ERROR utils - utils.c:safe_mount:1791 - > No such file or directory - Failed to mount /dev/pts/3 onto /dev/console > lxc-start 116 20190111094747.940 ERROR conf - > conf.c:lxc_setup_dev_console:1765 - Failed to mount "/dev/pts/3" on > "/dev/console" > lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup:3656 - > Failed to setup console > lxc-start 116 20190111094747.940 ERROR start - start.c:do_start:1246 - > Failed to setup container "116" > > Any hints how to fix this or what happens here? > > Best regards, > Marco > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mcooper at coopfire.com Fri Jan 11 12:12:38 2019 From: mcooper at coopfire.com (Michael Cooper) Date: Fri, 11 Jan 2019 06:12:38 -0500 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: References: <5c387889.1c69fb81.1226d.8ea1@mx.google.com> Message-ID: Ok I was just checking...... On Fri, Jan 11, 2019, 6:10 AM Marco M. Gabriel Yes, all the mount points are the same on each node. As the container > doesn't start, an existing /dev/pts/x on any other node wouldn't make a > difference. > > I also can create other virtual terminals (screen, tmux, ssh, etc.) and I > can see new /dev/pts/* devices being created, only lxc-start seems to have > a problem creating/mounting a console there. > > Am Fr., 11. Jan. 2019 um 12:05 Uhr schrieb Michael Cooper < > mcooper at coopfire.com>: > > > The reason I asked if this is the mount point (/dev/pts/x) on the other 3 > > nodes, then it has to be the same on the 4th node for it to migrate > > properly. > > > > > > Let me know, > > Michael A Cooper > > Infrastruture Engineer > > http://www.coopfire.com > > > > > > From: Marco M. Gabriel > > Sent: Friday, January 11, 2019 5:18 AM > > To: PVE User List > > Subject: [PVE-User] lxc container can't start on a newly added cluster > node > > > > Hi there, > > > > we recently added a 4th proxmox node to our 3-node proxmox cluster. When > > trying to migrate or start an lxc container to the new node, it doesn't > > start because it can't mount /dev/pts/x to /dev/console. I can move the > lxc > > container to each of the 3 old nodes and start it there, but it refuses > to > > start on the new node. > > > > Nodes 1 to 3 were installed with Version 4.4 and upgraded until the > > current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) > > Node 4 was installed a few days ago, but is on exactly the same version > > now. > > > > The lxc container is an ubuntu 16.04 based container from the proxmox > > repository and was installed about a month ago. > > > > Trace log of lxc-start (shortened): > > > > lxc-start 116 20190111094747.940 INFO conf - > > conf.c:lxc_fill_autodev:1220 - Populating "/dev" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" > > lxc-start 116 20190111094747.940 INFO conf - > > conf.c:lxc_fill_autodev:1297 - Populated "/dev" > > lxc-start 116 20190111094747.940 ERROR utils - > utils.c:safe_mount:1791 - > > No such file or directory - Failed to mount /dev/pts/3 onto /dev/console > > lxc-start 116 20190111094747.940 ERROR conf - > > conf.c:lxc_setup_dev_console:1765 - Failed to mount "/dev/pts/3" on > > "/dev/console" > > lxc-start 116 20190111094747.940 ERROR conf - conf.c:lxc_setup:3656 - > > Failed to setup console > > lxc-start 116 20190111094747.940 ERROR start - start.c:do_start:1246 - > > Failed to setup container "116" > > > > Any hints how to fix this or what happens here? > > > > Best regards, > > Marco > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From s.ivanov at proxmox.com Fri Jan 11 12:44:49 2019 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Fri, 11 Jan 2019 12:44:49 +0100 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: References: Message-ID: <20190111124449.07cc303d@rosa.proxmox.com> Hi, just a guess - does the container have an `/etc/fstab` inside? and does the `/etc/fstab` contain an entry for `/dev/pts`? If yes - I guess the container would start on PVE 5.3 when you remove the line (or the complete `/etc/fstab`). This might result in it not starting on PVE 4.x. Could you test this? Best regards, stoiko On Fri, 11 Jan 2019 11:18:09 +0100 "Marco M. Gabriel" wrote: > Hi there, > > we recently added a 4th proxmox node to our 3-node proxmox cluster. > When trying to migrate or start an lxc container to the new node, it > doesn't start because it can't mount /dev/pts/x to /dev/console. I > can move the lxc container to each of the 3 old nodes and start it > there, but it refuses to start on the new node. > > Nodes 1 to 3 were installed with Version 4.4 and upgraded until the > current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) > Node 4 was installed a few days ago, but is on exactly the same > version now. > > The lxc container is an ubuntu 16.04 based container from the proxmox > repository and was installed about a month ago. > > Trace log of lxc-start (shortened): > > lxc-start 116 20190111094747.940 INFO conf - > conf.c:lxc_fill_autodev:1220 - Populating "/dev" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" > lxc-start 116 20190111094747.940 DEBUG conf - > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" > lxc-start 116 20190111094747.940 INFO conf - > conf.c:lxc_fill_autodev:1297 - Populated "/dev" > lxc-start 116 20190111094747.940 ERROR utils - > utils.c:safe_mount:1791 - No such file or directory - Failed to > mount /dev/pts/3 onto /dev/console lxc-start 116 20190111094747.940 > ERROR conf - conf.c:lxc_setup_dev_console:1765 - Failed to mount > "/dev/pts/3" on "/dev/console" > lxc-start 116 20190111094747.940 ERROR conf - > conf.c:lxc_setup:3656 - Failed to setup console > lxc-start 116 20190111094747.940 ERROR start - > start.c:do_start:1246 - Failed to setup container "116" > > Any hints how to fix this or what happens here? > > Best regards, > Marco > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From marcomgabriel at gmail.com Fri Jan 11 12:48:33 2019 From: marcomgabriel at gmail.com (Marco M. Gabriel) Date: Fri, 11 Jan 2019 12:48:33 +0100 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: <20190111124449.07cc303d@rosa.proxmox.com> References: <20190111124449.07cc303d@rosa.proxmox.com> Message-ID: the /etc/fstab is there, but it is quite empty: cat fstab # UNCONFIGURED FSTAB FOR BASE SYSTEM thanks, Marco Am Fr., 11. Jan. 2019 um 12:44 Uhr schrieb Stoiko Ivanov < s.ivanov at proxmox.com>: > Hi, > > just a guess - does the container have an `/etc/fstab` inside? > and does the `/etc/fstab` contain an entry for `/dev/pts`? > > If yes - I guess the container would start on PVE 5.3 when you remove > the line (or the complete `/etc/fstab`). This might result in it not > starting on PVE 4.x. > > Could you test this? > > Best regards, > stoiko > > > > On Fri, 11 Jan 2019 11:18:09 +0100 > "Marco M. Gabriel" wrote: > > > Hi there, > > > > we recently added a 4th proxmox node to our 3-node proxmox cluster. > > When trying to migrate or start an lxc container to the new node, it > > doesn't start because it can't mount /dev/pts/x to /dev/console. I > > can move the lxc container to each of the 3 old nodes and start it > > there, but it refuses to start on the new node. > > > > Nodes 1 to 3 were installed with Version 4.4 and upgraded until the > > current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) > > Node 4 was installed a few days ago, but is on exactly the same > > version now. > > > > The lxc container is an ubuntu 16.04 based container from the proxmox > > repository and was installed about a month ago. > > > > Trace log of lxc-start (shortened): > > > > lxc-start 116 20190111094747.940 INFO conf - > > conf.c:lxc_fill_autodev:1220 - Populating "/dev" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" > > lxc-start 116 20190111094747.940 DEBUG conf - > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" > > lxc-start 116 20190111094747.940 INFO conf - > > conf.c:lxc_fill_autodev:1297 - Populated "/dev" > > lxc-start 116 20190111094747.940 ERROR utils - > > utils.c:safe_mount:1791 - No such file or directory - Failed to > > mount /dev/pts/3 onto /dev/console lxc-start 116 20190111094747.940 > > ERROR conf - conf.c:lxc_setup_dev_console:1765 - Failed to mount > > "/dev/pts/3" on "/dev/console" > > lxc-start 116 20190111094747.940 ERROR conf - > > conf.c:lxc_setup:3656 - Failed to setup console > > lxc-start 116 20190111094747.940 ERROR start - > > start.c:do_start:1246 - Failed to setup container "116" > > > > Any hints how to fix this or what happens here? > > > > Best regards, > > Marco > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > From s.ivanov at proxmox.com Fri Jan 11 14:21:46 2019 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Fri, 11 Jan 2019 14:21:46 +0100 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: References: <20190111124449.07cc303d@rosa.proxmox.com> Message-ID: <20190111142146.0733d29f@rosa.proxmox.com> Sorry - I misread your original mail (and thought 3 of your nodes were still running PVE 4.x). do you have the same versions of all relevant packages (pve-container, lxcfs, lxc-pve,..) on all nodes? does creating a fresh container work on the new node (also ubuntu 16.04)? FWIW - could not reproduce the issue locally (created 1 privileged and 1 unprivileged Ubuntu 16.04 container and both started). When starting with loglevel TRACE I do see a few more lines between the 'Populated "/dev"' and the error. Also the console and tty-creation does log a few lines further up - maybe they provide some hints where the problem lies. On Fri, 11 Jan 2019 12:48:33 +0100 "Marco M. Gabriel" wrote: > the /etc/fstab is there, but it is quite empty: > > cat fstab > # UNCONFIGURED FSTAB FOR BASE SYSTEM > > thanks, > Marco > > > Am Fr., 11. Jan. 2019 um 12:44 Uhr schrieb Stoiko Ivanov < > s.ivanov at proxmox.com>: > > > Hi, > > > > just a guess - does the container have an `/etc/fstab` inside? > > and does the `/etc/fstab` contain an entry for `/dev/pts`? > > > > If yes - I guess the container would start on PVE 5.3 when you > > remove the line (or the complete `/etc/fstab`). This might result > > in it not starting on PVE 4.x. > > > > Could you test this? > > > > Best regards, > > stoiko > > > > > > > > On Fri, 11 Jan 2019 11:18:09 +0100 > > "Marco M. Gabriel" wrote: > > > > > Hi there, > > > > > > we recently added a 4th proxmox node to our 3-node proxmox > > > cluster. When trying to migrate or start an lxc container to the > > > new node, it doesn't start because it can't mount /dev/pts/x > > > to /dev/console. I can move the lxc container to each of the 3 > > > old nodes and start it there, but it refuses to start on the new > > > node. > > > > > > Nodes 1 to 3 were installed with Version 4.4 and upgraded until > > > the current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) > > > Node 4 was installed a few days ago, but is on exactly the same > > > version now. > > > > > > The lxc container is an ubuntu 16.04 based container from the > > > proxmox repository and was installed about a month ago. > > > > > > Trace log of lxc-start (shortened): > > > > > > lxc-start 116 20190111094747.940 INFO conf - > > > conf.c:lxc_fill_autodev:1220 - Populating "/dev" > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" > > > lxc-start 116 20190111094747.940 INFO conf - > > > conf.c:lxc_fill_autodev:1297 - Populated "/dev" > > > lxc-start 116 20190111094747.940 ERROR utils - > > > utils.c:safe_mount:1791 - No such file or directory - Failed to > > > mount /dev/pts/3 onto /dev/console lxc-start 116 > > > 20190111094747.940 ERROR conf - > > > conf.c:lxc_setup_dev_console:1765 - Failed to mount "/dev/pts/3" > > > on "/dev/console" lxc-start 116 20190111094747.940 ERROR conf - > > > conf.c:lxc_setup:3656 - Failed to setup console > > > lxc-start 116 20190111094747.940 ERROR start - > > > start.c:do_start:1246 - Failed to setup container "116" > > > > > > Any hints how to fix this or what happens here? > > > > > > Best regards, > > > Marco > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > From marcomgabriel at gmail.com Fri Jan 11 15:58:03 2019 From: marcomgabriel at gmail.com (Marco M. Gabriel) Date: Fri, 11 Jan 2019 15:58:03 +0100 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: <20190111142146.0733d29f@rosa.proxmox.com> References: <20190111124449.07cc303d@rosa.proxmox.com> <20190111142146.0733d29f@rosa.proxmox.com> Message-ID: I just tried to set up a new ubuntu 16.04 and an ubuntu 18.04 container on the new node - and both are running fine. but the old one still doesn't start. After this, I wanted to trace a new and the old one and compare the logs of both, but it seems that neither the old one nor the new one start when I use "/usr/bin/lxc-start -n -F -f /etc/pve/lxc/.conf --logfile /tmp/lxc.log --logpriority TRACE" The new and the old one do not start when I use that command to get a log. The logs are the same for both containers. If that command above does not work, then how to debug a container that does not want to start? Thanks, Marco Am Fr., 11. Jan. 2019 um 14:21 Uhr schrieb Stoiko Ivanov < s.ivanov at proxmox.com>: > Sorry - I misread your original mail (and thought 3 of your nodes were > still running PVE 4.x). > > do you have the same versions of all relevant packages (pve-container, > lxcfs, lxc-pve,..) on all nodes? > does creating a fresh container work on the new node (also ubuntu > 16.04)? > > FWIW - could not reproduce the issue locally (created 1 privileged and > 1 unprivileged Ubuntu 16.04 container and both started). > > When starting with loglevel TRACE I do see a few more lines between > the 'Populated "/dev"' and the error. Also the console and tty-creation > does log a few lines further up - maybe they provide some hints where > the problem lies. > > > On Fri, 11 Jan 2019 12:48:33 +0100 > "Marco M. Gabriel" wrote: > > > the /etc/fstab is there, but it is quite empty: > > > > cat fstab > > # UNCONFIGURED FSTAB FOR BASE SYSTEM > > > > thanks, > > Marco > > > > > > Am Fr., 11. Jan. 2019 um 12:44 Uhr schrieb Stoiko Ivanov < > > s.ivanov at proxmox.com>: > > > > > Hi, > > > > > > just a guess - does the container have an `/etc/fstab` inside? > > > and does the `/etc/fstab` contain an entry for `/dev/pts`? > > > > > > If yes - I guess the container would start on PVE 5.3 when you > > > remove the line (or the complete `/etc/fstab`). This might result > > > in it not starting on PVE 4.x. > > > > > > Could you test this? > > > > > > Best regards, > > > stoiko > > > > > > > > > > > > On Fri, 11 Jan 2019 11:18:09 +0100 > > > "Marco M. Gabriel" wrote: > > > > > > > Hi there, > > > > > > > > we recently added a 4th proxmox node to our 3-node proxmox > > > > cluster. When trying to migrate or start an lxc container to the > > > > new node, it doesn't start because it can't mount /dev/pts/x > > > > to /dev/console. I can move the lxc container to each of the 3 > > > > old nodes and start it there, but it refuses to start on the new > > > > node. > > > > > > > > Nodes 1 to 3 were installed with Version 4.4 and upgraded until > > > > the current 5.3-6/37b3c8df (running kernel: 4.15.18-9-pve) > > > > Node 4 was installed a few days ago, but is on exactly the same > > > > version now. > > > > > > > > The lxc container is an ubuntu 16.04 based container from the > > > > proxmox repository and was installed about a month ago. > > > > > > > > Trace log of lxc-start (shortened): > > > > > > > > lxc-start 116 20190111094747.940 INFO conf - > > > > conf.c:lxc_fill_autodev:1220 - Populating "/dev" > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/random" > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/urandom" > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/zero" > > > > lxc-start 116 20190111094747.940 INFO conf - > > > > conf.c:lxc_fill_autodev:1297 - Populated "/dev" > > > > lxc-start 116 20190111094747.940 ERROR utils - > > > > utils.c:safe_mount:1791 - No such file or directory - Failed to > > > > mount /dev/pts/3 onto /dev/console lxc-start 116 > > > > 20190111094747.940 ERROR conf - > > > > conf.c:lxc_setup_dev_console:1765 - Failed to mount "/dev/pts/3" > > > > on "/dev/console" lxc-start 116 20190111094747.940 ERROR conf - > > > > conf.c:lxc_setup:3656 - Failed to setup console > > > > lxc-start 116 20190111094747.940 ERROR start - > > > > start.c:do_start:1246 - Failed to setup container "116" > > > > > > > > Any hints how to fix this or what happens here? > > > > > > > > Best regards, > > > > Marco > > > > _______________________________________________ > > > > pve-user mailing list > > > > pve-user at pve.proxmox.com > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > > > > > > From s.ivanov at proxmox.com Fri Jan 11 16:25:42 2019 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Fri, 11 Jan 2019 16:25:42 +0100 Subject: [PVE-User] lxc container can't start on a newly added cluster node In-Reply-To: References: <20190111124449.07cc303d@rosa.proxmox.com> <20190111142146.0733d29f@rosa.proxmox.com> Message-ID: <20190111162542.4812e4b1@rosa.proxmox.com> On Fri, 11 Jan 2019 15:58:03 +0100 "Marco M. Gabriel" wrote: > I just tried to set up a new ubuntu 16.04 and an ubuntu 18.04 > container on the new node - and both are running fine. but the old > one still doesn't start. > > After this, I wanted to trace a new and the old one and compare the > logs of both, but it seems that neither the old one nor the new one > start when I use > > "/usr/bin/lxc-start -n -F -f /etc/pve/lxc/.conf --logfile > /tmp/lxc.log --logpriority TRACE" > > The new and the old one do not start when I use that command to get a > log. The logs are the same for both containers. > > If that command above does not work, then how to debug a container > that does not want to start? lxc-start -n -l TRACE -o /tmp/.log works for me (and starts the container in the background (then you can run all `pct` commands for stopping/entering/configuring it... the problem is most likely the `-f` switch - that expects an lxc-config, and PVE uses its own config-format. see also: https://pve.proxmox.com/pve-docs/chapter-pct.html#_obtaining_debugging_logs Hope that helps! stoiko > > Thanks, > Marco > > Am Fr., 11. Jan. 2019 um 14:21 Uhr schrieb Stoiko Ivanov < > s.ivanov at proxmox.com>: > > > Sorry - I misread your original mail (and thought 3 of your nodes > > were still running PVE 4.x). > > > > do you have the same versions of all relevant packages > > (pve-container, lxcfs, lxc-pve,..) on all nodes? > > does creating a fresh container work on the new node (also ubuntu > > 16.04)? > > > > FWIW - could not reproduce the issue locally (created 1 privileged > > and 1 unprivileged Ubuntu 16.04 container and both started). > > > > When starting with loglevel TRACE I do see a few more lines between > > the 'Populated "/dev"' and the error. Also the console and > > tty-creation does log a few lines further up - maybe they provide > > some hints where the problem lies. > > > > > > On Fri, 11 Jan 2019 12:48:33 +0100 > > "Marco M. Gabriel" wrote: > > > > > the /etc/fstab is there, but it is quite empty: > > > > > > cat fstab > > > # UNCONFIGURED FSTAB FOR BASE SYSTEM > > > > > > thanks, > > > Marco > > > > > > > > > Am Fr., 11. Jan. 2019 um 12:44 Uhr schrieb Stoiko Ivanov < > > > s.ivanov at proxmox.com>: > > > > > > > Hi, > > > > > > > > just a guess - does the container have an `/etc/fstab` inside? > > > > and does the `/etc/fstab` contain an entry for `/dev/pts`? > > > > > > > > If yes - I guess the container would start on PVE 5.3 when you > > > > remove the line (or the complete `/etc/fstab`). This might > > > > result in it not starting on PVE 4.x. > > > > > > > > Could you test this? > > > > > > > > Best regards, > > > > stoiko > > > > > > > > > > > > > > > > On Fri, 11 Jan 2019 11:18:09 +0100 > > > > "Marco M. Gabriel" wrote: > > > > > > > > > Hi there, > > > > > > > > > > we recently added a 4th proxmox node to our 3-node proxmox > > > > > cluster. When trying to migrate or start an lxc container to > > > > > the new node, it doesn't start because it can't > > > > > mount /dev/pts/x to /dev/console. I can move the lxc > > > > > container to each of the 3 old nodes and start it there, but > > > > > it refuses to start on the new node. > > > > > > > > > > Nodes 1 to 3 were installed with Version 4.4 and upgraded > > > > > until the current 5.3-6/37b3c8df (running kernel: > > > > > 4.15.18-9-pve) Node 4 was installed a few days ago, but is on > > > > > exactly the same version now. > > > > > > > > > > The lxc container is an ubuntu 16.04 based container from the > > > > > proxmox repository and was installed about a month ago. > > > > > > > > > > Trace log of lxc-start (shortened): > > > > > > > > > > lxc-start 116 20190111094747.940 INFO conf - > > > > > conf.c:lxc_fill_autodev:1220 - Populating "/dev" > > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full" > > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null" > > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > > conf.c:lxc_fill_autodev:1235 - Created device node > > > > > "/dev/random" lxc-start 116 20190111094747.940 DEBUG conf - > > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty" > > > > > lxc-start 116 20190111094747.940 DEBUG conf - > > > > > conf.c:lxc_fill_autodev:1235 - Created device node > > > > > "/dev/urandom" lxc-start 116 20190111094747.940 DEBUG conf > > > > > - conf.c:lxc_fill_autodev:1235 - Created device node > > > > > "/dev/zero" lxc-start 116 20190111094747.940 INFO conf - > > > > > conf.c:lxc_fill_autodev:1297 - Populated "/dev" > > > > > lxc-start 116 20190111094747.940 ERROR utils - > > > > > utils.c:safe_mount:1791 - No such file or directory - Failed > > > > > to mount /dev/pts/3 onto /dev/console lxc-start 116 > > > > > 20190111094747.940 ERROR conf - > > > > > conf.c:lxc_setup_dev_console:1765 - Failed to mount > > > > > "/dev/pts/3" on "/dev/console" lxc-start 116 > > > > > 20190111094747.940 ERROR conf - conf.c:lxc_setup:3656 - > > > > > Failed to setup console lxc-start 116 20190111094747.940 > > > > > ERROR start - start.c:do_start:1246 - Failed to setup > > > > > container "116" > > > > > > > > > > Any hints how to fix this or what happens here? > > > > > > > > > > Best regards, > > > > > Marco > > > > > _______________________________________________ > > > > > pve-user mailing list > > > > > pve-user at pve.proxmox.com > > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > > > > > > > > > > > > From mszamocki at cirrus.pl Tue Jan 15 06:50:53 2019 From: mszamocki at cirrus.pl (=?utf-8?Q?Micha=C5=82?= Szamocki) Date: Tue, 15 Jan 2019 06:50:53 +0100 (CET) Subject: [PVE-User] Ubuntu 14.04 boot fail on PVE 5.3-7 Message-ID: <1080144377.17748.1547531453566.JavaMail.zimbra@cirrus.pl> Hello, since weekend I have problems with LXC containers (upstart based) booting on Proxmox 5.3-7. Today I have created new container and after starting it only few upstart processes are running: root at ubuntu14:~# ps ax PID TTY STAT TIME COMMAND 1 ? Ss 0:00 /sbin/init 37 ? S 0:00 @sbin/plymouthd --mode=boot --attach-to-session 39 ? Ss 0:00 plymouth-upstart-bridge 47 ? S 0:00 mountall --daemon 283 ? S 0:00 upstart-socket-bridge --daemon 1445 ? Ss 0:00 /bin/bash 1481 ? R+ 0:00 ps ax root at ubuntu14:~# ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) After switching to runlevel 2 container continues to boot: root at ubuntu14:~# ifup eth0; telinit 2 root at ubuntu14:~# ps ax; ifconfig PID TTY STAT TIME COMMAND 1 ? Ss 0:00 /sbin/init 47 ? S 0:00 mountall --daemon 283 ? S 0:00 upstart-socket-bridge --daemon 1445 ? Ss 0:00 /bin/bash 1550 ? Ss 0:00 /usr/sbin/sshd -D 1559 ? S 0:00 /bin/sh /etc/network/if-up.d/ntpdate 1562 ? S 0:00 lockfile-create /var/lock/ntpdate-ifup 1569 ? Ss 0:00 cron 1585 ? Ss 0:00 /usr/sbin/irqbalance 1687 ? Ss 0:00 /usr/lib/postfix/master 1691 ? S 0:00 pickup -l -t unix -u -c 1692 ? S 0:00 qmgr -l -t unix -u 1708 ? S 0:00 /bin/sh /etc/init.d/ondemand background 1714 ? S 0:00 sleep 60 1716 console Ss+ 0:00 /sbin/getty -8 38400 console 1718 lxc/tty2 Ss+ 0:00 /sbin/getty -8 38400 tty2 1719 lxc/tty1 Ss+ 0:00 /sbin/getty -8 38400 tty1 1734 ? R+ 0:00 ps ax eth0 Link encap:Ethernet HWaddr 6a:f7:05:0c:43:a4 inet addr:192.168.xxx.239 Bcast:192.168.xxx.255 Mask:255.255.255.0 inet6 addr: fe80::68f7:5ff:fe0c:43a4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:36 errors:0 dropped:0 overruns:0 frame:0 TX packets:29 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3647 (3.6 KB) TX bytes:2342 (2.3 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) What can I do to have fully booted container without runlevel switching? BR, Michal Szamocki Cirrus From sir_Misiek1 at o2.pl Tue Jan 15 11:46:23 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Tue, 15 Jan 2019 11:46:23 +0100 Subject: [PVE-User] Ubuntu 14.04 boot fail on PVE 5.3-7 In-Reply-To: <1080144377.17748.1547531453566.JavaMail.zimbra@cirrus.pl> References: <1080144377.17748.1547531453566.JavaMail.zimbra@cirrus.pl> Message-ID: > Hello, > > since weekend I have problems with LXC containers (upstart based) booting on Proxmox 5.3-7. Today I have created new container and after starting it only few upstart processes are running: > root at ubuntu14:~# ps ax > PID TTY STAT TIME COMMAND > 1 ? Ss 0:00 /sbin/init > 37 ? S 0:00 @sbin/plymouthd --mode=boot --attach-to-session > 39 ? Ss 0:00 plymouth-upstart-bridge > 47 ? S 0:00 mountall --daemon > 283 ? S 0:00 upstart-socket-bridge --daemon > 1445 ? Ss 0:00 /bin/bash > 1481 ? R+ 0:00 ps ax > root at ubuntu14:~# ifconfig > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:65536 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > After switching to runlevel 2 container continues to boot: > root at ubuntu14:~# ifup eth0; telinit 2 > root at ubuntu14:~# ps ax; ifconfig > PID TTY STAT TIME COMMAND > 1 ? Ss 0:00 /sbin/init > 47 ? S 0:00 mountall --daemon > 283 ? S 0:00 upstart-socket-bridge --daemon > 1445 ? Ss 0:00 /bin/bash > 1550 ? Ss 0:00 /usr/sbin/sshd -D > 1559 ? S 0:00 /bin/sh /etc/network/if-up.d/ntpdate > 1562 ? S 0:00 lockfile-create /var/lock/ntpdate-ifup > 1569 ? Ss 0:00 cron > 1585 ? Ss 0:00 /usr/sbin/irqbalance > 1687 ? Ss 0:00 /usr/lib/postfix/master > 1691 ? S 0:00 pickup -l -t unix -u -c > 1692 ? S 0:00 qmgr -l -t unix -u > 1708 ? S 0:00 /bin/sh /etc/init.d/ondemand background > 1714 ? S 0:00 sleep 60 > 1716 console Ss+ 0:00 /sbin/getty -8 38400 console > 1718 lxc/tty2 Ss+ 0:00 /sbin/getty -8 38400 tty2 > 1719 lxc/tty1 Ss+ 0:00 /sbin/getty -8 38400 tty1 > 1734 ? R+ 0:00 ps ax > eth0 Link encap:Ethernet HWaddr 6a:f7:05:0c:43:a4 > inet addr:192.168.xxx.239 Bcast:192.168.xxx.255 Mask:255.255.255.0 > inet6 addr: fe80::68f7:5ff:fe0c:43a4/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:36 errors:0 dropped:0 overruns:0 frame:0 > TX packets:29 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:3647 (3.6 KB) TX bytes:2342 (2.3 KB) > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:65536 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > What can I do to have fully booted container without runlevel switching? > > BR, > Michal Szamocki > Cirrus ??? Hi Micha?. I heave this same with fedora. Try this: chkconfig --levels 2345 network on #systemctl disable NetworkManager # in file /etc/sysconfig/network NETWORKING=yes HOSTNAME=wanwww5 GATEWAY=84.204.162.20 Best regards Grzegorz Misiek? ;-] From w.bumiller at proxmox.com Wed Jan 16 11:12:42 2019 From: w.bumiller at proxmox.com (Wolfgang Bumiller) Date: Wed, 16 Jan 2019 11:12:42 +0100 (CET) Subject: [PVE-User] Ubuntu 14.04 boot fail on PVE 5.3-7 In-Reply-To: <1080144377.17748.1547531453566.JavaMail.zimbra@cirrus.pl> References: <1080144377.17748.1547531453566.JavaMail.zimbra@cirrus.pl> Message-ID: <157930447.28.1547633562455@webmail.proxmox.com> > On January 15, 2019 at 6:50 AM Micha? Szamocki wrote: > > > Hello, > > since weekend I have problems with LXC containers (upstart based) booting on Proxmox 5.3-7. Today I have created new container and after starting it only few upstart processes are running: With a *privileged* 14.04 container I can reproduce this. A bisect revealed an lxc commit which causes no functional change but simply removes an unnecessary mount entry which seems to confuse upstart in some way. Considering the age and soon-to-be-EOL-ness of 14.04 I recommend moving away from it on the one hand, or, if absolutely required, use a VM (or at least an unprivileged container). Alternatively if you like digging into upstart scripts, you can try to figure out why it needs to find a read-write mounted /sys mount entry: Previously the container saw: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime) Now it only sees sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime) Link: https://github.com/lxc/lxc/issues/2602 Link: https://github.com/lxc/lxc/commit/51a922baf724689ff3a0df938ca8975601c9c815 From gilberto.nunes32 at gmail.com Wed Jan 16 13:11:06 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 10:11:06 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage Message-ID: Hi there Anybody else experiment hight memory usage in Proxmox CEPH Storage Server? I have a 6 node PVE CEPH and after upgrade, I have noticed this high memory usage... All server has 16GB of ram. I know this is not recomended, but that what I have at the moment. In fact, just 3 servers ran with about 90% of memory usage. All server is IBM x3200 m2 with SATA disks... Here's ceph osd tree ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 38.50000 root default -3 4.00000 host pve-ceph01 0 hdd 1.00000 osd.0 up 1.00000 0 1 hdd 1.00000 osd.1 up 1.00000 0 2 hdd 1.00000 osd.2 up 1.00000 0 3 hdd 1.00000 osd.3 up 1.00000 0 -5 8.00000 host pve-ceph02 4 hdd 2.00000 osd.4 up 1.00000 1.00000 5 hdd 2.00000 osd.5 up 1.00000 1.00000 6 hdd 2.00000 osd.6 up 1.00000 1.00000 7 hdd 2.00000 osd.7 up 1.00000 1.00000 -7 9.00000 host pve-ceph03 8 hdd 3.00000 osd.8 up 1.00000 1.00000 9 hdd 3.00000 osd.9 up 1.00000 1.00000 10 hdd 3.00000 osd.10 up 1.00000 1.00000 -9 12.00000 host pve-ceph04 11 hdd 3.00000 osd.11 up 1.00000 1.00000 12 hdd 3.00000 osd.12 up 1.00000 1.00000 13 hdd 3.00000 osd.13 up 1.00000 1.00000 14 hdd 3.00000 osd.14 up 1.00000 1.00000 -11 1.00000 host pve-ceph05 15 hdd 0.50000 osd.15 up 1.00000 0 16 hdd 0.50000 osd.16 up 1.00000 0 -13 4.50000 host pve-ceph06 17 hdd 1.00000 osd.17 up 1.00000 0 18 hdd 1.00000 osd.18 up 1.00000 0 20 hdd 1.00000 osd.20 up 1.00000 0 21 hdd 1.50000 osd.21 up 1.00000 1.00000 --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From elacunza at binovo.es Wed Jan 16 13:17:27 2019 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 16 Jan 2019 13:17:27 +0100 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: Message-ID: <484a69dd-d2ee-e42b-22c5-dea4784d1674@binovo.es> Hi Gilberto, Are you using Blustore? What version of Ceph? El 16/1/19 a las 13:11, Gilberto Nunes escribi?: > Hi there > > Anybody else experiment hight memory usage in Proxmox CEPH Storage Server? > I have a 6 node PVE CEPH and after upgrade, I have noticed this high memory > usage... > All server has 16GB of ram. I know this is not recomended, but that what I > have at the moment. > In fact, just 3 servers ran with about 90% of memory usage. > All server is IBM x3200 m2 with SATA disks... > Here's ceph osd tree > ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 38.50000 root default > -3 4.00000 host pve-ceph01 > 0 hdd 1.00000 osd.0 up 1.00000 0 > 1 hdd 1.00000 osd.1 up 1.00000 0 > 2 hdd 1.00000 osd.2 up 1.00000 0 > 3 hdd 1.00000 osd.3 up 1.00000 0 > -5 8.00000 host pve-ceph02 > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 > -7 9.00000 host pve-ceph03 > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 > -9 12.00000 host pve-ceph04 > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 > -11 1.00000 host pve-ceph05 > 15 hdd 0.50000 osd.15 up 1.00000 0 > 16 hdd 0.50000 osd.16 up 1.00000 0 > -13 4.50000 host pve-ceph06 > 17 hdd 1.00000 osd.17 up 1.00000 0 > 18 hdd 1.00000 osd.18 up 1.00000 0 > 20 hdd 1.00000 osd.20 up 1.00000 0 > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From gilberto.nunes32 at gmail.com Wed Jan 16 13:23:31 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 10:23:31 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: <484a69dd-d2ee-e42b-22c5-dea4784d1674@binovo.es> References: <484a69dd-d2ee-e42b-22c5-dea4784d1674@binovo.es> Message-ID: Hi... I am using BlueStore. pve-manager/5.3-7/e8ed1e22 (running kernel: 4.15.18-9-pve) ceph 12.2.10-pve1 Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 10:17, Eneko Lacunza escreveu: > Hi Gilberto, > > Are you using Blustore? What version of Ceph? > > El 16/1/19 a las 13:11, Gilberto Nunes escribi?: > > Hi there > > > > Anybody else experiment hight memory usage in Proxmox CEPH Storage > Server? > > I have a 6 node PVE CEPH and after upgrade, I have noticed this high > memory > > usage... > > All server has 16GB of ram. I know this is not recomended, but that what > I > > have at the moment. > > In fact, just 3 servers ran with about 90% of memory usage. > > All server is IBM x3200 m2 with SATA disks... > > Here's ceph osd tree > > ceph osd tree > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > > -1 38.50000 root default > > -3 4.00000 host pve-ceph01 > > 0 hdd 1.00000 osd.0 up 1.00000 0 > > 1 hdd 1.00000 osd.1 up 1.00000 0 > > 2 hdd 1.00000 osd.2 up 1.00000 0 > > 3 hdd 1.00000 osd.3 up 1.00000 0 > > -5 8.00000 host pve-ceph02 > > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 > > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 > > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 > > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 > > -7 9.00000 host pve-ceph03 > > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 > > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 > > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 > > -9 12.00000 host pve-ceph04 > > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 > > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 > > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 > > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 > > -11 1.00000 host pve-ceph05 > > 15 hdd 0.50000 osd.15 up 1.00000 0 > > 16 hdd 0.50000 osd.16 up 1.00000 0 > > -13 4.50000 host pve-ceph06 > > 17 hdd 1.00000 osd.17 up 1.00000 0 > > 18 hdd 1.00000 osd.18 up 1.00000 0 > > 20 hdd 1.00000 osd.20 up 1.00000 0 > > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 > > > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > -- > Zuzendari Teknikoa / Director T?cnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Wed Jan 16 13:28:56 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 10:28:56 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: <484a69dd-d2ee-e42b-22c5-dea4784d1674@binovo.es> Message-ID: pve-ceph01:~# ceph status cluster: id: e67534b4-0a66-48db-ad6f-aa0868e962d8 health: HEALTH_WARN nobackfill,norebalance,norecover,nodeep-scrub flag(s) set 394106/2589186 objects misplaced (15.221%) Degraded data redundancy: 124011/2589186 objects degraded (4.790%), 158 pgs degraded, 76 pgs undersized services: mon: 5 daemons, quorum pve-ceph01,pve-ceph02,pve-ceph03,pve-ceph04,pve-ceph05 mgr: pve-ceph05(active), standbys: pve-ceph01, pve-ceph03, pve-ceph04, pve-ceph02 osd: 21 osds: 21 up, 21 in; 230 remapped pgs flags nobackfill,norebalance,norecover,nodeep-scrub data: pools: 1 pools, 512 pgs objects: 863.06k objects, 3.17TiB usage: 9.73TiB used, 53.0TiB / 62.8TiB avail pgs: 124011/2589186 objects degraded (4.790%) 394106/2589186 objects misplaced (15.221%) 180 active+clean 76 active+remapped+backfill_wait 70 active+recovery_wait 63 active+undersized+degraded+remapped+backfill_wait 49 active+recovery_wait+degraded+remapped 32 active+recovery_wait+degraded 28 active+recovery_wait+remapped 12 active+recovery_wait+undersized+degraded+remapped 1 active+recovering+degraded+remapped 1 active+undersized+degraded+remapped+backfilling io: client: 694KiB/s rd, 172KiB/s wr, 118op/s rd, 38op/s wr recovery: 257KiB/s, 0objects/s --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 10:23, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > Hi... > I am using BlueStore. > pve-manager/5.3-7/e8ed1e22 (running kernel: 4.15.18-9-pve) > ceph 12.2.10-pve1 > > Thanks > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 16 de jan de 2019 ?s 10:17, Eneko Lacunza > escreveu: > >> Hi Gilberto, >> >> Are you using Blustore? What version of Ceph? >> >> El 16/1/19 a las 13:11, Gilberto Nunes escribi?: >> > Hi there >> > >> > Anybody else experiment hight memory usage in Proxmox CEPH Storage >> Server? >> > I have a 6 node PVE CEPH and after upgrade, I have noticed this high >> memory >> > usage... >> > All server has 16GB of ram. I know this is not recomended, but that >> what I >> > have at the moment. >> > In fact, just 3 servers ran with about 90% of memory usage. >> > All server is IBM x3200 m2 with SATA disks... >> > Here's ceph osd tree >> > ceph osd tree >> > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> > -1 38.50000 root default >> > -3 4.00000 host pve-ceph01 >> > 0 hdd 1.00000 osd.0 up 1.00000 0 >> > 1 hdd 1.00000 osd.1 up 1.00000 0 >> > 2 hdd 1.00000 osd.2 up 1.00000 0 >> > 3 hdd 1.00000 osd.3 up 1.00000 0 >> > -5 8.00000 host pve-ceph02 >> > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 >> > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 >> > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 >> > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 >> > -7 9.00000 host pve-ceph03 >> > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 >> > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 >> > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 >> > -9 12.00000 host pve-ceph04 >> > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 >> > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 >> > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 >> > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 >> > -11 1.00000 host pve-ceph05 >> > 15 hdd 0.50000 osd.15 up 1.00000 0 >> > 16 hdd 0.50000 osd.16 up 1.00000 0 >> > -13 4.50000 host pve-ceph06 >> > 17 hdd 1.00000 osd.17 up 1.00000 0 >> > 18 hdd 1.00000 osd.18 up 1.00000 0 >> > 20 hdd 1.00000 osd.20 up 1.00000 0 >> > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 >> > >> > --- >> > Gilberto Nunes Ferreira >> > >> > (47) 3025-5907 >> > (47) 99676-7530 - Whatsapp / Telegram >> > >> > Skype: gilberto.nunes36 >> > _______________________________________________ >> > pve-user mailing list >> > pve-user at pve.proxmox.com >> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> -- >> Zuzendari Teknikoa / Director T?cnico >> Binovo IT Human Project, S.L. >> Telf. 943569206 >> Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) >> www.binovo.es >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > From ronny+pve-user at aasen.cx Wed Jan 16 13:42:13 2019 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Wed, 16 Jan 2019 13:42:13 +0100 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: <484a69dd-d2ee-e42b-22c5-dea4784d1674@binovo.es> Message-ID: <146d78f4-61c0-332e-e694-e3be4175c80c@aasen.cx> the memory consumption of the machine is an aggregate of multiple consumers. identify what is using memory try commands like top -o VIRT and top -o RES to reduce VM memory usage, you can move or stop virtual machines, reconfigure them with less memory, or try to use KSM if you nave many identical vm's https://en.wikipedia.org/wiki/Kernel_same-page_merging to reduce ceoh osd memory consumption you can tweak the bluestore memory cache http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#automatic-cache-sizing ; with only 16GB i think you need to try to reduce cache (and hence performance) a bit here. also ceph memory usage increase quite a bit when recovering and backfilling, so when planning resource requirements, plan for the recovery situation, and have some free overhead. kind regards Ronny Aasen On 16.01.2019 13:28, Gilberto Nunes wrote: > pve-ceph01:~# ceph status > cluster: > id: e67534b4-0a66-48db-ad6f-aa0868e962d8 > health: HEALTH_WARN > nobackfill,norebalance,norecover,nodeep-scrub flag(s) set > 394106/2589186 objects misplaced (15.221%) > Degraded data redundancy: 124011/2589186 objects degraded > (4.790%), 158 pgs degraded, 76 pgs undersized > > services: > mon: 5 daemons, quorum > pve-ceph01,pve-ceph02,pve-ceph03,pve-ceph04,pve-ceph05 > mgr: pve-ceph05(active), standbys: pve-ceph01, pve-ceph03, pve-ceph04, > pve-ceph02 > osd: 21 osds: 21 up, 21 in; 230 remapped pgs > flags nobackfill,norebalance,norecover,nodeep-scrub > > data: > pools: 1 pools, 512 pgs > objects: 863.06k objects, 3.17TiB > usage: 9.73TiB used, 53.0TiB / 62.8TiB avail > pgs: 124011/2589186 objects degraded (4.790%) > 394106/2589186 objects misplaced (15.221%) > 180 active+clean > 76 active+remapped+backfill_wait > 70 active+recovery_wait > 63 active+undersized+degraded+remapped+backfill_wait > 49 active+recovery_wait+degraded+remapped > 32 active+recovery_wait+degraded > 28 active+recovery_wait+remapped > 12 active+recovery_wait+undersized+degraded+remapped > 1 active+recovering+degraded+remapped > 1 active+undersized+degraded+remapped+backfilling > > io: > client: 694KiB/s rd, 172KiB/s wr, 118op/s rd, 38op/s wr > recovery: 257KiB/s, 0objects/s > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 16 de jan de 2019 ?s 10:23, Gilberto Nunes < > gilberto.nunes32 at gmail.com> escreveu: > >> Hi... >> I am using BlueStore. >> pve-manager/5.3-7/e8ed1e22 (running kernel: 4.15.18-9-pve) >> ceph 12.2.10-pve1 >> >> Thanks >> --- >> Gilberto Nunes Ferreira >> >> (47) 3025-5907 >> (47) 99676-7530 - Whatsapp / Telegram >> >> Skype: gilberto.nunes36 >> >> >> >> >> >> Em qua, 16 de jan de 2019 ?s 10:17, Eneko Lacunza >> escreveu: >> >>> Hi Gilberto, >>> >>> Are you using Blustore? What version of Ceph? >>> >>> El 16/1/19 a las 13:11, Gilberto Nunes escribi?: >>>> Hi there >>>> >>>> Anybody else experiment hight memory usage in Proxmox CEPH Storage >>> Server? >>>> I have a 6 node PVE CEPH and after upgrade, I have noticed this high >>> memory >>>> usage... >>>> All server has 16GB of ram. I know this is not recomended, but that >>> what I >>>> have at the moment. >>>> In fact, just 3 servers ran with about 90% of memory usage. >>>> All server is IBM x3200 m2 with SATA disks... >>>> Here's ceph osd tree >>>> ceph osd tree >>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >>>> -1 38.50000 root default >>>> -3 4.00000 host pve-ceph01 >>>> 0 hdd 1.00000 osd.0 up 1.00000 0 >>>> 1 hdd 1.00000 osd.1 up 1.00000 0 >>>> 2 hdd 1.00000 osd.2 up 1.00000 0 >>>> 3 hdd 1.00000 osd.3 up 1.00000 0 >>>> -5 8.00000 host pve-ceph02 >>>> 4 hdd 2.00000 osd.4 up 1.00000 1.00000 >>>> 5 hdd 2.00000 osd.5 up 1.00000 1.00000 >>>> 6 hdd 2.00000 osd.6 up 1.00000 1.00000 >>>> 7 hdd 2.00000 osd.7 up 1.00000 1.00000 >>>> -7 9.00000 host pve-ceph03 >>>> 8 hdd 3.00000 osd.8 up 1.00000 1.00000 >>>> 9 hdd 3.00000 osd.9 up 1.00000 1.00000 >>>> 10 hdd 3.00000 osd.10 up 1.00000 1.00000 >>>> -9 12.00000 host pve-ceph04 >>>> 11 hdd 3.00000 osd.11 up 1.00000 1.00000 >>>> 12 hdd 3.00000 osd.12 up 1.00000 1.00000 >>>> 13 hdd 3.00000 osd.13 up 1.00000 1.00000 >>>> 14 hdd 3.00000 osd.14 up 1.00000 1.00000 >>>> -11 1.00000 host pve-ceph05 >>>> 15 hdd 0.50000 osd.15 up 1.00000 0 >>>> 16 hdd 0.50000 osd.16 up 1.00000 0 >>>> -13 4.50000 host pve-ceph06 >>>> 17 hdd 1.00000 osd.17 up 1.00000 0 >>>> 18 hdd 1.00000 osd.18 up 1.00000 0 >>>> 20 hdd 1.00000 osd.20 up 1.00000 0 >>>> 21 hdd 1.50000 osd.21 up 1.00000 1.00000 >>>> >>>> --- >>>> Gilberto Nunes Ferreira >>>> >>>> (47) 3025-5907 >>>> (47) 99676-7530 - Whatsapp / Telegram >>>> >>>> Skype: gilberto.nunes36 >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >>> >>> -- >>> Zuzendari Teknikoa / Director T?cnico >>> Binovo IT Human Project, S.L. >>> Telf. 943569206 >>> Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) >>> www.binovo.es >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Wed Jan 16 13:52:44 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 10:52:44 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: <146d78f4-61c0-332e-e694-e3be4175c80c@aasen.cx> References: <484a69dd-d2ee-e42b-22c5-dea4784d1674@binovo.es> <146d78f4-61c0-332e-e694-e3be4175c80c@aasen.cx> Message-ID: I already do that ceph config set bluestore_cache_size 536870912 ceph config set bluestore_cache_size_hdd 536870912 ceph config set bluestore_cache_size_ssd 1073741824 Any other clue you may have will be welcome. --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 10:42, Ronny Aasen escreveu: > the memory consumption of the machine is an aggregate of multiple > consumers. > > identify what is using memory try commands like > top -o VIRT and top -o RES > > > to reduce VM memory usage, you can move or stop virtual machines, > reconfigure them with less memory, or try to use KSM if you nave many > identical vm's https://en.wikipedia.org/wiki/Kernel_same-page_merging > > to reduce ceoh osd memory consumption you can tweak the bluestore memory > cache > > http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#automatic-cache-sizing > ; with only 16GB i think you need to try to reduce cache (and hence > performance) a bit here. > > also ceph memory usage increase quite a bit when recovering and > backfilling, so when planning resource requirements, plan for the > recovery situation, and have some free overhead. > > kind regards > Ronny Aasen > > On 16.01.2019 13:28, Gilberto Nunes wrote: > > pve-ceph01:~# ceph status > > cluster: > > id: e67534b4-0a66-48db-ad6f-aa0868e962d8 > > health: HEALTH_WARN > > nobackfill,norebalance,norecover,nodeep-scrub flag(s) set > > 394106/2589186 objects misplaced (15.221%) > > Degraded data redundancy: 124011/2589186 objects degraded > > (4.790%), 158 pgs degraded, 76 pgs undersized > > > > services: > > mon: 5 daemons, quorum > > pve-ceph01,pve-ceph02,pve-ceph03,pve-ceph04,pve-ceph05 > > mgr: pve-ceph05(active), standbys: pve-ceph01, pve-ceph03, > pve-ceph04, > > pve-ceph02 > > osd: 21 osds: 21 up, 21 in; 230 remapped pgs > > flags nobackfill,norebalance,norecover,nodeep-scrub > > > > data: > > pools: 1 pools, 512 pgs > > objects: 863.06k objects, 3.17TiB > > usage: 9.73TiB used, 53.0TiB / 62.8TiB avail > > pgs: 124011/2589186 objects degraded (4.790%) > > 394106/2589186 objects misplaced (15.221%) > > 180 active+clean > > 76 active+remapped+backfill_wait > > 70 active+recovery_wait > > 63 active+undersized+degraded+remapped+backfill_wait > > 49 active+recovery_wait+degraded+remapped > > 32 active+recovery_wait+degraded > > 28 active+recovery_wait+remapped > > 12 active+recovery_wait+undersized+degraded+remapped > > 1 active+recovering+degraded+remapped > > 1 active+undersized+degraded+remapped+backfilling > > > > io: > > client: 694KiB/s rd, 172KiB/s wr, 118op/s rd, 38op/s wr > > recovery: 257KiB/s, 0objects/s > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > > > > > > > > > > > Em qua, 16 de jan de 2019 ?s 10:23, Gilberto Nunes < > > gilberto.nunes32 at gmail.com> escreveu: > > > >> Hi... > >> I am using BlueStore. > >> pve-manager/5.3-7/e8ed1e22 (running kernel: 4.15.18-9-pve) > >> ceph 12.2.10-pve1 > >> > >> Thanks > >> --- > >> Gilberto Nunes Ferreira > >> > >> (47) 3025-5907 > >> (47) 99676-7530 - Whatsapp / Telegram > >> > >> Skype: gilberto.nunes36 > >> > >> > >> > >> > >> > >> Em qua, 16 de jan de 2019 ?s 10:17, Eneko Lacunza > >> escreveu: > >> > >>> Hi Gilberto, > >>> > >>> Are you using Blustore? What version of Ceph? > >>> > >>> El 16/1/19 a las 13:11, Gilberto Nunes escribi?: > >>>> Hi there > >>>> > >>>> Anybody else experiment hight memory usage in Proxmox CEPH Storage > >>> Server? > >>>> I have a 6 node PVE CEPH and after upgrade, I have noticed this high > >>> memory > >>>> usage... > >>>> All server has 16GB of ram. I know this is not recomended, but that > >>> what I > >>>> have at the moment. > >>>> In fact, just 3 servers ran with about 90% of memory usage. > >>>> All server is IBM x3200 m2 with SATA disks... > >>>> Here's ceph osd tree > >>>> ceph osd tree > >>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > >>>> -1 38.50000 root default > >>>> -3 4.00000 host pve-ceph01 > >>>> 0 hdd 1.00000 osd.0 up 1.00000 0 > >>>> 1 hdd 1.00000 osd.1 up 1.00000 0 > >>>> 2 hdd 1.00000 osd.2 up 1.00000 0 > >>>> 3 hdd 1.00000 osd.3 up 1.00000 0 > >>>> -5 8.00000 host pve-ceph02 > >>>> 4 hdd 2.00000 osd.4 up 1.00000 1.00000 > >>>> 5 hdd 2.00000 osd.5 up 1.00000 1.00000 > >>>> 6 hdd 2.00000 osd.6 up 1.00000 1.00000 > >>>> 7 hdd 2.00000 osd.7 up 1.00000 1.00000 > >>>> -7 9.00000 host pve-ceph03 > >>>> 8 hdd 3.00000 osd.8 up 1.00000 1.00000 > >>>> 9 hdd 3.00000 osd.9 up 1.00000 1.00000 > >>>> 10 hdd 3.00000 osd.10 up 1.00000 1.00000 > >>>> -9 12.00000 host pve-ceph04 > >>>> 11 hdd 3.00000 osd.11 up 1.00000 1.00000 > >>>> 12 hdd 3.00000 osd.12 up 1.00000 1.00000 > >>>> 13 hdd 3.00000 osd.13 up 1.00000 1.00000 > >>>> 14 hdd 3.00000 osd.14 up 1.00000 1.00000 > >>>> -11 1.00000 host pve-ceph05 > >>>> 15 hdd 0.50000 osd.15 up 1.00000 0 > >>>> 16 hdd 0.50000 osd.16 up 1.00000 0 > >>>> -13 4.50000 host pve-ceph06 > >>>> 17 hdd 1.00000 osd.17 up 1.00000 0 > >>>> 18 hdd 1.00000 osd.18 up 1.00000 0 > >>>> 20 hdd 1.00000 osd.20 up 1.00000 0 > >>>> 21 hdd 1.50000 osd.21 up 1.00000 1.00000 > >>>> > >>>> --- > >>>> Gilberto Nunes Ferreira > >>>> > >>>> (47) 3025-5907 > >>>> (47) 99676-7530 - Whatsapp / Telegram > >>>> > >>>> Skype: gilberto.nunes36 > >>>> _______________________________________________ > >>>> pve-user mailing list > >>>> pve-user at pve.proxmox.com > >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>> > >>> > >>> -- > >>> Zuzendari Teknikoa / Director T?cnico > >>> Binovo IT Human Project, S.L. > >>> Telf. 943569206 > >>> Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > >>> www.binovo.es > >>> > >>> _______________________________________________ > >>> pve-user mailing list > >>> pve-user at pve.proxmox.com > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>> > >> > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Wed Jan 16 14:04:01 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 11:04:01 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: <484a69dd-d2ee-e42b-22c5-dea4784d1674@binovo.es> <146d78f4-61c0-332e-e694-e3be4175c80c@aasen.cx> Message-ID: May I use this command: ceph osd primary-affinity In order to reduce slow requests? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 10:52, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > I already do that > ceph config set bluestore_cache_size 536870912 > ceph config set bluestore_cache_size_hdd 536870912 > ceph config set bluestore_cache_size_ssd 1073741824 > > Any other clue you may have will be welcome. > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 16 de jan de 2019 ?s 10:42, Ronny Aasen > escreveu: > >> the memory consumption of the machine is an aggregate of multiple >> consumers. >> >> identify what is using memory try commands like >> top -o VIRT and top -o RES >> >> >> to reduce VM memory usage, you can move or stop virtual machines, >> reconfigure them with less memory, or try to use KSM if you nave many >> identical vm's https://en.wikipedia.org/wiki/Kernel_same-page_merging >> >> to reduce ceoh osd memory consumption you can tweak the bluestore memory >> cache >> >> http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#automatic-cache-sizing >> ; with only 16GB i think you need to try to reduce cache (and hence >> performance) a bit here. >> >> also ceph memory usage increase quite a bit when recovering and >> backfilling, so when planning resource requirements, plan for the >> recovery situation, and have some free overhead. >> >> kind regards >> Ronny Aasen >> >> On 16.01.2019 13:28, Gilberto Nunes wrote: >> > pve-ceph01:~# ceph status >> > cluster: >> > id: e67534b4-0a66-48db-ad6f-aa0868e962d8 >> > health: HEALTH_WARN >> > nobackfill,norebalance,norecover,nodeep-scrub flag(s) set >> > 394106/2589186 objects misplaced (15.221%) >> > Degraded data redundancy: 124011/2589186 objects degraded >> > (4.790%), 158 pgs degraded, 76 pgs undersized >> > >> > services: >> > mon: 5 daemons, quorum >> > pve-ceph01,pve-ceph02,pve-ceph03,pve-ceph04,pve-ceph05 >> > mgr: pve-ceph05(active), standbys: pve-ceph01, pve-ceph03, >> pve-ceph04, >> > pve-ceph02 >> > osd: 21 osds: 21 up, 21 in; 230 remapped pgs >> > flags nobackfill,norebalance,norecover,nodeep-scrub >> > >> > data: >> > pools: 1 pools, 512 pgs >> > objects: 863.06k objects, 3.17TiB >> > usage: 9.73TiB used, 53.0TiB / 62.8TiB avail >> > pgs: 124011/2589186 objects degraded (4.790%) >> > 394106/2589186 objects misplaced (15.221%) >> > 180 active+clean >> > 76 active+remapped+backfill_wait >> > 70 active+recovery_wait >> > 63 active+undersized+degraded+remapped+backfill_wait >> > 49 active+recovery_wait+degraded+remapped >> > 32 active+recovery_wait+degraded >> > 28 active+recovery_wait+remapped >> > 12 active+recovery_wait+undersized+degraded+remapped >> > 1 active+recovering+degraded+remapped >> > 1 active+undersized+degraded+remapped+backfilling >> > >> > io: >> > client: 694KiB/s rd, 172KiB/s wr, 118op/s rd, 38op/s wr >> > recovery: 257KiB/s, 0objects/s >> > --- >> > Gilberto Nunes Ferreira >> > >> > (47) 3025-5907 >> > (47) 99676-7530 - Whatsapp / Telegram >> > >> > Skype: gilberto.nunes36 >> > >> > >> > >> > >> > >> > Em qua, 16 de jan de 2019 ?s 10:23, Gilberto Nunes < >> > gilberto.nunes32 at gmail.com> escreveu: >> > >> >> Hi... >> >> I am using BlueStore. >> >> pve-manager/5.3-7/e8ed1e22 (running kernel: 4.15.18-9-pve) >> >> ceph 12.2.10-pve1 >> >> >> >> Thanks >> >> --- >> >> Gilberto Nunes Ferreira >> >> >> >> (47) 3025-5907 >> >> (47) 99676-7530 - Whatsapp / Telegram >> >> >> >> Skype: gilberto.nunes36 >> >> >> >> >> >> >> >> >> >> >> >> Em qua, 16 de jan de 2019 ?s 10:17, Eneko Lacunza >> >> escreveu: >> >> >> >>> Hi Gilberto, >> >>> >> >>> Are you using Blustore? What version of Ceph? >> >>> >> >>> El 16/1/19 a las 13:11, Gilberto Nunes escribi?: >> >>>> Hi there >> >>>> >> >>>> Anybody else experiment hight memory usage in Proxmox CEPH Storage >> >>> Server? >> >>>> I have a 6 node PVE CEPH and after upgrade, I have noticed this high >> >>> memory >> >>>> usage... >> >>>> All server has 16GB of ram. I know this is not recomended, but that >> >>> what I >> >>>> have at the moment. >> >>>> In fact, just 3 servers ran with about 90% of memory usage. >> >>>> All server is IBM x3200 m2 with SATA disks... >> >>>> Here's ceph osd tree >> >>>> ceph osd tree >> >>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> >>>> -1 38.50000 root default >> >>>> -3 4.00000 host pve-ceph01 >> >>>> 0 hdd 1.00000 osd.0 up 1.00000 0 >> >>>> 1 hdd 1.00000 osd.1 up 1.00000 0 >> >>>> 2 hdd 1.00000 osd.2 up 1.00000 0 >> >>>> 3 hdd 1.00000 osd.3 up 1.00000 0 >> >>>> -5 8.00000 host pve-ceph02 >> >>>> 4 hdd 2.00000 osd.4 up 1.00000 1.00000 >> >>>> 5 hdd 2.00000 osd.5 up 1.00000 1.00000 >> >>>> 6 hdd 2.00000 osd.6 up 1.00000 1.00000 >> >>>> 7 hdd 2.00000 osd.7 up 1.00000 1.00000 >> >>>> -7 9.00000 host pve-ceph03 >> >>>> 8 hdd 3.00000 osd.8 up 1.00000 1.00000 >> >>>> 9 hdd 3.00000 osd.9 up 1.00000 1.00000 >> >>>> 10 hdd 3.00000 osd.10 up 1.00000 1.00000 >> >>>> -9 12.00000 host pve-ceph04 >> >>>> 11 hdd 3.00000 osd.11 up 1.00000 1.00000 >> >>>> 12 hdd 3.00000 osd.12 up 1.00000 1.00000 >> >>>> 13 hdd 3.00000 osd.13 up 1.00000 1.00000 >> >>>> 14 hdd 3.00000 osd.14 up 1.00000 1.00000 >> >>>> -11 1.00000 host pve-ceph05 >> >>>> 15 hdd 0.50000 osd.15 up 1.00000 0 >> >>>> 16 hdd 0.50000 osd.16 up 1.00000 0 >> >>>> -13 4.50000 host pve-ceph06 >> >>>> 17 hdd 1.00000 osd.17 up 1.00000 0 >> >>>> 18 hdd 1.00000 osd.18 up 1.00000 0 >> >>>> 20 hdd 1.00000 osd.20 up 1.00000 0 >> >>>> 21 hdd 1.50000 osd.21 up 1.00000 1.00000 >> >>>> >> >>>> --- >> >>>> Gilberto Nunes Ferreira >> >>>> >> >>>> (47) 3025-5907 >> >>>> (47) 99676-7530 - Whatsapp / Telegram >> >>>> >> >>>> Skype: gilberto.nunes36 >> >>>> _______________________________________________ >> >>>> pve-user mailing list >> >>>> pve-user at pve.proxmox.com >> >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >>> >> >>> >> >>> -- >> >>> Zuzendari Teknikoa / Director T?cnico >> >>> Binovo IT Human Project, S.L. >> >>> Telf. 943569206 >> >>> Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) >> >>> www.binovo.es >> >>> >> >>> _______________________________________________ >> >>> pve-user mailing list >> >>> pve-user at pve.proxmox.com >> >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >>> >> >> >> > _______________________________________________ >> > pve-user mailing list >> > pve-user at pve.proxmox.com >> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > From a.antreich at proxmox.com Wed Jan 16 14:07:26 2019 From: a.antreich at proxmox.com (Alwin Antreich) Date: Wed, 16 Jan 2019 14:07:26 +0100 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: Message-ID: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Hello Gilberto, On Wed, Jan 16, 2019 at 10:11:06AM -0200, Gilberto Nunes wrote: > Hi there > > Anybody else experiment hight memory usage in Proxmox CEPH Storage Server? > I have a 6 node PVE CEPH and after upgrade, I have noticed this high memory > usage... > All server has 16GB of ram. I know this is not recomended, but that what I > have at the moment. > In fact, just 3 servers ran with about 90% of memory usage. > All server is IBM x3200 m2 with SATA disks... > Here's ceph osd tree > ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 38.50000 root default > -3 4.00000 host pve-ceph01 > 0 hdd 1.00000 osd.0 up 1.00000 0 > 1 hdd 1.00000 osd.1 up 1.00000 0 > 2 hdd 1.00000 osd.2 up 1.00000 0 > 3 hdd 1.00000 osd.3 up 1.00000 0 > -5 8.00000 host pve-ceph02 > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 > -7 9.00000 host pve-ceph03 > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 > -9 12.00000 host pve-ceph04 > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 > -11 1.00000 host pve-ceph05 > 15 hdd 0.50000 osd.15 up 1.00000 0 > 16 hdd 0.50000 osd.16 up 1.00000 0 > -13 4.50000 host pve-ceph06 > 17 hdd 1.00000 osd.17 up 1.00000 0 > 18 hdd 1.00000 osd.18 up 1.00000 0 > 20 hdd 1.00000 osd.20 up 1.00000 0 > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 > Did you see the changle on package upgrade? It explains why it is using more memory then before. http://download.proxmox.com/debian/ceph-luminous/dists/stretch/main/binary-amd64/ceph_12.2.10-pve1.changelog -- Cheers, Alwin From gilberto.nunes32 at gmail.com Wed Jan 16 14:11:05 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 11:11:05 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Message-ID: Oh! I see it now! So what I need to change is osd_memory_target instead bluestore_cache_*, right? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 11:07, Alwin Antreich escreveu: > Hello Gilberto, > > On Wed, Jan 16, 2019 at 10:11:06AM -0200, Gilberto Nunes wrote: > > Hi there > > > > Anybody else experiment hight memory usage in Proxmox CEPH Storage > Server? > > I have a 6 node PVE CEPH and after upgrade, I have noticed this high > memory > > usage... > > All server has 16GB of ram. I know this is not recomended, but that what > I > > have at the moment. > > In fact, just 3 servers ran with about 90% of memory usage. > > All server is IBM x3200 m2 with SATA disks... > > Here's ceph osd tree > > ceph osd tree > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > > -1 38.50000 root default > > -3 4.00000 host pve-ceph01 > > 0 hdd 1.00000 osd.0 up 1.00000 0 > > 1 hdd 1.00000 osd.1 up 1.00000 0 > > 2 hdd 1.00000 osd.2 up 1.00000 0 > > 3 hdd 1.00000 osd.3 up 1.00000 0 > > -5 8.00000 host pve-ceph02 > > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 > > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 > > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 > > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 > > -7 9.00000 host pve-ceph03 > > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 > > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 > > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 > > -9 12.00000 host pve-ceph04 > > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 > > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 > > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 > > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 > > -11 1.00000 host pve-ceph05 > > 15 hdd 0.50000 osd.15 up 1.00000 0 > > 16 hdd 0.50000 osd.16 up 1.00000 0 > > -13 4.50000 host pve-ceph06 > > 17 hdd 1.00000 osd.17 up 1.00000 0 > > 18 hdd 1.00000 osd.18 up 1.00000 0 > > 20 hdd 1.00000 osd.20 up 1.00000 0 > > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 > > > Did you see the changle on package upgrade? It explains why it is using > more memory then before. > > http://download.proxmox.com/debian/ceph-luminous/dists/stretch/main/binary-amd64/ceph_12.2.10-pve1.changelog > > -- > Cheers, > Alwin > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Wed Jan 16 14:18:04 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 11:18:04 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Message-ID: SO I use this command ceph config set osd_memory_target 1073741824 And set it into /etc/pve/ceph.conf It's seems to me that have a positive effect... I am monitoring yet Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 11:11, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > Oh! I see it now! > So what I need to change is osd_memory_target instead bluestore_cache_*, > right? > > Thanks > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 16 de jan de 2019 ?s 11:07, Alwin Antreich > escreveu: > >> Hello Gilberto, >> >> On Wed, Jan 16, 2019 at 10:11:06AM -0200, Gilberto Nunes wrote: >> > Hi there >> > >> > Anybody else experiment hight memory usage in Proxmox CEPH Storage >> Server? >> > I have a 6 node PVE CEPH and after upgrade, I have noticed this high >> memory >> > usage... >> > All server has 16GB of ram. I know this is not recomended, but that >> what I >> > have at the moment. >> > In fact, just 3 servers ran with about 90% of memory usage. >> > All server is IBM x3200 m2 with SATA disks... >> > Here's ceph osd tree >> > ceph osd tree >> > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> > -1 38.50000 root default >> > -3 4.00000 host pve-ceph01 >> > 0 hdd 1.00000 osd.0 up 1.00000 0 >> > 1 hdd 1.00000 osd.1 up 1.00000 0 >> > 2 hdd 1.00000 osd.2 up 1.00000 0 >> > 3 hdd 1.00000 osd.3 up 1.00000 0 >> > -5 8.00000 host pve-ceph02 >> > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 >> > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 >> > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 >> > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 >> > -7 9.00000 host pve-ceph03 >> > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 >> > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 >> > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 >> > -9 12.00000 host pve-ceph04 >> > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 >> > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 >> > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 >> > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 >> > -11 1.00000 host pve-ceph05 >> > 15 hdd 0.50000 osd.15 up 1.00000 0 >> > 16 hdd 0.50000 osd.16 up 1.00000 0 >> > -13 4.50000 host pve-ceph06 >> > 17 hdd 1.00000 osd.17 up 1.00000 0 >> > 18 hdd 1.00000 osd.18 up 1.00000 0 >> > 20 hdd 1.00000 osd.20 up 1.00000 0 >> > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 >> > >> Did you see the changle on package upgrade? It explains why it is using >> more memory then before. >> >> http://download.proxmox.com/debian/ceph-luminous/dists/stretch/main/binary-amd64/ceph_12.2.10-pve1.changelog >> >> -- >> Cheers, >> Alwin >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > From gilberto.nunes32 at gmail.com Wed Jan 16 14:40:02 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 11:40:02 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Message-ID: My other question is if I could use affinity to prevent performance bottlenecks... I have 5 HDD which is 5 900 RPM.... So can I apply this affinity to this slow disks? --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 11:18, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > SO I use this command > > ceph config set osd_memory_target 1073741824 > And set it into /etc/pve/ceph.conf > It's seems to me that have a positive effect... > I am monitoring yet > > Thanks a lot > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 16 de jan de 2019 ?s 11:11, Gilberto Nunes < > gilberto.nunes32 at gmail.com> escreveu: > >> Oh! I see it now! >> So what I need to change is osd_memory_target instead bluestore_cache_*, >> right? >> >> Thanks >> --- >> Gilberto Nunes Ferreira >> >> (47) 3025-5907 >> (47) 99676-7530 - Whatsapp / Telegram >> >> Skype: gilberto.nunes36 >> >> >> >> >> >> Em qua, 16 de jan de 2019 ?s 11:07, Alwin Antreich < >> a.antreich at proxmox.com> escreveu: >> >>> Hello Gilberto, >>> >>> On Wed, Jan 16, 2019 at 10:11:06AM -0200, Gilberto Nunes wrote: >>> > Hi there >>> > >>> > Anybody else experiment hight memory usage in Proxmox CEPH Storage >>> Server? >>> > I have a 6 node PVE CEPH and after upgrade, I have noticed this high >>> memory >>> > usage... >>> > All server has 16GB of ram. I know this is not recomended, but that >>> what I >>> > have at the moment. >>> > In fact, just 3 servers ran with about 90% of memory usage. >>> > All server is IBM x3200 m2 with SATA disks... >>> > Here's ceph osd tree >>> > ceph osd tree >>> > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >>> > -1 38.50000 root default >>> > -3 4.00000 host pve-ceph01 >>> > 0 hdd 1.00000 osd.0 up 1.00000 0 >>> > 1 hdd 1.00000 osd.1 up 1.00000 0 >>> > 2 hdd 1.00000 osd.2 up 1.00000 0 >>> > 3 hdd 1.00000 osd.3 up 1.00000 0 >>> > -5 8.00000 host pve-ceph02 >>> > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 >>> > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 >>> > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 >>> > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 >>> > -7 9.00000 host pve-ceph03 >>> > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 >>> > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 >>> > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 >>> > -9 12.00000 host pve-ceph04 >>> > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 >>> > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 >>> > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 >>> > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 >>> > -11 1.00000 host pve-ceph05 >>> > 15 hdd 0.50000 osd.15 up 1.00000 0 >>> > 16 hdd 0.50000 osd.16 up 1.00000 0 >>> > -13 4.50000 host pve-ceph06 >>> > 17 hdd 1.00000 osd.17 up 1.00000 0 >>> > 18 hdd 1.00000 osd.18 up 1.00000 0 >>> > 20 hdd 1.00000 osd.20 up 1.00000 0 >>> > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 >>> > >>> Did you see the changle on package upgrade? It explains why it is using >>> more memory then before. >>> >>> http://download.proxmox.com/debian/ceph-luminous/dists/stretch/main/binary-amd64/ceph_12.2.10-pve1.changelog >>> >>> -- >>> Cheers, >>> Alwin >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> From gilberto.nunes32 at gmail.com Wed Jan 16 16:25:57 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 16 Jan 2019 13:25:57 -0200 Subject: [PVE-User] Proxmox Ceph high memory usage In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Message-ID: Well I realize that set memory to 1GB cause me a lot of trouble. Now set to 2GB and all is ok... --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 16 de jan de 2019 ?s 11:40, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > My other question is if I could use affinity to prevent performance > bottlenecks... > I have 5 HDD which is 5 900 RPM.... So can I apply this affinity to this > slow disks? > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 16 de jan de 2019 ?s 11:18, Gilberto Nunes < > gilberto.nunes32 at gmail.com> escreveu: > >> SO I use this command >> >> ceph config set osd_memory_target 1073741824 >> And set it into /etc/pve/ceph.conf >> It's seems to me that have a positive effect... >> I am monitoring yet >> >> Thanks a lot >> --- >> Gilberto Nunes Ferreira >> >> (47) 3025-5907 >> (47) 99676-7530 - Whatsapp / Telegram >> >> Skype: gilberto.nunes36 >> >> >> >> >> >> Em qua, 16 de jan de 2019 ?s 11:11, Gilberto Nunes < >> gilberto.nunes32 at gmail.com> escreveu: >> >>> Oh! I see it now! >>> So what I need to change is osd_memory_target instead bluestore_cache_*, >>> right? >>> >>> Thanks >>> --- >>> Gilberto Nunes Ferreira >>> >>> (47) 3025-5907 >>> (47) 99676-7530 - Whatsapp / Telegram >>> >>> Skype: gilberto.nunes36 >>> >>> >>> >>> >>> >>> Em qua, 16 de jan de 2019 ?s 11:07, Alwin Antreich < >>> a.antreich at proxmox.com> escreveu: >>> >>>> Hello Gilberto, >>>> >>>> On Wed, Jan 16, 2019 at 10:11:06AM -0200, Gilberto Nunes wrote: >>>> > Hi there >>>> > >>>> > Anybody else experiment hight memory usage in Proxmox CEPH Storage >>>> Server? >>>> > I have a 6 node PVE CEPH and after upgrade, I have noticed this high >>>> memory >>>> > usage... >>>> > All server has 16GB of ram. I know this is not recomended, but that >>>> what I >>>> > have at the moment. >>>> > In fact, just 3 servers ran with about 90% of memory usage. >>>> > All server is IBM x3200 m2 with SATA disks... >>>> > Here's ceph osd tree >>>> > ceph osd tree >>>> > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >>>> > -1 38.50000 root default >>>> > -3 4.00000 host pve-ceph01 >>>> > 0 hdd 1.00000 osd.0 up 1.00000 0 >>>> > 1 hdd 1.00000 osd.1 up 1.00000 0 >>>> > 2 hdd 1.00000 osd.2 up 1.00000 0 >>>> > 3 hdd 1.00000 osd.3 up 1.00000 0 >>>> > -5 8.00000 host pve-ceph02 >>>> > 4 hdd 2.00000 osd.4 up 1.00000 1.00000 >>>> > 5 hdd 2.00000 osd.5 up 1.00000 1.00000 >>>> > 6 hdd 2.00000 osd.6 up 1.00000 1.00000 >>>> > 7 hdd 2.00000 osd.7 up 1.00000 1.00000 >>>> > -7 9.00000 host pve-ceph03 >>>> > 8 hdd 3.00000 osd.8 up 1.00000 1.00000 >>>> > 9 hdd 3.00000 osd.9 up 1.00000 1.00000 >>>> > 10 hdd 3.00000 osd.10 up 1.00000 1.00000 >>>> > -9 12.00000 host pve-ceph04 >>>> > 11 hdd 3.00000 osd.11 up 1.00000 1.00000 >>>> > 12 hdd 3.00000 osd.12 up 1.00000 1.00000 >>>> > 13 hdd 3.00000 osd.13 up 1.00000 1.00000 >>>> > 14 hdd 3.00000 osd.14 up 1.00000 1.00000 >>>> > -11 1.00000 host pve-ceph05 >>>> > 15 hdd 0.50000 osd.15 up 1.00000 0 >>>> > 16 hdd 0.50000 osd.16 up 1.00000 0 >>>> > -13 4.50000 host pve-ceph06 >>>> > 17 hdd 1.00000 osd.17 up 1.00000 0 >>>> > 18 hdd 1.00000 osd.18 up 1.00000 0 >>>> > 20 hdd 1.00000 osd.20 up 1.00000 0 >>>> > 21 hdd 1.50000 osd.21 up 1.00000 1.00000 >>>> > >>>> Did you see the changle on package upgrade? It explains why it is using >>>> more memory then before. >>>> >>>> http://download.proxmox.com/debian/ceph-luminous/dists/stretch/main/binary-amd64/ceph_12.2.10-pve1.changelog >>>> >>>> -- >>>> Cheers, >>>> Alwin >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>> From sir_Misiek1 at o2.pl Tue Jan 22 15:24:02 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Tue, 22 Jan 2019 15:24:02 +0100 Subject: [PVE-User] aic94xx - problem In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Message-ID: modprobe aic94xx [? 600.202300] aic94xx: Adaptec aic94xx SAS/SATA driver version 1.0.3 loaded [? 600.202600] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to legacy IRQ 19 [? 600.203628] aic94xx: found Adaptec AIC-9405W SAS/SATA Host Adapter, device 0000:03:04.0 [? 600.203634] scsi host2: aic94xx [? 600.234870] aic94xx: Found sequencer Firmware version 1.1 (V17/10c6) [? 600.277468] aic94xx: device 0000:03:04.0: SAS addr 5005076a0144bd00, PCBA SN ORG, 4 phys, 4 enabled phys, flash present, BIOS build 1549 [? 600.277488] ------------[ cut here ]------------ [? 600.277490] sysfs: cannot create duplicate filename '/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/0000:03:04.0/revision' [? 600.277511] WARNING: CPU: 1 PID: 2281 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x56/0x70 [? 600.277513] Modules linked in: aic94xx(+) ip_set ip6table_filter ip6_tables iptable_filter softdog nfnetlink_log nfnetlink dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c gpio_ich radeon ttm snd_pcm drm_kms_helper intel_powerclamp input_leds snd_timer drm snd soundcore lpc_ich ipmi_si ipmi_devintf i2c_algo_bit fb_sys_fops pcspkr serio_raw ipmi_msghandler syscopyarea sysfillrect sysimgblt i3000_edac shpchp mac_hid zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq psmouse pata_acpi i2c_i801 libsas scsi_transport_sas e1000 tg3 ptp pps_core [last unloaded: aic94xx] [? 600.277595] CPU: 1 PID: 2281 Comm: modprobe Tainted: P??????? W O???? 4.15.17-1-pve #1 [? 600.277596] Hardware name: IBM IBM eServer 306m -[8491E6Y]-/M11ip/M11ix, BIOS IBM BIOS Version 1.29-[PAE129AUS-1.29]- 02/09/2006 [? 600.277600] RIP: 0010:sysfs_warn_dup+0x56/0x70 [? 600.277602] RSP: 0018:ffffad21c338f9d0 EFLAGS: 00010282 [? 600.277605] RAX: 0000000000000000 RBX: ffff950033c18000 RCX: 0000000000000006 [? 600.277606] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff95003fd16490 [? 600.277608] RBP: ffffad21c338f9e8 R08: 0000000000000001 R09: 0000000000000384 [? 600.277609] R10: 0000000000000001 R11: 0000000000000384 R12: ffffffffc01a07c4 [? 600.277611] R13: ffff950038c16908 R14: ffff950033314000 R15: 0000000000000004 [? 600.277613] FS:? 00007f9e245bb700(0000) GS:ffff95003fd00000(0000) knlGS:0000000000000000 [? 600.277615] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [? 600.277617] CR2: 00007fff0f58fff8 CR3: 00000001731b4000 CR4: 00000000000006e0 [? 600.277619] Call Trace: [? 600.277628]? sysfs_add_file_mode_ns+0x116/0x170 [? 600.277631]? sysfs_create_file_ns+0x2a/0x30 [? 600.277635]? device_create_file+0x42/0x80 [? 600.277643]? asd_pci_probe+0x91b/0xc10 [aic94xx] [? 600.277647]? local_pci_probe+0x4a/0xa0 [? 600.277650]? pci_device_probe+0x109/0x1b0 [? 600.277654]? driver_probe_device+0x2ba/0x4a0 [? 600.277657]? __driver_attach+0xe2/0xf0 [? 600.277660]? ? driver_probe_device+0x4a0/0x4a0 [? 600.277663]? bus_for_each_dev+0x72/0xc0 [? 600.277666]? driver_attach+0x1e/0x20 [? 600.277668]? bus_add_driver+0x170/0x260 [? 600.277671]? driver_register+0x60/0xe0 [? 600.277675]? ? 0xffffffffc09a1000 [? 600.277677]? __pci_register_driver+0x5a/0x60 [? 600.277684]? aic94xx_init+0xf8/0x1000 [aic94xx] [? 600.277686]? ? 0xffffffffc09a1000 [? 600.277689]? do_one_initcall+0x55/0x1ab [? 600.277693]? ? _cond_resched+0x1a/0x50 [? 600.277697]? ? kmem_cache_alloc_trace+0x108/0x1b0 [? 600.277700]? ? do_init_module+0x27/0x219 [? 600.277703]? do_init_module+0x5f/0x219 [? 600.277706]? load_module+0x28e6/0x2e00 [? 600.277710]? ? ima_post_read_file+0x83/0xa0 [? 600.277714]? SYSC_finit_module+0xe5/0x120 [? 600.277717]? ? SYSC_finit_module+0xe5/0x120 [? 600.277720]? SyS_finit_module+0xe/0x10 [? 600.277723]? do_syscall_64+0x73/0x130 [? 600.277726]? entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [? 600.277728] RIP: 0033:0x7f9e240eb229 [? 600.277730] RSP: 002b:00007ffc26fa9c48 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [? 600.277733] RAX: ffffffffffffffda RBX: 000055cec3fcf450 RCX: 00007f9e240eb229 [? 600.277734] RDX: 0000000000000000 RSI: 000055cec2823638 RDI: 0000000000000003 [? 600.277736] RBP: 000055cec2823638 R08: 0000000000000000 R09: 0000000000000000 [? 600.277737] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000 [? 600.277739] R13: 000055cec3fd1230 R14: 0000000000040000 R15: 0000000000000000 [? 600.277741] Code: 85 c0 48 89 c3 74 12 b9 00 10 00 00 48 89 c2 31 f6 4c 89 ef e8 0c c7 ff ff 4c 89 e2 48 89 de 48 c7 c7 18 72 2f 86 e8 ca 7f d8 ff <0f> 0b 48 89 df e8 90 0f f4 ff 5b 41 5c 41 5d 5d c3 66 0f 1f 84 [? 600.277798] ---[ end trace e693b63cde4c2a43 ]--- [? 600.293683] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to legacy IRQ 19 [? 600.293702] aic94xx: probe of 0000:03:04.0 failed with error -17 From miguel_3_gonzalez at yahoo.es Tue Jan 22 23:12:50 2019 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Tue, 22 Jan 2019 23:12:50 +0100 Subject: [PVE-User] pve-zsync issues Message-ID: <7845fb1c-f2c5-c2a1-21a6-73fb5335043e@yahoo.es> Hi, ? I have two servers running proxmox 5.3-6. Both run several VMs and I am using pve-zsync to sync two machines in server1 in server two for disaster recovery and offline backups. ? This has been working without issue with two proxmox servers running 5.1-46. I have just replaced them with two new servers. ? I have two jobs, one is reporting that is has to send the full batch and the other one reporting a failure. Snapshots in the backup server show? 0B. ? root at server1:~# pve-zsync status SOURCE?????????????????? NAME???????????????????? STATUS??? 100????????????????????? plesk1?????????????????? error 102????????????????????? cpanel1????????????????? ok root at server2:~# zfs list -t snapshot NAME?????????????????????????????????????????????????????? USED? AVAIL? REFER? MOUNTPOINT rpool/data/vm-100-disk-0 at rep_plesk1_2019-01-21_22:30:03????? 0B????? -? 20.4G? - rpool/data/vm-100-disk-1 at rep_plesk1_2019-01-21_22:30:03????? 0B????? -? 67.3G? - rpool/data/vm-100-disk-2 at rep_plesk1_2019-01-21_22:30:03????? 0B????? -? 92.9G? - rpool/data/vm-102-disk-0 at rep_cpanel1_2019-01-22_01:00:01???? 0B????? -? 20.0G? - rpool/data/vm-102-disk-1 at rep_cpanel1_2019-01-22_01:00:01???? 0B????? -? 60.4G? - root at server1:~# zfs list -t snapshot NAME????????????????????????????????????????????????? USED? AVAIL? REFER? MOUNTPOINT rpool/vm-100-disk-0 at rep_plesk1_2019-01-19_22:47:37??? 597M????? -? 20.0G? - rpool/vm-100-disk-0 at rep_plesk1_2019-01-20_11:22:21??? 482M????? -? 20.1G? - rpool/vm-100-disk-0 at rep_plesk1_2019-01-21_22:05:08??? 121M????? -? 20.4G? - rpool/vm-100-disk-0 at rep_plesk1_2019-01-21_22:30:03??? 117M????? -? 20.4G? - rpool/vm-100-disk-1 at rep_plesk1_2019-01-19_22:47:37?? 9.68G????? -? 67.1G? - rpool/vm-100-disk-1 at rep_plesk1_2019-01-20_11:22:21?? 9.49G????? -? 67.2G? - rpool/vm-100-disk-1 at rep_plesk1_2019-01-21_22:30:03?? 4.84G????? -? 67.3G? - rpool/vm-100-disk-2 at rep_plesk1_2019-01-19_22:47:37??? 519M????? -? 92.9G? - rpool/vm-100-disk-2 at rep_plesk1_2019-01-20_11:22:21??? 335M????? -? 92.9G? - rpool/vm-100-disk-2 at rep_plesk1_2019-01-21_22:30:03??? 517M????? -? 92.9G? - rpool/vm-102-disk-0 at rep_cpanel1_2019-01-20_01:00:01? 1.87G????? -? 20.1G? - rpool/vm-102-disk-0 at rep_cpanel1_2019-01-21_01:00:04? 1.21G????? -? 20.1G? - rpool/vm-102-disk-0 at rep_cpanel1_2019-01-22_01:00:01? 1.25G????? -? 20.0G? - rpool/vm-102-disk-1 at rep_cpanel1_2019-01-20_01:00:01? 4.94G????? -? 60.5G? - rpool/vm-102-disk-1 at rep_cpanel1_2019-01-21_01:00:04? 3.97G????? -? 60.5G? - rpool/vm-102-disk-1 at rep_cpanel1_2019-01-22_01:00:01? 3.31G????? -? 60.4G? - Nigthly jobs report different things: cpanel1 VM: WARN: COMMAND: ssh root at server2 -- zfs list -rt snapshot -Ho name rpool/data/vm-102-disk-0 at rep_cpanel1_2019-01-20_01:00:01 GET ERROR: cannot open 'rpool/data/vm-102-disk-0 at rep_cpanel1_2019-01-20_01:00:01': dataset does not exist full send of rpool/vm-102-disk-0 at rep_cpanel1_2019-01-22_01:00:01 estimated size is 29.7G total estimated size is 29.7G TIME SENT SNAPSHOT 01:00:03 23.8M rpool/vm-102-disk-0 at rep_cpanel1_2019-01-22_01:00:01 01:00:04 54.3M rpool/vm-102-disk-0 at rep_cpanel1_2019-01-22_01:00:01 01:00:05 84.7M rpool/vm-102-disk-0 at rep_cpanel1_2019-01-22_01:00:01 01:00:06 115M rpool/vm-102-disk-0 at rep_cpanel1_2019-01-22_01:00:01 and it has two set the full two disks, which I don?t understand why plesk1 VM: WARN: COMMAND: ssh root at server2 -- zfs list -rt snapshot -Ho name rpool/data/vm-100-disk-0 at rep_plesk1_2019-01-19_22:47:37 GET ERROR: cannot open 'rpool/data/vm-100-disk-0 at rep_plesk1_2019-01-19_22:47:37': dataset does not exist full send of rpool/vm-100-disk-0 at rep_plesk1_2019-01-22_01:58:55 estimated size is 28.4G total estimated size is 28.4G TIME SENT SNAPSHOT COMMAND: zfs send -v -- rpool/vm-100-disk-0 at rep_plesk1_2019-01-22_01:58:55 | ssh -o 'BatchMode=yes' root at 37.187.154.74 -- zfs recv -F -- rpool/data/vm-100-disk-0 GET ERROR: cannot receive new filesystem stream: destination has snapshots (eg. rpool/data/vm-100-disk-0) must destroy them to overwrite it Job --source 100 --name plesk1 got an ERROR!!! ERROR Message: --- This email has been checked for viruses by AVG. https://www.avg.com From smr at kmi.com Tue Jan 22 23:27:46 2019 From: smr at kmi.com (Stefan M. Radman) Date: Tue, 22 Jan 2019 22:27:46 +0000 Subject: [PVE-User] aic94xx - problem In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1443678 https://bugzilla.kernel.org/show_bug.cgi?id=201609 > On Jan 22, 2019, at 17:24, lord_Niedzwiedz wrote: > > modprobe aic94xx > > > [ 600.202300] aic94xx: Adaptec aic94xx SAS/SATA driver version 1.0.3 loaded > [ 600.202600] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to legacy IRQ 19 > [ 600.203628] aic94xx: found Adaptec AIC-9405W SAS/SATA Host Adapter, device 0000:03:04.0 > [ 600.203634] scsi host2: aic94xx > [ 600.234870] aic94xx: Found sequencer Firmware version 1.1 (V17/10c6) > [ 600.277468] aic94xx: device 0000:03:04.0: SAS addr 5005076a0144bd00, PCBA SN ORG, 4 phys, 4 enabled phys, flash present, BIOS build 1549 > [ 600.277488] ------------[ cut here ]------------ > [ 600.277490] sysfs: cannot create duplicate filename '/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/0000:03:04.0/revision' > [ 600.277511] WARNING: CPU: 1 PID: 2281 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x56/0x70 > [ 600.277513] Modules linked in: aic94xx(+) ip_set ip6table_filter ip6_tables iptable_filter softdog nfnetlink_log nfnetlink dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c gpio_ich radeon ttm snd_pcm drm_kms_helper intel_powerclamp input_leds snd_timer drm snd soundcore lpc_ich ipmi_si ipmi_devintf i2c_algo_bit fb_sys_fops pcspkr serio_raw ipmi_msghandler syscopyarea sysfillrect sysimgblt i3000_edac shpchp mac_hid zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq psmouse pata_acpi i2c_i801 libsas scsi_transport_sas e1000 tg3 ptp pps_core [last unloaded: aic94xx] > [ 600.277595] CPU: 1 PID: 2281 Comm: modprobe Tainted: P W O 4.15.17-1-pve #1 > [ 600.277596] Hardware name: IBM IBM eServer 306m -[8491E6Y]-/M11ip/M11ix, BIOS IBM BIOS Version 1.29-[PAE129AUS-1.29]- 02/09/2006 > [ 600.277600] RIP: 0010:sysfs_warn_dup+0x56/0x70 > [ 600.277602] RSP: 0018:ffffad21c338f9d0 EFLAGS: 00010282 > [ 600.277605] RAX: 0000000000000000 RBX: ffff950033c18000 RCX: 0000000000000006 > [ 600.277606] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff95003fd16490 > [ 600.277608] RBP: ffffad21c338f9e8 R08: 0000000000000001 R09: 0000000000000384 > [ 600.277609] R10: 0000000000000001 R11: 0000000000000384 R12: ffffffffc01a07c4 > [ 600.277611] R13: ffff950038c16908 R14: ffff950033314000 R15: 0000000000000004 > [ 600.277613] FS: 00007f9e245bb700(0000) GS:ffff95003fd00000(0000) knlGS:0000000000000000 > [ 600.277615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 600.277617] CR2: 00007fff0f58fff8 CR3: 00000001731b4000 CR4: 00000000000006e0 > [ 600.277619] Call Trace: > [ 600.277628] sysfs_add_file_mode_ns+0x116/0x170 > [ 600.277631] sysfs_create_file_ns+0x2a/0x30 > [ 600.277635] device_create_file+0x42/0x80 > [ 600.277643] asd_pci_probe+0x91b/0xc10 [aic94xx] > [ 600.277647] local_pci_probe+0x4a/0xa0 > [ 600.277650] pci_device_probe+0x109/0x1b0 > [ 600.277654] driver_probe_device+0x2ba/0x4a0 > [ 600.277657] __driver_attach+0xe2/0xf0 > [ 600.277660] ? driver_probe_device+0x4a0/0x4a0 > [ 600.277663] bus_for_each_dev+0x72/0xc0 > [ 600.277666] driver_attach+0x1e/0x20 > [ 600.277668] bus_add_driver+0x170/0x260 > [ 600.277671] driver_register+0x60/0xe0 > [ 600.277675] ? 0xffffffffc09a1000 > [ 600.277677] __pci_register_driver+0x5a/0x60 > [ 600.277684] aic94xx_init+0xf8/0x1000 [aic94xx] > [ 600.277686] ? 0xffffffffc09a1000 > [ 600.277689] do_one_initcall+0x55/0x1ab > [ 600.277693] ? _cond_resched+0x1a/0x50 > [ 600.277697] ? kmem_cache_alloc_trace+0x108/0x1b0 > [ 600.277700] ? do_init_module+0x27/0x219 > [ 600.277703] do_init_module+0x5f/0x219 > [ 600.277706] load_module+0x28e6/0x2e00 > [ 600.277710] ? ima_post_read_file+0x83/0xa0 > [ 600.277714] SYSC_finit_module+0xe5/0x120 > [ 600.277717] ? SYSC_finit_module+0xe5/0x120 > [ 600.277720] SyS_finit_module+0xe/0x10 > [ 600.277723] do_syscall_64+0x73/0x130 > [ 600.277726] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 > [ 600.277728] RIP: 0033:0x7f9e240eb229 > [ 600.277730] RSP: 002b:00007ffc26fa9c48 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 > [ 600.277733] RAX: ffffffffffffffda RBX: 000055cec3fcf450 RCX: 00007f9e240eb229 > [ 600.277734] RDX: 0000000000000000 RSI: 000055cec2823638 RDI: 0000000000000003 > [ 600.277736] RBP: 000055cec2823638 R08: 0000000000000000 R09: 0000000000000000 > [ 600.277737] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000 > [ 600.277739] R13: 000055cec3fd1230 R14: 0000000000040000 R15: 0000000000000000 > [ 600.277741] Code: 85 c0 48 89 c3 74 12 b9 00 10 00 00 48 89 c2 31 f6 4c 89 ef e8 0c c7 ff ff 4c 89 e2 48 89 de 48 c7 c7 18 72 2f 86 e8 ca 7f d8 ff <0f> 0b 48 89 df e8 90 0f f4 ff 5b 41 5c 41 5d 5d c3 66 0f 1f 84 > [ 600.277798] ---[ end trace e693b63cde4c2a43 ]--- > [ 600.293683] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to legacy IRQ 19 > [ 600.293702] aic94xx: probe of 0000:03:04.0 failed with error -17 > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From sir_Misiek1 at o2.pl Wed Jan 23 09:51:05 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 23 Jan 2019 09:51:05 +0100 Subject: [PVE-User] aic94xx - problem In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> Message-ID: <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> Question is, how i make downgrade and install in ProxMox old kernel 4.4.x.? ??!! Command ? > https://bugzilla.redhat.com/show_bug.cgi?id=1443678 > https://bugzilla.kernel.org/show_bug.cgi?id=201609 > >> On Jan 22, 2019, at 17:24, lord_Niedzwiedz wrote: >> >> modprobe aic94xx >> >> >> [ 600.202300] aic94xx: Adaptec aic94xx SAS/SATA driver version 1.0.3 loaded >> [ 600.202600] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to legacy IRQ 19 >> [ 600.203628] aic94xx: found Adaptec AIC-9405W SAS/SATA Host Adapter, device 0000:03:04.0 >> [ 600.203634] scsi host2: aic94xx >> [ 600.234870] aic94xx: Found sequencer Firmware version 1.1 (V17/10c6) >> [ 600.277468] aic94xx: device 0000:03:04.0: SAS addr 5005076a0144bd00, PCBA SN ORG, 4 phys, 4 enabled phys, flash present, BIOS build 1549 >> [ 600.277488] ------------[ cut here ]------------ >> [ 600.277490] sysfs: cannot create duplicate filename '/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/0000:03:04.0/revision' >> [ 600.277511] WARNING: CPU: 1 PID: 2281 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x56/0x70 >> [ 600.277513] Modules linked in: aic94xx(+) ip_set ip6table_filter ip6_tables iptable_filter softdog nfnetlink_log nfnetlink dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c gpio_ich radeon ttm snd_pcm drm_kms_helper intel_powerclamp input_leds snd_timer drm snd soundcore lpc_ich ipmi_si ipmi_devintf i2c_algo_bit fb_sys_fops pcspkr serio_raw ipmi_msghandler syscopyarea sysfillrect sysimgblt i3000_edac shpchp mac_hid zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq psmouse pata_acpi i2c_i801 libsas scsi_transport_sas e1000 tg3 ptp pps_core [last unloaded: aic94xx] >> [ 600.277595] CPU: 1 PID: 2281 Comm: modprobe Tainted: P W O 4.15.17-1-pve #1 >> [ 600.277596] Hardware name: IBM IBM eServer 306m -[8491E6Y]-/M11ip/M11ix, BIOS IBM BIOS Version 1.29-[PAE129AUS-1.29]- 02/09/2006 >> [ 600.277600] RIP: 0010:sysfs_warn_dup+0x56/0x70 >> [ 600.277602] RSP: 0018:ffffad21c338f9d0 EFLAGS: 00010282 >> [ 600.277605] RAX: 0000000000000000 RBX: ffff950033c18000 RCX: 0000000000000006 >> [ 600.277606] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff95003fd16490 >> [ 600.277608] RBP: ffffad21c338f9e8 R08: 0000000000000001 R09: 0000000000000384 >> [ 600.277609] R10: 0000000000000001 R11: 0000000000000384 R12: ffffffffc01a07c4 >> [ 600.277611] R13: ffff950038c16908 R14: ffff950033314000 R15: 0000000000000004 >> [ 600.277613] FS: 00007f9e245bb700(0000) GS:ffff95003fd00000(0000) knlGS:0000000000000000 >> [ 600.277615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 600.277617] CR2: 00007fff0f58fff8 CR3: 00000001731b4000 CR4: 00000000000006e0 >> [ 600.277619] Call Trace: >> [ 600.277628] sysfs_add_file_mode_ns+0x116/0x170 >> [ 600.277631] sysfs_create_file_ns+0x2a/0x30 >> [ 600.277635] device_create_file+0x42/0x80 >> [ 600.277643] asd_pci_probe+0x91b/0xc10 [aic94xx] >> [ 600.277647] local_pci_probe+0x4a/0xa0 >> [ 600.277650] pci_device_probe+0x109/0x1b0 >> [ 600.277654] driver_probe_device+0x2ba/0x4a0 >> [ 600.277657] __driver_attach+0xe2/0xf0 >> [ 600.277660] ? driver_probe_device+0x4a0/0x4a0 >> [ 600.277663] bus_for_each_dev+0x72/0xc0 >> [ 600.277666] driver_attach+0x1e/0x20 >> [ 600.277668] bus_add_driver+0x170/0x260 >> [ 600.277671] driver_register+0x60/0xe0 >> [ 600.277675] ? 0xffffffffc09a1000 >> [ 600.277677] __pci_register_driver+0x5a/0x60 >> [ 600.277684] aic94xx_init+0xf8/0x1000 [aic94xx] >> [ 600.277686] ? 0xffffffffc09a1000 >> [ 600.277689] do_one_initcall+0x55/0x1ab >> [ 600.277693] ? _cond_resched+0x1a/0x50 >> [ 600.277697] ? kmem_cache_alloc_trace+0x108/0x1b0 >> [ 600.277700] ? do_init_module+0x27/0x219 >> [ 600.277703] do_init_module+0x5f/0x219 >> [ 600.277706] load_module+0x28e6/0x2e00 >> [ 600.277710] ? ima_post_read_file+0x83/0xa0 >> [ 600.277714] SYSC_finit_module+0xe5/0x120 >> [ 600.277717] ? SYSC_finit_module+0xe5/0x120 >> [ 600.277720] SyS_finit_module+0xe/0x10 >> [ 600.277723] do_syscall_64+0x73/0x130 >> [ 600.277726] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 >> [ 600.277728] RIP: 0033:0x7f9e240eb229 >> [ 600.277730] RSP: 002b:00007ffc26fa9c48 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 >> [ 600.277733] RAX: ffffffffffffffda RBX: 000055cec3fcf450 RCX: 00007f9e240eb229 >> [ 600.277734] RDX: 0000000000000000 RSI: 000055cec2823638 RDI: 0000000000000003 >> [ 600.277736] RBP: 000055cec2823638 R08: 0000000000000000 R09: 0000000000000000 >> [ 600.277737] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000 >> [ 600.277739] R13: 000055cec3fd1230 R14: 0000000000040000 R15: 0000000000000000 >> [ 600.277741] Code: 85 c0 48 89 c3 74 12 b9 00 10 00 00 48 89 c2 31 f6 4c 89 ef e8 0c c7 ff ff 4c 89 e2 48 89 de 48 c7 c7 18 72 2f 86 e8 ca 7f d8 ff <0f> 0b 48 89 df e8 90 0f f4 ff 5b 41 5c 41 5d 5d c3 66 0f 1f 84 >> [ 600.277798] ---[ end trace e693b63cde4c2a43 ]--- >> [ 600.293683] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to legacy IRQ 19 >> [ 600.293702] aic94xx: probe of 0000:03:04.0 failed with error -17 >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From sir_Misiek1 at o2.pl Wed Jan 23 10:20:04 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 23 Jan 2019 10:20:04 +0100 Subject: [PVE-User] aic94xx - problem In-Reply-To: <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> Message-ID: <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> I heave only apt install pve-kernel-4.10....... Where is oldies repository ? > Question is, how i make downgrade and install in ProxMox old kernel > 4.4.x. ??!! > Command ? >> https://bugzilla.redhat.com/show_bug.cgi?id=1443678 >> https://bugzilla.kernel.org/show_bug.cgi?id=201609 >> >>> On Jan 22, 2019, at 17:24, lord_Niedzwiedz wrote: >>> >>> modprobe aic94xx >>> >>> >>> [? 600.202300] aic94xx: Adaptec aic94xx SAS/SATA driver version >>> 1.0.3 loaded >>> [? 600.202600] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to >>> legacy IRQ 19 >>> [? 600.203628] aic94xx: found Adaptec AIC-9405W SAS/SATA Host >>> Adapter, device 0000:03:04.0 >>> [? 600.203634] scsi host2: aic94xx >>> [? 600.234870] aic94xx: Found sequencer Firmware version 1.1 (V17/10c6) >>> [? 600.277468] aic94xx: device 0000:03:04.0: SAS addr >>> 5005076a0144bd00, PCBA SN ORG, 4 phys, 4 enabled phys, flash >>> present, BIOS build 1549 >>> [? 600.277488] ------------[ cut here ]------------ >>> [? 600.277490] sysfs: cannot create duplicate filename >>> '/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/0000:03:04.0/revision' >>> [? 600.277511] WARNING: CPU: 1 PID: 2281 at fs/sysfs/dir.c:31 >>> sysfs_warn_dup+0x56/0x70 >>> [? 600.277513] Modules linked in: aic94xx(+) ip_set ip6table_filter >>> ip6_tables iptable_filter softdog nfnetlink_log nfnetlink >>> dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c >>> gpio_ich radeon ttm snd_pcm drm_kms_helper intel_powerclamp >>> input_leds snd_timer drm snd soundcore lpc_ich ipmi_si ipmi_devintf >>> i2c_algo_bit fb_sys_fops pcspkr serio_raw ipmi_msghandler >>> syscopyarea sysfillrect sysimgblt i3000_edac shpchp mac_hid zfs(PO) >>> zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) >>> vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp >>> libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables >>> autofs4 btrfs xor zstd_compress raid6_pq psmouse pata_acpi i2c_i801 >>> libsas scsi_transport_sas e1000 tg3 ptp pps_core [last unloaded: >>> aic94xx] >>> [? 600.277595] CPU: 1 PID: 2281 Comm: modprobe Tainted: P??????? W >>> O???? 4.15.17-1-pve #1 >>> [? 600.277596] Hardware name: IBM IBM eServer 306m >>> -[8491E6Y]-/M11ip/M11ix, BIOS IBM BIOS Version >>> 1.29-[PAE129AUS-1.29]- 02/09/2006 >>> [? 600.277600] RIP: 0010:sysfs_warn_dup+0x56/0x70 >>> [? 600.277602] RSP: 0018:ffffad21c338f9d0 EFLAGS: 00010282 >>> [? 600.277605] RAX: 0000000000000000 RBX: ffff950033c18000 RCX: >>> 0000000000000006 >>> [? 600.277606] RDX: 0000000000000007 RSI: 0000000000000096 RDI: >>> ffff95003fd16490 >>> [? 600.277608] RBP: ffffad21c338f9e8 R08: 0000000000000001 R09: >>> 0000000000000384 >>> [? 600.277609] R10: 0000000000000001 R11: 0000000000000384 R12: >>> ffffffffc01a07c4 >>> [? 600.277611] R13: ffff950038c16908 R14: ffff950033314000 R15: >>> 0000000000000004 >>> [? 600.277613] FS:? 00007f9e245bb700(0000) GS:ffff95003fd00000(0000) >>> knlGS:0000000000000000 >>> [? 600.277615] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [? 600.277617] CR2: 00007fff0f58fff8 CR3: 00000001731b4000 CR4: >>> 00000000000006e0 >>> [? 600.277619] Call Trace: >>> [? 600.277628]? sysfs_add_file_mode_ns+0x116/0x170 >>> [? 600.277631]? sysfs_create_file_ns+0x2a/0x30 >>> [? 600.277635]? device_create_file+0x42/0x80 >>> [? 600.277643]? asd_pci_probe+0x91b/0xc10 [aic94xx] >>> [? 600.277647]? local_pci_probe+0x4a/0xa0 >>> [? 600.277650]? pci_device_probe+0x109/0x1b0 >>> [? 600.277654]? driver_probe_device+0x2ba/0x4a0 >>> [? 600.277657]? __driver_attach+0xe2/0xf0 >>> [? 600.277660]? ? driver_probe_device+0x4a0/0x4a0 >>> [? 600.277663]? bus_for_each_dev+0x72/0xc0 >>> [? 600.277666]? driver_attach+0x1e/0x20 >>> [? 600.277668]? bus_add_driver+0x170/0x260 >>> [? 600.277671]? driver_register+0x60/0xe0 >>> [? 600.277675]? ? 0xffffffffc09a1000 >>> [? 600.277677]? __pci_register_driver+0x5a/0x60 >>> [? 600.277684]? aic94xx_init+0xf8/0x1000 [aic94xx] >>> [? 600.277686]? ? 0xffffffffc09a1000 >>> [? 600.277689]? do_one_initcall+0x55/0x1ab >>> [? 600.277693]? ? _cond_resched+0x1a/0x50 >>> [? 600.277697]? ? kmem_cache_alloc_trace+0x108/0x1b0 >>> [? 600.277700]? ? do_init_module+0x27/0x219 >>> [? 600.277703]? do_init_module+0x5f/0x219 >>> [? 600.277706]? load_module+0x28e6/0x2e00 >>> [? 600.277710]? ? ima_post_read_file+0x83/0xa0 >>> [? 600.277714]? SYSC_finit_module+0xe5/0x120 >>> [? 600.277717]? ? SYSC_finit_module+0xe5/0x120 >>> [? 600.277720]? SyS_finit_module+0xe/0x10 >>> [? 600.277723]? do_syscall_64+0x73/0x130 >>> [? 600.277726]? entry_SYSCALL_64_after_hwframe+0x3d/0xa2 >>> [? 600.277728] RIP: 0033:0x7f9e240eb229 >>> [? 600.277730] RSP: 002b:00007ffc26fa9c48 EFLAGS: 00000246 ORIG_RAX: >>> 0000000000000139 >>> [? 600.277733] RAX: ffffffffffffffda RBX: 000055cec3fcf450 RCX: >>> 00007f9e240eb229 >>> [? 600.277734] RDX: 0000000000000000 RSI: 000055cec2823638 RDI: >>> 0000000000000003 >>> [? 600.277736] RBP: 000055cec2823638 R08: 0000000000000000 R09: >>> 0000000000000000 >>> [? 600.277737] R10: 0000000000000003 R11: 0000000000000246 R12: >>> 0000000000000000 >>> [? 600.277739] R13: 000055cec3fd1230 R14: 0000000000040000 R15: >>> 0000000000000000 >>> [? 600.277741] Code: 85 c0 48 89 c3 74 12 b9 00 10 00 00 48 89 c2 31 >>> f6 4c 89 ef e8 0c c7 ff ff 4c 89 e2 48 89 de 48 c7 c7 18 72 2f 86 e8 >>> ca 7f d8 ff <0f> 0b 48 89 df e8 90 0f f4 ff 5b 41 5c 41 5d 5d c3 66 >>> 0f 1f 84 >>> [? 600.277798] ---[ end trace e693b63cde4c2a43 ]--- >>> [? 600.293683] aic94xx 0000:03:04.0: PCI IRQ 19 -> rerouted to >>> legacy IRQ 19 >>> [? 600.293702] aic94xx: probe of 0000:03:04.0 failed with error -17 >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From f.gruenbichler at proxmox.com Wed Jan 23 10:27:36 2019 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?Q?Gr=FCnbichler?=) Date: Wed, 23 Jan 2019 10:27:36 +0100 Subject: [PVE-User] APT CVE-2019-3462 (please read before upgrading!) Message-ID: <20190123092736.cmvrpdlc4uwite43@nora.maurer-it.com> The APT package manager used by Proxmox VE and Proxmox Mail Gateway was recently discovered to be affected by CVE-2019-3462, allowing a Man-In-The-Middle or malicious mirror server to execute arbitrary code with root privileges when affected systems attempt to install upgrades. To securely upgrade your systems, run the following commands as root: # apt -o Acquire::http::AllowRedirect=false update # apt -o Acquire::http::AllowRedirect=false full-upgrade and verify that apt is now at least version 1.4.9 on Debian Stretch: $ apt -v apt 1.4.9 (amd64) Please see the Debian Security Advisory for details: https://www.debian.org/security/2019/dsa-4371 From trojahn+proxmox at pluspol.info Wed Jan 23 13:25:18 2019 From: trojahn+proxmox at pluspol.info (Falko Trojahn) Date: Wed, 23 Jan 2019 13:25:18 +0100 Subject: [PVE-User] aic94xx - problem In-Reply-To: <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> Message-ID: <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> lord_Niedzwiedz schrieb am 23.01.19 um 10:20: > I heave only apt install pve-kernel-4.10....... > Where is oldies repository ? >> Question is, how i make downgrade and install in ProxMox old kernel >> 4.4.x. ??!! >> Command ? apt-cache search pve-kernel -> do you find what you need? Then: apt-get install pve-kernel-4.4-... From sir_Misiek1 at o2.pl Wed Jan 23 15:33:54 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 23 Jan 2019 15:33:54 +0100 Subject: [PVE-User] aic94xx - problem In-Reply-To: <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> Message-ID: <6ca09a6d-80b0-c957-614c-c72a5af80645@o2.pl> >> I heave only apt install pve-kernel-4.10....... >> Where is oldies repository ? >>> Question is, how i make downgrade and install in ProxMox old kernel >>> 4.4.x. ??!! >>> Command ? > > apt-cache search pve-kernel > > -> do you find what you need? Then: > > apt-get install pve-kernel-4.4-... > I dont heave in repos pve-kernel-4.4- The minimal is pve-kernel-4.10.1-2-pve I must install proxmox 4.4 Or how I make get this apt repos ? From sir_Misiek1 at o2.pl Wed Jan 23 15:37:38 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 23 Jan 2019 15:37:38 +0100 Subject: [PVE-User] Join cluster first time - problem In-Reply-To: <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> Message-ID: <447f7e38-f938-20cb-cf38-738c6fd182d6@o2.pl> ??? ??? I do it first time. I create cluster ok on host1: pvecm create klaster1 pvecm status And on host2 i try: pvecm add tomas.komandor.pl detected the following error(s): * this host already contains virtual guests Check if node may join a cluster failed! what i do wrong ?? From elacunza at binovo.es Wed Jan 23 15:45:44 2019 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 23 Jan 2019 15:45:44 +0100 Subject: [PVE-User] Join cluster first time - problem In-Reply-To: <447f7e38-f938-20cb-cf38-738c6fd182d6@o2.pl> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> <447f7e38-f938-20cb-cf38-738c6fd182d6@o2.pl> Message-ID: <4cd61f28-11c3-4983-dce0-be7aabeb9da1@binovo.es> Hi, Seems you have VMs on host2. Please read: https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Adding_nodes_to_the_Cluster Cheers El 23/1/19 a las 15:37, lord_Niedzwiedz escribi?: > ??? ??? I do it first time. > > I create cluster ok on host1: > > pvecm create klaster1 > pvecm status > > And on host2 i try: > > pvecm add tomas.komandor.pl > > detected the following error(s): > * this host already contains virtual guests > Check if node may join a cluster failed! > > what i do wrong ?? > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From sir_Misiek1 at o2.pl Wed Jan 23 15:56:37 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 23 Jan 2019 15:56:37 +0100 Subject: [PVE-User] Join cluster first time - problem In-Reply-To: <4cd61f28-11c3-4983-dce0-be7aabeb9da1@binovo.es> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> <447f7e38-f938-20cb-cf38-738c6fd182d6@o2.pl> <4cd61f28-11c3-4983-dce0-be7aabeb9da1@binovo.es> Message-ID: ??? Hi, OK, thank you I thought so, I'm doing a test cluster on 2 test machines, for a moment and remove, the first time. I'm learning. I've created. On host2 (the one he adds) can not be CT ora VM; / Regards ;) Gregory > Hi, > > Seems you have VMs on host2. Please read: > https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Adding_nodes_to_the_Cluster > > > Cheers > > El 23/1/19 a las 15:37, lord_Niedzwiedz escribi?: >> ??? ??? I do it first time. >> >> I create cluster ok on host1: >> >> pvecm create klaster1 >> pvecm status >> >> And on host2 i try: >> >> pvecm add toms.komndr.pl >> >> detected the following error(s): >> * this host already contains virtual guests >> Check if node may join a cluster failed! >> >> what i do wrong ?? >> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From sir_Misiek1 at o2.pl Wed Jan 23 16:29:22 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 23 Jan 2019 16:29:22 +0100 Subject: [PVE-User] Join cluster first time - problem In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> <447f7e38-f938-20cb-cf38-738c6fd182d6@o2.pl> <4cd61f28-11c3-4983-dce0-be7aabeb9da1@binovo.es> Message-ID: <11514b43-ad88-97d6-98c4-5fa6f045263f@o2.pl> "WARNING: Adding a node to the cluster will delete it's current /etc/pve/storage.cfg. If you have VMs stored on the node, be prepared to add back your storage locations if necessary. Even though the storage locations disappear from the GUI, your data is still there." I add node to cluster. But lost local-zfs. Now i heave only "local" and nothing in added node. "could not activate storage 'local-zfs', zfs error: cannot import 'rpool': no such pool available (500)" How i make recover this. At node 1 I had local-zfs. On node 2? (including added) i had local-lvm I do not see the latter, but I have changed the configuration in /etc/pve/storages.cfd for node2 from local-lvm to local-zfs? ;-/ > Regards ;) > Gregory >> Hi, >> >> Seems you have VMs on host2. Please read: >> https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Adding_nodes_to_the_Cluster >> >> >> Cheers >> >> El 23/1/19 a las 15:37, lord_Niedzwiedz escribi?: >>> ??? ??? I do it first time. >>> >>> I create cluster ok on host1: >>> >>> pvecm create klaster1 >>> pvecm status >>> >>> And on host2 i try: >>> >>> pvecm add toms.komndr.pl >>> >>> detected the following error(s): >>> * this host already contains virtual guests >>> Check if node may join a cluster failed! >>> >>> what i do wrong ?? >>> >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> From sir_Misiek1 at o2.pl Wed Jan 23 16:37:07 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 23 Jan 2019 16:37:07 +0100 Subject: [PVE-User] Join cluster first time - problem In-Reply-To: <11514b43-ad88-97d6-98c4-5fa6f045263f@o2.pl> References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> <447f7e38-f938-20cb-cf38-738c6fd182d6@o2.pl> <4cd61f28-11c3-4983-dce0-be7aabeb9da1@binovo.es> <11514b43-ad88-97d6-98c4-5fa6f045263f@o2.pl> Message-ID: Ok, when I added in node2 this: lvmthin: local-lvm ??????? thinpool data ??????? vgname pve ??????? content rootdir,images I see local-vm in node2?? ;-) But i see to local-zfs (not active) "could not activate storage 'local-zfs', zfs error: cannot import 'rpool': no such pool available (500)" I w drug? stron?. Widz? na w??le 1 local-lvm te? nie aktywne. Sorry for spamming on the group (my first cluster ;-D) I will not be anymore? ;-) W dniu 23.01.2019 o?16:29, lord_Niedzwiedz pisze: > "WARNING: Adding a node to the cluster will delete it's current > /etc/pve/storage.cfg. If you have VMs stored on the node, be prepared > to add back your storage locations if necessary. Even though the > storage locations disappear from the GUI, your data is still there." > I add node to cluster. > But lost local-zfs. > Now i heave only "local" and nothing in added node. > "could not activate storage 'local-zfs', zfs error: cannot import > 'rpool': no such pool available (500)" > How i make recover this. > > At node 1 I had local-zfs. > On node 2? (including added) i had local-lvm > I do not see the latter, but I have changed the configuration in > /etc/pve/storages.cfd for node2 from local-lvm to local-zfs? ;-/ >> Regards ;) >> Gregory >>> Hi, >>> >>> Seems you have VMs on host2. Please read: >>> https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Adding_nodes_to_the_Cluster >>> >>> >>> Cheers >>> >>> El 23/1/19 a las 15:37, lord_Niedzwiedz escribi?: >>>> ??? ??? I do it first time. >>>> >>>> I create cluster ok on host1: >>>> >>>> pvecm create klaster1 >>>> pvecm status >>>> >>>> And on host2 i try: >>>> >>>> pvecm add toms.komndr.pl >>>> >>>> detected the following error(s): >>>> * this host already contains virtual guests >>>> Check if node may join a cluster failed! >>>> >>>> what i do wrong ?? >>>> >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >>> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From elacunza at binovo.es Wed Jan 23 16:42:45 2019 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 23 Jan 2019 16:42:45 +0100 Subject: [PVE-User] Join cluster first time - problem In-Reply-To: References: <20190116130726.om4nz4aeudywfujf@dona.proxmox.com> <3e1b3594-dd43-9845-02a7-c92ca2a8f359@o2.pl> <711ff24e-6116-2ade-fae3-83c9f8777d85@o2.pl> <62a9b217-04f7-20cb-6b17-6690206e1983@pluspol.info> <447f7e38-f938-20cb-cf38-738c6fd182d6@o2.pl> <4cd61f28-11c3-4983-dce0-be7aabeb9da1@binovo.es> <11514b43-ad88-97d6-98c4-5fa6f045263f@o2.pl> Message-ID: <4a59b0e8-8e1b-a9b8-4e91-70294279ac33@binovo.es> Just restrict "local-zfs" storage to node1 (can be done from WebGUI) El 23/1/19 a las 16:37, lord_Niedzwiedz escribi?: > > Ok, when I added in node2 this: > lvmthin: local-lvm > ??????? thinpool data > ??????? vgname pve > ??????? content rootdir,images > I see local-vm in node2?? ;-) > But i see to local-zfs (not active) > "could not activate storage 'local-zfs', zfs error: cannot import > 'rpool': no such pool available (500)" > > I w drug? stron?. > Widz? na w??le 1 local-lvm te? nie aktywne. > > Sorry for spamming on the group (my first cluster ;-D) > I will not be anymore? ;-) > > W dniu 23.01.2019 o?16:29, lord_Niedzwiedz pisze: >> "WARNING: Adding a node to the cluster will delete it's current >> /etc/pve/storage.cfg. If you have VMs stored on the node, be prepared >> to add back your storage locations if necessary. Even though the >> storage locations disappear from the GUI, your data is still there." >> I add node to cluster. >> But lost local-zfs. >> Now i heave only "local" and nothing in added node. >> "could not activate storage 'local-zfs', zfs error: cannot import >> 'rpool': no such pool available (500)" >> How i make recover this. >> >> At node 1 I had local-zfs. >> On node 2? (including added) i had local-lvm >> I do not see the latter, but I have changed the configuration in >> /etc/pve/storages.cfd for node2 from local-lvm to local-zfs? ;-/ >>> Regards ;) >>> Gregory >>>> Hi, >>>> >>>> Seems you have VMs on host2. Please read: >>>> https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Adding_nodes_to_the_Cluster >>>> >>>> >>>> Cheers >>>> >>>> El 23/1/19 a las 15:37, lord_Niedzwiedz escribi?: >>>>> ??? ??? I do it first time. >>>>> >>>>> I create cluster ok on host1: >>>>> >>>>> pvecm create klaster1 >>>>> pvecm status >>>>> >>>>> And on host2 i try: >>>>> >>>>> pvecm add toms.komndr.pl >>>>> >>>>> detected the following error(s): >>>>> * this host already contains virtual guests >>>>> Check if node may join a cluster failed! >>>>> >>>>> what i do wrong ?? >>>>> >>>>> >>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user at pve.proxmox.com >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From gaio at sv.lnf.it Wed Jan 23 18:51:03 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Wed, 23 Jan 2019 18:51:03 +0100 Subject: [PVE-User] Backup stalls, on server where it is not needed to run... Message-ID: <20190123175103.GP3440@sv.lnf.it> I'v a 5 node cluster of proxmox 4.4. I've setup a backup task that do some daily backup of some VM. The backup task use a storage that is phisically on one of the server, mounted to all other server via NFS. In the server where the task actually have the NFS share, the task run without trouble. In all the other 4 servers, stalls: root at thor:~# ps aux | grep vzdump root 3748 0.0 0.0 4340 728 ? Ss Jan20 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 3749 0.0 0.1 236772 65620 ? D Jan20 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 6596 0.0 0.0 4340 764 ? Ss Jan15 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 6597 0.0 0.1 236852 65644 ? D Jan15 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 7240 0.0 0.0 12736 2120 pts/5 S+ 18:44 0:00 grep vzdump root 15055 0.0 0.0 4340 764 ? Ss Jan21 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 15056 0.0 0.1 236780 65532 ? D Jan21 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 18046 0.0 0.0 4340 724 ? Ss Jan16 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 18047 0.0 0.1 236804 65504 ? D Jan16 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 25072 0.0 0.0 4340 724 ? Ss Jan22 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 25073 0.0 0.1 236716 65632 ? D Jan22 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 29237 0.0 0.0 4340 760 ? Ss Jan17 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 29238 0.0 0.1 236812 65540 ? D Jan17 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 57286 0.0 0.0 4340 816 ? Ss Jan18 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 root 57287 0.0 0.1 236772 65716 ? D Jan18 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto ced at pp.lnf.it --compress lzo --storage Backup --quiet 1 I've tried to restart nfs-common in a client, and nfs-kernel-server in the server but seems does not change. Processes are unkillable, even with 'kill -9'. Boh... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gilberto.nunes32 at gmail.com Wed Jan 23 19:03:45 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 23 Jan 2019 16:03:45 -0200 Subject: [PVE-User] Kill LXC... Message-ID: Hi there I am facing some trouble with 2 LXC that cannot access either kill it. Already try lxc-stop -k -n but no effect. Any advice will be welcome... Best --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From t.lamprecht at proxmox.com Wed Jan 23 19:28:12 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 23 Jan 2019 19:28:12 +0100 Subject: [PVE-User] Kill LXC... In-Reply-To: References: Message-ID: <9f479315-6efc-eb0a-c675-65f15bb70cef@proxmox.com> Hi, On 1/23/19 7:03 PM, Gilberto Nunes wrote: > I am facing some trouble with 2 LXC that cannot access either kill it. > Already try lxc-stop -k -n but no effect. > Any advice will be welcome... does it have processes in the "D" (uninterruptible) state? Probably because some network mount where it has IO pending. In this case, where whatever storage device/mount does not comes back to complete the IO your single way out may be a reboot... You could check with ps either on the host (lots of processes) or from inside the container, e.g. with: # pct exec VMID ps faux cheers, Thomas From sleemburg at it-functions.nl Wed Jan 23 20:22:40 2019 From: sleemburg at it-functions.nl (Stephan Leemburg) Date: Wed, 23 Jan 2019 20:22:40 +0100 Subject: [PVE-User] Kill LXC... In-Reply-To: <9f479315-6efc-eb0a-c675-65f15bb70cef@proxmox.com> References: <9f479315-6efc-eb0a-c675-65f15bb70cef@proxmox.com> Message-ID: It could be you are looking at a bug we are experiencing weekly for months. It is some race condition on copying namespaces in netns. I posted earlier on it, but no reaction. This seems to be solved in kernel 4.17 and up, so that needs an ubuntu 18.10 kernel. The only way to resolve it is - as far as we know - to reboot the system. Met vriendelijke groet, Stephan Leemburg IT Functions e: sleemburg at it-functions.nl p: +31 (0)71 889 23 33 m: +31(0)6 83 22 30 69 kvk: 27313647 On 23-01-19 19:28, Thomas Lamprecht wrote: > Hi, > > On 1/23/19 7:03 PM, Gilberto Nunes wrote: >> I am facing some trouble with 2 LXC that cannot access either kill it. >> Already try lxc-stop -k -n but no effect. >> Any advice will be welcome... > does it have processes in the "D" (uninterruptible) state? > Probably because some network mount where it has IO pending. > > In this case, where whatever storage device/mount does not comes > back to complete the IO your single way out may be a reboot... > > You could check with ps either on the host (lots of processes) or from > inside the container, e.g. with: > # pct exec VMID ps faux > > cheers, > Thomas > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gilberto.nunes32 at gmail.com Wed Jan 23 20:46:19 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 23 Jan 2019 17:46:19 -0200 Subject: [PVE-User] Kill LXC... In-Reply-To: References: <9f479315-6efc-eb0a-c675-65f15bb70cef@proxmox.com> Message-ID: Oh OMG! We'll planne for that! I mean we'll reboot the system! Sad! --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 23 de jan de 2019 ?s 17:22, Stephan Leemburg < sleemburg at it-functions.nl> escreveu: > It could be you are looking at a bug we are experiencing weekly for months. > > It is some race condition on copying namespaces in netns. > > I posted earlier on it, but no reaction. > > This seems to be solved in kernel 4.17 and up, so that needs an ubuntu > 18.10 kernel. > > The only way to resolve it is - as far as we know - to reboot the system. > > > Met vriendelijke groet, > Stephan Leemburg > IT Functions > > e: sleemburg at it-functions.nl > p: +31 (0)71 889 23 33 > m: +31(0)6 83 22 30 69 > kvk: 27313647 > > On 23-01-19 19:28, Thomas Lamprecht wrote: > > Hi, > > > > On 1/23/19 7:03 PM, Gilberto Nunes wrote: > >> I am facing some trouble with 2 LXC that cannot access either kill it. > >> Already try lxc-stop -k -n but no effect. > >> Any advice will be welcome... > > does it have processes in the "D" (uninterruptible) state? > > Probably because some network mount where it has IO pending. > > > > In this case, where whatever storage device/mount does not comes > > back to complete the IO your single way out may be a reboot... > > > > You could check with ps either on the host (lots of processes) or from > > inside the container, e.g. with: > > # pct exec VMID ps faux > > > > cheers, > > Thomas > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gaio at sv.lnf.it Thu Jan 24 09:19:20 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Thu, 24 Jan 2019 09:19:20 +0100 Subject: [PVE-User] Backup stalls, on server where it is not needed to run... In-Reply-To: <20190123175103.GP3440@sv.lnf.it> References: <20190123175103.GP3440@sv.lnf.it> Message-ID: <20190124081920.GA3938@sv.lnf.it> > I've tried to restart nfs-common in a client, and nfs-kernel-server in > the server but seems does not change. Ok, STOPping and the STARTing nfs-kernel-server seems have resolved the issue, or at least after some hours all accumulated tasks go away. On a node corosinc get crazy: Jan 23 22:29:38 thor pmxcfs[30679]: [status] notice: cpg_send_message retried 59 times Jan 23 22:29:39 thor pmxcfs[30679]: [status] notice: cpg_send_message retry 10 but a restart of 'pve-cluster' fixed that. FYI. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gilberto.nunes32 at gmail.com Thu Jan 24 14:21:53 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 24 Jan 2019 11:21:53 -0200 Subject: [PVE-User] Error trying to dist-upgrade a fresh installation Message-ID: Hi list I have a fresh installation here, and when I try to updagre it I get some errors: apt dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n] Setting up pve-manager (5.3-8) ... Job for pvedaemon.service failed because the control process exited with error code. See "systemctl status pvedaemon.service" and "journalctl -xe" for details. dpkg: error processing package pve-manager (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: pve-manager E: Sub-process /usr/bin/dpkg returned an error code (1) systemctl status pvedaemon.service * pvedaemon.service - PVE API Daemon Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled) Active: active (running) (Result: exit-code) since Thu 2019-01-24 11:15:52 -02; 5min ago Process: 2634 ExecReload=/usr/bin/pvedaemon restart (code=exited, status=2) Main PID: 1366 (pvedaemon) Tasks: 4 (limit: 4915) Memory: 115.6M CPU: 6.251s CGroup: /system.slice/pvedaemon.service |-1366 pvedaemon |-1369 pvedaemon worker |-1370 pvedaemon worker `-1371 pvedaemon worker Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 13, line 755. Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 13, line Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 13, line 755. Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 13, line 755. Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, line Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, l Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at /usr/bin/pvedaemon line 11, line 755. Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11, line 755. Jan 24 11:19:38 pve01 systemd[1]: pvedaemon.service: Control process exited, code=exited status=2 Jan 24 11:19:38 pve01 systemd[1]: Reload failed for PVE API Daemon. I am not use subscription. Is there something I can do to solve this issue? Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From d.csapak at proxmox.com Thu Jan 24 14:38:25 2019 From: d.csapak at proxmox.com (Dominik Csapak) Date: Thu, 24 Jan 2019 14:38:25 +0100 Subject: [PVE-User] Error trying to dist-upgrade a fresh installation In-Reply-To: References: Message-ID: <79991892-2d05-3ae3-2876-f2d13be356b9@proxmox.com> On 1/24/19 2:21 PM, Gilberto Nunes wrote: > Hi list > > I have a fresh installation here, and when I try to updagre it I get some > errors: > apt dist-upgrade > Reading package lists... Done > Building dependency tree > Reading state information... Done > Calculating upgrade... Done > 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. > 1 not fully installed or removed. > After this operation, 0 B of additional disk space will be used. > Do you want to continue? [Y/n] > Setting up pve-manager (5.3-8) ... > Job for pvedaemon.service failed because the control process exited with > error code. > See "systemctl status pvedaemon.service" and "journalctl -xe" for details. > dpkg: error processing package pve-manager (--configure): > subprocess installed post-installation script returned error exit status 1 > Errors were encountered while processing: > pve-manager > E: Sub-process /usr/bin/dpkg returned an error code (1) > > systemctl status pvedaemon.service > * pvedaemon.service - PVE API Daemon > Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor > preset: enabled) > Active: active (running) (Result: exit-code) since Thu 2019-01-24 > 11:15:52 -02; 5min ago > Process: 2634 ExecReload=/usr/bin/pvedaemon restart (code=exited, > status=2) > Main PID: 1366 (pvedaemon) > Tasks: 4 (limit: 4915) > Memory: 115.6M > CPU: 6.251s > CGroup: /system.slice/pvedaemon.service > |-1366 pvedaemon > |-1369 pvedaemon worker > |-1370 pvedaemon worker > `-1371 pvedaemon worker > > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > /usr/share/perl5/PVE/API2/Cluster.pm line 13, line 755. > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at > /usr/share/perl5/PVE/API2/Cluster.pm line 13, line > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > /usr/share/perl5/PVE/API2.pm line 13, line 755. > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at > /usr/share/perl5/PVE/API2.pm line 13, line 755. > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, line > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at > /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, l > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > /usr/bin/pvedaemon line 11, line 755. > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted at > /usr/bin/pvedaemon line 11, line 755. > Jan 24 11:19:38 pve01 systemd[1]: pvedaemon.service: Control process > exited, code=exited status=2 > Jan 24 11:19:38 pve01 systemd[1]: Reload failed for PVE API Daemon. > > can you post the complete error from the journal? also the complete output of apt update apt dist-upgrade could be helpful From gilberto.nunes32 at gmail.com Thu Jan 24 15:15:00 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 24 Jan 2019 12:15:00 -0200 Subject: [PVE-User] Error trying to dist-upgrade a fresh installation In-Reply-To: <79991892-2d05-3ae3-2876-f2d13be356b9@proxmox.com> References: <79991892-2d05-3ae3-2876-f2d13be356b9@proxmox.com> Message-ID: journal -xe Jan 24 12:12:50 pve01 systemd[1]: Reloading PVE API Daemon. -- Subject: Unit pvedaemon.service has begun reloading its configuration -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit pvedaemon.service has begun reloading its configuration Jan 24 12:12:52 pve01 pvedaemon[2766]: Can't load '/usr/lib/x86_64-linux-gnu/perl5/5.24/auto/PVE/RADOS/RADOS.so' for module PVE::RADOS Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 10, line Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 10, Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at /usr/share/perl5/PVE/Storage.pm line 30, line 755. Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Storage.pm line 30, line 755. Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 13, line 755. Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 13, line Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 13, line 755. Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 13, line 755. Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, line Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, l Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at /usr/bin/pvedaemon line 11, line 755. Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11, line 755. Jan 24 12:12:52 pve01 systemd[1]: pvedaemon.service: Control process exited, code=exited status=2 Jan 24 12:12:52 pve01 systemd[1]: Reload failed for PVE API Daemon. -- Subject: Unit pvedaemon.service has finished reloading its configuration -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit pvedaemon.service has finished reloading its configuration -- -- The result is failed. lines 2003-2030/2030 (END) apt update Ign:1 http://mirror.pop-sc.rnp.br/debian stretch InRelease Hit:2 http://security.debian.org stretch/updates InRelease Hit:3 http://mirror.pop-sc.rnp.br/debian stretch-updates InRelease Hit:4 http://mirror.pop-sc.rnp.br/debian stretch Release Hit:5 http://download.proxmox.com/debian stretch InRelease Reading package lists... Done Building dependency tree Reading state information... Done All packages are up to date. apt dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n] Setting up pve-manager (5.3-8) ... Job for pvedaemon.service failed because the control process exited with error code. See "systemctl status pvedaemon.service" and "journalctl -xe" for details. dpkg: error processing package pve-manager (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: pve-manager E: Sub-process /usr/bin/dpkg returned an error code (1) --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qui, 24 de jan de 2019 ?s 11:38, Dominik Csapak escreveu: > On 1/24/19 2:21 PM, Gilberto Nunes wrote: > > Hi list > > > > I have a fresh installation here, and when I try to updagre it I get some > > errors: > > apt dist-upgrade > > Reading package lists... Done > > Building dependency tree > > Reading state information... Done > > Calculating upgrade... Done > > 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. > > 1 not fully installed or removed. > > After this operation, 0 B of additional disk space will be used. > > Do you want to continue? [Y/n] > > Setting up pve-manager (5.3-8) ... > > Job for pvedaemon.service failed because the control process exited with > > error code. > > See "systemctl status pvedaemon.service" and "journalctl -xe" for > details. > > dpkg: error processing package pve-manager (--configure): > > subprocess installed post-installation script returned error exit > status 1 > > Errors were encountered while processing: > > pve-manager > > E: Sub-process /usr/bin/dpkg returned an error code (1) > > > > systemctl status pvedaemon.service > > * pvedaemon.service - PVE API Daemon > > Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; > vendor > > preset: enabled) > > Active: active (running) (Result: exit-code) since Thu 2019-01-24 > > 11:15:52 -02; 5min ago > > Process: 2634 ExecReload=/usr/bin/pvedaemon restart (code=exited, > > status=2) > > Main PID: 1366 (pvedaemon) > > Tasks: 4 (limit: 4915) > > Memory: 115.6M > > CPU: 6.251s > > CGroup: /system.slice/pvedaemon.service > > |-1366 pvedaemon > > |-1369 pvedaemon worker > > |-1370 pvedaemon worker > > `-1371 pvedaemon worker > > > > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > > /usr/share/perl5/PVE/API2/Cluster.pm line 13, line 755. > > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted > at > > /usr/share/perl5/PVE/API2/Cluster.pm line 13, line > > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > > /usr/share/perl5/PVE/API2.pm line 13, line 755. > > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted > at > > /usr/share/perl5/PVE/API2.pm line 13, line 755. > > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > > /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, line > > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted > at > > /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, l > > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at > > /usr/bin/pvedaemon line 11, line 755. > > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation aborted > at > > /usr/bin/pvedaemon line 11, line 755. > > Jan 24 11:19:38 pve01 systemd[1]: pvedaemon.service: Control process > > exited, code=exited status=2 > > Jan 24 11:19:38 pve01 systemd[1]: Reload failed for PVE API Daemon. > > > > > > can you post the complete error from the journal? > also the complete output of > > apt update > apt dist-upgrade > > could be helpful > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Thu Jan 24 15:21:05 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 24 Jan 2019 12:21:05 -0200 Subject: [PVE-User] Error trying to dist-upgrade a fresh installation In-Reply-To: References: <79991892-2d05-3ae3-2876-f2d13be356b9@proxmox.com> Message-ID: A note: I am using ZFS as root file system... I don't if this is relevant or not... --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qui, 24 de jan de 2019 ?s 12:15, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > journal -xe > > Jan 24 12:12:50 pve01 systemd[1]: Reloading PVE API Daemon. > -- Subject: Unit pvedaemon.service has begun reloading its configuration > -- Defined-By: systemd > -- Support: https://www.debian.org/support > -- > -- Unit pvedaemon.service has begun reloading its configuration > Jan 24 12:12:52 pve01 pvedaemon[2766]: Can't load > '/usr/lib/x86_64-linux-gnu/perl5/5.24/auto/PVE/RADOS/RADOS.so' for module > PVE::RADOS > Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at > /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 10, line > Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted > at /usr/share/perl5/PVE/Storage/RBDPlugin.pm line 10, > Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at > /usr/share/perl5/PVE/Storage.pm line 30, line 755. > Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted > at /usr/share/perl5/PVE/Storage.pm line 30, line 755. > Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at > /usr/share/perl5/PVE/API2/Cluster.pm line 13, line 755. > Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted > at /usr/share/perl5/PVE/API2/Cluster.pm line 13, line > Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at > /usr/share/perl5/PVE/API2.pm line 13, line 755. > Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted > at /usr/share/perl5/PVE/API2.pm line 13, line 755. > Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at > /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, line > Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted > at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, l > Jan 24 12:12:52 pve01 pvedaemon[2766]: Compilation failed in require at > /usr/bin/pvedaemon line 11, line 755. > Jan 24 12:12:52 pve01 pvedaemon[2766]: BEGIN failed--compilation aborted > at /usr/bin/pvedaemon line 11, line 755. > Jan 24 12:12:52 pve01 systemd[1]: pvedaemon.service: Control process > exited, code=exited status=2 > Jan 24 12:12:52 pve01 systemd[1]: Reload failed for PVE API Daemon. > -- Subject: Unit pvedaemon.service has finished reloading its configuration > -- Defined-By: systemd > -- Support: https://www.debian.org/support > -- > -- Unit pvedaemon.service has finished reloading its configuration > -- > -- The result is failed. > lines 2003-2030/2030 (END) > > apt update > Ign:1 http://mirror.pop-sc.rnp.br/debian stretch InRelease > Hit:2 http://security.debian.org stretch/updates InRelease > Hit:3 http://mirror.pop-sc.rnp.br/debian stretch-updates InRelease > > Hit:4 http://mirror.pop-sc.rnp.br/debian stretch Release > > Hit:5 http://download.proxmox.com/debian stretch InRelease > Reading package lists... Done > Building dependency tree > Reading state information... Done > All packages are up to date. > > apt dist-upgrade > Reading package lists... Done > Building dependency tree > Reading state information... Done > Calculating upgrade... Done > 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. > 1 not fully installed or removed. > After this operation, 0 B of additional disk space will be used. > Do you want to continue? [Y/n] > Setting up pve-manager (5.3-8) ... > Job for pvedaemon.service failed because the control process exited with > error code. > See "systemctl status pvedaemon.service" and "journalctl -xe" for details. > dpkg: error processing package pve-manager (--configure): > subprocess installed post-installation script returned error exit status 1 > Errors were encountered while processing: > pve-manager > E: Sub-process /usr/bin/dpkg returned an error code (1) > > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qui, 24 de jan de 2019 ?s 11:38, Dominik Csapak > escreveu: > >> On 1/24/19 2:21 PM, Gilberto Nunes wrote: >> > Hi list >> > >> > I have a fresh installation here, and when I try to updagre it I get >> some >> > errors: >> > apt dist-upgrade >> > Reading package lists... Done >> > Building dependency tree >> > Reading state information... Done >> > Calculating upgrade... Done >> > 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. >> > 1 not fully installed or removed. >> > After this operation, 0 B of additional disk space will be used. >> > Do you want to continue? [Y/n] >> > Setting up pve-manager (5.3-8) ... >> > Job for pvedaemon.service failed because the control process exited with >> > error code. >> > See "systemctl status pvedaemon.service" and "journalctl -xe" for >> details. >> > dpkg: error processing package pve-manager (--configure): >> > subprocess installed post-installation script returned error exit >> status 1 >> > Errors were encountered while processing: >> > pve-manager >> > E: Sub-process /usr/bin/dpkg returned an error code (1) >> > >> > systemctl status pvedaemon.service >> > * pvedaemon.service - PVE API Daemon >> > Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; >> vendor >> > preset: enabled) >> > Active: active (running) (Result: exit-code) since Thu 2019-01-24 >> > 11:15:52 -02; 5min ago >> > Process: 2634 ExecReload=/usr/bin/pvedaemon restart (code=exited, >> > status=2) >> > Main PID: 1366 (pvedaemon) >> > Tasks: 4 (limit: 4915) >> > Memory: 115.6M >> > CPU: 6.251s >> > CGroup: /system.slice/pvedaemon.service >> > |-1366 pvedaemon >> > |-1369 pvedaemon worker >> > |-1370 pvedaemon worker >> > `-1371 pvedaemon worker >> > >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at >> > /usr/share/perl5/PVE/API2/Cluster.pm line 13, line 755. >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation >> aborted at >> > /usr/share/perl5/PVE/API2/Cluster.pm line 13, line >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at >> > /usr/share/perl5/PVE/API2.pm line 13, line 755. >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation >> aborted at >> > /usr/share/perl5/PVE/API2.pm line 13, line 755. >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at >> > /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, line >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation >> aborted at >> > /usr/share/perl5/PVE/Service/pvedaemon.pm line 8, l >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: Compilation failed in require at >> > /usr/bin/pvedaemon line 11, line 755. >> > Jan 24 11:19:38 pve01 pvedaemon[2634]: BEGIN failed--compilation >> aborted at >> > /usr/bin/pvedaemon line 11, line 755. >> > Jan 24 11:19:38 pve01 systemd[1]: pvedaemon.service: Control process >> > exited, code=exited status=2 >> > Jan 24 11:19:38 pve01 systemd[1]: Reload failed for PVE API Daemon. >> > >> > >> >> can you post the complete error from the journal? >> also the complete output of >> >> apt update >> apt dist-upgrade >> >> could be helpful >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > From d.csapak at proxmox.com Thu Jan 24 15:22:39 2019 From: d.csapak at proxmox.com (Dominik Csapak) Date: Thu, 24 Jan 2019 15:22:39 +0100 Subject: [PVE-User] Error trying to dist-upgrade a fresh installation In-Reply-To: References: <79991892-2d05-3ae3-2876-f2d13be356b9@proxmox.com> Message-ID: On 1/24/19 3:15 PM, Gilberto Nunes wrote: > Jan 24 12:12:52 pve01 pvedaemon[2766]: Can't load > '/usr/lib/x86_64-linux-gnu/perl5/5.24/auto/PVE/RADOS/RADOS.so' for module > PVE::RADOS does that file exist? it should be contained in the package librados2-perl From gilberto.nunes32 at gmail.com Thu Jan 24 15:30:38 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 24 Jan 2019 12:30:38 -0200 Subject: [PVE-User] Error trying to dist-upgrade a fresh installation In-Reply-To: References: <79991892-2d05-3ae3-2876-f2d13be356b9@proxmox.com> Message-ID: dpkg -l |grep librados2-perl ii librados2-perl 1.0-6 amd64 Perl bindings for librados --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qui, 24 de jan de 2019 ?s 12:22, Dominik Csapak escreveu: > On 1/24/19 3:15 PM, Gilberto Nunes wrote: > > Jan 24 12:12:52 pve01 pvedaemon[2766]: Can't load > > '/usr/lib/x86_64-linux-gnu/perl5/5.24/auto/PVE/RADOS/RADOS.so' for module > > PVE::RADOS > > does that file exist? > > it should be contained in the package > > librados2-perl > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Thu Jan 24 16:20:20 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 24 Jan 2019 13:20:20 -0200 Subject: [PVE-User] Error trying to dist-upgrade a fresh installation In-Reply-To: References: <79991892-2d05-3ae3-2876-f2d13be356b9@proxmox.com> Message-ID: Well guys! I download de ISO again from PVE site, reinstall everything using XFS as root file system, and now I was able to upgrade. Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qui, 24 de jan de 2019 ?s 12:30, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > dpkg -l |grep librados2-perl > ii librados2-perl 1.0-6 > amd64 Perl bindings for librados > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qui, 24 de jan de 2019 ?s 12:22, Dominik Csapak > escreveu: > >> On 1/24/19 3:15 PM, Gilberto Nunes wrote: >> > Jan 24 12:12:52 pve01 pvedaemon[2766]: Can't load >> > '/usr/lib/x86_64-linux-gnu/perl5/5.24/auto/PVE/RADOS/RADOS.so' for >> module >> > PVE::RADOS >> >> does that file exist? >> >> it should be contained in the package >> >> librados2-perl >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > From lists at merit.unu.edu Fri Jan 25 13:40:48 2019 From: lists at merit.unu.edu (mj) Date: Fri, 25 Jan 2019 13:40:48 +0100 Subject: [PVE-User] win 2012 | resize disk, new size not seen from windows Message-ID: <7e07cc03-dda1-4fc0-0a04-db74b09e915c@merit.unu.edu> Hi, Running a win2012 guest on proxmox 5.3 with ceph storage backend. The disks are added as scsi0 and scsi1. I have enlarged scsi1 (the data disk, non-boot device) using the proxmox gui from 900GB to 1800GB. I have shutdown/restartd the win2012 guest, but it's Disk Management is not recognising the new disk size. I have refreshed and rescanned: no change. No empty space at the end of the device is showing up and extend Volume is greyed out. Am I missing a step somewhere..? MJ From alfiomunoz at gmail.com Fri Jan 25 13:47:09 2019 From: alfiomunoz at gmail.com (Alfio munoz) Date: Fri, 25 Jan 2019 08:47:09 -0400 Subject: [PVE-User] win 2012 | resize disk, new size not seen from windows In-Reply-To: <7e07cc03-dda1-4fc0-0a04-db74b09e915c@merit.unu.edu> References: <7e07cc03-dda1-4fc0-0a04-db74b09e915c@merit.unu.edu> Message-ID: Hi, Is a Windows task, not a Proxmox problem, go to computer management and right click over the disk you need to increase and choose expand. El vie., 25 de enero de 2019 8:40 a. m., mj escribi?: > Hi, > > Running a win2012 guest on proxmox 5.3 with ceph storage backend. The > disks are added as scsi0 and scsi1. I have enlarged scsi1 (the data > disk, non-boot device) using the proxmox gui from 900GB to 1800GB. > > I have shutdown/restartd the win2012 guest, but it's Disk Management is > not recognising the new disk size. I have refreshed and rescanned: no > change. No empty space at the end of the device is showing up and extend > Volume is greyed out. > > Am I missing a step somewhere..? > > MJ > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From t.lamprecht at proxmox.com Fri Jan 25 15:05:04 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Fri, 25 Jan 2019 15:05:04 +0100 Subject: [PVE-User] APT CVE-2019-3462 (please read before upgrading!) In-Reply-To: <20190123092736.cmvrpdlc4uwite43@nora.maurer-it.com> References: <20190123092736.cmvrpdlc4uwite43@nora.maurer-it.com> Message-ID: On 1/23/19 10:27 AM, Fabian Gr?nbichler wrote: > The APT package manager used by Proxmox VE and Proxmox Mail Gateway was > recently discovered to be affected by CVE-2019-3462, allowing a > Man-In-The-Middle or malicious mirror server to execute arbitrary code > with root privileges when affected systems attempt to install upgrades. > > To securely upgrade your systems, run the following commands as root: > > # apt -o Acquire::http::AllowRedirect=false update > # apt -o Acquire::http::AllowRedirect=false full-upgrade > > and verify that apt is now at least version 1.4.9 on Debian Stretch: > > $ apt -v > apt 1.4.9 (amd64) > > Please see the Debian Security Advisory for details: > https://www.debian.org/security/2019/dsa-4371 > To allow you to install Proxmox VE with a package management system version not affected by this issue, we additionally released a new Proxmox VE 5.3 ISO containing the fix for CVE-2019-3462 and all other security fixes since the first 5.3 ISO. Get it from: https://www.proxmox.com/en/downloads/category/iso-images-pve http://download.proxmox.com/iso/proxmox-ve_5.3-2.iso All container templates based on apt (Debian and Ubuntu) got also updated yesterday. cheers, Thomas From lists at merit.unu.edu Sat Jan 26 13:05:44 2019 From: lists at merit.unu.edu (mj) Date: Sat, 26 Jan 2019 13:05:44 +0100 Subject: [PVE-User] win 2012 | resize disk, new size not seen from windows In-Reply-To: References: <7e07cc03-dda1-4fc0-0a04-db74b09e915c@merit.unu.edu> Message-ID: Hi Alfio, Yes, I realise that, but as written: I have done all within windows: rebooted, opened Disk Management, refreshed, rescanned, it just keeps displaying the old size, without empty space at the end of the disk. Windows simply does not notice that the disk has changed. Hence my question: am I missing anything? Could there be an obvious reason fo this behaviour? Am I missing a step? Any tips, ideas..? MJ On 1/25/19 1:47 PM, Alfio munoz wrote: > Hi, Is a Windows task, not a Proxmox problem, go to computer management and > right click over the disk you need to increase and choose expand. > > > > El vie., 25 de enero de 2019 8:40 a. m., mj escribi?: > >> Hi, >> >> Running a win2012 guest on proxmox 5.3 with ceph storage backend. The >> disks are added as scsi0 and scsi1. I have enlarged scsi1 (the data >> disk, non-boot device) using the proxmox gui from 900GB to 1800GB. >> >> I have shutdown/restartd the win2012 guest, but it's Disk Management is >> not recognising the new disk size. I have refreshed and rescanned: no >> change. No empty space at the end of the device is showing up and extend >> Volume is greyed out. >> >> Am I missing a step somewhere..? >> >> MJ >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From lists at merit.unu.edu Sat Jan 26 15:56:41 2019 From: lists at merit.unu.edu (mj) Date: Sat, 26 Jan 2019 15:56:41 +0100 Subject: [PVE-User] win 2012 | resize disk, new size not seen from windows In-Reply-To: References: <7e07cc03-dda1-4fc0-0a04-db74b09e915c@merit.unu.edu> Message-ID: <9bf9272f-d8d1-0889-f580-8a037092283c@merit.unu.edu> ok, I added a second disk to the win2012 machine, tried enlarging that, which worked as it should. So I ended up moving all contents from my old data-disk to this new (now larger) disk, and now I'm happy. Have a nice weekend all, MJ On 1/26/19 1:05 PM, mj wrote: > Hi Alfio, > > Yes, I realise that, but as written: I have done all within windows: > rebooted, opened Disk Management, refreshed, rescanned, it just keeps > displaying the old size, without empty space at the end of the disk. > Windows simply does not notice that the disk has changed. > > Hence my question: am I missing anything? Could there be an obvious > reason fo this behaviour? Am I missing a step? Any tips, ideas..? > > MJ > > On 1/25/19 1:47 PM, Alfio munoz wrote: >> Hi, Is a Windows task, not a Proxmox problem, go to computer >> management and >> right click over the disk you need to increase and choose expand. >> >> >> >> El vie., 25 de enero de 2019 8:40 a. m., mj >> escribi?: >> >>> Hi, >>> >>> Running a win2012 guest on proxmox 5.3 with ceph storage backend. The >>> disks are added as scsi0 and scsi1. I have enlarged scsi1 (the data >>> disk, non-boot device) using the proxmox gui from 900GB to 1800GB. >>> >>> I have shutdown/restartd the win2012 guest, but it's Disk Management is >>> not recognising the new disk size. I have refreshed and rescanned: no >>> change. No empty space at the end of the device is showing up and extend >>> Volume is greyed out. >>> >>> Am I missing a step somewhere..? >>> >>> MJ >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From daniel at linux-nerd.de Sat Jan 26 21:17:09 2019 From: daniel at linux-nerd.de (Daniel) Date: Sat, 26 Jan 2019 21:17:09 +0100 Subject: [PVE-User] LVM usage Message-ID: Hi there, i have a question. Proxmox is telling me that my THIN-LVM getting more and more full. I checked some virtual machines and I saw something weird (also on different hosts) --- Logical volume --- LV Path /dev/pve/vm-196-disk-1 LV Name vm-196-disk-1 VG Name pve LV UUID IOLFz3-QmhU-VqFr-nJlo-Ulrr-bsYX-PzJpgW LV Write Access read/write LV Creation host, time host09, 2017-11-18 14:18:47 +0100 LV Pool name data LV Status available # open 1 LV Size 500.00 GiB Mapped size 94.22% Current LE 128000 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:7 As I understand the LVM is using 94,22% of the available Storage (500GB). When I check in the VPS it self it show me Filesystem Size Used Avail Use% Mounted on /dev/mapper/pve-vm--196--disk--1 492G 32G 435G 7% / So only 32GB is used. What is happened there? There are no? Snapshots or something like this. Cheers Daniel From mityapetuhov at gmail.com Sat Jan 26 21:25:03 2019 From: mityapetuhov at gmail.com (Dmitry Petuhov) Date: Sat, 26 Jan 2019 23:25:03 +0300 Subject: [PVE-User] LVM usage In-Reply-To: References: Message-ID: <17b018eb-2934-343f-bda8-44a4a13c8235@gmail.com> This means that this 94.22% was once used. LVM knows when guest uses block first time, and allocates it at this omoment, but does not knows when it is freed in guest. You need to enable discard both in disk config on host and on filesystem in guest (via mount option of FS, or regulary run fstrim or its analog in your guest OS) to free space on host's LVM level. 26.01.2019 23:17, Daniel ?????: > Hi there, > > > > i have a question. Proxmox is telling me that my THIN-LVM getting more and more full. > > I checked some virtual machines and I saw something weird (also on different hosts) > > > > --- Logical volume --- > > LV Path /dev/pve/vm-196-disk-1 > > LV Name vm-196-disk-1 > > VG Name pve > > LV UUID IOLFz3-QmhU-VqFr-nJlo-Ulrr-bsYX-PzJpgW > > LV Write Access read/write > > LV Creation host, time host09, 2017-11-18 14:18:47 +0100 > > LV Pool name data > > LV Status available > > # open 1 > > LV Size 500.00 GiB > > Mapped size 94.22% > > Current LE 128000 > > Segments 1 > > Allocation inherit > > Read ahead sectors auto > > - currently set to 256 > > Block device 253:7 > > > > As I understand the LVM is using 94,22% of the available Storage (500GB). When I check in the VPS it self it show me > > > > Filesystem Size Used Avail Use% Mounted on > > /dev/mapper/pve-vm--196--disk--1 492G 32G 435G 7% / > > > > So only 32GB is used. What is happened there? There are no? Snapshots or something like this. > > > > Cheers > > > > Daniel > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From daniel at linux-nerd.de Sat Jan 26 21:56:08 2019 From: daniel at linux-nerd.de (Daniel) Date: Sat, 26 Jan 2019 21:56:08 +0100 Subject: [PVE-User] LVM usage In-Reply-To: <17b018eb-2934-343f-bda8-44a4a13c8235@gmail.com> References: <17b018eb-2934-343f-bda8-44a4a13c8235@gmail.com> Message-ID: <85A5984B-BC6E-45AD-8F77-5E8F0EFE780F@linux-nerd.de> But when this is not enbaled by default in Proxmox it sounds like a "bug" inside Proxmox? Problem is that I have in general no access to mount options inside Proxmox. Fstrim is working for me at the moment. But this cant be a working solution. ?Am 26.01.19, 21:25 schrieb "pve-user im Auftrag von Dmitry Petuhov" : This means that this 94.22% was once used. LVM knows when guest uses block first time, and allocates it at this omoment, but does not knows when it is freed in guest. You need to enable discard both in disk config on host and on filesystem in guest (via mount option of FS, or regulary run fstrim or its analog in your guest OS) to free space on host's LVM level. 26.01.2019 23:17, Daniel ?????: > Hi there, > > > > i have a question. Proxmox is telling me that my THIN-LVM getting more and more full. > > I checked some virtual machines and I saw something weird (also on different hosts) > > > > --- Logical volume --- > > LV Path /dev/pve/vm-196-disk-1 > > LV Name vm-196-disk-1 > > VG Name pve > > LV UUID IOLFz3-QmhU-VqFr-nJlo-Ulrr-bsYX-PzJpgW > > LV Write Access read/write > > LV Creation host, time host09, 2017-11-18 14:18:47 +0100 > > LV Pool name data > > LV Status available > > # open 1 > > LV Size 500.00 GiB > > Mapped size 94.22% > > Current LE 128000 > > Segments 1 > > Allocation inherit > > Read ahead sectors auto > > - currently set to 256 > > Block device 253:7 > > > > As I understand the LVM is using 94,22% of the available Storage (500GB). When I check in the VPS it self it show me > > > > Filesystem Size Used Avail Use% Mounted on > > /dev/mapper/pve-vm--196--disk--1 492G 32G 435G 7% / > > > > So only 32GB is used. What is happened there? There are no Snapshots or something like this. > > > > Cheers > > > > Daniel > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mityapetuhov at gmail.com Sun Jan 27 13:19:27 2019 From: mityapetuhov at gmail.com (Dmitry Petuhov) Date: Sun, 27 Jan 2019 15:19:27 +0300 Subject: [PVE-User] LVM usage In-Reply-To: <85A5984B-BC6E-45AD-8F77-5E8F0EFE780F@linux-nerd.de> References: <17b018eb-2934-343f-bda8-44a4a13c8235@gmail.com> <85A5984B-BC6E-45AD-8F77-5E8F0EFE780F@linux-nerd.de> Message-ID: No, that's not bug, that's feature. If fstrim frees space, then you have discard enabled on host (in VM config). You need to enable discard for guest filesystems on mount (usually in /etc/fstab), or just put fstrim to crontab. 26.01.2019 23:56, Daniel wrote: > But when this is not enbaled by default in Proxmox it sounds like a "bug" inside Proxmox? > Problem is that I have in general no access to mount options inside Proxmox. > > Fstrim is working for me at the moment. But this cant be a working solution. > > ?Am 26.01.19, 21:25 schrieb "pve-user im Auftrag von Dmitry Petuhov" : > > This means that this 94.22% was once used. LVM knows when guest uses > block first time, and allocates it at this omoment, but does not knows > when it is freed in guest. You need to enable discard both in disk > config on host and on filesystem in guest (via mount option of FS, or > regulary run fstrim or its analog in your guest OS) to free space on > host's LVM level. > > 26.01.2019 23:17, Daniel ?????: > > Hi there, > > > > > > > > i have a question. Proxmox is telling me that my THIN-LVM getting more and more full. > > > > I checked some virtual machines and I saw something weird (also on different hosts) > > > > > > > > --- Logical volume --- > > > > LV Path /dev/pve/vm-196-disk-1 > > > > LV Name vm-196-disk-1 > > > > VG Name pve > > > > LV UUID IOLFz3-QmhU-VqFr-nJlo-Ulrr-bsYX-PzJpgW > > > > LV Write Access read/write > > > > LV Creation host, time host09, 2017-11-18 14:18:47 +0100 > > > > LV Pool name data > > > > LV Status available > > > > # open 1 > > > > LV Size 500.00 GiB > > > > Mapped size 94.22% > > > > Current LE 128000 > > > > Segments 1 > > > > Allocation inherit > > > > Read ahead sectors auto > > > > - currently set to 256 > > > > Block device 253:7 > > > > > > > > As I understand the LVM is using 94,22% of the available Storage (500GB). When I check in the VPS it self it show me > > > > > > > > Filesystem Size Used Avail Use% Mounted on > > > > /dev/mapper/pve-vm--196--disk--1 492G 32G 435G 7% / > > > > > > > > So only 32GB is used. What is happened there? There are no Snapshots or something like this. > > > > > > > > Cheers > > > > > > > > Daniel > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From daniel at linux-nerd.de Sun Jan 27 13:34:59 2019 From: daniel at linux-nerd.de (Daniel) Date: Sun, 27 Jan 2019 13:34:59 +0100 Subject: [PVE-User] LVM usage In-Reply-To: References: <17b018eb-2934-343f-bda8-44a4a13c8235@gmail.com> <85A5984B-BC6E-45AD-8F77-5E8F0EFE780F@linux-nerd.de> Message-ID: <75BBA513-715C-45CB-B6CA-B174052680CF@linux-nerd.de> Yes, I run it via Cron on the Host. I found a Script in the Proxmox Forum. That?s working for me fine. Thx ?Am 27.01.19, 13:19 schrieb "pve-user im Auftrag von Dmitry Petuhov" : No, that's not bug, that's feature. If fstrim frees space, then you have discard enabled on host (in VM config). You need to enable discard for guest filesystems on mount (usually in /etc/fstab), or just put fstrim to crontab. 26.01.2019 23:56, Daniel wrote: > But when this is not enbaled by default in Proxmox it sounds like a "bug" inside Proxmox? > Problem is that I have in general no access to mount options inside Proxmox. > > Fstrim is working for me at the moment. But this cant be a working solution. > > ?Am 26.01.19, 21:25 schrieb "pve-user im Auftrag von Dmitry Petuhov" : > > This means that this 94.22% was once used. LVM knows when guest uses > block first time, and allocates it at this omoment, but does not knows > when it is freed in guest. You need to enable discard both in disk > config on host and on filesystem in guest (via mount option of FS, or > regulary run fstrim or its analog in your guest OS) to free space on > host's LVM level. > > 26.01.2019 23:17, Daniel ?????: > > Hi there, > > > > > > > > i have a question. Proxmox is telling me that my THIN-LVM getting more and more full. > > > > I checked some virtual machines and I saw something weird (also on different hosts) > > > > > > > > --- Logical volume --- > > > > LV Path /dev/pve/vm-196-disk-1 > > > > LV Name vm-196-disk-1 > > > > VG Name pve > > > > LV UUID IOLFz3-QmhU-VqFr-nJlo-Ulrr-bsYX-PzJpgW > > > > LV Write Access read/write > > > > LV Creation host, time host09, 2017-11-18 14:18:47 +0100 > > > > LV Pool name data > > > > LV Status available > > > > # open 1 > > > > LV Size 500.00 GiB > > > > Mapped size 94.22% > > > > Current LE 128000 > > > > Segments 1 > > > > Allocation inherit > > > > Read ahead sectors auto > > > > - currently set to 256 > > > > Block device 253:7 > > > > > > > > As I understand the LVM is using 94,22% of the available Storage (500GB). When I check in the VPS it self it show me > > > > > > > > Filesystem Size Used Avail Use% Mounted on > > > > /dev/mapper/pve-vm--196--disk--1 492G 32G 435G 7% / > > > > > > > > So only 32GB is used. What is happened there? There are no Snapshots or something like this. > > > > > > > > Cheers > > > > > > > > Daniel > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From wzyboy at wzyboy.org Mon Jan 28 15:47:22 2019 From: wzyboy at wzyboy.org (Zhuoyun Wei) Date: Mon, 28 Jan 2019 09:47:22 -0500 Subject: [PVE-User] Non-configurable options in cloud-init config Message-ID: <62ae2fbc-a0cc-4291-ab29-7acf98304235@www.fastmail.com> Hi, I am trying to set up our first Proxmox server. I am using the GenericCloud pre-built image by CentOS [1]. We use the exact same image already in a public cloud provider. By re-using the same image, we could replicate our production VMs in our office. After importing and starting the first VM in Proxmox, I found that something was not quite right: the packages in the Proxmox VM were newer than the packages in the cloud provider VM. After checking logs, it turned out that cloud-init did a full system upgrade on first boot (it upgraded CentOS 7.5 to CentOS 7.6). A further inspection revealed the roots of the issue: there was a hard-coded "package_upgrade: true" config key in the generated cloud-init config iso [2]. There was no way to override this config key as of cloud-init 0.7.9 [3], which was used in the latest CentOS GenericCloud image. Could Proxmox make a little improvement here and allow user to opt out the full system upgrade after the first boot of the VM? The only workaround I could see now is edit the CloudInit.pm file and comment out that line. Thanks. Links: [1] https://cloud.centos.org/centos/7/images/ [2] https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QemuServer/Cloudinit.pm;h=5be820c22d58b0cc0014e7dab6412f8d2db9132d;hb=refs/heads/master#l135 [3] https://cloudinit.readthedocs.io/en/0.7.9/topics/modules.html#package-update-upgrade-install -- Zhuoyun Wei From d.limbeck at proxmox.com Mon Jan 28 16:28:17 2019 From: d.limbeck at proxmox.com (David Limbeck) Date: Mon, 28 Jan 2019 16:28:17 +0100 Subject: [PVE-User] Non-configurable options in cloud-init config In-Reply-To: <62ae2fbc-a0cc-4291-ab29-7acf98304235@www.fastmail.com> References: <62ae2fbc-a0cc-4291-ab29-7acf98304235@www.fastmail.com> Message-ID: Yes, that's currently the only way to disable package upgrades. Feel free to open an enhancement in our bugtracker. (https://bugzilla.proxmox.com/) Just a quick note, the cloud-init version in CentOS is old and some features don't work correctly or at all. On 1/28/19 3:47 PM, Zhuoyun Wei wrote: > Hi, > > I am trying to set up our first Proxmox server. I am using the GenericCloud pre-built image by CentOS [1]. We use the exact same image already in a public cloud provider. By re-using the same image, we could replicate our production VMs in our office. > > After importing and starting the first VM in Proxmox, I found that something was not quite right: the packages in the Proxmox VM were newer than the packages in the cloud provider VM. After checking logs, it turned out that cloud-init did a full system upgrade on first boot (it upgraded CentOS 7.5 to CentOS 7.6). > > A further inspection revealed the roots of the issue: there was a hard-coded "package_upgrade: true" config key in the generated cloud-init config iso [2]. > > There was no way to override this config key as of cloud-init 0.7.9 [3], which was used in the latest CentOS GenericCloud image. Could Proxmox make a little improvement here and allow user to opt out the full system upgrade after the first boot of the VM? The only workaround I could see now is edit the CloudInit.pm file and comment out that line. > > Thanks. > > > Links: > > [1] https://cloud.centos.org/centos/7/images/ > [2] https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QemuServer/Cloudinit.pm;h=5be820c22d58b0cc0014e7dab6412f8d2db9132d;hb=refs/heads/master#l135 > [3] https://cloudinit.readthedocs.io/en/0.7.9/topics/modules.html#package-update-upgrade-install > From wzyboy at wzyboy.org Mon Jan 28 16:45:08 2019 From: wzyboy at wzyboy.org (Zhuoyun Wei) Date: Mon, 28 Jan 2019 10:45:08 -0500 Subject: [PVE-User] Non-configurable options in cloud-init config In-Reply-To: References: <62ae2fbc-a0cc-4291-ab29-7acf98304235@www.fastmail.com> Message-ID: <4b6227eb-7de0-4354-a3ab-58b0b821ce42@www.fastmail.com> Thanks. I have filed a bug. [1] I am aware that cloud-init 0.7.9 is quite an old version. The good news is, CentOS Team has rebased its cloud-init source to upstream version 18.2 as of 2018-10 [2]. The new cloud-init package is already in the repo (could be yum updated). But the latest pre-built cloud image is 1809, so it is stuck at 0.7.9 for now. Highly likely cloud-init 18.2 would be available in the next pre-built cloud image release. Links: [1] https://bugzilla.proxmox.com/show_bug.cgi?id=2068 [2] https://git.centos.org/blobdiff/rpms!cloud-init.git/c60dcdee662fa585e0ef611c5cd5c48078259a68/SPECS!cloud-init.spec -- Zhuoyun Wei On Mon, Jan 28, 2019, at 23:28, David Limbeck wrote: > Yes, that's currently the only way to disable package upgrades. Feel > free to open an enhancement in our bugtracker. > (https://bugzilla.proxmox.com/) > > Just a quick note, the cloud-init version in CentOS is old and some > features don't work correctly or at all. > > On 1/28/19 3:47 PM, Zhuoyun Wei wrote: > > Hi, > > > > I am trying to set up our first Proxmox server. I am using the GenericCloud pre-built image by CentOS [1]. We use the exact same image already in a public cloud provider. By re-using the same image, we could replicate our production VMs in our office. > > > > After importing and starting the first VM in Proxmox, I found that something was not quite right: the packages in the Proxmox VM were newer than the packages in the cloud provider VM. After checking logs, it turned out that cloud-init did a full system upgrade on first boot (it upgraded CentOS 7.5 to CentOS 7.6). > > > > A further inspection revealed the roots of the issue: there was a hard-coded "package_upgrade: true" config key in the generated cloud-init config iso [2]. > > > > There was no way to override this config key as of cloud-init 0.7.9 [3], which was used in the latest CentOS GenericCloud image. Could Proxmox make a little improvement here and allow user to opt out the full system upgrade after the first boot of the VM? The only workaround I could see now is edit the CloudInit.pm file and comment out that line. > > > > Thanks. > > > > > > Links: > > > > [1] https://cloud.centos.org/centos/7/images/ > > [2] https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QemuServer/Cloudinit.pm;h=5be820c22d58b0cc0014e7dab6412f8d2db9132d;hb=refs/heads/master#l135 > > [3] https://cloudinit.readthedocs.io/en/0.7.9/topics/modules.html#package-update-upgrade-install > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >