From elacunza at binovo.es Wed Aug 1 11:02:18 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 1 Aug 2018 11:02:18 +0200 Subject: [PVE-User] Cluster doesn't recover automatically after blackout Message-ID: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> Hi all, This morning there was a quite long blackout which powered off a cluster of 3 proxmox 5.1 servers. All 3 servers the same make and model, so they need the same amount of time to boot. When the power came back, servers started correctly but corosync couldn't set up a quorum. Events timing: 07:57:10 corosync start 07:57:15 first pmxcfs error quorum_initialize_failed: 2 07:57:52 network up 07:58:40 Corosync timeout 07:59:57 time sync works What I can see is that network switch boot was slower than server's, but nonetheless network was operational about 45s before corosync gives up trying to set up a quorum. I also can see that internet access wasn't back until 1 minute after corosync timeout (the time sync event). A simple restart of pve-cluster at about 9:50 restored the cluster to normal state. Is this expected? I expected that corosync would set up a quorum after network was operational.... # pveversion -v proxmox-ve: 5.1-41 (running kernel: 4.13.13-6-pve) pve-manager: 5.1-46 (running version: 5.1-46/ae8241d4) pve-kernel-4.13.13-6-pve: 4.13.13-41 pve-kernel-4.13.13-2-pve: 4.13.13-33 ceph: 12.2.2-pve1 corosync: 2.4.2-pve3 criu: 2.11.1-1~bpo90 glusterfs-client: 3.8.8-1 ksm-control-daemon: 1.2-2 libjs-extjs: 6.0.1-2 libpve-access-control: 5.0-8 libpve-common-perl: 5.0-28 libpve-guest-common-perl: 2.0-14 libpve-http-server-perl: 2.0-8 libpve-storage-perl: 5.0-17 libqb0: 1.0.1-1 lvm2: 2.02.168-pve6 lxc-pve: 2.1.1-2 lxcfs: 2.0.8-2 novnc-pve: 0.6-4 proxmox-widget-toolkit: 1.0-11 pve-cluster: 5.0-20 pve-container: 2.0-19 pve-docs: 5.1-16 pve-firewall: 3.0-5 pve-firmware: 2.0-3 pve-ha-manager: 2.0-5 pve-i18n: 1.0-4 pve-libspice-server1: 0.12.8-3 pve-qemu-kvm: 2.9.1-9 pve-xtermjs: 1.0-2 qemu-server: 5.0-22 smartmontools: 6.5+svn4324-1 spiceterm: 3.0-5 vncterm: 1.5-3 zfsutils-linux: 0.7.6-pve1~bpo9 Thanks a lot Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From a.antreich at proxmox.com Wed Aug 1 12:56:05 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Wed, 1 Aug 2018 12:56:05 +0200 Subject: [PVE-User] Cluster doesn't recover automatically after blackout In-Reply-To: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> References: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> Message-ID: <20180801105605.qsojkpnicmewv33t@dona.proxmox.com> Hi, On Wed, Aug 01, 2018 at 11:02:18AM +0200, Eneko Lacunza wrote: > Hi all, > > This morning there was a quite long blackout which powered off a cluster of > 3 proxmox 5.1 servers. > > All 3 servers the same make and model, so they need the same amount of time > to boot. > > When the power came back, servers started correctly but corosync couldn't > set up a quorum. Events timing: I recommend against, servers returning automatically to previous power state after a power loss. A manual start up is better, as by then the admin made sure power is back to normal operation. This will also reduce the chance of breakage if there are subsequent power or hardware failures. > > 07:57:10 corosync start > 07:57:15 first pmxcfs error quorum_initialize_failed: 2 > 07:57:52 network up > 07:58:40 Corosync timeout > 07:59:57 time sync works > > What I can see is that network switch boot was slower than server's, but > nonetheless network was operational about 45s before corosync gives up > trying to set up a quorum. > > I also can see that internet access wasn't back until 1 minute after > corosync timeout (the time sync event). > > A simple restart of pve-cluster at about 9:50 restored the cluster to normal > state. > > Is this expected? I expected that corosync would set up a quorum after > network was operational.... When was multicast working again? That might have taken longer, as IGMP snooping and the querier on the switch might just take longer to get operating again. -- Cheers, Alwin From elacunza at binovo.es Wed Aug 1 13:40:34 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 1 Aug 2018 13:40:34 +0200 Subject: [PVE-User] Cluster doesn't recover automatically after blackout In-Reply-To: <20180801105605.qsojkpnicmewv33t@dona.proxmox.com> References: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> <20180801105605.qsojkpnicmewv33t@dona.proxmox.com> Message-ID: <3c8c095a-a9d2-57a0-366b-c0388e021ac8@binovo.es> Hi Alwin, El 01/08/18 a las 12:56, Alwin Antreich escribi?: > On Wed, Aug 01, 2018 at 11:02:18AM +0200, Eneko Lacunza wrote: >> Hi all, >> >> This morning there was a quite long blackout which powered off a cluster of >> 3 proxmox 5.1 servers. >> >> All 3 servers the same make and model, so they need the same amount of time >> to boot. >> >> When the power came back, servers started correctly but corosync couldn't >> set up a quorum. Events timing: > I recommend against, servers returning automatically to previous power > state after a power loss. A manual start up is better, as by then the > admin made sure power is back to normal operation. This will also reduce > the chance of breakage if there are subsequent power or hardware > failures. This is an off-site place with no knowledgeable sysadmins and servers don't have remote control cards. I'm sure they would screw the boot up? :) I'm afraid we have to take the risk. :) > >> 07:57:10 corosync start >> 07:57:15 first pmxcfs error quorum_initialize_failed: 2 >> 07:57:52 network up >> 07:58:40 Corosync timeout >> 07:59:57 time sync works >> >> What I can see is that network switch boot was slower than server's, but >> nonetheless network was operational about 45s before corosync gives up >> trying to set up a quorum. >> >> I also can see that internet access wasn't back until 1 minute after >> corosync timeout (the time sync event). >> >> A simple restart of pve-cluster at about 9:50 restored the cluster to normal >> state. >> >> Is this expected? I expected that corosync would set up a quorum after >> network was operational.... > When was multicast working again? That might have taken longer, as IGMP > snooping and the querier on the switch might just take longer to get > operating again. I don't have that info (or I don't know how to look that in the logs, /var/log/corosync is empty). I'm trying to plan an intentional blackout to test things again with technicians onsite, we could get more info that day. Switch is HPE 1820-24G J9980A, it's L2 but quite dumb; we have serveral 18x0 switches deployed with good results so far. Thanks a lot, Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From a.antreich at proxmox.com Wed Aug 1 13:57:09 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Wed, 1 Aug 2018 13:57:09 +0200 Subject: [PVE-User] Cluster doesn't recover automatically after blackout In-Reply-To: <3c8c095a-a9d2-57a0-366b-c0388e021ac8@binovo.es> References: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> <20180801105605.qsojkpnicmewv33t@dona.proxmox.com> <3c8c095a-a9d2-57a0-366b-c0388e021ac8@binovo.es> Message-ID: <20180801115709.mizr2ow4jvhuhwbu@dona.proxmox.com> On Wed, Aug 01, 2018 at 01:40:34PM +0200, Eneko Lacunza wrote: > Hi Alwin, > > El 01/08/18 a las 12:56, Alwin Antreich escribi?: > > On Wed, Aug 01, 2018 at 11:02:18AM +0200, Eneko Lacunza wrote: > > > Hi all, > > > > > > This morning there was a quite long blackout which powered off a cluster of > > > 3 proxmox 5.1 servers. > > > > > > All 3 servers the same make and model, so they need the same amount of time > > > to boot. > > > > > > When the power came back, servers started correctly but corosync couldn't > > > set up a quorum. Events timing: > > I recommend against, servers returning automatically to previous power > > state after a power loss. A manual start up is better, as by then the > > admin made sure power is back to normal operation. This will also reduce > > the chance of breakage if there are subsequent power or hardware > > failures. > This is an off-site place with no knowledgeable sysadmins and servers don't > have remote control cards. I'm sure they would screw the boot up? :) > > I'm afraid we have to take the risk. :) A boot delay, if the server have such a setting or switchable UPS power plugs might help. :) > > > > > 07:57:10 corosync start > > > 07:57:15 first pmxcfs error quorum_initialize_failed: 2 > > > 07:57:52 network up > > > 07:58:40 Corosync timeout > > > 07:59:57 time sync works > > > > > > What I can see is that network switch boot was slower than server's, but > > > nonetheless network was operational about 45s before corosync gives up > > > trying to set up a quorum. > > > > > > I also can see that internet access wasn't back until 1 minute after > > > corosync timeout (the time sync event). > > > > > > A simple restart of pve-cluster at about 9:50 restored the cluster to normal > > > state. > > > > > > Is this expected? I expected that corosync would set up a quorum after > > > network was operational.... > > When was multicast working again? That might have taken longer, as IGMP > > snooping and the querier on the switch might just take longer to get > > operating again. > I don't have that info (or I don't know how to look that in the logs, > /var/log/corosync is empty). I'm trying to plan an intentional blackout to > test things again with technicians onsite, we could get more info that day. Corosync writes into the syslog, there should be more to find. > > Switch is HPE 1820-24G J9980A, it's L2 but quite dumb; we have serveral 18x0 > switches deployed with good results so far. The switch may hold a log that shows its startup process. From elacunza at binovo.es Wed Aug 1 16:12:19 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 1 Aug 2018 16:12:19 +0200 Subject: [PVE-User] Cluster doesn't recover automatically after blackout In-Reply-To: <20180801115709.mizr2ow4jvhuhwbu@dona.proxmox.com> References: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> <20180801105605.qsojkpnicmewv33t@dona.proxmox.com> <3c8c095a-a9d2-57a0-366b-c0388e021ac8@binovo.es> <20180801115709.mizr2ow4jvhuhwbu@dona.proxmox.com> Message-ID: Hi, El 01/08/18 a las 13:57, Alwin Antreich escribi?: > On Wed, Aug 01, 2018 at 01:40:34PM +0200, Eneko Lacunza wrote: >> El 01/08/18 a las 12:56, Alwin Antreich escribi?: >>> On Wed, Aug 01, 2018 at 11:02:18AM +0200, Eneko Lacunza wrote: >>>> Hi all, >>>> >>>> This morning there was a quite long blackout which powered off a cluster of >>>> 3 proxmox 5.1 servers. >>>> >>>> All 3 servers the same make and model, so they need the same amount of time >>>> to boot. >>>> >>>> When the power came back, servers started correctly but corosync couldn't >>>> set up a quorum. Events timing: >>> I recommend against, servers returning automatically to previous power >>> state after a power loss. A manual start up is better, as by then the >>> admin made sure power is back to normal operation. This will also reduce >>> the chance of breakage if there are subsequent power or hardware >>> failures. >> This is an off-site place with no knowledgeable sysadmins and servers don't >> have remote control cards. I'm sure they would screw the boot up? :) >> >> I'm afraid we have to take the risk. :) > A boot delay, if the server have such a setting or switchable UPS power > plugs might help. :) Yes, I can do that at grub level, that's no problem. But I have to know first the correct amount for the delay ;) > >>> >>>> 07:57:10 corosync start >>>> 07:57:15 first pmxcfs error quorum_initialize_failed: 2 >>>> 07:57:52 network up >>>> 07:58:40 Corosync timeout >>>> 07:59:57 time sync works >>>> >>>> What I can see is that network switch boot was slower than server's, but >>>> nonetheless network was operational about 45s before corosync gives up >>>> trying to set up a quorum. >>>> >>>> I also can see that internet access wasn't back until 1 minute after >>>> corosync timeout (the time sync event). >>>> >>>> A simple restart of pve-cluster at about 9:50 restored the cluster to normal >>>> state. >>>> >>>> Is this expected? I expected that corosync would set up a quorum after >>>> network was operational.... >>> When was multicast working again? That might have taken longer, as IGMP >>> snooping and the querier on the switch might just take longer to get >>> operating again. >> I don't have that info (or I don't know how to look that in the logs, >> /var/log/corosync is empty). I'm trying to plan an intentional blackout to >> test things again with technicians onsite, we could get more info that day. > Corosync writes into the syslog, there should be more to find. Doesn't seem there is any more to me: # grep corosync /var/log/syslog Aug? 1 07:57:11 proxmox1 corosync[1697]:? [MAIN? ] Corosync Cluster Engine ('2.4.2-dirty'): started and ready to provide service. Aug? 1 07:57:11 proxmox1 corosync[1697]: notice? [MAIN? ] Corosync Cluster Engine ('2.4.2-dirty'): started and ready to provide service. Aug? 1 07:57:11 proxmox1 corosync[1697]: info??? [MAIN? ] Corosync built-in features: dbus rdma monitoring watchdog augeas systemd upstart xmlconf qdevices qnetd snmp pie relro bindnow Aug? 1 07:57:11 proxmox1 corosync[1697]:? [MAIN? ] Corosync built-in features: dbus rdma monitoring watchdog augeas systemd upstart xmlconf qdevices qnetd snmp pie relro bindnow Aug? 1 07:58:40 proxmox1 systemd[1]: corosync.service: Start operation timed out. Terminating. Aug? 1 07:58:40 proxmox1 systemd[1]: corosync.service: Unit entered failed state. Aug? 1 07:58:40 proxmox1 systemd[1]: corosync.service: Failed with result 'timeout'. Aug? 1 09:51:35 proxmox1 corosync[32220]:? [MAIN? ] Corosync Cluster Engine ('2.4.2-dirty'): started and ready to provide service. This last line is our manual pve-cluster restart . > >> Switch is HPE 1820-24G J9980A, it's L2 but quite dumb; we have serveral 18x0 >> switches deployed with good results so far. > The switch may hold a log that shows its startup process. Seems it was disabled, we have enabled it. Thanks Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From lyt_yudi at icloud.com Fri Aug 3 09:58:39 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Fri, 03 Aug 2018 15:58:39 +0800 Subject: [PVE-User] This parameter is the length of characters OR a file size? Message-ID: Hi, Sorry by this :( # cat test.txt |wc 9000 9000 129699 # pvesh create nodes/localhost/qemu/131/agent/file-write --content "$(cat test.txt)" --file /root/test.txt 400 Parameter verification failed. content: value may only be 61440 characters long pvesh create --content --file [OPTIONS] [FORMAT_OPTIONS] Can increase it? 442 __PACKAGE__->register_method({ 443 name => 'file-write', 444 path => 'file-write', 445 method => 'POST', 446 protected => 1, 447 proxyto => 'node', 448 description => "Writes the given file via guest agent.", 449 permissions => { check => [ 'perm', '/vms/{vmid}', [ 'VM.Monitor' ]]}, 450 parameters => { 451 additionalProperties => 0, 452 properties => { 453 node => get_standard_option('pve-node'), 454 vmid => get_standard_option('pve-vmid', { 455 completion => \&PVE::QemuServer::complete_vmid_running }), 456 file => { 457 type => 'string', 458 description => 'The path to the file.' 459 }, 460 content => { 461 type => 'string', 462 maxLength => 60*1024, # 60k, smaller than our 64k POST limit 463 description => "The content to write into the file." 464 } 465 }, 466 }, From renato at aster-lab.com Fri Aug 3 14:45:07 2018 From: renato at aster-lab.com (Renato Gallo) Date: Fri, 3 Aug 2018 14:45:07 +0200 (CEST) Subject: [PVE-User] vm----proxmox----switch---proxmox2---vm2 Message-ID: <291978734.22728.1533300307429.JavaMail.zimbra@aster-lab.com> Hello, I need to separate completely the routing and traffic of WAN and LAN I have vm1 on proxmox1 I have vm2 on proxmox2 each proxmox has a two nics one for WAN and one for LAN the two proxmoxes are connected to each other on the LAN interface via a switch The two vm's have two nics vmbr0 connected to WAN and vmbr1 connected to LAN each of the vm's can ping both proxmoxes LAN address I cannot ping the other vm from none of each vm what am I missing ? Renato Gallo System Engineer sede legale e operativa: Via San brunone, 13 - 20156 - Milano (MI) Tel. +39 02 - 87049490 Fax +39 02 - 48677349 Mobile. +39 342 - 6350524 Wi | FreeNumbers: https://freenumbers.way-interactive.com Wi | SMS: https://sms.way-interactive.com Wi | Voip: https://voip.way-interactive.com Asterweb: http://www.asterweb.org Le informazioni contenute in questo messaggio e negli eventuali allegati sono riservate e per uso esclusivo del destinatario. Persone diverse dallo stesso non possono copiare o distribuire il messaggio a terzi. Chiunque riceva questo messaggio per errore ? pregato di distruggerlo e di informare immediatamente [ mailto:info at sigmaware.it | info@ ] asterweb.org From josh at noobbox.com Fri Aug 3 16:31:56 2018 From: josh at noobbox.com (Josh Knight) Date: Fri, 3 Aug 2018 10:31:56 -0400 Subject: [PVE-User] vm----proxmox----switch---proxmox2---vm2 In-Reply-To: <291978734.22728.1533300307429.JavaMail.zimbra@aster-lab.com> References: <291978734.22728.1533300307429.JavaMail.zimbra@aster-lab.com> Message-ID: for vm1 and vm2, are the LAN interfaces on the same subnet/vlan? A diagram with sample subnets might help, my initial suspect would be a ip route issue if they're not on the same vlan. Josh Knight On Fri, Aug 3, 2018 at 8:45 AM, Renato Gallo wrote: > Hello, > > I need to separate completely the routing and traffic of WAN and LAN > > I have vm1 on proxmox1 > I have vm2 on proxmox2 > > each proxmox has a two nics one for WAN and one for LAN > the two proxmoxes are connected to each other on the LAN interface via a > switch > > The two vm's have two nics vmbr0 connected to WAN and vmbr1 connected to > LAN > > each of the vm's can ping both proxmoxes LAN address > > I cannot ping the other vm from none of each vm > > what am I missing ? > > Renato Gallo > > System Engineer > sede legale e operativa: Via San brunone, 13 - 20156 - Milano (MI) > Tel. +39 02 - 87049490 > Fax +39 02 - 48677349 > Mobile. +39 342 - 6350524 > Wi | FreeNumbers: https://freenumbers.way-interactive.com > Wi | SMS: https://sms.way-interactive.com > Wi | Voip: https://voip.way-interactive.com > Asterweb: http://www.asterweb.org > > Le informazioni contenute in questo messaggio e negli eventuali allegati > sono riservate e per uso esclusivo del destinatario. > Persone diverse dallo stesso non possono copiare o distribuire il > messaggio a terzi. > Chiunque riceva questo messaggio per errore ? pregato di distruggerlo e di > informare immediatamente [ mailto:info at sigmaware.it | info@ ] asterweb.org > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From yannis.milios at gmail.com Fri Aug 3 19:11:10 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Fri, 3 Aug 2018 18:11:10 +0100 Subject: [PVE-User] vm----proxmox----switch---proxmox2---vm2 In-Reply-To: References: <291978734.22728.1533300307429.JavaMail.zimbra@aster-lab.com> Message-ID: If both VMs have their vnic attached to vmbr1, then it might be also worth having a look at the firewall on the o/s inside the VM. If that's enabled on both sides, it could potentially block ping requests... On Friday, August 3, 2018, Josh Knight wrote: > for vm1 and vm2, are the LAN interfaces on the same subnet/vlan? A diagram > with sample subnets might help, my initial suspect would be a ip route > issue if they're not on the same vlan. > > Josh Knight > > > On Fri, Aug 3, 2018 at 8:45 AM, Renato Gallo wrote: > > > Hello, > > > > I need to separate completely the routing and traffic of WAN and LAN > > > > I have vm1 on proxmox1 > > I have vm2 on proxmox2 > > > > each proxmox has a two nics one for WAN and one for LAN > > the two proxmoxes are connected to each other on the LAN interface via a > > switch > > > > The two vm's have two nics vmbr0 connected to WAN and vmbr1 connected to > > LAN > > > > each of the vm's can ping both proxmoxes LAN address > > > > I cannot ping the other vm from none of each vm > > > > what am I missing ? > > > > Renato Gallo > > > > System Engineer > > sede legale e operativa: Via San brunone, 13 - 20156 - Milano (MI) > > Tel. +39 02 - 87049490 > > Fax +39 02 - 48677349 > > Mobile. +39 342 - 6350524 > > Wi | FreeNumbers: https://freenumbers.way-interactive.com > > Wi | SMS: https://sms.way-interactive.com > > Wi | Voip: https://voip.way-interactive.com > > Asterweb: http://www.asterweb.org > > > > Le informazioni contenute in questo messaggio e negli eventuali allegati > > sono riservate e per uso esclusivo del destinatario. > > Persone diverse dallo stesso non possono copiare o distribuire il > > messaggio a terzi. > > Chiunque riceva questo messaggio per errore ? pregato di distruggerlo e > di > > informare immediatamente [ mailto:info at sigmaware.it | info@ ] > asterweb.org > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Sent from Gmail Mobile From klaus.mailinglists at pernau.at Mon Aug 6 17:25:36 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Mon, 6 Aug 2018 17:25:36 +0200 Subject: [PVE-User] How to specify VLANs for a trunk interface into the VM Message-ID: <75b30778-f206-a1a4-412f-d706a52283cd@pernau.at> Hello! I have a trunk into my Proxmox server with plenty of vlans, into OVS bridge vmbr0. How can I configure a trunk into a VM and specify the allowed VLANs? The GUI dialog only allows to specify a single VLAN tag (which causes the interface to be an OVS access port). I need to configure an OVS port like: ovs-vsctl set port XXX trunks=20,30,40 Any chance to configure this via GUI or console? Thanks Klaus From josh at noobbox.com Mon Aug 6 18:57:40 2018 From: josh at noobbox.com (Josh Knight) Date: Mon, 6 Aug 2018 12:57:40 -0400 Subject: [PVE-User] How to specify VLANs for a trunk interface into the VM In-Reply-To: <75b30778-f206-a1a4-412f-d706a52283cd@pernau.at> References: <75b30778-f206-a1a4-412f-d706a52283cd@pernau.at> Message-ID: I don't see a way to do this via the GUI for the VM interface. You can use that same command you listed but on the tap interface tapXiY where X is the VM ID and Y is the interface number. E.g. tap100i1 would be interface 1 of VMID 100. That will restrict the allowed VLANs on that port, but of course that doesn't stick around after a reboot, ovs has no 'startup config'. But it should be obvious which IDs to use if you take a look at the list of interfaces on the host with `ip link | grep tap`. Or you can do something like this. But either way, the ovs-vsctl command will work. ovs-vsctl add port tapXiY trunks 20 ovs-vsctl add port tapXiY trunks 30 ovs-vsctl add port tapXiY trunks 40 Another thing to consider however, is by default, the ovs trunk port will allow all VLANs. In your VM if you create subinterfaces like eth0.20, eth0.30, eth0.40, then you'll have access to each specific VLAN without having to configure allowed vlans in ovs. It depends on your use case. Josh Josh Knight On Mon, Aug 6, 2018 at 11:25 AM, Klaus Darilion < klaus.mailinglists at pernau.at> wrote: > Hello! > > I have a trunk into my Proxmox server with plenty of vlans, into OVS > bridge vmbr0. > > How can I configure a trunk into a VM and specify the allowed VLANs? The > GUI dialog only allows to specify a single VLAN tag (which causes the > interface to be an OVS access port). > > I need to configure an OVS port like: > ovs-vsctl set port XXX trunks=20,30,40 > > Any chance to configure this via GUI or console? > > Thanks > Klaus > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From klaus.mailinglists at pernau.at Mon Aug 6 21:19:55 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Mon, 6 Aug 2018 21:19:55 +0200 Subject: [PVE-User] How to specify VLANs for a trunk interface into the VM In-Reply-To: References: <75b30778-f206-a1a4-412f-d706a52283cd@pernau.at> Message-ID: <92c3153d-e992-f7ca-d00b-1c8a4bf56552@pernau.at> Hi Josh! Am 06.08.2018 um 18:57 schrieb Josh Knight: > I don't see a way to do this via the GUI for the VM interface. > > You can use that same command you listed but on the tap interface tapXiY > where X is the VM ID and Y is the interface number. E.g. tap100i1 would be > interface 1 of VMID 100. That will restrict the allowed VLANs on that > port, but of course that doesn't stick around after a reboot, ovs has no > 'startup config'. But it should be obvious which IDs to use if you take a > look at the list of interfaces on the host with `ip link | grep tap`. > > Or you can do something like this. But either way, the ovs-vsctl command > will work. > ovs-vsctl add port tapXiY trunks 20 > ovs-vsctl add port tapXiY trunks 30 > ovs-vsctl add port tapXiY trunks 40 That's not reasonable. The port config has to be reboot save and stick to the VM config (ie VM is migrated to another host). > Another thing to consider however, is by default, the ovs trunk port will > allow all VLANs. In your VM if you create subinterfaces like eth0.20, > eth0.30, eth0.40, then you'll have access to each specific VLAN without > having to configure allowed vlans in ovs. It depends on your use case. I want to avoid that - for security reasons and to not confuse Linux. I had seen strange things in Linux ie. where it answered to ARP on VLANs available on the trunk but not explicitely configured as eth0.XX interface. So, the VM should only see the required VLANs. I hoped that the port config is flexible to configure trunks=...., but it is not available in the GUI. The workaround would be to convert every trunk into access ports. That would be 3 more interfaces - and probably the best solution at the moment. I just was looking for a more beautiful solution. regards Klaus From klaus.mailinglists at pernau.at Mon Aug 6 21:25:20 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Mon, 6 Aug 2018 21:25:20 +0200 Subject: [PVE-User] How to specify VLANs for a trunk interface into the VM In-Reply-To: <92c3153d-e992-f7ca-d00b-1c8a4bf56552@pernau.at> References: <75b30778-f206-a1a4-412f-d706a52283cd@pernau.at> <92c3153d-e992-f7ca-d00b-1c8a4bf56552@pernau.at> Message-ID: <514da49d-da0b-5a11-ad38-0d4afbc9219a@pernau.at> Am 06.08.2018 um 21:19 schrieb Klaus Darilion: > I hoped that the port config is flexible to configure trunks=...., but > it is not available in the GUI. I think I just found it at https://pve.proxmox.com/wiki/Manual:_qm.conf: net[n]: [model=] [,bridge=] [,firewall=<1|0>] [,link_down=<1|0>] [,macaddr=] [,queues=] [,rate=] [,tag=] [,trunks=] [,=] regards Klaus From vadim.bulst at bbz.uni-leipzig.de Tue Aug 7 12:13:11 2018 From: vadim.bulst at bbz.uni-leipzig.de (Vadim Bulst) Date: Tue, 7 Aug 2018 12:13:11 +0200 Subject: [PVE-User] Cephfs starting 2nd MDS Message-ID: <1cd7d091-eedb-6bd8-2443-c8b3827b109d@bbz.uni-leipzig.de> Dear list, I'm trying to bring up a second mds with no luck. This is what my ceph.conf looks like: [global] ???????? auth client required = cephx ???????? auth cluster required = cephx ???????? auth service required = cephx ???????? cluster network = 10.10.144.0/24 ???????? filestore xattr use omap = true ???????? fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7 ???????? keyring = /etc/pve/priv/$cluster.$name.keyring ???????? osd journal size = 5120 ???????? osd pool default min size = 1 ???????? public network = 172.18.144.0/24 ???????? mon allow pool delete = true [osd] ???????? keyring = /var/lib/ceph/osd/ceph-$id/keyring [mon.2] ???????? host = scvirt03 ???????? mon addr = 172.18.144.243:6789 [mon.0] ???????? host = scvirt01 ???????? mon addr = 172.18.144.241:6789 [mon.1] ???????? host = scvirt02 ???????? mon addr = 172.18.144.242:6789 [mds.0] ??????? host = scvirt02 [mds.1] ??????? host = scvirt03 I did the following to set up the service: apt install ceph-mds mkdir /var/lib/ceph/mds mkdir /var/lib/ceph/mds/ceph-$(hostname -s) chown -R ceph:ceph /var/lib/ceph/mds chmod -R 0750 /var/lib/ceph/mds ceph auth get-or-create mds.$(hostname -s) mon 'allow profile mds' mgr 'allow profile mds' osd 'allow rwx' mds 'allow' > /var/lib/ceph/mds/ceph-$(hostname -s)/keyring chmod -R 0600 /var/lib/ceph/mds/ceph-$(hostname -s)/keyring systemctl enable ceph-mds@$(hostname -s).service systemctl start ceph-mds@$(hostname -s).service The service will not start. I also did the same procedure with the first mds which is running with no problems. 1st mds: root at scvirt02:/home/urzadmin# systemctl status -l ceph-mds@$(hostname -s).service ? ceph-mds at scvirt02.service - Ceph metadata server daemon ?? Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; vendor preset: enabled) ? Drop-In: /lib/systemd/system/ceph-mds at .service.d ?????????? ??ceph-after-pve-cluster.conf ?? Active: active (running) since Thu 2018-06-07 13:08:58 CEST; 2 months 0 days ago ?Main PID: 612704 (ceph-mds) ?? CGroup: /system.slice/system-ceph\x2dmds.slice/ceph-mds at scvirt02.service ?????????? ??612704 /usr/bin/ceph-mds -f --cluster ceph --id scvirt02 --setuser ceph --setgroup ceph Jul 29 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-29 06:25:01.792601 7f6e4bae0700 -1 received? signal: Hangup from? PID: 3831071 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Jul 30 06:25:02 scvirt02 ceph-mds[612704]: 2018-07-30 06:25:02.081591 7f6e4bae0700 -1 received? signal: Hangup from? PID: 184355 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Jul 31 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-31 06:25:01.448571 7f6e4bae0700 -1 received? signal: Hangup from? PID: 731440 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Aug 01 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-01 06:25:01.274541 7f6e4bae0700 -1 received? signal: Hangup from? PID: 1278492 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Aug 02 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-02 06:25:02.009054 7f6e4bae0700 -1 received? signal: Hangup from? PID: 1825500 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Aug 03 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-03 06:25:02.042845 7f6e4bae0700 -1 received? signal: Hangup from? PID: 2372815 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Aug 04 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-04 06:25:01.404619 7f6e4bae0700 -1 received? signal: Hangup from? PID: 2919837 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Aug 05 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-05 06:25:01.214749 7f6e4bae0700 -1 received? signal: Hangup from? PID: 3467000 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Aug 06 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-06 06:25:01.149512 7f6e4bae0700 -1 received? signal: Hangup from? PID: 4014197 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 Aug 07 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-07 06:25:01.863104 7f6e4bae0700 -1 received? signal: Hangup from? PID: 367698 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw? UID: 0 2nd mds: root at scvirt03:/home/urzadmin# systemctl status -l ceph-mds@$(hostname -s).service ? ceph-mds at scvirt03.service - Ceph metadata server daemon ?? Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; vendor preset: enabled) ? Drop-In: /lib/systemd/system/ceph-mds at .service.d ?????????? ??ceph-after-pve-cluster.conf ?? Active: inactive (dead) since Tue 2018-08-07 10:27:18 CEST; 1h 38min ago ? Process: 3620063 ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id scvirt03 --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS) ?Main PID: 3620063 (code=exited, status=0/SUCCESS) Aug 07 10:27:17 scvirt03 systemd[1]: Started Ceph metadata server daemon. Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: starting mds.scvirt03 at - Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008338 7f6be03816c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.mds.scvirt03.keyring: (13) Permission denied Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008351 7f6be03816c0 -1 mds.scvirt03 ERROR: failed to get monmap: (13) Permission denied content of /etc/pve/priv root at scvirt03:/home/urzadmin# ls -la /etc/pve/priv/ total 5 drwx------ 2 root www-data??? 0 Apr 15? 2017 . drwxr-xr-x 2 root www-data??? 0 Jan? 1? 1970 .. -rw------- 1 root www-data 1675 Apr 15? 2017 authkey.key -rw------- 1 root www-data 1976 Jul? 6 15:41 authorized_keys drwx------ 2 root www-data??? 0 Apr 16? 2017 ceph -rw------- 1 root www-data?? 63 Apr 15? 2017 ceph.client.admin.keyring -rw------- 1 root www-data? 214 Apr 15? 2017 ceph.mon.keyring -rw------- 1 root www-data 4224 Jul? 6 15:41 known_hosts drwx------ 2 root www-data??? 0 Apr 15? 2017 lock -rw------- 1 root www-data 3243 Apr 15? 2017 pve-root-ca.key -rw------- 1 root www-data??? 3 Jul? 6 15:41 pve-root-ca.srl -rw------- 1 root www-data?? 36 May 23 13:03 urzbackup.cred What could be the reason this failure? Cheers, Vadim -- Vadim Bulst Universit?t Leipzig / URZ 04109 Leipzig, Augustusplatz 10 phone: ++49-341-97-33380 mail: vadim.bulst at uni-leipzig.de From denis.morejon at etecsa.cu Tue Aug 7 14:13:50 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Tue, 7 Aug 2018 08:13:50 -0400 Subject: [PVE-User] How to use lvm on zfs ? Message-ID: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> Hi: I installed a Proxmox 5.1 server with 4 sata hdds. I built a Raidz1 (Raid 5 aquivalent) to introduce storage redundance. But no lvm is present. I want to use lvm storage on top of zpool. What should I do ? From mark at tuxis.nl Tue Aug 7 14:19:03 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 07 Aug 2018 14:19:03 +0200 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> Message-ID: <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> On Tue, 2018-08-07 at 08:13 -0400, Denis Morejon wrote: > I installed a Proxmox 5.1 server with 4 sata hdds. I built a Raidz1 > (Raid 5 aquivalent) to introduce storage redundance. But no lvm is > present. I want to use lvm storage on top of zpool. What should I do > ? I'm curious about why you would want to do that.. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From denis.morejon at etecsa.cu Tue Aug 7 15:32:17 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Tue, 7 Aug 2018 09:32:17 -0400 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> Message-ID: El 07/08/18 a las 08:19, Mark Schouten escribi?: > On Tue, 2018-08-07 at 08:13 -0400, Denis Morejon wrote: >> I installed a Proxmox 5.1 server with 4 sata hdds. I built a Raidz1 >> (Raid 5 aquivalent) to introduce storage redundance. But no lvm is >> present. I want to use lvm storage on top of zpool. What should I do >> ? > I'm curious about why you would want to do that.. > I don't understand your question. Do you refer to either using zfs or lvm/zfs ? From denis.morejon at etecsa.cu Tue Aug 7 15:42:14 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Tue, 7 Aug 2018 09:42:14 -0400 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> Message-ID: <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> When I install proxmox 5.1 with Raidz1, It mounts this zfs device "rpool/ROOT/pve-1" on "/". This device is the ENTIRE pool with the 4 hdds in raid 5. So that there is no extra space to conform any volumen group, not even pve!!! So local-lvm is not active by default. Then, when you add this node to others with local-lvm storage active, And you try to migrate VMs between them, there are problems... That's why I pretend to use lvm on all cluster nodes!!! Some ones with just lvm and others with lvm/zfs. El 07/08/18 a las 08:19, Mark Schouten escribi?: > On Tue, 2018-08-07 at 08:13 -0400, Denis Morejon wrote: >> I installed a Proxmox 5.1 server with 4 sata hdds. I built a Raidz1 >> (Raid 5 aquivalent) to introduce storage redundance. But no lvm is >> present. I want to use lvm storage on top of zpool. What should I do >> ? > I'm curious about why you would want to do that.. > From mark at tuxis.nl Tue Aug 7 14:49:46 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 07 Aug 2018 14:49:46 +0200 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> Message-ID: <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> On Tue, 2018-08-07 at 09:42 -0400, Denis Morejon wrote: > So local-lvm is not active by default. Then, when you add this node > to > others with local-lvm storage active, And you try to migrate VMs > between > > them, there are problems... I don't think the actual technique used is relevant for migrating local storage, but just the name of the storage.. You can create images on ZFS, you don't need LVM to be able to create VM's with storage. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From aa at ipnerd.net Tue Aug 7 15:30:07 2018 From: aa at ipnerd.net (Alwin Antreich) Date: Tue, 7 Aug 2018 15:30:07 +0200 Subject: [PVE-User] Cephfs starting 2nd MDS In-Reply-To: <1cd7d091-eedb-6bd8-2443-c8b3827b109d@bbz.uni-leipzig.de> References: <1cd7d091-eedb-6bd8-2443-c8b3827b109d@bbz.uni-leipzig.de> Message-ID: Hello Vadim, On Tue, Aug 7, 2018, 12:13 Vadim Bulst wrote: > Dear list, > > I'm trying to bring up a second mds with no luck. > > This is what my ceph.conf looks like: > > [global] > > auth client required = cephx > auth cluster required = cephx > auth service required = cephx > cluster network = 10.10.144.0/24 > filestore xattr use omap = true > fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7 > keyring = /etc/pve/priv/$cluster.$name.keyring > osd journal size = 5120 > osd pool default min size = 1 > public network = 172.18.144.0/24 > mon allow pool delete = true > > [osd] > keyring = /var/lib/ceph/osd/ceph-$id/keyring > > [mon.2] > host = scvirt03 > mon addr = 172.18.144.243:6789 > > [mon.0] > host = scvirt01 > mon addr = 172.18.144.241:6789 > [mon.1] > host = scvirt02 > mon addr = 172.18.144.242:6789 > > [mds.0] > host = scvirt02 > [mds.1] > host = scvirt03 > > > I did the following to set up the service: > > apt install ceph-mds > > mkdir /var/lib/ceph/mds > > mkdir /var/lib/ceph/mds/ceph-$(hostname -s) > > chown -R ceph:ceph /var/lib/ceph/mds > > chmod -R 0750 /var/lib/ceph/mds > > ceph auth get-or-create mds.$(hostname -s) mon 'allow profile mds' mgr > 'allow profile mds' osd 'allow rwx' mds 'allow' > > /var/lib/ceph/mds/ceph-$(hostname -s)/keyring > > chmod -R 0600 /var/lib/ceph/mds/ceph-$(hostname -s)/keyring > > systemctl enable ceph-mds@$(hostname -s).service > > systemctl start ceph-mds@$(hostname -s).service > > > The service will not start. I also did the same procedure with the first > mds which is running with no problems. > > 1st mds: > > root at scvirt02:/home/urzadmin# systemctl status -l ceph-mds@$(hostname > -s).service > ? ceph-mds at scvirt02.service - Ceph metadata server daemon > Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; > vendor preset: enabled) > Drop-In: /lib/systemd/system/ceph-mds at .service.d > ??ceph-after-pve-cluster.conf > Active: active (running) since Thu 2018-06-07 13:08:58 CEST; 2 > months 0 days ago > Main PID: 612704 (ceph-mds) > CGroup: > /system.slice/system-ceph\x2dmds.slice/ceph-mds at scvirt02.service > ??612704 /usr/bin/ceph-mds -f --cluster ceph --id scvirt02 > --setuser ceph --setgroup ceph > > Jul 29 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-29 06:25:01.792601 > 7f6e4bae0700 -1 received signal: Hangup from PID: 3831071 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Jul 30 06:25:02 scvirt02 ceph-mds[612704]: 2018-07-30 06:25:02.081591 > 7f6e4bae0700 -1 received signal: Hangup from PID: 184355 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Jul 31 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-31 06:25:01.448571 > 7f6e4bae0700 -1 received signal: Hangup from PID: 731440 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Aug 01 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-01 06:25:01.274541 > 7f6e4bae0700 -1 received signal: Hangup from PID: 1278492 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Aug 02 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-02 06:25:02.009054 > 7f6e4bae0700 -1 received signal: Hangup from PID: 1825500 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Aug 03 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-03 06:25:02.042845 > 7f6e4bae0700 -1 received signal: Hangup from PID: 2372815 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Aug 04 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-04 06:25:01.404619 > 7f6e4bae0700 -1 received signal: Hangup from PID: 2919837 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Aug 05 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-05 06:25:01.214749 > 7f6e4bae0700 -1 received signal: Hangup from PID: 3467000 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Aug 06 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-06 06:25:01.149512 > 7f6e4bae0700 -1 received signal: Hangup from PID: 4014197 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > Aug 07 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-07 06:25:01.863104 > 7f6e4bae0700 -1 received signal: Hangup from PID: 367698 task name: > killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 > > 2nd mds: > > root at scvirt03:/home/urzadmin# systemctl status -l ceph-mds@$(hostname > -s).service > ? ceph-mds at scvirt03.service - Ceph metadata server daemon > Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; > vendor preset: enabled) > Drop-In: /lib/systemd/system/ceph-mds at .service.d > ??ceph-after-pve-cluster.conf > Active: inactive (dead) since Tue 2018-08-07 10:27:18 CEST; 1h > 38min ago > Process: 3620063 ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} > --id scvirt03 --setuser ceph --setgroup ceph (code=exited, > status=0/SUCCESS) > Main PID: 3620063 (code=exited, status=0/SUCCESS) > > Aug 07 10:27:17 scvirt03 systemd[1]: Started Ceph metadata server daemon. > Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: starting mds.scvirt03 at - > Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008338 > 7f6be03816c0 -1 auth: unable to find a keyring on > /etc/pve/priv/ceph.mds.scvirt03.keyring: (13) Permission denied > Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008351 > 7f6be03816c0 -1 mds.scvirt03 ERROR: failed to get monmap: (13) > Permission denied > > > content of /etc/pve/priv > > root at scvirt03:/home/urzadmin# ls -la /etc/pve/priv/ > total 5 > drwx------ 2 root www-data 0 Apr 15 2017 . > drwxr-xr-x 2 root www-data 0 Jan 1 1970 .. > -rw------- 1 root www-data 1675 Apr 15 2017 authkey.key > -rw------- 1 root www-data 1976 Jul 6 15:41 authorized_keys > drwx------ 2 root www-data 0 Apr 16 2017 ceph > -rw------- 1 root www-data 63 Apr 15 2017 ceph.client.admin.keyring > -rw------- 1 root www-data 214 Apr 15 2017 ceph.mon.keyring > -rw------- 1 root www-data 4224 Jul 6 15:41 known_hosts > drwx------ 2 root www-data 0 Apr 15 2017 lock > -rw------- 1 root www-data 3243 Apr 15 2017 pve-root-ca.key > -rw------- 1 root www-data 3 Jul 6 15:41 pve-root-ca.srl > -rw------- 1 root www-data 36 May 23 13:03 urzbackup.cred > > > What could be the reason this failure? > The ceph user has no permissions to access the the keyring under /etc/pve. Add a section for [mds] into the ceph.conf pointing to the keyring, similar to the OSD one. This way the MDS will find the key in It's working directory. Cheers, Alwin > From klaus.mailinglists at pernau.at Tue Aug 7 15:44:14 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Tue, 7 Aug 2018 15:44:14 +0200 Subject: [PVE-User] How to specify VLANs for a trunk interface into the VM In-Reply-To: <514da49d-da0b-5a11-ad38-0d4afbc9219a@pernau.at> References: <75b30778-f206-a1a4-412f-d706a52283cd@pernau.at> <92c3153d-e992-f7ca-d00b-1c8a4bf56552@pernau.at> <514da49d-da0b-5a11-ad38-0d4afbc9219a@pernau.at> Message-ID: Am 06.08.2018 um 21:25 schrieb Klaus Darilion: > Am 06.08.2018 um 21:19 schrieb Klaus Darilion: >> I hoped that the port config is flexible to configure trunks=...., but >> it is not available in the GUI. > > I think I just found it at https://pve.proxmox.com/wiki/Manual:_qm.conf: > > net[n]: [model=] [,bridge=] [,firewall=<1|0>] > [,link_down=<1|0>] [,macaddr=] [,queues=] > [,rate=] [,tag=] [,trunks=] > [,=] For the records - it works fine - - but probably will be overwritten if the NIC would be editet in the GUI. regards Klaus From denis.morejon at etecsa.cu Tue Aug 7 16:49:43 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Tue, 7 Aug 2018 10:49:43 -0400 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> Message-ID: El 07/08/18 a las 08:49, Mark Schouten escribi?: > On Tue, 2018-08-07 at 09:42 -0400, Denis Morejon wrote: >> So local-lvm is not active by default. Then, when you add this node >> to >> others with local-lvm storage active, And you try to migrate VMs >> between >> >> them, there are problems... > I don't think the actual technique used is relevant for migrating local > storage, but just the name of the storage.. You can create images on > ZFS, you don't need LVM to be able to create VM's with storage. > I don't have a cluster wide ZFS storage. If I did, I would define a ZFS storage better, So that the cluster uses it, and that's all. I have a mixture of 4 professional servers with hardware Raid controllers, and 4 not professional servers (PCs) with 4 hdds each one. I want to create just one proxmox cluster with all these 8 servers. In the 4 PCs, I had to install proxmox using the zfs Raidz1 advanced option, So that Proxmox to be installed? on a zfs Raid5. Up to this stage,? I have redundancy, in such a way that I can remove any sata hdd and the proxmox can startup ok. And that's what I need ZFS. So, on 4 nodes of the cluster I am able to use local-lvm to put CTs and VMs over there, but I am not able to put VMs and CTs on local-lvm in the others. That's why I want to create pve VGs on the zfs nodes. From klaus.mailinglists at pernau.at Tue Aug 7 15:54:02 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Tue, 7 Aug 2018 15:54:02 +0200 Subject: [PVE-User] Cluster doesn't recover automatically after blackout In-Reply-To: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> References: <34ef0138-352b-e9c4-8bc8-2e10c4f17912@binovo.es> Message-ID: <675b77a4-4c60-d225-3a08-cfab3c2e5599@pernau.at> Just some rant: I do think that the presented solutions are the wrong approach to this problem. A HA cluster should recover automatically from such simple failures (power loss, network outage) to achieve HA. If there is manual intervention necessary - then the whole thing should not be called "HA" cluster. I know corosync is picky and for example does not start when a configured network interface is not available yet. Hence, corosync should be automatically restarted if it fails. Intrudocing a "sleep" until the network is available is also a dirty workaround - the problem is the cluster software - the software should try to form a cluster endlessly (why should a "HA" software give up?). Would a mars rover give up and shutdown when it could not ping the earth for some days? Probably no, it would try endlessly. regards Klaus Am 01.08.2018 um 11:02 schrieb Eneko Lacunza: > Hi all, > > This morning there was a quite long blackout which powered off a cluster > of 3 proxmox 5.1 servers. > > All 3 servers the same make and model, so they need the same amount of > time to boot. > > When the power came back, servers started correctly but corosync > couldn't set up a quorum. Events timing: > > 07:57:10 corosync start > 07:57:15 first pmxcfs error quorum_initialize_failed: 2 > 07:57:52 network up > 07:58:40 Corosync timeout > 07:59:57 time sync works > > What I can see is that network switch boot was slower than server's, but > nonetheless network was operational about 45s before corosync gives up > trying to set up a quorum. > > I also can see that internet access wasn't back until 1 minute after > corosync timeout (the time sync event). > > A simple restart of pve-cluster at about 9:50 restored the cluster to > normal state. > > Is this expected? I expected that corosync would set up a quorum after > network was operational.... > > # pveversion -v > proxmox-ve: 5.1-41 (running kernel: 4.13.13-6-pve) > pve-manager: 5.1-46 (running version: 5.1-46/ae8241d4) > pve-kernel-4.13.13-6-pve: 4.13.13-41 > pve-kernel-4.13.13-2-pve: 4.13.13-33 > ceph: 12.2.2-pve1 > corosync: 2.4.2-pve3 > criu: 2.11.1-1~bpo90 > glusterfs-client: 3.8.8-1 > ksm-control-daemon: 1.2-2 > libjs-extjs: 6.0.1-2 > libpve-access-control: 5.0-8 > libpve-common-perl: 5.0-28 > libpve-guest-common-perl: 2.0-14 > libpve-http-server-perl: 2.0-8 > libpve-storage-perl: 5.0-17 > libqb0: 1.0.1-1 > lvm2: 2.02.168-pve6 > lxc-pve: 2.1.1-2 > lxcfs: 2.0.8-2 > novnc-pve: 0.6-4 > proxmox-widget-toolkit: 1.0-11 > pve-cluster: 5.0-20 > pve-container: 2.0-19 > pve-docs: 5.1-16 > pve-firewall: 3.0-5 > pve-firmware: 2.0-3 > pve-ha-manager: 2.0-5 > pve-i18n: 1.0-4 > pve-libspice-server1: 0.12.8-3 > pve-qemu-kvm: 2.9.1-9 > pve-xtermjs: 1.0-2 > qemu-server: 5.0-22 > smartmontools: 6.5+svn4324-1 > spiceterm: 3.0-5 > vncterm: 1.5-3 > zfsutils-linux: 0.7.6-pve1~bpo9 > > Thanks a lot > Eneko > From mark at tuxis.nl Tue Aug 7 15:57:19 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 07 Aug 2018 15:57:19 +0200 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> Message-ID: <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> On Tue, 2018-08-07 at 10:49 -0400, Denis Morejon wrote: > So, on 4 nodes of the cluster I am able to use local-lvm to put CTs > and > VMs over there, but I am not able to put VMs and CTs on local-lvm in > the > others. > That's why I want to create pve VGs on the zfs nodes. Like I said, I think you should be able to rename 'local-zfs' in 'local-lvm' in /etc/pve/storage.cfg, and not worry about LVM anymore. But maybe we're misunderstanding each other. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From aheinlein at gmx.com Tue Aug 7 16:19:50 2018 From: aheinlein at gmx.com (Andreas Heinlein) Date: Tue, 7 Aug 2018 16:19:50 +0200 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> Message-ID: <93bd59e0-cbb7-5dd6-c512-d124bdf612d4@gmx.com> Am 07.08.2018 um 16:49 schrieb Denis Morejon: > I don't have a cluster wide ZFS storage. If I did, I would define a > ZFS storage better, So that the cluster uses it, and that's all. > I have a mixture of 4 professional servers with hardware Raid > controllers, and 4 not professional servers (PCs) with 4 hdds each one. > I want to create just one proxmox cluster with all these 8 servers. > In the 4 PCs, I had to install proxmox using the zfs Raidz1 advanced > option, So that Proxmox to be installed? on a zfs Raid5. > Up to this stage,? I have redundancy, in such a way that I can remove > any sata hdd and the proxmox can startup ok. And that's what I need ZFS. > > So, on 4 nodes of the cluster I am able to use local-lvm to put CTs > and VMs over there, but I am not able to put VMs and CTs on local-lvm > in the others. > That's why I want to create pve VGs on the zfs nodes. Hello, if I understand you correctly, you essentially don't need/want to use ZFS as a storage/file system, but used it anyway as a replacement for the missing hardware RAID controller in 4 of your machines. This way you aim to achieve the same configuration on all 8 machines despite different hardware. Correct? I am no expert here, but I guess this approach is not good. You may think about using classic linux software RAID (aka mdadm RAID) here. This is, according to this page: https://pve.proxmox.com/wiki/Software_RAID not officially supported. But I know it works, I did that some time ago on a test install. IIRC I even did that using the Proxmox installer, hidden somewhere under "Advanced options". If not, it is still possible to install a plain debian system on RAID+LVM and "convert" that to a proxmox node later: https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch Andreas From denis.morejon at etecsa.cu Tue Aug 7 17:30:00 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Tue, 7 Aug 2018 11:30:00 -0400 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> Message-ID: <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> El 07/08/18 a las 09:57, Mark Schouten escribi?: > On Tue, 2018-08-07 at 10:49 -0400, Denis Morejon wrote: >> So, on 4 nodes of the cluster I am able to use local-lvm to put CTs >> and >> VMs over there, but I am not able to put VMs and CTs on local-lvm in >> the >> others. >> That's why I want to create pve VGs on the zfs nodes. > Like I said, I think you should be able to rename 'local-zfs' in > 'local-lvm' in /etc/pve/storage.cfg, and not worry about LVM anymore. > But maybe we're misunderstanding each other. > These is not possible because I don't use zfs on the eight nodes ? Just in 4 of them and modifying /etc/pve/storage.cfg is a cluster wide operation! More over. I use zfs under the Proxmox (In the S.O level) to join the internal disks. So that proxmox node NEVER sees any zfs, just the local storage (Dir /var/lib/vz) that is mounted on the internal zpool (rpool/ROOT/pve-1) that is the zfs pool (The 4 disks together seen as one). Now the problem is that I have 4 nodes with local-lvm storage active and 4 nodes with just local storage active, because in these? last nodes the local-lvm is disabled! (Due to the non-existence of any lmv volumen group) So, the migrations of MVs between all these nodes cause problems. From mark at tuxis.nl Tue Aug 7 16:45:45 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 07 Aug 2018 16:45:45 +0200 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> Message-ID: On Tue, 2018-08-07 at 11:30 -0400, Denis Morejon wrote: > These is not possible because I don't use zfs on the eight nodes ? > Just > in 4 of them and modifying /etc/pve/storage.cfg is a cluster wide > operation! Ah yes, crap. That's right.. > Now the problem is that I have 4 nodes with local-lvm storage active > and > 4 nodes with just local storage active, because in these last nodes > the > local-lvm is disabled! (Due to the non-existence of any lmv volumen > group) > > So, the migrations of MVs between all these nodes cause problems. Ok. Not sure if this is supposed to work, but what if you create a ZFS Volume (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate /dev/zvol/rpool/lvm) and make a VG (vgcreate pve /dev/zvol/rpool/lvm) and then a LV (lvcreate -L100% pve/data) ? Does that allow you to use local-lvm? (Not 100% sure about the commands, check before executing) -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From denis.morejon at etecsa.cu Tue Aug 7 19:36:09 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Tue, 7 Aug 2018 13:36:09 -0400 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> Message-ID: El 07/08/18 a las 10:45, Mark Schouten escribi?: > On Tue, 2018-08-07 at 11:30 -0400, Denis Morejon wrote: >> These is not possible because I don't use zfs on the eight nodes ? >> Just >> in 4 of them and modifying /etc/pve/storage.cfg is a cluster wide >> operation! > Ah yes, crap. That's right.. > >> Now the problem is that I have 4 nodes with local-lvm storage active >> and >> 4 nodes with just local storage active, because in these last nodes >> the >> local-lvm is disabled! (Due to the non-existence of any lmv volumen >> group) >> >> So, the migrations of MVs between all these nodes cause problems. > Ok. Not sure if this is supposed to work, but what if you create a ZFS > Volume (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate > /dev/zvol/rpool/lvm) and make a VG (vgcreate pve /dev/zvol/rpool/lvm) > and then a LV (lvcreate -L100% pve/data) ? Does that allow you to use > local-lvm? > > (Not 100% sure about the commands, check before executing) > This is the idea Mark! But I suspect I have no space to create an additional zfs volume since the one mounted on "/" occupied all the space. So I have to know how reduce It first, and then create the new one on the remaining space. From yannis.milios at gmail.com Tue Aug 7 23:51:12 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Tue, 7 Aug 2018 22:51:12 +0100 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> Message-ID: > > (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate >> > /dev/zvol/rpool/lvm) and make a VG (vgcreate pve /dev/zvol/rpool/lvm) >> > and then a LV (lvcreate -L100% pve/data) >> >> Try the above as it was suggested to you ... > >But I suspect I have no space to create an >> >additional zfs volume since the one mounted on "/" occupied all the space > > No, that's a wrong assumption, zfs does not pre-allocate the whole space of the pool, even if looks like it does so. In short there is no need to "shrink" the pool in order to create a zvol as it was suggested above... Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but if you insist, it's up to you ... A combination of Linux RAID + LVM would look much more elegant in your case, but for that you have to reinstall PVE by using the Debian iso. During the installation create a linux raid array with lvm on top and then add PVE repos ass described in the wiki: https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie From ken.woods at alaska.gov Wed Aug 8 00:49:30 2018 From: ken.woods at alaska.gov (Woods, Ken A (DNR)) Date: Tue, 7 Aug 2018 22:49:30 +0000 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> Message-ID: Because this really is a really bad idea, I just want to echo Yannis and say: ...the whole idea of having LVM on top of ZFS/zvol is a mess.... kw -----Original Message----- From: pve-user [mailto:pve-user-bounces at pve.proxmox.com] On Behalf Of Yannis Milios Sent: Tuesday, August 07, 2018 1:51 PM To: PVE User List Subject: Re: [PVE-User] How to use lvm on zfs ? > > (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate >> > /dev/zvol/rpool/lvm) and make a VG (vgcreate pve >> > /dev/zvol/rpool/lvm) and then a LV (lvcreate -L100% pve/data) >> >> Try the above as it was suggested to you ... > >But I suspect I have no space to create an >> >additional zfs volume since the one mounted on "/" occupied all the >> >space > > No, that's a wrong assumption, zfs does not pre-allocate the whole space of the pool, even if looks like it does so. In short there is no need to "shrink" the pool in order to create a zvol as it was suggested above... Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but if you insist, it's up to you ... A combination of Linux RAID + LVM would look much more elegant in your case, but for that you have to reinstall PVE by using the Debian iso. During the installation create a linux raid array with lvm on top and then add PVE repos ass described in the wiki: https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From vadim.bulst at bbz.uni-leipzig.de Wed Aug 8 07:54:45 2018 From: vadim.bulst at bbz.uni-leipzig.de (Vadim Bulst) Date: Wed, 8 Aug 2018 07:54:45 +0200 Subject: [PVE-User] Cephfs starting 2nd MDS In-Reply-To: References: <1cd7d091-eedb-6bd8-2443-c8b3827b109d@bbz.uni-leipzig.de> Message-ID: Hi Alwin, thanks for your advise. But no success. Still same error. mds-section: [mds.1] ??????? host = scvirt03 ??????? keyring = /var/lib/ceph/mds/ceph-scvirt03/keyring Vadim On 07.08.2018 15:30, Alwin Antreich wrote: > Hello Vadim, > > On Tue, Aug 7, 2018, 12:13 Vadim Bulst > wrote: > >> Dear list, >> >> I'm trying to bring up a second mds with no luck. >> >> This is what my ceph.conf looks like: >> >> [global] >> >> auth client required = cephx >> auth cluster required = cephx >> auth service required = cephx >> cluster network = 10.10.144.0/24 >> filestore xattr use omap = true >> fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7 >> keyring = /etc/pve/priv/$cluster.$name.keyring >> osd journal size = 5120 >> osd pool default min size = 1 >> public network = 172.18.144.0/24 >> mon allow pool delete = true >> >> [osd] >> keyring = /var/lib/ceph/osd/ceph-$id/keyring >> >> [mon.2] >> host = scvirt03 >> mon addr = 172.18.144.243:6789 >> >> [mon.0] >> host = scvirt01 >> mon addr = 172.18.144.241:6789 >> [mon.1] >> host = scvirt02 >> mon addr = 172.18.144.242:6789 >> >> [mds.0] >> host = scvirt02 >> [mds.1] >> host = scvirt03 >> >> >> I did the following to set up the service: >> >> apt install ceph-mds >> >> mkdir /var/lib/ceph/mds >> >> mkdir /var/lib/ceph/mds/ceph-$(hostname -s) >> >> chown -R ceph:ceph /var/lib/ceph/mds >> >> chmod -R 0750 /var/lib/ceph/mds >> >> ceph auth get-or-create mds.$(hostname -s) mon 'allow profile mds' mgr >> 'allow profile mds' osd 'allow rwx' mds 'allow' > >> /var/lib/ceph/mds/ceph-$(hostname -s)/keyring >> >> chmod -R 0600 /var/lib/ceph/mds/ceph-$(hostname -s)/keyring >> >> systemctl enable ceph-mds@$(hostname -s).service >> >> systemctl start ceph-mds@$(hostname -s).service >> >> >> The service will not start. I also did the same procedure with the first >> mds which is running with no problems. >> >> 1st mds: >> >> root at scvirt02:/home/urzadmin# systemctl status -l ceph-mds@$(hostname >> -s).service >> ? ceph-mds at scvirt02.service - Ceph metadata server daemon >> Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; >> vendor preset: enabled) >> Drop-In: /lib/systemd/system/ceph-mds at .service.d >> ??ceph-after-pve-cluster.conf >> Active: active (running) since Thu 2018-06-07 13:08:58 CEST; 2 >> months 0 days ago >> Main PID: 612704 (ceph-mds) >> CGroup: >> /system.slice/system-ceph\x2dmds.slice/ceph-mds at scvirt02.service >> ??612704 /usr/bin/ceph-mds -f --cluster ceph --id scvirt02 >> --setuser ceph --setgroup ceph >> >> Jul 29 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-29 06:25:01.792601 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 3831071 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Jul 30 06:25:02 scvirt02 ceph-mds[612704]: 2018-07-30 06:25:02.081591 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 184355 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Jul 31 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-31 06:25:01.448571 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 731440 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Aug 01 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-01 06:25:01.274541 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 1278492 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Aug 02 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-02 06:25:02.009054 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 1825500 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Aug 03 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-03 06:25:02.042845 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 2372815 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Aug 04 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-04 06:25:01.404619 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 2919837 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Aug 05 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-05 06:25:01.214749 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 3467000 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Aug 06 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-06 06:25:01.149512 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 4014197 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> Aug 07 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-07 06:25:01.863104 >> 7f6e4bae0700 -1 received signal: Hangup from PID: 367698 task name: >> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >> >> 2nd mds: >> >> root at scvirt03:/home/urzadmin# systemctl status -l ceph-mds@$(hostname >> -s).service >> ? ceph-mds at scvirt03.service - Ceph metadata server daemon >> Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; >> vendor preset: enabled) >> Drop-In: /lib/systemd/system/ceph-mds at .service.d >> ??ceph-after-pve-cluster.conf >> Active: inactive (dead) since Tue 2018-08-07 10:27:18 CEST; 1h >> 38min ago >> Process: 3620063 ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} >> --id scvirt03 --setuser ceph --setgroup ceph (code=exited, >> status=0/SUCCESS) >> Main PID: 3620063 (code=exited, status=0/SUCCESS) >> >> Aug 07 10:27:17 scvirt03 systemd[1]: Started Ceph metadata server daemon. >> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: starting mds.scvirt03 at - >> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008338 >> 7f6be03816c0 -1 auth: unable to find a keyring on >> /etc/pve/priv/ceph.mds.scvirt03.keyring: (13) Permission denied >> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008351 >> 7f6be03816c0 -1 mds.scvirt03 ERROR: failed to get monmap: (13) >> Permission denied >> >> >> content of /etc/pve/priv >> >> root at scvirt03:/home/urzadmin# ls -la /etc/pve/priv/ >> total 5 >> drwx------ 2 root www-data 0 Apr 15 2017 . >> drwxr-xr-x 2 root www-data 0 Jan 1 1970 .. >> -rw------- 1 root www-data 1675 Apr 15 2017 authkey.key >> -rw------- 1 root www-data 1976 Jul 6 15:41 authorized_keys >> drwx------ 2 root www-data 0 Apr 16 2017 ceph >> -rw------- 1 root www-data 63 Apr 15 2017 ceph.client.admin.keyring >> -rw------- 1 root www-data 214 Apr 15 2017 ceph.mon.keyring >> -rw------- 1 root www-data 4224 Jul 6 15:41 known_hosts >> drwx------ 2 root www-data 0 Apr 15 2017 lock >> -rw------- 1 root www-data 3243 Apr 15 2017 pve-root-ca.key >> -rw------- 1 root www-data 3 Jul 6 15:41 pve-root-ca.srl >> -rw------- 1 root www-data 36 May 23 13:03 urzbackup.cred >> >> >> What could be the reason this failure? >> > The ceph user has no permissions to access the the keyring under /etc/pve. > Add a section for [mds] into the ceph.conf pointing to the keyring, similar > to the OSD one. This way the MDS will find the key in It's working > directory. > > Cheers, > Alwin > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From ronny+pve-user at aasen.cx Wed Aug 8 09:20:02 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Wed, 8 Aug 2018 09:20:02 +0200 Subject: [PVE-User] Cephfs starting 2nd MDS In-Reply-To: References: <1cd7d091-eedb-6bd8-2443-c8b3827b109d@bbz.uni-leipzig.de> Message-ID: <4396515a-1242-3bc6-1bad-e31ab7b2a46e@aasen.cx> your .conf references mds.1 (id =1) but your command starts the mds with id=scvirt03 so the block in ceph.conf is not used. replace [mds.1] with [mds.scvirt03] btw: iirc you can not have just numerical id's for mds's for some versions now, so mds.1 would not be valid either. kind regards Ronny Aasen On 08. aug. 2018 07:54, Vadim Bulst wrote: > Hi Alwin, > > thanks for your advise. But no success. Still same error. > > mds-section: > > [mds.1] > ??????? host = scvirt03 > ??????? keyring = /var/lib/ceph/mds/ceph-scvirt03/keyring > > Vadim > > > On 07.08.2018 15:30, Alwin Antreich wrote: >> Hello Vadim, >> >> On Tue, Aug 7, 2018, 12:13 Vadim Bulst >> wrote: >> >>> Dear list, >>> >>> I'm trying to bring up a second mds with no luck. >>> >>> This is what my ceph.conf looks like: >>> >>> [global] >>> >>> ??????????? auth client required = cephx >>> ??????????? auth cluster required = cephx >>> ??????????? auth service required = cephx >>> ??????????? cluster network = 10.10.144.0/24 >>> ??????????? filestore xattr use omap = true >>> ??????????? fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7 >>> ??????????? keyring = /etc/pve/priv/$cluster.$name.keyring >>> ??????????? osd journal size = 5120 >>> ??????????? osd pool default min size = 1 >>> ??????????? public network = 172.18.144.0/24 >>> ??????????? mon allow pool delete = true >>> >>> [osd] >>> ??????????? keyring = /var/lib/ceph/osd/ceph-$id/keyring >>> >>> [mon.2] >>> ??????????? host = scvirt03 >>> ??????????? mon addr = 172.18.144.243:6789 >>> >>> [mon.0] >>> ??????????? host = scvirt01 >>> ??????????? mon addr = 172.18.144.241:6789 >>> [mon.1] >>> ??????????? host = scvirt02 >>> ??????????? mon addr = 172.18.144.242:6789 >>> >>> [mds.0] >>> ?????????? host = scvirt02 >>> [mds.1] >>> ?????????? host = scvirt03 >>> >>> >>> I did the following to set up the service: >>> >>> apt install ceph-mds >>> >>> mkdir /var/lib/ceph/mds >>> >>> mkdir /var/lib/ceph/mds/ceph-$(hostname -s) >>> >>> chown -R ceph:ceph /var/lib/ceph/mds >>> >>> chmod -R 0750 /var/lib/ceph/mds >>> >>> ceph auth get-or-create mds.$(hostname -s) mon 'allow profile mds' mgr >>> 'allow profile mds' osd 'allow rwx' mds 'allow' > >>> /var/lib/ceph/mds/ceph-$(hostname -s)/keyring >>> >>> chmod -R 0600 /var/lib/ceph/mds/ceph-$(hostname -s)/keyring >>> >>> systemctl enable ceph-mds@$(hostname -s).service >>> >>> systemctl start ceph-mds@$(hostname -s).service >>> >>> >>> The service will not start. I also did the same procedure with the first >>> mds which is running with no problems. >>> >>> 1st mds: >>> >>> root at scvirt02:/home/urzadmin# systemctl status -l ceph-mds@$(hostname >>> -s).service >>> ? ceph-mds at scvirt02.service - Ceph metadata server daemon >>> ????? Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; >>> vendor preset: enabled) >>> ???? Drop-In: /lib/systemd/system/ceph-mds at .service.d >>> ????????????? ??ceph-after-pve-cluster.conf >>> ????? Active: active (running) since Thu 2018-06-07 13:08:58 CEST; 2 >>> months 0 days ago >>> ??? Main PID: 612704 (ceph-mds) >>> ????? CGroup: >>> /system.slice/system-ceph\x2dmds.slice/ceph-mds at scvirt02.service >>> ????????????? ??612704 /usr/bin/ceph-mds -f --cluster ceph --id scvirt02 >>> --setuser ceph --setgroup ceph >>> >>> Jul 29 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-29 06:25:01.792601 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 3831071 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Jul 30 06:25:02 scvirt02 ceph-mds[612704]: 2018-07-30 06:25:02.081591 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 184355 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Jul 31 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-31 06:25:01.448571 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 731440 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Aug 01 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-01 06:25:01.274541 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 1278492 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Aug 02 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-02 06:25:02.009054 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 1825500 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Aug 03 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-03 06:25:02.042845 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 2372815 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Aug 04 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-04 06:25:01.404619 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 2919837 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Aug 05 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-05 06:25:01.214749 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 3467000 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Aug 06 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-06 06:25:01.149512 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 4014197 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> Aug 07 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-07 06:25:01.863104 >>> 7f6e4bae0700 -1 received? signal: Hangup from? PID: 367698 task name: >>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw >>> UID: 0 >>> >>> 2nd mds: >>> >>> root at scvirt03:/home/urzadmin# systemctl status -l ceph-mds@$(hostname >>> -s).service >>> ? ceph-mds at scvirt03.service - Ceph metadata server daemon >>> ????? Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; >>> vendor preset: enabled) >>> ???? Drop-In: /lib/systemd/system/ceph-mds at .service.d >>> ????????????? ??ceph-after-pve-cluster.conf >>> ????? Active: inactive (dead) since Tue 2018-08-07 10:27:18 CEST; 1h >>> 38min ago >>> ???? Process: 3620063 ExecStart=/usr/bin/ceph-mds -f --cluster >>> ${CLUSTER} >>> --id scvirt03 --setuser ceph --setgroup ceph (code=exited, >>> status=0/SUCCESS) >>> ??? Main PID: 3620063 (code=exited, status=0/SUCCESS) >>> >>> Aug 07 10:27:17 scvirt03 systemd[1]: Started Ceph metadata server >>> daemon. >>> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: starting mds.scvirt03 at - >>> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008338 >>> 7f6be03816c0 -1 auth: unable to find a keyring on >>> /etc/pve/priv/ceph.mds.scvirt03.keyring: (13) Permission denied >>> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008351 >>> 7f6be03816c0 -1 mds.scvirt03 ERROR: failed to get monmap: (13) >>> Permission denied >>> >>> >>> content of /etc/pve/priv >>> >>> root at scvirt03:/home/urzadmin# ls -la /etc/pve/priv/ >>> total 5 >>> drwx------ 2 root www-data??? 0 Apr 15? 2017 . >>> drwxr-xr-x 2 root www-data??? 0 Jan? 1? 1970 .. >>> -rw------- 1 root www-data 1675 Apr 15? 2017 authkey.key >>> -rw------- 1 root www-data 1976 Jul? 6 15:41 authorized_keys >>> drwx------ 2 root www-data??? 0 Apr 16? 2017 ceph >>> -rw------- 1 root www-data?? 63 Apr 15? 2017 ceph.client.admin.keyring >>> -rw------- 1 root www-data? 214 Apr 15? 2017 ceph.mon.keyring >>> -rw------- 1 root www-data 4224 Jul? 6 15:41 known_hosts >>> drwx------ 2 root www-data??? 0 Apr 15? 2017 lock >>> -rw------- 1 root www-data 3243 Apr 15? 2017 pve-root-ca.key >>> -rw------- 1 root www-data??? 3 Jul? 6 15:41 pve-root-ca.srl >>> -rw------- 1 root www-data?? 36 May 23 13:03 urzbackup.cred >>> >>> >>> What could be the reason this failure? >>> >> The ceph user has no permissions to access the the keyring under >> /etc/pve. >> Add a section for [mds] into the ceph.conf pointing to the keyring, >> similar >> to the OSD one. This way the MDS will find the key in It's working >> directory. >> >> Cheers, >> Alwin >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From a.antreich at proxmox.com Wed Aug 8 09:22:38 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Wed, 8 Aug 2018 09:22:38 +0200 Subject: [PVE-User] Cephfs starting 2nd MDS In-Reply-To: References: <1cd7d091-eedb-6bd8-2443-c8b3827b109d@bbz.uni-leipzig.de> Message-ID: <20180808072238.52utnqtq6hfoy6za@dona.proxmox.com> Hi, On Wed, Aug 08, 2018 at 07:54:45AM +0200, Vadim Bulst wrote: > Hi Alwin, > > thanks for your advise. But no success. Still same error. > > mds-section: > > [mds.1] > ??????? host = scvirt03 > ??????? keyring = /var/lib/ceph/mds/ceph-scvirt03/keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring So it will work for every MDS that you setup. Besides that, no extra options would be needed for MDS to start. More then the bellow lines should not be needed to get the mds started. mkdir -p /var/lib/ceph/mds/ceph-$SERVER chown -R ceph:ceph /var/lib/ceph/mds/ceph-$SERVER ceph --cluster ceph --name client.bootstrap-mds \ --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth \ get-or-create mds.$SERVER osd 'allow rwx' mds 'allow' mon 'allow profile mds' \ -o /var/lib/ceph/mds/ceph-$SERVER/keyring If it's not working, whats the output of 'systemctl status ceph-mds@'? -- Cheers, Alwin From vadim.bulst at bbz.uni-leipzig.de Wed Aug 8 11:29:45 2018 From: vadim.bulst at bbz.uni-leipzig.de (Vadim Bulst) Date: Wed, 8 Aug 2018 11:29:45 +0200 Subject: [PVE-User] Cephfs starting 2nd MDS In-Reply-To: <20180808072238.52utnqtq6hfoy6za@dona.proxmox.com> References: <1cd7d091-eedb-6bd8-2443-c8b3827b109d@bbz.uni-leipzig.de> <20180808072238.52utnqtq6hfoy6za@dona.proxmox.com> Message-ID: <1db6fe69-b986-1669-25ab-22b6cca4a7f6@bbz.uni-leipzig.de> Thanks guys - great help! All up and running :-) On 08.08.2018 09:22, Alwin Antreich wrote: > Hi, > > On Wed, Aug 08, 2018 at 07:54:45AM +0200, Vadim Bulst wrote: >> Hi Alwin, >> >> thanks for your advise. But no success. Still same error. >> >> mds-section: >> >> [mds.1] >> ??????? host = scvirt03 >> ??????? keyring = /var/lib/ceph/mds/ceph-scvirt03/keyring > [mds] > keyring = /var/lib/ceph/mds/ceph-$id/keyring > > So it will work for every MDS that you setup. Besides that, no extra > options would be needed for MDS to start. > > More then the bellow lines should not be needed to get the mds started. > > mkdir -p /var/lib/ceph/mds/ceph-$SERVER > chown -R ceph:ceph /var/lib/ceph/mds/ceph-$SERVER > ceph --cluster ceph --name client.bootstrap-mds \ > --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth \ > get-or-create mds.$SERVER osd 'allow rwx' mds 'allow' mon 'allow profile mds' \ > -o /var/lib/ceph/mds/ceph-$SERVER/keyring > > If it's not working, whats the output of 'systemctl status > ceph-mds@'? > > -- > Cheers, > Alwin > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Vadim Bulst Universit?t Leipzig / URZ 04109 Leipzig, Augustusplatz 10 phone: ++49-341-97-33380 mail: vadim.bulst at uni-leipzig.de From denis.morejon at etecsa.cu Wed Aug 8 15:23:16 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Wed, 8 Aug 2018 09:23:16 -0400 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> Message-ID: <855155c1-e7fe-7bf4-116f-9253c49551d5@etecsa.cu> El 07/08/18 a las 17:51, Yannis Milios escribi?: >> (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate >>>> /dev/zvol/rpool/lvm) and make a VG (vgcreate pve /dev/zvol/rpool/lvm) >>>> and then a LV (lvcreate -L100% pve/data) >>> > Try the above as it was suggested to you ... > > >>> But I suspect I have no space to create an >>>> additional zfs volume since the one mounted on "/" occupied all the space >> > No, that's a wrong assumption, zfs does not pre-allocate the whole space of > the pool, even if looks like it does so. In short there is no need to > "shrink" the pool in order to create a zvol as it was suggested above... > Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but if you > insist, it's up to you ... > A combination of Linux RAID + LVM would look much more elegant in your > case, but for that you have to reinstall PVE by using the Debian iso. > During the installation create a linux raid array with lvm on top and then > add PVE repos ass described in the wiki: > > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie That's right. Now I understand that lvm/zfs would be a mess. Mainly because zfs doesn't create a block devices such as partitions on which I could do pvcreate ... and make It part of a lvm volumen group. After a (zfs create -V 100G rpool/lvm) a have to do a losetup to create a loop device an so on... Instead, I will keep zfs Raid mounted on "/" (local storage) on the last 4 Proxmox, remove the local-lvm storage from all Proxmox, and resize the local storage of the first 4 Proxmox . In such a way that all the 8 Proxmox have just local storage making the migration of VMs between nodes easy. > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From denis.morejon at etecsa.cu Wed Aug 8 15:32:48 2018 From: denis.morejon at etecsa.cu (Denis Morejon) Date: Wed, 8 Aug 2018 09:32:48 -0400 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <855155c1-e7fe-7bf4-116f-9253c49551d5@etecsa.cu> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> <855155c1-e7fe-7bf4-116f-9253c49551d5@etecsa.cu> Message-ID: <9c41b2c7-963f-cf92-81df-51eb028ef74a@etecsa.cu> Why does Proxmox team have not incorporated a software Raid in the install process ? So that we could include redundancy and lvm advantages when using local disks. El 08/08/18 a las 09:23, Denis Morejon escribi?: > > > El 07/08/18 a las 17:51, Yannis Milios escribi?: >>> ? (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate >>>>> /dev/zvol/rpool/lvm) and make a VG (vgcreate pve /dev/zvol/rpool/lvm) >>>>> and then a LV (lvcreate -L100% pve/data) >>>> >> Try the above as it was suggested to you ... >> >> >>>> But I suspect I have no space to create an >>>>> additional zfs volume since the one mounted on "/" occupied all >>>>> the space >>> >> No, that's a wrong assumption, zfs does not pre-allocate the whole >> space of >> the pool, even if looks like it does so. In short there is no need to >> "shrink" the pool in order to create a zvol as it was suggested above... >> Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but >> if you >> insist, it's up to you ... >> A combination of Linux RAID + LVM would look much more elegant in your >> case, but for that you have to reinstall PVE by using the Debian iso. >> During the installation create a linux raid array with lvm on top and >> then >> add PVE repos ass described in the wiki: >> >> https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie > That's right. Now I understand that lvm/zfs would be a mess. Mainly > because > zfs doesn't create a block devices such as partitions on which I could > do pvcreate ... > and make It part of a lvm volumen group. > > After a (zfs create -V 100G rpool/lvm) a have to do a losetup to > create a loop device an so on... > > Instead, I will keep zfs Raid mounted on "/" (local storage) on the > last 4 Proxmox, remove the local-lvm storage from all Proxmox, and > resize the local storage of the first 4 Proxmox . In such a way that > all the 8 Proxmox have just local storage making the migration of VMs > between nodes easy. > > >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From aheinlein at gmx.com Wed Aug 8 14:40:38 2018 From: aheinlein at gmx.com (Andreas Heinlein) Date: Wed, 8 Aug 2018 14:40:38 +0200 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <9c41b2c7-963f-cf92-81df-51eb028ef74a@etecsa.cu> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> <855155c1-e7fe-7bf4-116f-9253c49551d5@etecsa.cu> <9c41b2c7-963f-cf92-81df-51eb028ef74a@etecsa.cu> Message-ID: <00046e31-3c92-08d9-c00e-1def6d5f03e0@gmx.com> Am 08.08.2018 um 15:32 schrieb Denis Morejon: > Why does Proxmox team have not incorporated a software Raid in the > install process ? So that we could include redundancy and lvm > advantages when using local disks. Because ZFS offers redundancy and LVM features (and much more) in a more modern way, e.g. during a rebuild only used blocks need to be resilvered, resulting in much greater speed. ZFS is intended to entirely replace MD-RAID and LVM. Only drawback of ZFS is that it needs bare metal disk access and must not (or at least should not) be used with hardware RAID controllers. This makes it difficult to use with older hardware, e.g. HP ProLiants which only have HP SmartArray controllers as disk controllers. It is possible to put some RAID controllers in HBA mode, though ZFS docs advise against it. Andreas From dorsyka at yahoo.com Wed Aug 8 14:55:58 2018 From: dorsyka at yahoo.com (dorsy) Date: Wed, 8 Aug 2018 14:55:58 +0200 Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <9c41b2c7-963f-cf92-81df-51eb028ef74a@etecsa.cu> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> <855155c1-e7fe-7bf4-116f-9253c49551d5@etecsa.cu> <9c41b2c7-963f-cf92-81df-51eb028ef74a@etecsa.cu> Message-ID: <4ca90484-40a1-502e-459a-1194b9f97e7f@yahoo.com> I'd say that it is more convinient to support one method. Also mentioned in this thread that zfs could be considered to be a successor of MDraid+LVM. It is still a debian system with cutom kernel and some pve packages on top, so You could do anything just like on any standard debian system. On 8/8/18 3:32 PM, Denis Morejon wrote: > Why does Proxmox team have not incorporated a software Raid in the > install process ? So that we could include redundancy and lvm > advantages when using local disks. > > > > > El 08/08/18 a las 09:23, Denis Morejon escribi?: >> >> >> El 07/08/18 a las 17:51, Yannis Milios escribi?: >>>> ? (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate >>>>>> /dev/zvol/rpool/lvm) and make a VG (vgcreate pve >>>>>> /dev/zvol/rpool/lvm) >>>>>> and then a LV (lvcreate -L100% pve/data) >>>>> >>> Try the above as it was suggested to you ... >>> >>> >>>>> But I suspect I have no space to create an >>>>>> additional zfs volume since the one mounted on "/" occupied all >>>>>> the space >>>> >>> No, that's a wrong assumption, zfs does not pre-allocate the whole >>> space of >>> the pool, even if looks like it does so. In short there is no need to >>> "shrink" the pool in order to create a zvol as it was suggested >>> above... >>> Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but >>> if you >>> insist, it's up to you ... >>> A combination of Linux RAID + LVM would look much more elegant in your >>> case, but for that you have to reinstall PVE by using the Debian iso. >>> During the installation create a linux raid array with lvm on top >>> and then >>> add PVE repos ass described in the wiki: >>> >>> https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie >> That's right. Now I understand that lvm/zfs would be a mess. Mainly >> because >> zfs doesn't create a block devices such as partitions on which I >> could do pvcreate ... >> and make It part of a lvm volumen group. >> >> After a (zfs create -V 100G rpool/lvm) a have to do a losetup to >> create a loop device an so on... >> >> Instead, I will keep zfs Raid mounted on "/" (local storage) on the >> last 4 Proxmox, remove the local-lvm storage from all Proxmox, and >> resize the local storage of the first 4 Proxmox . In such a way that >> all the 8 Proxmox have just local storage making the migration of VMs >> between nodes easy. >> >> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From dietmar at proxmox.com Wed Aug 8 15:04:55 2018 From: dietmar at proxmox.com (Dietmar Maurer) Date: Wed, 8 Aug 2018 15:04:55 +0200 (CEST) Subject: [PVE-User] How to use lvm on zfs ? In-Reply-To: <9c41b2c7-963f-cf92-81df-51eb028ef74a@etecsa.cu> References: <336e66c7-c59d-43b6-d6fa-f533775dfe0e@etecsa.cu> <599556307a99e66197bf997a0c695390fe9f18cd.camel@tuxis.nl> <0322b7ca-39d6-aecd-0fee-221e65f365ca@etecsa.cu> <786c9288bd317da591f28844c34a0eaea40c3b34.camel@tuxis.nl> <79cc72b6252734322b4778019c812400130d32b7.camel@tuxis.nl> <81ba3555-d188-5924-1e4d-148e108b196d@etecsa.cu> <855155c1-e7fe-7bf4-116f-9253c49551d5@etecsa.cu> <9c41b2c7-963f-cf92-81df-51eb028ef74a@etecsa.cu> Message-ID: <1204557606.32.1533733495414@webmail.proxmox.com> > Why does Proxmox team have not incorporated a software Raid in the > install process ? Because we consider mdraid unreliable and dangerous. > So that we could include redundancy and lvm advantages > when using local disks. Sorry, but we have software raid included - ZFS provides that. From atokovenko at gmail.com Sun Aug 12 19:55:10 2018 From: atokovenko at gmail.com (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KLQvtC60L7QstC10L3QutC+?=) Date: Sun, 12 Aug 2018 20:55:10 +0300 Subject: [PVE-User] pve-user Digest, Vol 125, Issue 10 In-Reply-To: References: Message-ID: unsubscribe 2018-08-09 13:00 GMT+03:00 : > Send pve-user mailing list submissions to > pve-user at pve.proxmox.com > > To subscribe or unsubscribe via the World Wide Web, visit > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > or, via email, send a message with subject or body 'help' to > pve-user-request at pve.proxmox.com > > You can reach the person managing the list at > pve-user-owner at pve.proxmox.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of pve-user digest..." > > > Today's Topics: > > 1. Re: How to use lvm on zfs ? (Denis Morejon) > 2. Re: How to use lvm on zfs ? (Denis Morejon) > 3. Re: How to use lvm on zfs ? (Andreas Heinlein) > 4. Re: How to use lvm on zfs ? (dorsy) > 5. Re: How to use lvm on zfs ? (Dietmar Maurer) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 8 Aug 2018 09:23:16 -0400 > From: Denis Morejon > To: pve-user at pve.proxmox.com > Subject: Re: [PVE-User] How to use lvm on zfs ? > Message-ID: <855155c1-e7fe-7bf4-116f-9253c49551d5 at etecsa.cu> > Content-Type: text/plain; charset=utf-8; format=flowed > > > > El 07/08/18 a las 17:51, Yannis Milios escribi?: > >> (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate > >>>> /dev/zvol/rpool/lvm) and make a VG (vgcreate pve /dev/zvol/rpool/lvm) > >>>> and then a LV (lvcreate -L100% pve/data) > >>> > > Try the above as it was suggested to you ... > > > > > >>> But I suspect I have no space to create an > >>>> additional zfs volume since the one mounted on "/" occupied all the > space > >> > > No, that's a wrong assumption, zfs does not pre-allocate the whole space > of > > the pool, even if looks like it does so. In short there is no need to > > "shrink" the pool in order to create a zvol as it was suggested above... > > Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but if > you > > insist, it's up to you ... > > A combination of Linux RAID + LVM would look much more elegant in your > > case, but for that you have to reinstall PVE by using the Debian iso. > > During the installation create a linux raid array with lvm on top and > then > > add PVE repos ass described in the wiki: > > > > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie > That's right. Now I understand that lvm/zfs would be a mess. Mainly because > zfs doesn't create a block devices such as partitions on which I could > do pvcreate ... > and make It part of a lvm volumen group. > > After a (zfs create -V 100G rpool/lvm) a have to do a losetup to create > a loop device an so on... > > Instead, I will keep zfs Raid mounted on "/" (local storage) on the last > 4 Proxmox, remove the local-lvm storage from all Proxmox, and resize the > local storage of the first 4 Proxmox . In such a way that all the 8 > Proxmox have just local storage making the migration of VMs between > nodes easy. > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > ------------------------------ > > Message: 2 > Date: Wed, 8 Aug 2018 09:32:48 -0400 > From: Denis Morejon > To: pve-user at pve.proxmox.com > Subject: Re: [PVE-User] How to use lvm on zfs ? > Message-ID: <9c41b2c7-963f-cf92-81df-51eb028ef74a at etecsa.cu> > Content-Type: text/plain; charset=utf-8; format=flowed > > Why does Proxmox team have not incorporated a software Raid in the > install process ? So that we could include redundancy and lvm advantages > when using local disks. > > > > > El 08/08/18 a las 09:23, Denis Morejon escribi?: > > > > > > El 07/08/18 a las 17:51, Yannis Milios escribi?: > >>> (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate > >>>>> /dev/zvol/rpool/lvm) and make a VG (vgcreate pve /dev/zvol/rpool/lvm) > >>>>> and then a LV (lvcreate -L100% pve/data) > >>>> > >> Try the above as it was suggested to you ... > >> > >> > >>>> But I suspect I have no space to create an > >>>>> additional zfs volume since the one mounted on "/" occupied all > >>>>> the space > >>> > >> No, that's a wrong assumption, zfs does not pre-allocate the whole > >> space of > >> the pool, even if looks like it does so. In short there is no need to > >> "shrink" the pool in order to create a zvol as it was suggested above... > >> Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but > >> if you > >> insist, it's up to you ... > >> A combination of Linux RAID + LVM would look much more elegant in your > >> case, but for that you have to reinstall PVE by using the Debian iso. > >> During the installation create a linux raid array with lvm on top and > >> then > >> add PVE repos ass described in the wiki: > >> > >> https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie > > That's right. Now I understand that lvm/zfs would be a mess. Mainly > > because > > zfs doesn't create a block devices such as partitions on which I could > > do pvcreate ... > > and make It part of a lvm volumen group. > > > > After a (zfs create -V 100G rpool/lvm) a have to do a losetup to > > create a loop device an so on... > > > > Instead, I will keep zfs Raid mounted on "/" (local storage) on the > > last 4 Proxmox, remove the local-lvm storage from all Proxmox, and > > resize the local storage of the first 4 Proxmox . In such a way that > > all the 8 Proxmox have just local storage making the migration of VMs > > between nodes easy. > > > > > >> _______________________________________________ > >> pve-user mailing list > >> pve-user at pve.proxmox.com > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > ------------------------------ > > Message: 3 > Date: Wed, 8 Aug 2018 14:40:38 +0200 > From: Andreas Heinlein > To: PVE User List , Denis Morejon > > Subject: Re: [PVE-User] How to use lvm on zfs ? > Message-ID: <00046e31-3c92-08d9-c00e-1def6d5f03e0 at gmx.com> > Content-Type: text/plain; charset=utf-8 > > Am 08.08.2018 um 15:32 schrieb Denis Morejon: > > Why does Proxmox team have not incorporated a software Raid in the > > install process ? So that we could include redundancy and lvm > > advantages when using local disks. > Because ZFS offers redundancy and LVM features (and much more) in a more > modern way, e.g. during a rebuild only used blocks need to be > resilvered, resulting in much greater speed. ZFS is intended to entirely > replace MD-RAID and LVM. > > Only drawback of ZFS is that it needs bare metal disk access and must > not (or at least should not) be used with hardware RAID controllers. > This makes it difficult to use with older hardware, e.g. HP ProLiants > which only have HP SmartArray controllers as disk controllers. It is > possible to put some RAID controllers in HBA mode, though ZFS docs > advise against it. > > Andreas > > > ------------------------------ > > Message: 4 > Date: Wed, 8 Aug 2018 14:55:58 +0200 > From: dorsy > To: pve-user at pve.proxmox.com > Subject: Re: [PVE-User] How to use lvm on zfs ? > Message-ID: <4ca90484-40a1-502e-459a-1194b9f97e7f at yahoo.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > I'd say that it is more convinient to support one method. > Also mentioned in this thread that zfs could be considered to be a > successor of MDraid+LVM. > > It is still a debian system with cutom kernel and some pve packages on > top, so You could do anything just like on any standard debian system. > > On 8/8/18 3:32 PM, Denis Morejon wrote: > > Why does Proxmox team have not incorporated a software Raid in the > > install process ? So that we could include redundancy and lvm > > advantages when using local disks. > > > > > > > > > > El 08/08/18 a las 09:23, Denis Morejon escribi?: > >> > >> > >> El 07/08/18 a las 17:51, Yannis Milios escribi?: > >>>> (zfs create -V 100G rpool/lvm) and make that a PV (pvcreate > >>>>>> /dev/zvol/rpool/lvm) and make a VG (vgcreate pve > >>>>>> /dev/zvol/rpool/lvm) > >>>>>> and then a LV (lvcreate -L100% pve/data) > >>>>> > >>> Try the above as it was suggested to you ... > >>> > >>> > >>>>> But I suspect I have no space to create an > >>>>>> additional zfs volume since the one mounted on "/" occupied all > >>>>>> the space > >>>> > >>> No, that's a wrong assumption, zfs does not pre-allocate the whole > >>> space of > >>> the pool, even if looks like it does so. In short there is no need to > >>> "shrink" the pool in order to create a zvol as it was suggested > >>> above... > >>> Still, the whole idea of having LVM ontop of ZFS/zvol is a mess, but > >>> if you > >>> insist, it's up to you ... > >>> A combination of Linux RAID + LVM would look much more elegant in your > >>> case, but for that you have to reinstall PVE by using the Debian iso. > >>> During the installation create a linux raid array with lvm on top > >>> and then > >>> add PVE repos ass described in the wiki: > >>> > >>> https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie > >> That's right. Now I understand that lvm/zfs would be a mess. Mainly > >> because > >> zfs doesn't create a block devices such as partitions on which I > >> could do pvcreate ... > >> and make It part of a lvm volumen group. > >> > >> After a (zfs create -V 100G rpool/lvm) a have to do a losetup to > >> create a loop device an so on... > >> > >> Instead, I will keep zfs Raid mounted on "/" (local storage) on the > >> last 4 Proxmox, remove the local-lvm storage from all Proxmox, and > >> resize the local storage of the first 4 Proxmox . In such a way that > >> all the 8 Proxmox have just local storage making the migration of VMs > >> between nodes easy. > >> > >> > >>> _______________________________________________ > >>> pve-user mailing list > >>> pve-user at pve.proxmox.com > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > ------------------------------ > > Message: 5 > Date: Wed, 8 Aug 2018 15:04:55 +0200 (CEST) > From: Dietmar Maurer > To: PVE User List , Denis Morejon > > Subject: Re: [PVE-User] How to use lvm on zfs ? > Message-ID: <1204557606.32.1533733495414 at webmail.proxmox.com> > Content-Type: text/plain; charset=UTF-8 > > > Why does Proxmox team have not incorporated a software Raid in the > > install process ? > > Because we consider mdraid unreliable and dangerous. > > > So that we could include redundancy and lvm advantages > > when using local disks. > > Sorry, but we have software raid included - ZFS provides that. > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > ------------------------------ > > End of pve-user Digest, Vol 125, Issue 10 > ***************************************** > -- ? ?????????, ????????? ??????? ?????????? From gilberto.nunes32 at gmail.com Tue Aug 14 23:09:19 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Tue, 14 Aug 2018 18:09:19 -0300 Subject: [PVE-User] Migrate XenServer to Proxmox Message-ID: Hi there... I have a environment, running XenServer 7.1 So we decide to make a Ceph cluster with 3 IBM servers and with an HP Server the first Proxmox node, for a planned cluster. Now, my first question is: How can I migrate some VM without downtime? This VM are Windows Server with SQL Server, that we haven't way to stop it, because the production of the factory depend on it. Second question is: Can I attached Ceph Server based on Proxmox into a running XenServer? I found this project that claim make it possible: https://github.com/rposudnevskiy/RBDSR/tree/v3.0 Just seeking for some advice. If someone already face the same scenario, I will appreciated any idea! Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From ghamlin at needlesolutions.com Wed Aug 15 02:22:45 2018 From: ghamlin at needlesolutions.com (Garrett Hamlin) Date: Tue, 14 Aug 2018 20:22:45 -0400 Subject: [PVE-User] Migrate XenServer to Proxmox In-Reply-To: References: Message-ID: <1534292135.local-6dee90e7-9eb3-v1.4.0-549b7968@getmailspring.com> you might want to look into setting up a VM for the database and then replicating the databases using SQL itself. Will take some time to sync up but if performed correctly the transition should have close to zero downtime. Article for reference. https://dba.stackexchange.com/questions/100686/move-live-sql-server-database-to-a-new-server Sent from Mailspring (https://getmailspring.com/), the best free email app for work On Aug 14 2018, at 5:09 pm, Gilberto Nunes wrote: > > Hi there... > I have a environment, running XenServer 7.1 > So we decide to make a Ceph cluster with 3 IBM servers and with an HP > Server the first Proxmox node, for a planned cluster. > Now, my first question is: How can I migrate some VM without downtime? > This VM are Windows Server with SQL Server, that we haven't way to stop it, > because the production of the factory depend on it. > Second question is: Can I attached Ceph Server based on Proxmox into a > running XenServer? > I found this project that claim make it possible: > https://github.com/rposudnevskiy/RBDSR/tree/v3.0 > > Just seeking for some advice. If someone already face the same scenario, I > will appreciated any idea! > > Thanks a lot > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mityapetuhov at gmail.com Wed Aug 15 10:41:43 2018 From: mityapetuhov at gmail.com (Dmitry Petuhov) Date: Wed, 15 Aug 2018 11:41:43 +0300 Subject: [PVE-User] node not rebooted after corosync crash Message-ID: <7dced6e1-eaa0-fc62-4b19-8c4c5405c104@gmail.com> Week ago on one of my PVE nodes suddenly crashed corosync. -------------->8========= corosync[4701]: error?? [TOTEM ] FAILED TO RECEIVE corosync[4701]:? [TOTEM ] FAILED TO RECEIVE corosync[4701]: notice? [TOTEM ] A new membership (10.19.92.53:1992) was formed. Members left: 1 2 4 corosync[4701]: notice? [TOTEM ] Failed to receive the leave message. failed: 1 2 4 corosync[4701]:? [TOTEM ] A new membership (10.19.92.53:1992) was formed. Members left: 1 2 4 corosync[4701]:? [TOTEM ] Failed to receive the leave message. failed: 1 2 4 corosync[4701]: notice? [QUORUM] This node is within the non-primary component and will NOT provide any services. corosync[4701]: notice? [QUORUM] Members[1]: 3 corosync[4701]: notice? [MAIN? ] Completed service synchronization, ready to provide service. corosync[4701]:? [QUORUM] This node is within the non-primary component and will NOT provide any services. corosync[4701]:? [QUORUM] Members[1]: 3 corosync[4701]:? [MAIN? ] Completed service synchronization, ready to provide service. kernel: [29187555.500409] dlm: closing connection to node 2 corosync[4701]: notice? [TOTEM ] A new membership (10.19.92.51:2000) was formed. Members joined: 1 2 4 corosync[4701]:? [TOTEM ] A new membership (10.19.92.51:2000) was formed. Members joined: 1 2 4 corosync[4701]: notice? [QUORUM] This node is within the primary component and will provide service. corosync[4701]: notice? [QUORUM] Members[4]: 1 2 3 4 corosync[4701]: notice? [MAIN? ] Completed service synchronization, ready to provide service. corosync[4701]:? [QUORUM] This node is within the primary component and will provide service. corosync[4701]: notice? [CFG?? ] Killed by node 1: dlm_controld corosync[4701]: error?? [MAIN? ] Corosync Cluster Engine exiting with status -1 at cfg.c:530. corosync[4701]:? [QUORUM] Members[4]: 1 2 3 4 corosync[4701]:? [MAIN? ] Completed service synchronization, ready to provide service. dlm_controld[688]: 29187298 daemon node 4 stateful merge dlm_controld[688]: 29187298 receive_start 4:6 add node with started_count 2 dlm_controld[688]: 29187298 daemon node 1 stateful merge dlm_controld[688]: 29187298 receive_start 1:5 add node with started_count 4 dlm_controld[688]: 29187298 daemon node 2 stateful merge dlm_controld[688]: 29187298 receive_start 2:17 add node with started_count 13 corosync[4701]:? [CFG?? ] Killed by node 1: dlm_controld corosync[4701]:? [MAIN? ] Corosync Cluster Engine exiting with status -1 at cfg.c:530. dlm_controld[688]: 29187298 cpg_dispatch error 2 dlm_controld[688]: 29187298 process_cluster_cfg cfg_dispatch 2 dlm_controld[688]: 29187298 cluster is down, exiting dlm_controld[688]: 29187298 process_cluster quorum_dispatch 2 dlm_controld[688]: 29187298 daemon cpg_dispatch error 2 systemd[1]: corosync.service: Main process exited, code=exited, status=255/n/a systemd[1]: corosync.service: Unit entered failed state. systemd[1]: corosync.service: Failed with result 'exit-code'. kernel: [29187556.903177] dlm: closing connection to node 4 kernel: [29187556.906730] dlm: closing connection to node 3 dlm_controld[688]: 29187298 abandoned lockspace hp-big-gfs kernel: [29187556.924279] dlm: dlm user daemon left 1 lockspaces -------------->8========= But node did not rebooted. I use WATCHDOG_MODULE=ipmi_watchdog. Watchdog still running: -------------->8========= # ipmitool mc watchdog get Watchdog Timer Use:???? SMS/OS (0x44) Watchdog Timer Is:????? Started/Running Watchdog Timer Actions: Hard Reset (0x01) Pre-timeout interval:?? 0 seconds Timer Expiration Flags: 0x10 Initial Countdown:????? 10 sec Present Countdown:????? 9 sec -------------->8========= The only down service is corosync. -------------->8========= # pveversion --verbose proxmox-ve: 5.0-21 (running kernel: 4.10.17-2-pve) pve-manager: 5.0-31 (running version: 5.0-31/27769b1f) pve-kernel-4.10.17-2-pve: 4.10.17-20 pve-kernel-4.10.17-3-pve: 4.10.17-21 libpve-http-server-perl: 2.0-6 lvm2: 2.02.168-pve3 corosync: 2.4.2-pve3 libqb0: 1.0.1-1 pve-cluster: 5.0-12 qemu-server: 5.0-15 pve-firmware: 2.0-2 libpve-common-perl: 5.0-16 libpve-guest-common-perl: 2.0-11 libpve-access-control: 5.0-6 libpve-storage-perl: 5.0-14 pve-libspice-server1: 0.12.8-3 vncterm: 1.5-2 pve-docs: 5.0-9 pve-qemu-kvm: 2.9.0-5 pve-container: 2.0-15 pve-firewall: 3.0-2 pve-ha-manager: 2.0-2 ksm-control-daemon: 1.2-2 glusterfs-client: 3.8.8-1 lxc-pve: 2.0.8-3 lxcfs: 2.0.7-pve4 criu: 2.11.1-1~bpo90 novnc-pve: 0.6-4 smartmontools: 6.5+svn4324-1 zfsutils-linux: 0.6.5.11-pve17~bpo90 gfs2-utils: 3.1.9-2 openvswitch-switch: 2.7.0-2 ceph: 12.2.0-pve1 -------------->8========= I also have GFS2 in this cluster, which did not stop work after corosync crash (which scares me most). Shouldn't node reboot on corosync fail, and why it can still run? Or shall node have HA VMs to reboot, and just stay as it is if there's only regular autostarted VMs and no HA machines present? From gilberto.nunes32 at gmail.com Wed Aug 15 12:23:25 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 15 Aug 2018 07:23:25 -0300 Subject: [PVE-User] Migrate XenServer to Proxmox In-Reply-To: <1534292135.local-6dee90e7-9eb3-v1.4.0-549b7968@getmailspring.com> References: <1534292135.local-6dee90e7-9eb3-v1.4.0-549b7968@getmailspring.com> Message-ID: Hi Yes! That's what we looking forward. Tha Em ter, 14 de ago de 2018 21:23, Garrett Hamlin escreveu: > you might want to look into setting up a VM for the database and then > replicating the databases using SQL itself. Will take some time to sync up > but if performed correctly the transition should have close to zero > downtime. > > Article for reference. > > https://dba.stackexchange.com/questions/100686/move-live-sql-server-database-to-a-new-server > > Sent from Mailspring (https://getmailspring.com/), the best free email > app for work > On Aug 14 2018, at 5:09 pm, Gilberto Nunes > wrote: > > > > Hi there... > > I have a environment, running XenServer 7.1 > > So we decide to make a Ceph cluster with 3 IBM servers and with an HP > > Server the first Proxmox node, for a planned cluster. > > Now, my first question is: How can I migrate some VM without downtime? > > This VM are Windows Server with SQL Server, that we haven't way to stop > it, > > because the production of the factory depend on it. > > Second question is: Can I attached Ceph Server based on Proxmox into a > > running XenServer? > > I found this project that claim make it possible: > > https://github.com/rposudnevskiy/RBDSR/tree/v3.0 > > > > Just seeking for some advice. If someone already face the same scenario, > I > > will appreciated any idea! > > > > Thanks a lot > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From ronny+pve-user at aasen.cx Wed Aug 15 13:00:48 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Wed, 15 Aug 2018 13:00:48 +0200 Subject: [PVE-User] Migrate XenServer to Proxmox In-Reply-To: References: Message-ID: On 14. aug. 2018 23:09, Gilberto Nunes wrote: > Hi there... > > I have a environment, running XenServer 7.1 > So we decide to make a Ceph cluster with 3 IBM servers and with an HP > Server the first Proxmox node, for a planned cluster. > Now, my first question is: How can I migrate some VM without downtime? > This VM are Windows Server with SQL Server, that we haven't way to stop it, > because the production of the factory depend on it. > Second question is: Can I attached Ceph Server based on Proxmox into a > running XenServer? > I found this project that claim make it possible: > https://github.com/rposudnevskiy/RBDSR/tree/v3.0 > > Just seeking for some advice. If someone already face the same scenario, I > will appreciated any idea! > > Thanks a lot > --- > Gilberto Nunes Ferreira > afaik you can not live migrate a vm from xen to proxmox directly. easiest is to stop vm -> copy image file -> start vm. to speed this up you can do a initial transfer, and then the last changes after you shutdown the vm. how depends on the origin of the vm if you have zfs you can use zfs send, if you have lvm you can use lvm snapshots etc. once you have the vm on proxmox in some form or other (imagefile or lvm lv) , you can live migrate it into ceph. But if you can get xen to put the vm on ceph directly, you should be able to just shutdown on xen, and boot it with the same disk in proxmox. but that is probably more a xen question, that i can not comment on. PS: btw with ceph and only 3 machines, you do not have a machine available when a machine eventualy dies or needs longer maintainance. ceph's self healing can not relocate objects when you have 3x replication and only 3 hosts. I would really recommend having a 4th host, so you have a failuredomain. kind regards Ronny Aasen From josh at noobbox.com Fri Aug 17 16:30:34 2018 From: josh at noobbox.com (Josh Knight) Date: Fri, 17 Aug 2018 10:30:34 -0400 Subject: [PVE-User] node not rebooted after corosync crash In-Reply-To: <7dced6e1-eaa0-fc62-4b19-8c4c5405c104@gmail.com> References: <7dced6e1-eaa0-fc62-4b19-8c4c5405c104@gmail.com> Message-ID: The ipmi_watchdog is a hardware watchdog which the OS pokes to keep happy. If the OS hangs/crashes and therefore fails to poke it, then the IPMI watchdog will reset the system. It will not catch the case of an individual daemon/process, like corosync, hanging/crashing on the system. On Wed, Aug 15, 2018 at 4:41 AM Dmitry Petuhov wrote: > Week ago on one of my PVE nodes suddenly crashed corosync. > > -------------->8========= > corosync[4701]: error [TOTEM ] FAILED TO RECEIVE > corosync[4701]: [TOTEM ] FAILED TO RECEIVE > corosync[4701]: notice [TOTEM ] A new membership (10.19.92.53:1992) was > formed. Members left: 1 2 4 > corosync[4701]: notice [TOTEM ] Failed to receive the leave message. > failed: 1 2 4 > corosync[4701]: [TOTEM ] A new membership (10.19.92.53:1992) was > formed. Members left: 1 2 4 > corosync[4701]: [TOTEM ] Failed to receive the leave message. failed: 1 2 > 4 > corosync[4701]: notice [QUORUM] This node is within the non-primary > component and will NOT provide any services. > corosync[4701]: notice [QUORUM] Members[1]: 3 > corosync[4701]: notice [MAIN ] Completed service synchronization, > ready to provide service. > corosync[4701]: [QUORUM] This node is within the non-primary component > and will NOT provide any services. > corosync[4701]: [QUORUM] Members[1]: 3 > corosync[4701]: [MAIN ] Completed service synchronization, ready to > provide service. > kernel: [29187555.500409] dlm: closing connection to node 2 > corosync[4701]: notice [TOTEM ] A new membership (10.19.92.51:2000) was > formed. Members joined: 1 2 4 > corosync[4701]: [TOTEM ] A new membership (10.19.92.51:2000) was > formed. Members joined: 1 2 4 > corosync[4701]: notice [QUORUM] This node is within the primary > component and will provide service. > corosync[4701]: notice [QUORUM] Members[4]: 1 2 3 4 > corosync[4701]: notice [MAIN ] Completed service synchronization, > ready to provide service. > corosync[4701]: [QUORUM] This node is within the primary component and > will provide service. > corosync[4701]: notice [CFG ] Killed by node 1: dlm_controld > corosync[4701]: error [MAIN ] Corosync Cluster Engine exiting with > status -1 at cfg.c:530. > corosync[4701]: [QUORUM] Members[4]: 1 2 3 4 > corosync[4701]: [MAIN ] Completed service synchronization, ready to > provide service. > dlm_controld[688]: 29187298 daemon node 4 stateful merge > dlm_controld[688]: 29187298 receive_start 4:6 add node with started_count 2 > dlm_controld[688]: 29187298 daemon node 1 stateful merge > dlm_controld[688]: 29187298 receive_start 1:5 add node with started_count 4 > dlm_controld[688]: 29187298 daemon node 2 stateful merge > dlm_controld[688]: 29187298 receive_start 2:17 add node with > started_count 13 > corosync[4701]: [CFG ] Killed by node 1: dlm_controld > corosync[4701]: [MAIN ] Corosync Cluster Engine exiting with status -1 > at cfg.c:530. > dlm_controld[688]: 29187298 cpg_dispatch error 2 > dlm_controld[688]: 29187298 process_cluster_cfg cfg_dispatch 2 > dlm_controld[688]: 29187298 cluster is down, exiting > dlm_controld[688]: 29187298 process_cluster quorum_dispatch 2 > dlm_controld[688]: 29187298 daemon cpg_dispatch error 2 > systemd[1]: corosync.service: Main process exited, code=exited, > status=255/n/a > systemd[1]: corosync.service: Unit entered failed state. > systemd[1]: corosync.service: Failed with result 'exit-code'. > kernel: [29187556.903177] dlm: closing connection to node 4 > kernel: [29187556.906730] dlm: closing connection to node 3 > dlm_controld[688]: 29187298 abandoned lockspace hp-big-gfs > kernel: [29187556.924279] dlm: dlm user daemon left 1 lockspaces > -------------->8========= > > > But node did not rebooted. > > I use WATCHDOG_MODULE=ipmi_watchdog. Watchdog still running: > > > -------------->8========= > > # ipmitool mc watchdog get > Watchdog Timer Use: SMS/OS (0x44) > Watchdog Timer Is: Started/Running > Watchdog Timer Actions: Hard Reset (0x01) > Pre-timeout interval: 0 seconds > Timer Expiration Flags: 0x10 > Initial Countdown: 10 sec > Present Countdown: 9 sec > > -------------->8========= > > > The only down service is corosync. > > > -------------->8========= > > # pveversion --verbose > proxmox-ve: 5.0-21 (running kernel: 4.10.17-2-pve) > pve-manager: 5.0-31 (running version: 5.0-31/27769b1f) > pve-kernel-4.10.17-2-pve: 4.10.17-20 > pve-kernel-4.10.17-3-pve: 4.10.17-21 > libpve-http-server-perl: 2.0-6 > lvm2: 2.02.168-pve3 > corosync: 2.4.2-pve3 > libqb0: 1.0.1-1 > pve-cluster: 5.0-12 > qemu-server: 5.0-15 > pve-firmware: 2.0-2 > libpve-common-perl: 5.0-16 > libpve-guest-common-perl: 2.0-11 > libpve-access-control: 5.0-6 > libpve-storage-perl: 5.0-14 > pve-libspice-server1: 0.12.8-3 > vncterm: 1.5-2 > pve-docs: 5.0-9 > pve-qemu-kvm: 2.9.0-5 > pve-container: 2.0-15 > pve-firewall: 3.0-2 > pve-ha-manager: 2.0-2 > ksm-control-daemon: 1.2-2 > glusterfs-client: 3.8.8-1 > lxc-pve: 2.0.8-3 > lxcfs: 2.0.7-pve4 > criu: 2.11.1-1~bpo90 > novnc-pve: 0.6-4 > smartmontools: 6.5+svn4324-1 > zfsutils-linux: 0.6.5.11-pve17~bpo90 > gfs2-utils: 3.1.9-2 > openvswitch-switch: 2.7.0-2 > ceph: 12.2.0-pve1 > > -------------->8========= > > > I also have GFS2 in this cluster, which did not stop work after corosync > crash (which scares me most). > > > Shouldn't node reboot on corosync fail, and why it can still run? Or > shall node have HA VMs to reboot, and just stay as it is if there's only > regular autostarted VMs and no HA machines present? > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From klaus.mailinglists at pernau.at Fri Aug 17 22:22:30 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Fri, 17 Aug 2018 22:22:30 +0200 Subject: [PVE-User] Proxmox and DRBD Message-ID: Hi! Reading the archives I learnt that Proxmox removed DRBD as the consequence of license issues (which were reverted). As far as is know this was about DRBD9 and older Proxmox releases had support for DRBD8. Checking the Proxmox sources (storage modules) it seems that there is still some code for DRBD left, but apperently not used. IMO it is a real pitty that DRBD is not supported anymore. There are several use case where DRBD is a perfect and cost efficient solution avoiding additional external storages. Debian 9 comes with DRBD8 out of the - so why not use it (without needed any additonal software from Linbit)? Is there any technical reason why DRBD(8) is not supported anymore? Is it possible to activate and use the leftover DRBD code in Proxmox? If not, I think the simple "manual" solution would be a DRBD-backed LVM storage in active-active mode. Any experiences with such a setup (except that it is not nice to have a single DRBD ressource for all VMs instead of one DRBD ressource per virtual disk)? Thanks Klaus From aa at ipnerd.net Sat Aug 18 00:19:57 2018 From: aa at ipnerd.net (Alwin Antreich) Date: Sat, 18 Aug 2018 00:19:57 +0200 Subject: [PVE-User] Proxmox and DRBD In-Reply-To: References: Message-ID: <20180817221957.4awlei3y4566tygx@ipnerd.net> Hello Klaus, On Fri, Aug 17, 2018 at 10:22:30PM +0200, Klaus Darilion wrote: > Hi! > > Reading the archives I learnt that Proxmox removed DRBD as the consequence > of license issues (which were reverted). As far as is know this was about > DRBD9 and older Proxmox releases had support for DRBD8. Linbit itself took on them to support PVE, they have a repository and manual for that. > > Checking the Proxmox sources (storage modules) it seems that there is still > some code for DRBD left, but apperently not used. > > IMO it is a real pitty that DRBD is not supported anymore. There are several > use case where DRBD is a perfect and cost efficient solution avoiding > additional external storages. Debian 9 comes with DRBD8 out of the - so why > not use it (without needed any additonal software from Linbit)? Check out the pvesr for storage replication. https://pve.proxmox.com/pve-docs/chapter-pvesr.html > > Is there any technical reason why DRBD(8) is not supported anymore? The storage plugin is, AFAIR only for DRBD9. https://docs.linbit.com/docs/users-guide-9.0/#ch-proxmox > > Is it possible to activate and use the leftover DRBD code in Proxmox? > > If not, I think the simple "manual" solution would be a DRBD-backed LVM > storage in active-active mode. Any experiences with such a setup (except > that it is not nice to have a single DRBD ressource for all VMs instead of > one DRBD ressource per virtual disk)? Try pvesr, as mentioned above, it uses ZFS and snapshots to replicate volumes, less hassle then with DRBD, IMHO. Or give Ceph a go, integrated into PVE. -- Cheers, Alwin From dietmar at proxmox.com Sat Aug 18 07:33:12 2018 From: dietmar at proxmox.com (Dietmar Maurer) Date: Sat, 18 Aug 2018 07:33:12 +0200 (CEST) Subject: [PVE-User] Proxmox and DRBD In-Reply-To: References: Message-ID: <995339290.5.1534570392891@webmail.proxmox.com> > IMO it is a real pitty that DRBD is not supported anymore. You can get DRBD support from Linbit. From klaus.mailinglists at pernau.at Sat Aug 18 15:09:47 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Sat, 18 Aug 2018 15:09:47 +0200 Subject: [PVE-User] Proxmox and DRBD In-Reply-To: <20180817221957.4awlei3y4566tygx@ipnerd.net> References: <20180817221957.4awlei3y4566tygx@ipnerd.net> Message-ID: <74c4a83e-4897-fdd5-6325-6081d3c0c218@pernau.at> >> Is it possible to activate and use the leftover DRBD code in Proxmox? >> >> If not, I think the simple "manual" solution would be a DRBD-backed LVM >> storage in active-active mode. Any experiences with such a setup (except >> that it is not nice to have a single DRBD ressource for all VMs instead of >> one DRBD ressource per virtual disk)? > Try pvesr, as mentioned above, it uses ZFS and snapshots to replicate > volumes, less hassle then with DRBD, IMHO. Or give Ceph a go, integrated > into PVE. We currently use ZFS replication, but it does not support live migration. Further, I think it does not support HA - does it? Klaus From yannis.milios at gmail.com Sat Aug 18 16:14:20 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Sat, 18 Aug 2018 15:14:20 +0100 Subject: [PVE-User] Proxmox and DRBD In-Reply-To: <74c4a83e-4897-fdd5-6325-6081d3c0c218@pernau.at> References: <20180817221957.4awlei3y4566tygx@ipnerd.net> <74c4a83e-4897-fdd5-6325-6081d3c0c218@pernau.at> Message-ID: Both drbd8 and drbd9 can be configured in pve without issues. The former, can be configured in pairs of two pve nodes, where for the latter, three pve nodes is the minimum.IIRC drbd8 kmod is included by default in the pve kernel, so you will only need to install drbd-utils to get started. For drbd9, follow the instructions as described in Linbit's documentation.In my opinion enabling HA in just 2 pve nodes, with drbd8 in dual primary mode, is just too dangerous.As a minimum, proper (hardware) fencing must be configured in both drbd and pve to avoid data corruption. From aa at ipnerd.net Sun Aug 19 12:21:53 2018 From: aa at ipnerd.net (Alwin Antreich) Date: Sun, 19 Aug 2018 12:21:53 +0200 Subject: [PVE-User] Proxmox and DRBD In-Reply-To: <74c4a83e-4897-fdd5-6325-6081d3c0c218@pernau.at> References: <20180817221957.4awlei3y4566tygx@ipnerd.net> <74c4a83e-4897-fdd5-6325-6081d3c0c218@pernau.at> Message-ID: On Sat, Aug 18, 2018, 15:09 Klaus Darilion wrote: > > >> Is it possible to activate and use the leftover DRBD code in Proxmox? > >> > >> If not, I think the simple "manual" solution would be a DRBD-backed LVM > >> storage in active-active mode. Any experiences with such a setup (except > >> that it is not nice to have a single DRBD ressource for all VMs instead > of > >> one DRBD ressource per virtual disk)? > > Try pvesr, as mentioned above, it uses ZFS and snapshots to replicate > > volumes, less hassle then with DRBD, IMHO. Or give Ceph a go, integrated > > into PVE. > We currently use ZFS replication, but it does not support live > migration. Further, I think it does not support HA - does it? > The storage replication doesn't work with live migration yet but HA is possible, see first section of the docs. > https://pve.proxmox.com/pve-docs/chapter-pvesr.html -- Cheers, Alwin > From jm at ginernet.com Mon Aug 20 09:36:10 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Mon, 20 Aug 2018 09:36:10 +0200 Subject: [PVE-User] DHCP for non cloudinit VM Message-ID: Hello, there any plan to implement DHCP for non cloudinit VMs? Thanks! -- Jos? Manuel Giner http://ginernet.com From d.csapak at proxmox.com Mon Aug 20 11:14:32 2018 From: d.csapak at proxmox.com (Dominik Csapak) Date: Mon, 20 Aug 2018 11:14:32 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: Message-ID: On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: > Hello, > > there any plan to implement DHCP for non cloudinit VMs? > > Thanks! > > the question does not really make sense as we did not implement dhcp for cloudinit, the config there only tells the vm how to configure its network (same as with containers, were we also don't implement a dhcp server) but where is the problem in having a host/vm in your network serving dhcp? From jm at ginernet.com Mon Aug 20 13:43:14 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Mon, 20 Aug 2018 13:43:14 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: Message-ID: The possibility of being able to define IPs directly from the Proxmox/API interface. Just like with Cloud-init or containers. On 20/08/2018 11:14, Dominik Csapak wrote: > On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: >> Hello, >> >> there any plan to implement DHCP for non cloudinit VMs? >> >> Thanks! >> >> > > the question does not really make sense as we did not implement dhcp for > cloudinit, the config there only tells the vm how to configure its > network (same as with containers, were we also don't implement a dhcp > server) > > but where is the problem in having a host/vm in your network serving > dhcp? > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Jos? Manuel Giner http://ginernet.com From proxmox at iancoetzee.za.net Mon Aug 20 15:21:50 2018 From: proxmox at iancoetzee.za.net (Ian Coetzee) Date: Mon, 20 Aug 2018 15:21:50 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: Message-ID: Hi Jos?, Using the Qemu Agent you are able to determine the IP of the VM. Kind regards On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner wrote: > The possibility of being able to define IPs directly from the > Proxmox/API interface. Just like with Cloud-init or containers. > > > > On 20/08/2018 11:14, Dominik Csapak wrote: > > On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: > >> Hello, > >> > >> there any plan to implement DHCP for non cloudinit VMs? > >> > >> Thanks! > >> > >> > > > > the question does not really make sense as we did not implement dhcp for > > cloudinit, the config there only tells the vm how to configure its > > network (same as with containers, were we also don't implement a dhcp > > server) > > > > but where is the problem in having a host/vm in your network serving > > dhcp? > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > -- > Jos? Manuel Giner > http://ginernet.com > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From jm at ginernet.com Mon Aug 20 15:25:04 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Mon, 20 Aug 2018 15:25:04 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: Message-ID: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> I mean, when you install the ISO of an operating system on a VM, when configuring the network, that the user can choose the DHCP option instead of defining the values by hand. On 20/08/2018 15:21, Ian Coetzee wrote: > Hi Jos?, > > Using the Qemu Agent you are able to determine the IP of the VM. > > Kind regards > > On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner wrote: > >> The possibility of being able to define IPs directly from the >> Proxmox/API interface. Just like with Cloud-init or containers. >> >> >> >> On 20/08/2018 11:14, Dominik Csapak wrote: >>> On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: >>>> Hello, >>>> >>>> there any plan to implement DHCP for non cloudinit VMs? >>>> >>>> Thanks! >>>> >>>> >>> >>> the question does not really make sense as we did not implement dhcp for >>> cloudinit, the config there only tells the vm how to configure its >>> network (same as with containers, were we also don't implement a dhcp >>> server) >>> >>> but where is the problem in having a host/vm in your network serving >>> dhcp? >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> -- >> Jos? Manuel Giner >> http://ginernet.com >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Jos? Manuel Giner http://ginernet.com From d.csapak at proxmox.com Mon Aug 20 16:28:04 2018 From: d.csapak at proxmox.com (Dominik Csapak) Date: Mon, 20 Aug 2018 16:28:04 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> Message-ID: <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> but that cannot happen without an agent/cloud init like program in the vm, so what and why (if we have cloud-init) should we implement there? On 08/20/2018 03:25 PM, Jos? Manuel Giner wrote: > I mean, when you install the ISO of an operating system on a VM, when > configuring the network, that the user can choose the DHCP option > instead of defining the values by hand. > > > > On 20/08/2018 15:21, Ian Coetzee wrote: >> Hi Jos?, >> >> Using the Qemu Agent you are able to determine the IP of the VM. >> >> Kind regards >> >> On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner wrote: >> >>> The possibility of being able to define IPs directly from the >>> Proxmox/API interface. Just like with Cloud-init or containers. >>> >>> >>> >>> On 20/08/2018 11:14, Dominik Csapak wrote: >>>> On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: >>>>> Hello, >>>>> >>>>> there any plan to implement DHCP for non cloudinit VMs? >>>>> >>>>> Thanks! >>>>> >>>>> >>>> >>>> the question does not really make sense as we did not implement dhcp >>>> for >>>> cloudinit, the config there only tells the vm how to configure its >>>> network (same as with containers, were we also don't implement a dhcp >>>> server) >>>> >>>> but where is the problem in having a host/vm in your network serving >>>> dhcp? >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >>> >>> -- >>> Jos? Manuel Giner >>> http://ginernet.com >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > From jm at ginernet.com Mon Aug 20 16:35:24 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Mon, 20 Aug 2018 16:35:24 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> Message-ID: <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> I thought cloud-init was connecting against a DHCP server. Does Proxmox have any plan to provide a DHCP service so that VMs can receive IP auto-configuration, whether they use cloud-init or not? On 20/08/2018 16:28, Dominik Csapak wrote: > but that cannot happen without an agent/cloud init like program in the > vm, so what and why (if we have cloud-init) should we implement there? > > On 08/20/2018 03:25 PM, Jos? Manuel Giner wrote: >> I mean, when you install the ISO of an operating system on a VM, when >> configuring the network, that the user can choose the DHCP option >> instead of defining the values by hand. >> >> >> >> On 20/08/2018 15:21, Ian Coetzee wrote: >>> Hi Jos?, >>> >>> Using the Qemu Agent you are able to determine the IP of the VM. >>> >>> Kind regards >>> >>> On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner wrote: >>> >>>> The possibility of being able to define IPs directly from the >>>> Proxmox/API interface. Just like with Cloud-init or containers. >>>> >>>> >>>> >>>> On 20/08/2018 11:14, Dominik Csapak wrote: >>>>> On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: >>>>>> Hello, >>>>>> >>>>>> there any plan to implement DHCP for non cloudinit VMs? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>> >>>>> the question does not really make sense as we did not implement >>>>> dhcp for >>>>> cloudinit, the config there only tells the vm how to configure its >>>>> network (same as with containers, were we also don't implement a dhcp >>>>> server) >>>>> >>>>> but where is the problem in having a host/vm in your network serving >>>>> dhcp? >>>>> >>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user at pve.proxmox.com >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>>> >>>> -- >>>> Jos? Manuel Giner >>>> http://ginernet.com >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> >> > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Jos? Manuel Giner http://ginernet.com From viniciuschagas2 at gmail.com Tue Aug 21 00:16:12 2018 From: viniciuschagas2 at gmail.com (Vinicius Barreto) Date: Mon, 20 Aug 2018 19:16:12 -0300 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: Message-ID: Hello please could you tell which command what qemu agent do you use to configure the IP settings of the VM? Thank you! Em seg, 20 de ago de 2018 ?s 10:22, Ian Coetzee escreveu: > Hi Jos?, > > Using the Qemu Agent you are able to determine the IP of the VM. > > Kind regards > > On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner wrote: > > > The possibility of being able to define IPs directly from the > > Proxmox/API interface. Just like with Cloud-init or containers. > > > > > > > > On 20/08/2018 11:14, Dominik Csapak wrote: > > > On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: > > >> Hello, > > >> > > >> there any plan to implement DHCP for non cloudinit VMs? > > >> > > >> Thanks! > > >> > > >> > > > > > > the question does not really make sense as we did not implement dhcp > for > > > cloudinit, the config there only tells the vm how to configure its > > > network (same as with containers, were we also don't implement a dhcp > > > server) > > > > > > but where is the problem in having a host/vm in your network serving > > > dhcp? > > > > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > -- > > Jos? Manuel Giner > > http://ginernet.com > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- *Marcos Vin?cius Barreto das Chagas* From mark at openvs.co.uk Tue Aug 21 01:21:46 2018 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 21 Aug 2018 00:21:46 +0100 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> Message-ID: Just install your own DHCP server on proxmox if you want. I don't see this as a feature many people would want, as in any "normal" network you always have a dhcp server already? On 20 August 2018 at 15:35, Jos? Manuel Giner wrote: > I thought cloud-init was connecting against a DHCP server. > > Does Proxmox have any plan to provide a DHCP service so that VMs can > receive IP auto-configuration, whether they use cloud-init or not? > > > > > > On 20/08/2018 16:28, Dominik Csapak wrote: > >> but that cannot happen without an agent/cloud init like program in the >> vm, so what and why (if we have cloud-init) should we implement there? >> >> On 08/20/2018 03:25 PM, Jos? Manuel Giner wrote: >> >>> I mean, when you install the ISO of an operating system on a VM, when >>> configuring the network, that the user can choose the DHCP option instead >>> of defining the values by hand. >>> >>> >>> >>> On 20/08/2018 15:21, Ian Coetzee wrote: >>> >>>> Hi Jos?, >>>> >>>> Using the Qemu Agent you are able to determine the IP of the VM. >>>> >>>> Kind regards >>>> >>>> On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner >>>> wrote: >>>> >>>> The possibility of being able to define IPs directly from the >>>>> Proxmox/API interface. Just like with Cloud-init or containers. >>>>> >>>>> >>>>> >>>>> On 20/08/2018 11:14, Dominik Csapak wrote: >>>>> >>>>>> On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> there any plan to implement DHCP for non cloudinit VMs? >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>> >>>>>> the question does not really make sense as we did not implement dhcp >>>>>> for >>>>>> cloudinit, the config there only tells the vm how to configure its >>>>>> network (same as with containers, were we also don't implement a dhcp >>>>>> server) >>>>>> >>>>>> but where is the problem in having a host/vm in your network serving >>>>>> dhcp? >>>>>> >>>>>> _______________________________________________ >>>>>> pve-user mailing list >>>>>> pve-user at pve.proxmox.com >>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>>> >>>>> >>>>> >>>>> -- >>>>> Jos? Manuel Giner >>>>> http://ginernet.com >>>>> >>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user at pve.proxmox.com >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>> >>>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>>> >>> >>> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > > -- > Jos? Manuel Giner > http://ginernet.com > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mark at openvs.co.uk Tue Aug 21 01:23:38 2018 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 21 Aug 2018 00:23:38 +0100 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: Message-ID: Maybe lost in translation? He said "determine" not "configure". That means installing qemu agent in the guest will allow the proxmox interface to show you what IP it is using. On 20 August 2018 at 23:16, Vinicius Barreto wrote: > Hello please could you tell which command what qemu agent do you use to > configure the IP settings of the VM? > > Thank you! > > Em seg, 20 de ago de 2018 ?s 10:22, Ian Coetzee > > escreveu: > > > Hi Jos?, > > > > Using the Qemu Agent you are able to determine the IP of the VM. > > > > Kind regards > > > > On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner wrote: > > > > > The possibility of being able to define IPs directly from the > > > Proxmox/API interface. Just like with Cloud-init or containers. > > > > > > > > > > > > On 20/08/2018 11:14, Dominik Csapak wrote: > > > > On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: > > > >> Hello, > > > >> > > > >> there any plan to implement DHCP for non cloudinit VMs? > > > >> > > > >> Thanks! > > > >> > > > >> > > > > > > > > the question does not really make sense as we did not implement dhcp > > for > > > > cloudinit, the config there only tells the vm how to configure its > > > > network (same as with containers, were we also don't implement a dhcp > > > > server) > > > > > > > > but where is the problem in having a host/vm in your network serving > > > > dhcp? > > > > > > > > _______________________________________________ > > > > pve-user mailing list > > > > pve-user at pve.proxmox.com > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > > > -- > > > Jos? Manuel Giner > > > http://ginernet.com > > > > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > -- > *Marcos Vin?cius Barreto das Chagas* > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From jm at ginernet.com Tue Aug 21 08:18:14 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Tue, 21 Aug 2018 08:18:14 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> Message-ID: It's not that simple when IP allocation depends on the VM MAC addres, if you have a few VMs it may not be a problem, but hosting companies that handle thousands of VMs do not want to assign any random IP to a VPS and would be better to have a native integration, at least at the API level. On 21/08/2018 1:21, Mark Adams wrote: > Just install your own DHCP server on proxmox if you want. I don't see this > as a feature many people would want, as in any "normal" network you always > have a dhcp server already? > > On 20 August 2018 at 15:35, Jos? Manuel Giner wrote: > >> I thought cloud-init was connecting against a DHCP server. >> >> Does Proxmox have any plan to provide a DHCP service so that VMs can >> receive IP auto-configuration, whether they use cloud-init or not? >> >> >> >> >> >> On 20/08/2018 16:28, Dominik Csapak wrote: >> >>> but that cannot happen without an agent/cloud init like program in the >>> vm, so what and why (if we have cloud-init) should we implement there? >>> >>> On 08/20/2018 03:25 PM, Jos? Manuel Giner wrote: >>> >>>> I mean, when you install the ISO of an operating system on a VM, when >>>> configuring the network, that the user can choose the DHCP option instead >>>> of defining the values by hand. >>>> >>>> >>>> >>>> On 20/08/2018 15:21, Ian Coetzee wrote: >>>> >>>>> Hi Jos?, >>>>> >>>>> Using the Qemu Agent you are able to determine the IP of the VM. >>>>> >>>>> Kind regards >>>>> >>>>> On Mon, 20 Aug 2018 at 13:43, Jos? Manuel Giner >>>>> wrote: >>>>> >>>>> The possibility of being able to define IPs directly from the >>>>>> Proxmox/API interface. Just like with Cloud-init or containers. >>>>>> >>>>>> >>>>>> >>>>>> On 20/08/2018 11:14, Dominik Csapak wrote: >>>>>> >>>>>>> On 08/20/2018 09:36 AM, Jos? Manuel Giner wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> there any plan to implement DHCP for non cloudinit VMs? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> the question does not really make sense as we did not implement dhcp >>>>>>> for >>>>>>> cloudinit, the config there only tells the vm how to configure its >>>>>>> network (same as with containers, were we also don't implement a dhcp >>>>>>> server) >>>>>>> >>>>>>> but where is the problem in having a host/vm in your network serving >>>>>>> dhcp? >>>>>>> >>>>>>> _______________________________________________ >>>>>>> pve-user mailing list >>>>>>> pve-user at pve.proxmox.com >>>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jos? Manuel Giner >>>>>> http://ginernet.com >>>>>> >>>>>> _______________________________________________ >>>>>> pve-user mailing list >>>>>> pve-user at pve.proxmox.com >>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>>> >>>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user at pve.proxmox.com >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>> >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> >> >> -- >> Jos? Manuel Giner >> http://ginernet.com >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Jos? Manuel Giner http://ginernet.com From mir at miras.org Tue Aug 21 08:27:32 2018 From: mir at miras.org (Michael Rasmussen) Date: Tue, 21 Aug 2018 08:27:32 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> Message-ID: <20180821082732.78df79f8@sleipner.datanom.net> On Tue, 21 Aug 2018 08:18:14 +0200 Jos? Manuel Giner wrote: > It's not that simple when IP allocation depends on the VM MAC addres, if you have a few VMs it may not be a problem, but hosting companies that handle thousands of VMs do not want to assign any random IP to a VPS and would be better to have a native integration, at least at the API level. > The MAC address is configurable for a VM to match a IP <-> MAC relation with fixed addresses. -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- /usr/games/fortune -es says: Hodie natus est radici frater. [ Unto the root is born a brother ] -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From jm at ginernet.com Tue Aug 21 08:51:31 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Tue, 21 Aug 2018 08:51:31 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <20180821082732.78df79f8@sleipner.datanom.net> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> Message-ID: I know that already :) and it doesn't change anything because the management difficulty still exists. Everything would be simpler with native integration. On 21/08/2018 8:27, Michael Rasmussen wrote: > On Tue, 21 Aug 2018 08:18:14 +0200 > Jos? Manuel Giner wrote: > >> It's not that simple when IP allocation depends on the VM MAC addres, if you have a few VMs it may not be a problem, but hosting companies that handle thousands of VMs do not want to assign any random IP to a VPS and would be better to have a native integration, at least at the API level. >> > The MAC address is configurable for a VM to match a IP <-> MAC relation > with fixed addresses. > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Jos? Manuel Giner http://ginernet.com From mark at tuxis.nl Tue Aug 21 09:40:55 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 21 Aug 2018 09:40:55 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> Message-ID: <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> On Tue, 2018-08-21 at 08:51 +0200, Jos? Manuel Giner wrote: > I know that already :) and it doesn't change anything because the > management difficulty still exists. > > Everything would be simpler with native integration. I disagree. As would many people. But, nothing stops you from writing your own qemu-guest-agent script to configure the IP address on the VM, I think. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From jm at ginernet.com Tue Aug 21 09:48:25 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Tue, 21 Aug 2018 09:48:25 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> Message-ID: <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> We are talking about auto-configuring the network on the VM, and therefore you cannot install the qemu-guest-agent on the VM if you do not have a network yet. On 21/08/2018 9:40, Mark Schouten wrote: > On Tue, 2018-08-21 at 08:51 +0200, Jos? Manuel Giner wrote: >> I know that already :) and it doesn't change anything because the >> management difficulty still exists. >> >> Everything would be simpler with native integration. > > I disagree. As would many people. But, nothing stops you from writing > your own qemu-guest-agent script to configure the IP address on the VM, > I think. > -- Jos? Manuel Giner http://ginernet.com From mark at tuxis.nl Tue Aug 21 09:54:08 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 21 Aug 2018 09:54:08 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> Message-ID: <358fe6fdfa9956f4c7f632cb3cdc3dbd0bef5cce.camel@tuxis.nl> On Tue, 2018-08-21 at 09:48 +0200, Jos? Manuel Giner wrote: > We are talking about auto-configuring the network on the VM, and > therefore you cannot install the qemu-guest-agent on the VM if you > do > not have a network yet. I deploy using templates, with packages required already installed om them. So this would not be an issue for me, personally. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From jm at ginernet.com Tue Aug 21 09:58:46 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Tue, 21 Aug 2018 09:58:46 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <358fe6fdfa9956f4c7f632cb3cdc3dbd0bef5cce.camel@tuxis.nl> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> <358fe6fdfa9956f4c7f632cb3cdc3dbd0bef5cce.camel@tuxis.nl> Message-ID: <19d64b5a-9c9a-8cf0-dc10-369582aa6573@ginernet.com> I mentioned at the beginning of the conversation that this is about installing ISO images. On 21/08/2018 9:54, Mark Schouten wrote: > On Tue, 2018-08-21 at 09:48 +0200, Jos? Manuel Giner wrote: >> We are talking about auto-configuring the network on the VM, and >> therefore you cannot install the qemu-guest-agent on the VM if you >> do >> not have a network yet. > > I deploy using templates, with packages required already installed om > them. So this would not be an issue for me, personally. > -- Jos? Manuel Giner http://ginernet.com From mark at openvs.co.uk Tue Aug 21 10:08:08 2018 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 21 Aug 2018 09:08:08 +0100 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> Message-ID: What sort of OS are you using for VM's, that does not default to having DHCP enabled? I personally can't think of one that isn't DHCP out of the box. As for using specific IP's based on MAC address, this would be easily set in the dhcp server config? On 21 August 2018 at 08:48, Jos? Manuel Giner wrote: > We are talking about auto-configuring the network on the VM, and therefore > you cannot install the qemu-guest-agent on the VM if you do not have a > network yet. > > > > On 21/08/2018 9:40, Mark Schouten wrote: > >> On Tue, 2018-08-21 at 08:51 +0200, Jos? Manuel Giner wrote: >> >>> I know that already :) and it doesn't change anything because the >>> management difficulty still exists. >>> >>> Everything would be simpler with native integration. >>> >> >> I disagree. As would many people. But, nothing stops you from writing >> your own qemu-guest-agent script to configure the IP address on the VM, >> I think. >> >> > > -- > Jos? Manuel Giner > http://ginernet.com > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From jm at ginernet.com Tue Aug 21 10:35:19 2018 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Tue, 21 Aug 2018 10:35:19 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> Message-ID: What I ask is that Proxmox offer us a native DHCP server integration, so that we can configure VMs as easily as in cloud-init based VMs or containers. What we want is to avoid fighting with alternative solutions: https://forum.proxmox.com/threads/dhcp-server-proxmox-and-ovh.43420/ https://www.docs.modulesgarden.com/Proxmox_VPS_For_WHMCS#Automatic_IP_Address_Assignment If we Google, we find that this request has been made many times: https://forum.proxmox.com/threads/dhcp-server.5892/ https://forum.proxmox.com/threads/proxmox-ve4-3-dhcp-setup-for-kvm-vms.30574/ and more... On 21/08/2018 10:08, Mark Adams wrote: > What sort of OS are you using for VM's, that does not default to having > DHCP enabled? I personally can't think of one that isn't DHCP out of the > box. -- Jos? Manuel Giner http://ginernet.com From lists at merit.unu.edu Tue Aug 21 12:18:20 2018 From: lists at merit.unu.edu (lists) Date: Tue, 21 Aug 2018 12:18:20 +0200 Subject: [PVE-User] removing vm snapshot (on ceph) fails Message-ID: <335dde85-452a-d90a-10ef-410899979f53@merit.unu.edu> Hi, We were trying to remove some old snapshots from our proxmox install, and we're getting the following task error: > TASK ERROR: VM 103 qmp command 'delete-drive-snapshot' failed - Device 'drive-virtio0' not found This is also true: the disk is not configured as virtio0, but as scsi0. Probably when the snapshot was created, the disk was virtio0. But I would like to remove this old snapshot anyway. :-) Ceph image deletion itself appears to have been successful: "Removing image: 100% complete...done." but the snapshot still appears in the proxmox gui, and now my VM is locked (snapshot-delete) Suggestions how to proceed? MJ From ralf.storm at konzept-is.de Tue Aug 21 12:55:40 2018 From: ralf.storm at konzept-is.de (Ralf Storm) Date: Tue, 21 Aug 2018 12:55:40 +0200 Subject: [PVE-User] removing vm snapshot (on ceph) fails In-Reply-To: <335dde85-452a-d90a-10ef-410899979f53@merit.unu.edu> References: <335dde85-452a-d90a-10ef-410899979f53@merit.unu.edu> Message-ID: <9900003d-0449-3fd5-0646-0d3c9eaaf25b@konzept-is.de> manually remove the snapshot and the lock from the config - file and proceed > Hi, > > We were trying to remove some old snapshots from our proxmox install, > and we're getting the following task error: > >> TASK ERROR: VM 103 qmp command 'delete-drive-snapshot' failed - >> Device 'drive-virtio0' not found > > This is also true: the disk is not configured as virtio0, but as > scsi0. Probably when the snapshot was created, the disk was virtio0. > > But I would like to remove this old snapshot anyway. :-) > > Ceph image deletion itself appears to have been successful: "Removing > image: 100% complete...done." but the snapshot still appears in the > proxmox gui, and now my VM is locked (snapshot-delete) > > Suggestions how to proceed? > > MJ > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mark at tuxis.nl Tue Aug 21 13:14:28 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 21 Aug 2018 13:14:28 +0200 Subject: [PVE-User] removing vm snapshot (on ceph) fails In-Reply-To: <9900003d-0449-3fd5-0646-0d3c9eaaf25b@konzept-is.de> References: <335dde85-452a-d90a-10ef-410899979f53@merit.unu.edu> <9900003d-0449-3fd5-0646-0d3c9eaaf25b@konzept-is.de> Message-ID: On Tue, 2018-08-21 at 12:55 +0200, Ralf Storm wrote: > manually remove the snapshot and the lock from the config - file and > proceed Yes. Also, check if Ceph has no snapshots left that you think you deleted :) -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From lists at merit.unu.edu Tue Aug 21 13:38:18 2018 From: lists at merit.unu.edu (lists) Date: Tue, 21 Aug 2018 13:38:18 +0200 Subject: [PVE-User] removing vm snapshot (on ceph) fails In-Reply-To: References: <335dde85-452a-d90a-10ef-410899979f53@merit.unu.edu> <9900003d-0449-3fd5-0646-0d3c9eaaf25b@konzept-is.de> Message-ID: <25d999b8-a27d-9a33-77db-c8fe2be4f73a@merit.unu.edu> Hi Ralf and Mark, Yes that worked. Thanks! On 21-8-2018 13:14, Mark Schouten wrote: > On Tue, 2018-08-21 at 12:55 +0200, Ralf Storm wrote: >> manually remove the snapshot and the lock from the config - file and >> proceed > > Yes. Also, check if Ceph has no snapshots left that you think you > deleted :) > From gilberto.nunes32 at gmail.com Tue Aug 21 21:01:01 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Tue, 21 Aug 2018 16:01:01 -0300 Subject: [PVE-User] Mainline kernel Message-ID: Hi there Can I download a kernel from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/ And use it with proxmox? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From uwe.sauter.de at gmail.com Wed Aug 22 09:48:19 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Wed, 22 Aug 2018 09:48:19 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) Message-ID: Hi all, some quick questions: * As far as I can tell the PVE kernel is a modified version of Ubuntu kernels, correct? Modifications can be viewed in the pve-kernel.git repository (https://git.proxmox.com/?p=pve-kernel.git;a=tree). * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? Thanks, Uwe From t.lamprecht at proxmox.com Wed Aug 22 09:55:10 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 22 Aug 2018 09:55:10 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: References: Message-ID: Hi Uwe, On 8/22/18 9:48 AM, Uwe Sauter wrote: > Hi all, > > some quick questions: > > * As far as I can tell the PVE kernel is a modified version of Ubuntu kernels, correct? > Modifications can be viewed in the pve-kernel.git repository ( https://git.proxmox.com/?p=pve-kernel.git;a=tree ). > Yes, in the respective git branch (master is currently 4.13 and pve-kernel-4.15 is, you guess it, 4.15 patches/ includes on-top bug/security fixes also some out of tree modules get included (ZFS, igb, e1000e) > * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? > Yes. (Note that this may not get much updates anymore) > * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? > Yes. We're normally on the latest stable release tagged on the master branch. cheers, Thomas From marcus.haarmann at midoco.de Wed Aug 22 09:57:02 2018 From: marcus.haarmann at midoco.de (Marcus Haarmann) Date: Wed, 22 Aug 2018 09:57:02 +0200 (CEST) Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: References: Message-ID: <1761885320.6470076.1534924622698.JavaMail.zimbra@midoco.de> Hi, yes we tried to work with a standard kernel (Debian) and this was a very wrong decision. All the CT did not work any more.... You need to apply the patches. The process of compiling a new one is documented, but you will need some time and lots of additional packages to do this. (and of course some programming knowledge) Marcus Haarmann Von: "Uwe Sauter" An: "pve-user" Gesendet: Mittwoch, 22. August 2018 09:48:19 Betreff: [PVE-User] PVE kernel (sources, build process, etc.) Hi all, some quick questions: * As far as I can tell the PVE kernel is a modified version of Ubuntu kernels, correct? Modifications can be viewed in the pve-kernel.git repository (https://git.proxmox.com/?p=pve-kernel.git;a=tree). * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? Thanks, Uwe _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From uwe.sauter.de at gmail.com Wed Aug 22 09:58:53 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Wed, 22 Aug 2018 09:58:53 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: References: Message-ID: Hi Thomas, Am 22.08.18 um 09:55 schrieb Thomas Lamprecht: > Hi Uwe, > > On 8/22/18 9:48 AM, Uwe Sauter wrote: >> Hi all, >> >> some quick questions: >> >> * As far as I can tell the PVE kernel is a modified version of Ubuntu kernels, correct? >> Modifications can be viewed in the pve-kernel.git repository ( https://git.proxmox.com/?p=pve-kernel.git;a=tree ). >> > > Yes, in the respective git branch (master is currently 4.13 and pve-kernel-4.15 is, you guess it, 4.15 > patches/ includes on-top bug/security fixes also some out of tree modules get included (ZFS, igb, e1000e) I'm mostly interested in the myri10ge driver right now. From what I can tell, you do ship this particular driver without modification? > >> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? >> > > Yes. (Note that this may not get much updates anymore) > >> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? >> > > Yes. We're normally on the latest stable release tagged on the master branch. > I'll checkout both and compare the myri10ge drivers? Thanks, Uwe > cheers, > Thomas > From uwe.sauter.de at gmail.com Wed Aug 22 10:02:38 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Wed, 22 Aug 2018 10:02:38 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: <1761885320.6470076.1534924622698.JavaMail.zimbra@midoco.de> References: <1761885320.6470076.1534924622698.JavaMail.zimbra@midoco.de> Message-ID: <251bd3eb-3275-730f-c409-0477512621eb@gmail.com> Marcus, thanks. Hopefully it won't come as far as compiling my own pve-kernel package. I'm currently hunting a networking issue and need to compare driver versions? Regards, Uwe Am 22.08.18 um 09:57 schrieb Marcus Haarmann: > Hi, > > yes we tried to work with a standard kernel (Debian) and this was a very wrong decision. > All the CT did not work any more.... > You need to apply the patches. The process of compiling a new one is documented, > but you will need some time and lots of additional packages to do this. > (and of course some programming knowledge) > > Marcus Haarmann > > ---------------------------------------------------------------------------------------------------------------------------------- > *Von: *"Uwe Sauter" > *An: *"pve-user" > *Gesendet: *Mittwoch, 22. August 2018 09:48:19 > *Betreff: *[PVE-User] PVE kernel (sources, build process, etc.) > > Hi all, > > some quick questions: > > * As far as I can tell the PVE kernel is a modified version of Ubuntu kernels, correct? > ??Modifications can be viewed in the pve-kernel.git repository (https://git.proxmox.com/?p=pve-kernel.git;a=tree). > > * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? > > * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? > > > Thanks, > > ????????Uwe > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From t.lamprecht at proxmox.com Wed Aug 22 10:09:22 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 22 Aug 2018 10:09:22 +0200 Subject: [PVE-User] Mainline kernel In-Reply-To: References: Message-ID: Hi, On 8/21/18 9:01 PM, Gilberto Nunes wrote: > Hi there > > Can I download a kernel from here: > > http://kernel.ubuntu.com/~kernel-ppa/mainline/ > > And use it with proxmox? > You can, just install the .deb with dpkg, but you won't have ZFS and a few other things included. It may work but your totally on your own, as Proxmox VE is only tested and tailored to our latest shipped kernel(s). You may get a fix not yet backported but you also may run into new bugs. I heavily advise against doing this in production, at least not until a lot of testing on similar test setups. From t.lamprecht at proxmox.com Wed Aug 22 10:24:15 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 22 Aug 2018 10:24:15 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: References: Message-ID: <704a770d-9e9a-f35b-4c35-a0e16d974e97@proxmox.com> On 8/22/18 9:58 AM, Uwe Sauter wrote: > Am 22.08.18 um 09:55 schrieb Thomas Lamprecht: >> On 8/22/18 9:48 AM, Uwe Sauter wrote: >>> Hi all, >>> >>> some quick questions: >>> >>> * As far as I can tell the PVE kernel is a modified version of Ubuntu kernels, correct? >>> Modifications can be viewed in the pve-kernel.git repository ( https://git.proxmox.com/?p=pve-kernel.git;a=tree ). >>> >> >> Yes, in the respective git branch (master is currently 4.13 and pve-kernel-4.15 is, you guess it, 4.15 >> patches/ includes on-top bug/security fixes also some out of tree modules get included (ZFS, igb, e1000e) > > I'm mostly interested in the myri10ge driver right now. From what I can tell, you do ship this particular driver without modification? > You're right, no modifications of this module. # modinfo myri10ge filename: /lib/modules/4.15.18-2-pve/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko firmware: myri10ge_rss_eth_z8e.dat firmware: myri10ge_rss_ethp_z8e.dat firmware: myri10ge_eth_z8e.dat firmware: myri10ge_ethp_z8e.dat license: Dual BSD/GPL version: 1.5.3-1.534 author: Maintainer: help at myri.com description: Myricom 10G driver (10GbE) srcversion: 46526E4E4E82667CBFF2D7C alias: pci:v000014C1d00000009sv*sd*bc*sc*i* alias: pci:v000014C1d00000008sv*sd*bc*sc*i* depends: dca retpoline: Y intree: Y name: myri10ge vermagic: 4.15.18-2-pve SMP mod_unload modversions parm: myri10ge_fw_name:Firmware image name (charp) parm: myri10ge_fw_names:Firmware image names per board (array of charp) parm: myri10ge_ecrc_enable:Enable Extended CRC on PCI-E (int) parm: myri10ge_small_bytes:Threshold of small packets (int) parm: myri10ge_msi:Enable Message Signalled Interrupts (int) parm: myri10ge_intr_coal_delay:Interrupt coalescing delay (int) parm: myri10ge_flow_control:Pause parameter (int) parm: myri10ge_deassert_wait:Wait when deasserting legacy interrupts (int) parm: myri10ge_force_firmware:Force firmware to assume aligned completions (int) parm: myri10ge_initial_mtu:Initial MTU (int) parm: myri10ge_napi_weight:Set NAPI weight (int) parm: myri10ge_watchdog_timeout:Set watchdog timeout (int) parm: myri10ge_max_irq_loops:Set stuck legacy IRQ detection threshold (int) parm: myri10ge_debug:Debug level (0=none,...,16=all) (int) parm: myri10ge_fill_thresh:Number of empty rx slots allowed (int) parm: myri10ge_max_slices:Max tx/rx queues (int) parm: myri10ge_rss_hash:Type of RSS hashing to do (int) parm: myri10ge_dca:Enable DCA if possible (int) > >> >>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? >>> >> >> Yes. (Note that this may not get much updates anymore) >> >>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? >>> >> >> Yes. We're normally on the latest stable release tagged on the master branch. >> > > I'll checkout both and compare the myri10ge drivers? > What's your exact issue, if I may ask? cheers, Thomas From uwe.sauter.de at gmail.com Wed Aug 22 10:50:19 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Wed, 22 Aug 2018 10:50:19 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: <704a770d-9e9a-f35b-4c35-a0e16d974e97@proxmox.com> References: <704a770d-9e9a-f35b-4c35-a0e16d974e97@proxmox.com> Message-ID: <679df401-dccb-9cab-da34-4a198b1fa615@gmail.com> >>> >>>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? >>>> >>> >>> Yes. (Note that this may not get much updates anymore) >>> >>>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? >>>> >>> >>> Yes. We're normally on the latest stable release tagged on the master branch. >>> >> >> I'll checkout both and compare the myri10ge drivers? >> > > What's your exact issue, if I may ask? > > cheers, > Thomas > > > Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those are gone. Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs. Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well. Long story is here: https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html https://marc.info/?l=ceph-devel&m=153449419830984&w=2 From marcus.haarmann at midoco.de Wed Aug 22 11:08:20 2018 From: marcus.haarmann at midoco.de (Marcus Haarmann) Date: Wed, 22 Aug 2018 11:08:20 +0200 (CEST) Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: <679df401-dccb-9cab-da34-4a198b1fa615@gmail.com> References: <704a770d-9e9a-f35b-4c35-a0e16d974e97@proxmox.com> <679df401-dccb-9cab-da34-4a198b1fa615@gmail.com> Message-ID: <898309527.6474207.1534928900759.JavaMail.zimbra@midoco.de> Hi, did you try this: (taken from the ceph list) This is PTI, I think.? Try to add "noibrs noibpb nopti nospectre_v2" to kernel cmdline and reboot. Did this make a difference ? We are struggeling with proxmox/ceph too and we have the suspect that it is kernel related or network related, but could not narrow it down to a specific reason. But the effects are different... We encountered stuck I/O on rdb devices. And kernel says it is losing a mon connection and hunting for a new mon all the time (when backup takes place and heavy I/O is done). Marcus Haarmann Von: "uwe sauter de" An: "Thomas Lamprecht" , "pve-user" Gesendet: Mittwoch, 22. August 2018 10:50:19 Betreff: Re: [PVE-User] PVE kernel (sources, build process, etc.) >>> >>>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? >>>> >>> >>> Yes. (Note that this may not get much updates anymore) >>> >>>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? >>>> >>> >>> Yes. We're normally on the latest stable release tagged on the master branch. >>> >> >> I'll checkout both and compare the myri10ge drivers? >> > > What's your exact issue, if I may ask? > > cheers, > Thomas > > > Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those are gone. Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs. Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well. Long story is here: https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html https://marc.info/?l=ceph-devel&m=153449419830984&w=2 _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From uwe.sauter.de at gmail.com Wed Aug 22 11:45:15 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Wed, 22 Aug 2018 11:45:15 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: <898309527.6474207.1534928900759.JavaMail.zimbra@midoco.de> References: <704a770d-9e9a-f35b-4c35-a0e16d974e97@proxmox.com> <679df401-dccb-9cab-da34-4a198b1fa615@gmail.com> <898309527.6474207.1534928900759.JavaMail.zimbra@midoco.de> Message-ID: Hi Marcus, no, I haven't disabled Spectre/Meltdown mitigations (yet). For 4 of my hosts I have the Intel microcode from 2018-05-08 running which is the most up-to-date version from Ubuntu. Using the spectre-meltdown-checker.sh script from https://github.com/speed47/spectre-meltdown-checker gives below output which let's me believe that performace should be goot with mitigations enabled. #### output for sandybridge based hosts #### CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) * Kernel has the Red Hat/Ubuntu patch: NO * Kernel has mask_nospec64 (arm64): NO > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW) * Mitigation 1 * Kernel is compiled with IBRS support: YES * IBRS enabled and active: YES (for kernel and firmware code) * Kernel is compiled with IBPB support: YES * IBPB enabled and active: YES * Mitigation 2 * Kernel has branch predictor hardening (arm): NO * Kernel compiled with retpoline option: YES * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) > STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' * Mitigated according to the /sys interface: YES (Mitigation: PTI) * Kernel supports Page Table Isolation (PTI): YES * PTI enabled and active: YES * Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced) * Running as a Xen PV DomU: NO > STATUS: NOT VULNERABLE (Mitigation: PTI) CVE-2018-3640 [rogue system register read] aka 'Variant 3a' * CPU microcode mitigates the vulnerability: YES > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) CVE-2018-3639 [speculative store bypass] aka 'Variant 4' * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) * Kernel supports speculation store bypass: YES (found in /proc/self/status) > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG' * Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable) > STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable) ##### end ##### The other 2 hosts are base on Westmere architecture and won't get microcode updates as fas as I know. Disabling mitigations on these hosts might be worth a try. Same script from above gives: #### output of westmere based hosts #### CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) * Kernel has the Red Hat/Ubuntu patch: NO * Kernel has mask_nospec64 (arm64): NO > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline) * Mitigation 1 * Kernel is compiled with IBRS support: YES * IBRS enabled and active: NO * Kernel is compiled with IBPB support: YES * IBPB enabled and active: NO * Mitigation 2 * Kernel has branch predictor hardening (arm): NO * Kernel compiled with retpoline option: YES * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) > STATUS: NOT VULNERABLE (Full retpoline is mitigating the vulnerability) IBPB is considered as a good addition to retpoline for Variant 2 mitigation, but your CPU microcode doesn't support it CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' * Mitigated according to the /sys interface: YES (Mitigation: PTI) * Kernel supports Page Table Isolation (PTI): YES * PTI enabled and active: YES * Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced) * Running as a Xen PV DomU: NO > STATUS: NOT VULNERABLE (Mitigation: PTI) CVE-2018-3640 [rogue system register read] aka 'Variant 3a' * CPU microcode mitigates the vulnerability: NO > STATUS: VULNERABLE (an up-to-date CPU microcode is needed to mitigate this vulnerability) CVE-2018-3639 [speculative store bypass] aka 'Variant 4' * Mitigated according to the /sys interface: NO (Vulnerable) * Kernel supports speculation store bypass: YES (found in /proc/self/status) > STATUS: VULNERABLE (Your CPU doesn't support SSBD) CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG' * Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled) > STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled) ##### end ##### Am 22.08.18 um 11:08 schrieb Marcus Haarmann: > Hi, > did you try this: > (taken from the ceph list) > > This is PTI, I think.? Try to add "noibrs noibpb nopti nospectre_v2" to > kernel cmdline and reboot. > > > Did this make a difference ? > We are struggeling with proxmox/ceph too and we have the suspect that it is kernel related or network related, > but could not narrow it down to a specific reason. > But the effects are different... We encountered stuck I/O on rdb devices. > And kernel says it is losing a mon connection and hunting for a new mon all the time (when backup takes > place and heavy I/O is done). > > Marcus Haarmann > > ---------------------------------------------------------------------------------------------------------------------------------- > *Von: *"uwe sauter de" > *An: *"Thomas Lamprecht" , "pve-user" > *Gesendet: *Mittwoch, 22. August 2018 10:50:19 > *Betreff: *Re: [PVE-User] PVE kernel (sources, build process, etc.) > >>>> >>>>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? >>>>> >>>> >>>> Yes. (Note that this may not get much updates anymore) >>>> >>>>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? >>>>> >>>> >>>> Yes. We're normally on the latest stable release tagged on the master branch. >>>> >>> >>> I'll checkout both and compare the myri10ge drivers? >>> >> >> What's your exact issue, if I may ask? >> >> cheers, >> Thomas >> >> >> > > Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those > are gone. > > Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs. > > > Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different > settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well. > > > > Long story is here: > > https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html > > https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html > > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html > > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html > > https://marc.info/?l=ceph-devel&m=153449419830984&w=2 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From uwe.sauter.de at gmail.com Wed Aug 22 14:14:48 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Wed, 22 Aug 2018 14:14:48 +0200 Subject: [PVE-User] PVE kernel (sources, build process, etc.) In-Reply-To: References: <704a770d-9e9a-f35b-4c35-a0e16d974e97@proxmox.com> <679df401-dccb-9cab-da34-4a198b1fa615@gmail.com> <898309527.6474207.1534928900759.JavaMail.zimbra@midoco.de> Message-ID: One thing speaks againts this being PTI is that both types of nodes have secondary OSDs causing slow requests. Though it still is an option to try before giving up completely. Am 22.08.18 um 11:45 schrieb Uwe Sauter: > Hi Marcus, > > no, I haven't disabled Spectre/Meltdown mitigations (yet). > > For 4 of my hosts I have the Intel microcode from 2018-05-08 running which is the most up-to-date version from Ubuntu. > > Using the spectre-meltdown-checker.sh script from https://github.com/speed47/spectre-meltdown-checker gives below output which > let's me believe that performace should be goot with mitigations enabled. > > #### output for sandybridge based hosts #### > CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' > * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) > * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) > * Kernel has the Red Hat/Ubuntu patch: NO > * Kernel has mask_nospec64 (arm64): NO >> STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) > > CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' > * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW) > * Mitigation 1 > * Kernel is compiled with IBRS support: YES > * IBRS enabled and active: YES (for kernel and firmware code) > * Kernel is compiled with IBPB support: YES > * IBPB enabled and active: YES > * Mitigation 2 > * Kernel has branch predictor hardening (arm): NO > * Kernel compiled with retpoline option: YES > * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) >> STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) > > CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' > * Mitigated according to the /sys interface: YES (Mitigation: PTI) > * Kernel supports Page Table Isolation (PTI): YES > * PTI enabled and active: YES > * Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced) > * Running as a Xen PV DomU: NO >> STATUS: NOT VULNERABLE (Mitigation: PTI) > > CVE-2018-3640 [rogue system register read] aka 'Variant 3a' > * CPU microcode mitigates the vulnerability: YES >> STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) > > CVE-2018-3639 [speculative store bypass] aka 'Variant 4' > * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) > * Kernel supports speculation store bypass: YES (found in /proc/self/status) >> STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) > > CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG' > * Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable) >> STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable) > > > ##### end ##### > > > The other 2 hosts are base on Westmere architecture and won't get microcode updates as fas as I know. Disabling mitigations on > these hosts might be worth a try. Same script from above gives: > > #### output of westmere based hosts #### > CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' > * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) > * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) > * Kernel has the Red Hat/Ubuntu patch: NO > * Kernel has mask_nospec64 (arm64): NO >> STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) > > CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' > * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline) > * Mitigation 1 > * Kernel is compiled with IBRS support: YES > * IBRS enabled and active: NO > * Kernel is compiled with IBPB support: YES > * IBPB enabled and active: NO > * Mitigation 2 > * Kernel has branch predictor hardening (arm): NO > * Kernel compiled with retpoline option: YES > * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) >> STATUS: NOT VULNERABLE (Full retpoline is mitigating the vulnerability) > IBPB is considered as a good addition to retpoline for Variant 2 mitigation, but your CPU microcode doesn't support it > > CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' > * Mitigated according to the /sys interface: YES (Mitigation: PTI) > * Kernel supports Page Table Isolation (PTI): YES > * PTI enabled and active: YES > * Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced) > * Running as a Xen PV DomU: NO >> STATUS: NOT VULNERABLE (Mitigation: PTI) > > CVE-2018-3640 [rogue system register read] aka 'Variant 3a' > * CPU microcode mitigates the vulnerability: NO >> STATUS: VULNERABLE (an up-to-date CPU microcode is needed to mitigate this vulnerability) > > CVE-2018-3639 [speculative store bypass] aka 'Variant 4' > * Mitigated according to the /sys interface: NO (Vulnerable) > * Kernel supports speculation store bypass: YES (found in /proc/self/status) >> STATUS: VULNERABLE (Your CPU doesn't support SSBD) > > CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG' > * Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled) >> STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled) > > ##### end ##### > > > > > > > Am 22.08.18 um 11:08 schrieb Marcus Haarmann: >> Hi, >> did you try this: >> (taken from the ceph list) >> >> This is PTI, I think.? Try to add "noibrs noibpb nopti nospectre_v2" to >> kernel cmdline and reboot. >> >> >> Did this make a difference ? >> We are struggeling with proxmox/ceph too and we have the suspect that it is kernel related or network related, >> but could not narrow it down to a specific reason. >> But the effects are different... We encountered stuck I/O on rdb devices. >> And kernel says it is losing a mon connection and hunting for a new mon all the time (when backup takes >> place and heavy I/O is done). >> >> Marcus Haarmann >> >> ---------------------------------------------------------------------------------------------------------------------------------- >> *Von: *"uwe sauter de" >> *An: *"Thomas Lamprecht" , "pve-user" >> *Gesendet: *Mittwoch, 22. August 2018 10:50:19 >> *Betreff: *Re: [PVE-User] PVE kernel (sources, build process, etc.) >> >>>>> >>>>>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ? >>>>>> >>>>> >>>>> Yes. (Note that this may not get much updates anymore) >>>>> >>>>>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ? >>>>>> >>>>> >>>>> Yes. We're normally on the latest stable release tagged on the master branch. >>>>> >>>> >>>> I'll checkout both and compare the myri10ge drivers? >>>> >>> >>> What's your exact issue, if I may ask? >>> >>> cheers, >>> Thomas >>> >>> >>> >> >> Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those >> are gone. >> >> Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs. >> >> >> Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different >> settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well. >> >> >> >> Long story is here: >> >> https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html >> >> https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html >> >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html >> >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html >> >> https://marc.info/?l=ceph-devel&m=153449419830984&w=2 >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Wed Aug 22 14:33:19 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 22 Aug 2018 09:33:19 -0300 Subject: [PVE-User] SSD and Proxmox CEPH Message-ID: Hi there... I have 128GB SSD here, and wanna install Proxmox in it, and create a separated partition, lets say /dev/sda8, to make cache to CEPH? I mean, use the same SSD to install Proxmox AND a separated partition to make cache to CEPH! Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From ronny+pve-user at aasen.cx Wed Aug 22 15:31:51 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Wed, 22 Aug 2018 15:31:51 +0200 Subject: [PVE-User] Mainline kernel In-Reply-To: References: Message-ID: <87276724-2506-15e4-3dc2-79a30e0a163f@aasen.cx> On 21. aug. 2018 21:01, Gilberto Nunes wrote: > Hi there > > Can I download a kernel from here: > > http://kernel.ubuntu.com/~kernel-ppa/mainline/ > > And use it with proxmox? > > Thanks > if you need a more recent kernel you could try the ones from pvetest repo. but i would not reccomend doing that in production. kind regards Ronny Aasen From gilberto.nunes32 at gmail.com Wed Aug 22 21:04:32 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 22 Aug 2018 16:04:32 -0300 Subject: [PVE-User] Proxmox Ceph with differents HDD Size Message-ID: Hi there It's possible create a Ceph cluster with 4 servers, which has differents disk sizes: Server A - 2x 4TB Server B, C - 2x 8TB Server D - 2x 4TB This is ok? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From chance_ellis at yahoo.com Wed Aug 22 21:22:59 2018 From: chance_ellis at yahoo.com (Chance Ellis) Date: Wed, 22 Aug 2018 15:22:59 -0400 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: Message-ID: Yes, you can mix and match drive sizes on ceph. Caution: heterogeneous environments do provide challenges. You will want to set your osd weight on the 8TB drives to 2x what the 4TB drives are. In doing so, however, realize the 8TB drives will be expected to "perform" 2x as much as the 4TB drives. If the 8TB are not 2x "faster" the cluster will slow down as the 8TB drives are over worked. To resolve this phenomenon, look into primary_affinity. Primary_affinity allows you to adjust the amount of load on a disk without reducing the amount of data it can contain. References: https://ceph.com/geen-categorie/ceph-primary-affinity/ https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/html/storage_strategies/primary-affinity ?On 8/22/18, 3:05 PM, "pve-user on behalf of Gilberto Nunes" wrote: Hi there It's possible create a Ceph cluster with 4 servers, which has differents disk sizes: Server A - 2x 4TB Server B, C - 2x 8TB Server D - 2x 4TB This is ok? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From brians at iptel.co Thu Aug 23 00:19:41 2018 From: brians at iptel.co (Brian :) Date: Wed, 22 Aug 2018 23:19:41 +0100 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: Message-ID: Its really not a great idea because the larger drives will tend to get more writes so your performance won't be as good as all the same size where the writes will be distributed more evenly. On Wed, Aug 22, 2018 at 8:05 PM Gilberto Nunes wrote: > > Hi there > > > It's possible create a Ceph cluster with 4 servers, which has differents > disk sizes: > > Server A - 2x 4TB > Server B, C - 2x 8TB > Server D - 2x 4TB > > This is ok? > > Thanks > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mliebherr99 at googlemail.com Thu Aug 23 08:57:48 2018 From: mliebherr99 at googlemail.com (Ml Ml) Date: Thu, 23 Aug 2018 08:57:48 +0200 Subject: [PVE-User] periodic Node Crash/freeze Message-ID: Hello, i could need some hint/help since one cluster is letting me down since 29.07.2018 . Thats when one of my three nodes started to freeze and stop. In syslog the last entries are: Aug 21 02:33:00 node10 systemd[1]: Starting Proxmox VE replication runner... Aug 21 02:33:01 node10 systemd[1]: Started Proxmox VE replication runner. Aug 21 02:33:01 node10 CRON[1870491]: (root) CMD (/usr/bin/puppet agent -vt --color false --logdest /var/log/puppet/agent.log 1>/dev/null) ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ or: Aug 22 16:11:12 node08 pmxcfs[5227]: [dcdb] notice: cpg_send_message retried 1 times Aug 22 16:11:12 node08 pmxcfs[5227]: [status] notice: members: 1/5227, 2/5058 Aug 22 16:11:12 node08 pmxcfs[5227]: [status] notice: starting data syncronisation ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ I already posted it here: https://forum.proxmox.com/threads/periodic-node-crash-freeze.46407/ It happened at: 29.07.2018 node09 / pve 4.4 07.08.2018 node08 / pve 4.4 ( then i decided to upgrade) 21.08.2018 node10 / pve 5.2 22.08.2018 node08 / pve 5.2 ...and i am getting nervous now since there are 60 important VMs on it. As you can see it happened across multiple nodes with diffrent PVE Versions. Memtest is okay. As far as i googled the "^@^@^@^@^@^" appear is syslog because i can not fully write the file to disk? Maybe something triggers some totem/watchdog stuff which then ends in a disaster? My Ideas from here: - disable corosync/totem and see if the problems stop Have you any ideas which could narrow my problem down? My Setup is a 3 Node Cluster (node08, node09, node10) with ceph. I have 4 other 3-NodeCluster running just fine. Thanks a lot. Mario From ken.woods at alaska.gov Thu Aug 23 12:19:32 2018 From: ken.woods at alaska.gov (Woods, Ken A (DNR)) Date: Thu, 23 Aug 2018 10:19:32 +0000 Subject: [PVE-User] periodic Node Crash/freeze In-Reply-To: References: Message-ID: <122D7934-F47D-47AD-8A42-6CA230946B0D@alaska.gov> Why did you decide to not use multicast? > On Aug 22, 2018, at 22:58, Ml Ml wrote: > > Hello, > > i could need some hint/help since one cluster is letting me down since > 29.07.2018 . > Thats when one of my three nodes started to freeze and stop. > > In syslog the last entries are: > > Aug 21 02:33:00 node10 systemd[1]: Starting Proxmox VE replication runner... > Aug 21 02:33:01 node10 systemd[1]: Started Proxmox VE replication runner. > Aug 21 02:33:01 node10 CRON[1870491]: (root) CMD (/usr/bin/puppet > agent -vt --color false --logdest /var/log/puppet/agent.log > 1>/dev/null) > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ > > > or: > > Aug 22 16:11:12 node08 pmxcfs[5227]: [dcdb] notice: cpg_send_message > retried 1 times > Aug 22 16:11:12 node08 pmxcfs[5227]: [status] notice: members: 1/5227, 2/5058 > Aug 22 16:11:12 node08 pmxcfs[5227]: [status] notice: starting data > syncronisation > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > > I already posted it here: > https://urldefense.proofpoint.com/v2/url?u=https-3A__forum.proxmox.com_threads_periodic-2Dnode-2Dcrash-2Dfreeze.46407_&d=DwIGaQ&c=teXCf5DW4bHgLDM-H5_GmQ&r=THf3d3FQjCY5FQHo3goSprNAh9vsOWPUM7J0jwvvVwM&m=zpOdKmRPAro1hJw-CO0lkGqmzXn8fQ4Ye5aJvsC8lbk&s=fRGRq_-sMJvikzFr6peWj3oZxkZ5eHY434Re48Mv9mI&e= > > It happened at: > 29.07.2018 node09 / pve 4.4 > 07.08.2018 node08 / pve 4.4 ( then i decided to upgrade) > 21.08.2018 node10 / pve 5.2 > 22.08.2018 node08 / pve 5.2 > > ...and i am getting nervous now since there are 60 important VMs on it. > As you can see it happened across multiple nodes with diffrent PVE Versions. > > Memtest is okay. > > As far as i googled the "^@^@^@^@^@^" appear is syslog because i can > not fully write the file to disk? > > Maybe something triggers some totem/watchdog stuff which then ends in > a disaster? > > My Ideas from here: > - disable corosync/totem and see if the problems stop > > Have you any ideas which could narrow my problem down? > > > My Setup is a 3 Node Cluster (node08, node09, node10) with ceph. > > I have 4 other 3-NodeCluster running just fine. > > Thanks a lot. > > Mario > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://urldefense.proofpoint.com/v2/url?u=https-3A__pve.proxmox.com_cgi-2Dbin_mailman_listinfo_pve-2Duser&d=DwIGaQ&c=teXCf5DW4bHgLDM-H5_GmQ&r=THf3d3FQjCY5FQHo3goSprNAh9vsOWPUM7J0jwvvVwM&m=zpOdKmRPAro1hJw-CO0lkGqmzXn8fQ4Ye5aJvsC8lbk&s=8K2XEB3Soz8V0JMR6hzvc78bjDExInI2vC2LC_FfljI&e= From klaus.mailinglists at pernau.at Thu Aug 23 21:20:45 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Thu, 23 Aug 2018 21:20:45 +0200 Subject: [PVE-User] DHCP for non cloudinit VM In-Reply-To: References: <1608324e-1221-d6fc-712a-9b266cc4d49a@ginernet.com> <7bbd7d41-a7aa-be70-d5d7-cd65895ed501@proxmox.com> <93716113-4f97-a33d-eb61-a16795a493c2@ginernet.com> <20180821082732.78df79f8@sleipner.datanom.net> <16c186cd6964f5a5252b952b64114b1a9f7dbca9.camel@tuxis.nl> <36008b84-6c13-aad1-ff05-aee4b8bf0863@ginernet.com> Message-ID: Am 21.08.2018 um 10:35 schrieb Jos? Manuel Giner: > What I ask is that Proxmox offer us a native DHCP server integration, > so that we can configure VMs as easily as in cloud-init based VMs or > containers. I do not have a solution but I see the point of Jos?. The thing is about having untrusted VMs where a user can install any OS and can hack around in the VM as he like (e.g. setting any MAC addresses on the interfaces). Hence, any agent-based solution does not work. So, if the VM uses DHCP and the DHCP is integrated into Proxmox/qemu (if possible) that would be a great feature. Assigning an IP address as any other VM parameter (disk ...) and then beeing sure that the VM got this IP address offered by DHCP (and any other IP addresses, MAC addresses configured in the VM will be refused) would be a very very nice feature. So why having DHCP integrated into Proxmox instead of adding it outside? Why is storage replication added to Proxmox instead of doing it manually with cronjobs? Why is User management integrated inside Proxmox although could could implement it yourself with some reverse proxy in front of the GUI?? Because it is very useful. Honestly, the first time I used Proxmox and provisioned a VM I asked myself why I can not configure the network settings of the VM. regards Klaus From gilberto.nunes32 at gmail.com Fri Aug 24 03:33:39 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 23 Aug 2018 22:33:39 -0300 Subject: [PVE-User] Confusing about Bond 802.3ad Message-ID: Hello I have a T-Link switch T1600G-28TS and wanna use 4 NIC, all 1GB, to create a LAG... I have success to create it, indeed, however, when use iperf3 to check the performance, found that the traffic do not transpass 1GB/s??? This 802.3ad do no suppose to agrengate the speed of all available NIC?? In the switch, I also have VLAN ( no trunk) which are used to communicate only with the clusters and storages... The iperf3 performance was like this: Server: iperf3 --bind 10.10.10.100 -s Client: iperf3 -c 10.10.10.100 Need same advice. Thanks a lot. --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From chance_ellis at yahoo.com Fri Aug 24 04:54:19 2018 From: chance_ellis at yahoo.com (Chance Ellis) Date: Thu, 23 Aug 2018 22:54:19 -0400 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: Message-ID: iperf will open a single stream by default. An 802.3ad bond will only send a specific stream across a single link in the lag. Try the -P flag to add parallel streams with iperf and see how the performance looks. > On Aug 23, 2018, at 9:33 PM, Gilberto Nunes wrote: > > Hello > > I have a T-Link switch T1600G-28TS and wanna use 4 NIC, all 1GB, to create > a LAG... > I have success to create it, indeed, however, when use iperf3 to check the > performance, found that the traffic do not transpass 1GB/s??? > This 802.3ad do no suppose to agrengate the speed of all available NIC?? > > In the switch, I also have VLAN ( no trunk) which are used to communicate > only with the clusters and storages... > > The iperf3 performance was like this: > Server: > iperf3 --bind 10.10.10.100 -s > Client: > iperf3 -c 10.10.10.100 > > Need same advice. > > Thanks a lot. > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From dietmar at proxmox.com Fri Aug 24 08:01:35 2018 From: dietmar at proxmox.com (Dietmar Maurer) Date: Fri, 24 Aug 2018 08:01:35 +0200 (CEST) Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: Message-ID: <825167043.4.1535090495766@webmail.proxmox.com> > This 802.3ad do no suppose to agrengate the speed of all available NIC?? No, not really. One connection is limited to 1GB. If you start more parallel connections you can gain more speed. From Jan.Dreyer at arvato.com Fri Aug 24 11:51:02 2018 From: Jan.Dreyer at arvato.com (Dreyer, Jan, SCM-IT) Date: Fri, 24 Aug 2018 09:51:02 +0000 Subject: [PVE-User] Problems with HP Smart Array P400 and ZFS Message-ID: <8B7B44EF-AE80-4F19-BEAD-7EC6F6837EC5@arvato.com> Hi, my configuration: HP DL380 G5 with Smart Array P400 Proxmox VE 5.2-1 name: 4.4.128-1-pve #1 SMP PVE 4.4.128-111 (Wed, 23 May 2018 14:00:02 +0000) x86_64 GNU/Linux This system is currently running ZFS filesystem version 5. My problem: When trying to update to a higher kernel (I tried 4.10 and 4.15 series), the initrd is not able to detect the cciss devices, and as such not able to load the ZFS pools, including the root pool. Falling back to kernel 4.4 doesn?t let me use the ZFS cache file system though. :-( Any hints on how to detect the raid controller device in initrd? Gru? / Regards ? Jan Dreyer IT Administrator ? Operations ? A-SCM-IT IOX Expert -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From gilberto.nunes32 at gmail.com Fri Aug 24 12:01:28 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 24 Aug 2018 07:01:28 -0300 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: <825167043.4.1535090495766@webmail.proxmox.com> References: <825167043.4.1535090495766@webmail.proxmox.com> Message-ID: So what bond mode I suppose to use in order to get more speed? I mean how to join the nic to get 4 GB? I will use Ceph! I know I should use 10gb but I dont have it right now. Thanks Em 24/08/2018 03:01, "Dietmar Maurer" escreveu: > > This 802.3ad do no suppose to agrengate the speed of all available NIC?? > > No, not really. One connection is limited to 1GB. If you start more > parallel connections you can gain more speed. > > From t.lamprecht at proxmox.com Fri Aug 24 12:15:06 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Fri, 24 Aug 2018 12:15:06 +0200 Subject: [PVE-User] Problems with HP Smart Array P400 and ZFS In-Reply-To: <8B7B44EF-AE80-4F19-BEAD-7EC6F6837EC5@arvato.com> References: <8B7B44EF-AE80-4F19-BEAD-7EC6F6837EC5@arvato.com> Message-ID: <3518a8e3-ce55-6bee-e0e7-d9aa68ab3f21@proxmox.com> Hi, On 8/24/18 11:51 AM, Dreyer, Jan, SCM-IT wrote: > Hi, > > my configuration: > HP DL380 G5 with Smart Array P400 > Proxmox VE 5.2-1 > name: 4.4.128-1-pve #1 SMP PVE 4.4.128-111 (Wed, 23 May 2018 14:00:02 +0000) x86_64 GNU/Linux > This system is currently running ZFS filesystem version 5. > > My problem: When trying to update to a higher kernel (I tried 4.10 and 4.15 series), the initrd is not able to detect the cciss devices, and as such not able to load the ZFS pools, including the root pool. FYI cciss is an alias to hpsa as since 4.14: > commit 253d2464df446456c0bba5ed4137a7be0b278aa8 > Author: Hannes Reinecke > Date: Tue Aug 15 08:58:08 2017 +0200 > > scsi: cciss: Drop obsolete driver > > The hpsa driver now has support for all boards the cciss driver > used to support, so this patch removes the cciss driver and > make hpsa an alias to cciss. > Falling back to kernel 4.4 doesn?t let me use the ZFS cache file system though. :-( > > Any hints on how to detect the raid controller device in initrd? > Sounds a bit like: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1765105 Are your firmware versions all up to date? I did not find related issues searching the LKML, Ubuntu's kernel devel or related lists. (We very recently ported back a fix for hpsa but it was related to clean shutdown, no changes for detection/bring up AFAICT). regards, Thomas From uwe.sauter.de at gmail.com Fri Aug 24 12:45:11 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Fri, 24 Aug 2018 12:45:11 +0200 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> Message-ID: <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> If using standard 802.3ad (LACP) you will always get only the performance of a single link between one host and another. Using "bond-xmit-hash-policy layer3+4" might get you a better performance but is not standard LACP. Am 24.08.18 um 12:01 schrieb Gilberto Nunes: > So what bond mode I suppose to use in order to get more speed? I mean how > to join the nic to get 4 GB? I will use Ceph! > I know I should use 10gb but I dont have it right now. > > Thanks > Em 24/08/2018 03:01, "Dietmar Maurer" escreveu: > >>> This 802.3ad do no suppose to agrengate the speed of all available NIC?? >> >> No, not really. One connection is limited to 1GB. If you start more >> parallel connections you can gain more speed. >> >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From lists at merit.unu.edu Fri Aug 24 12:52:10 2018 From: lists at merit.unu.edu (mj) Date: Fri, 24 Aug 2018 12:52:10 +0200 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> Message-ID: Hi, Yes, it is our undertanding that if the hardware (switch) supports it, "bond-xmit-hash-policy layer3+4" gives you best spread. But it will still give you 4 'lanes' of 1GB. Ceph will connect using different ports, ip's etc, en each connection should use a different lane, so altogether, you should see a network throughput that (theoretically) could be as high as 4GB. That is how we understand it. You can also try something on the switch, like we did on our ProCurve: > Procurve chassis(config)# show trunk > > Load Balancing Method: L3-based (default) > > Port | Name Type | Group Type > ---- + -------------------------------- --------- + ------ -------- > D1 | Link to prn004 - 1 10GbE-T | Trk1 LACP > D2 | Link to prn004 - 2 10GbE-T | Trk1 LACP > D3 | Link to prn005 - 1 10GbE-T | Trk2 LACP > D4 | Link to prn005 - 2 10GbE-T | Trk2 LACP Namely: change the load balancing method to: > Procurve chassis(config)# trunk-load-balance L4 So the load balance is now based on Layer4 instead of L3. Besides these details, I think what you are doing should work nicely. MJ On 08/24/2018 12:45 PM, Uwe Sauter wrote: > If using standard 802.3ad (LACP) you will always get only the performance of a single link between one host and another. > > Using "bond-xmit-hash-policy layer3+4" might get you a better performance but is not standard LACP. > > > > > Am 24.08.18 um 12:01 schrieb Gilberto Nunes: >> So what bond mode I suppose to use in order to get more speed? I mean how >> to join the nic to get 4 GB? I will use Ceph! >> I know I should use 10gb but I dont have it right now. >> >> Thanks >> Em 24/08/2018 03:01, "Dietmar Maurer" escreveu: >> >>>> This 802.3ad do no suppose to agrengate the speed of all available NIC?? >>> >>> No, not really. One connection is limited to 1GB. If you start more >>> parallel connections you can gain more speed. >>> >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From s.ivanov at proxmox.com Fri Aug 24 13:36:44 2018 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Fri, 24 Aug 2018 13:36:44 +0200 Subject: [PVE-User] Problems with HP Smart Array P400 and ZFS In-Reply-To: <3518a8e3-ce55-6bee-e0e7-d9aa68ab3f21@proxmox.com> References: <8B7B44EF-AE80-4F19-BEAD-7EC6F6837EC5@arvato.com> <3518a8e3-ce55-6bee-e0e7-d9aa68ab3f21@proxmox.com> Message-ID: <20180824133644.140ab328@rosa.proxmox.com> Hi, On Fri, 24 Aug 2018 12:15:06 +0200 Thomas Lamprecht wrote: > Hi, > > On 8/24/18 11:51 AM, Dreyer, Jan, SCM-IT wrote: > > Hi, > > > > my configuration: > > HP DL380 G5 with Smart Array P400 > > Proxmox VE 5.2-1 > > name: 4.4.128-1-pve #1 SMP PVE 4.4.128-111 (Wed, 23 May 2018 > > 14:00:02 +0000) x86_64 GNU/Linux This system is currently running > > ZFS filesystem version 5. > > > > My problem: When trying to update to a higher kernel (I tried 4.10 > > and 4.15 series), the initrd is not able to detect the cciss > > devices, and as such not able to load the ZFS pools, including the > > root pool. > > FYI cciss is an alias to hpsa as since 4.14: AFAIR, the switch from cciss to hpsa also changed the way the 'disks' were reported to the OS (cciss -> /dev/cciss/cXdYpZ, hpsa -> /dev/sdX). Maybe your disks are present as /dev/sdX? regards, stoiko > > > commit 253d2464df446456c0bba5ed4137a7be0b278aa8 > > Author: Hannes Reinecke > > Date: Tue Aug 15 08:58:08 2017 +0200 > > > > scsi: cciss: Drop obsolete driver > > > > The hpsa driver now has support for all boards the cciss driver > > used to support, so this patch removes the cciss driver and > > make hpsa an alias to cciss. > > > > Falling back to kernel 4.4 doesn?t let me use the ZFS cache file > > system though. :-( > > > > Any hints on how to detect the raid controller device in initrd? > > > > Sounds a bit like: > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1765105 > > Are your firmware versions all up to date? > > I did not find related issues searching the LKML, Ubuntu's kernel > devel or related lists. (We very recently ported back a fix for hpsa > but it was related to clean shutdown, no changes for detection/bring > up AFAICT). > > regards, > Thomas > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From josh at noobbox.com Fri Aug 24 17:02:04 2018 From: josh at noobbox.com (Josh Knight) Date: Fri, 24 Aug 2018 11:02:04 -0400 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> Message-ID: Depending on your topology/configuration, you could try to use bond-rr mode in Linux instead of 802.3ad. Bond-rr mode is the only mode that will put pkts for the same mac/ip/port tuple across multiple interfaces. This will work well for UDP but TCP may suffer performance issues because pkts can end up out of order and trigger TCP retransmits. There are some examples on this page, you may need to do some testing before deploying it to ensure it does what you want. https://wiki.linuxfoundation.org/networking/bonding#bonding-driver-options As others have stated, you can adjust the hashing, but a single flow (mac/ip/port combination) will still end up limited to 1Gbps without using round robin mode. On Fri, Aug 24, 2018 at 6:52 AM mj wrote: > Hi, > > Yes, it is our undertanding that if the hardware (switch) supports it, > "bond-xmit-hash-policy layer3+4" gives you best spread. > > But it will still give you 4 'lanes' of 1GB. Ceph will connect using > different ports, ip's etc, en each connection should use a different > lane, so altogether, you should see a network throughput that > (theoretically) could be as high as 4GB. > > That is how we understand it. > > You can also try something on the switch, like we did on our ProCurve: > > > Procurve chassis(config)# show trunk > > > > Load Balancing Method: L3-based (default) > > > > Port | Name Type | Group Type > > ---- + -------------------------------- --------- + ------ -------- > > D1 | Link to prn004 - 1 10GbE-T | Trk1 LACP > > D2 | Link to prn004 - 2 10GbE-T | Trk1 LACP > > D3 | Link to prn005 - 1 10GbE-T | Trk2 LACP > > D4 | Link to prn005 - 2 10GbE-T | Trk2 LACP > > Namely: change the load balancing method to: > > > Procurve chassis(config)# trunk-load-balance L4 > > So the load balance is now based on Layer4 instead of L3. > > Besides these details, I think what you are doing should work nicely. > > MJ > > > > On 08/24/2018 12:45 PM, Uwe Sauter wrote: > > If using standard 802.3ad (LACP) you will always get only the > performance of a single link between one host and another. > > > > Using "bond-xmit-hash-policy layer3+4" might get you a better > performance but is not standard LACP. > > > > > > > > > > Am 24.08.18 um 12:01 schrieb Gilberto Nunes: > >> So what bond mode I suppose to use in order to get more speed? I mean > how > >> to join the nic to get 4 GB? I will use Ceph! > >> I know I should use 10gb but I dont have it right now. > >> > >> Thanks > >> Em 24/08/2018 03:01, "Dietmar Maurer" escreveu: > >> > >>>> This 802.3ad do no suppose to agrengate the speed of all available > NIC?? > >>> > >>> No, not really. One connection is limited to 1GB. If you start more > >>> parallel connections you can gain more speed. > >>> > >>> > >> _______________________________________________ > >> pve-user mailing list > >> pve-user at pve.proxmox.com > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Fri Aug 24 17:19:50 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 24 Aug 2018 12:19:50 -0300 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> Message-ID: So I try balance-rr with LAG in the switch and still get 1 GB pve-ceph02:~# iperf3 -c 10.10.10.100 Connecting to host 10.10.10.100, port 5201 [ 4] local 10.10.10.110 port 52674 connected to 10.10.10.100 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 116 MBytes 974 Mbits/sec 32 670 KBytes [ 4] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 3 597 KBytes [ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 3 509 KBytes [ 4] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 660 KBytes [ 4] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 6 585 KBytes [ 4] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 720 KBytes [ 4] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 3 650 KBytes [ 4] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 4 570 KBytes [ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 708 KBytes [ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 8 635 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec 59 sender [ 4] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec receiver iperf Done. --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-24 12:02 GMT-03:00 Josh Knight : > Depending on your topology/configuration, you could try to use bond-rr mode > in Linux instead of 802.3ad. > > Bond-rr mode is the only mode that will put pkts for the same mac/ip/port > tuple across multiple interfaces. This will work well for UDP but TCP may > suffer performance issues because pkts can end up out of order and trigger > TCP retransmits. There are some examples on this page, you may need to do > some testing before deploying it to ensure it does what you want. > > https://wiki.linuxfoundation.org/networking/bonding#bonding-driver-options > > As others have stated, you can adjust the hashing, but a single flow > (mac/ip/port combination) will still end up limited to 1Gbps without using > round robin mode. > > > On Fri, Aug 24, 2018 at 6:52 AM mj wrote: > > > Hi, > > > > Yes, it is our undertanding that if the hardware (switch) supports it, > > "bond-xmit-hash-policy layer3+4" gives you best spread. > > > > But it will still give you 4 'lanes' of 1GB. Ceph will connect using > > different ports, ip's etc, en each connection should use a different > > lane, so altogether, you should see a network throughput that > > (theoretically) could be as high as 4GB. > > > > That is how we understand it. > > > > You can also try something on the switch, like we did on our ProCurve: > > > > > Procurve chassis(config)# show trunk > > > > > > Load Balancing Method: L3-based (default) > > > > > > Port | Name Type | Group Type > > > ---- + -------------------------------- --------- + ------ -------- > > > D1 | Link to prn004 - 1 10GbE-T | Trk1 LACP > > > D2 | Link to prn004 - 2 10GbE-T | Trk1 LACP > > > D3 | Link to prn005 - 1 10GbE-T | Trk2 LACP > > > D4 | Link to prn005 - 2 10GbE-T | Trk2 LACP > > > > Namely: change the load balancing method to: > > > > > Procurve chassis(config)# trunk-load-balance L4 > > > > So the load balance is now based on Layer4 instead of L3. > > > > Besides these details, I think what you are doing should work nicely. > > > > MJ > > > > > > > > On 08/24/2018 12:45 PM, Uwe Sauter wrote: > > > If using standard 802.3ad (LACP) you will always get only the > > performance of a single link between one host and another. > > > > > > Using "bond-xmit-hash-policy layer3+4" might get you a better > > performance but is not standard LACP. > > > > > > > > > > > > > > > Am 24.08.18 um 12:01 schrieb Gilberto Nunes: > > >> So what bond mode I suppose to use in order to get more speed? I mean > > how > > >> to join the nic to get 4 GB? I will use Ceph! > > >> I know I should use 10gb but I dont have it right now. > > >> > > >> Thanks > > >> Em 24/08/2018 03:01, "Dietmar Maurer" escreveu: > > >> > > >>>> This 802.3ad do no suppose to agrengate the speed of all available > > NIC?? > > >>> > > >>> No, not really. One connection is limited to 1GB. If you start more > > >>> parallel connections you can gain more speed. > > >>> > > >>> > > >> _______________________________________________ > > >> pve-user mailing list > > >> pve-user at pve.proxmox.com > > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >> > > > > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From josh at noobbox.com Fri Aug 24 17:57:48 2018 From: josh at noobbox.com (Josh Knight) Date: Fri, 24 Aug 2018 11:57:48 -0400 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> Message-ID: I don't know your topology, I'm assuming you're going from nodeA -> switch -> nodeB ? Make sure that entire path is using RR. You could verify this with interface counters on the various hops. If a single hop is not doing it correctly, it will limit the throughput. On Fri, Aug 24, 2018 at 11:20 AM Gilberto Nunes wrote: > So I try balance-rr with LAG in the switch and still get 1 GB > > pve-ceph02:~# iperf3 -c 10.10.10.100 > Connecting to host 10.10.10.100, port 5201 > [ 4] local 10.10.10.110 port 52674 connected to 10.10.10.100 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 116 MBytes 974 Mbits/sec 32 670 > KBytes > [ 4] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 3 597 > KBytes > [ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 3 509 > KBytes > [ 4] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 660 > KBytes > [ 4] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 6 585 > KBytes > [ 4] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 720 > KBytes > [ 4] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 3 650 > KBytes > [ 4] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 4 570 > KBytes > [ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 708 > KBytes > [ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 8 635 > KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec 59 > sender > [ 4] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec > receiver > > iperf Done. > > > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > 2018-08-24 12:02 GMT-03:00 Josh Knight : > > > Depending on your topology/configuration, you could try to use bond-rr > mode > > in Linux instead of 802.3ad. > > > > Bond-rr mode is the only mode that will put pkts for the same mac/ip/port > > tuple across multiple interfaces. This will work well for UDP but TCP > may > > suffer performance issues because pkts can end up out of order and > trigger > > TCP retransmits. There are some examples on this page, you may need to > do > > some testing before deploying it to ensure it does what you want. > > > > > https://wiki.linuxfoundation.org/networking/bonding#bonding-driver-options > > > > As others have stated, you can adjust the hashing, but a single flow > > (mac/ip/port combination) will still end up limited to 1Gbps without > using > > round robin mode. > > > > > > On Fri, Aug 24, 2018 at 6:52 AM mj wrote: > > > > > Hi, > > > > > > Yes, it is our undertanding that if the hardware (switch) supports it, > > > "bond-xmit-hash-policy layer3+4" gives you best spread. > > > > > > But it will still give you 4 'lanes' of 1GB. Ceph will connect using > > > different ports, ip's etc, en each connection should use a different > > > lane, so altogether, you should see a network throughput that > > > (theoretically) could be as high as 4GB. > > > > > > That is how we understand it. > > > > > > You can also try something on the switch, like we did on our ProCurve: > > > > > > > Procurve chassis(config)# show trunk > > > > > > > > Load Balancing Method: L3-based (default) > > > > > > > > Port | Name Type | Group Type > > > > ---- + -------------------------------- --------- + ------ -------- > > > > D1 | Link to prn004 - 1 10GbE-T | Trk1 LACP > > > > D2 | Link to prn004 - 2 10GbE-T | Trk1 LACP > > > > D3 | Link to prn005 - 1 10GbE-T | Trk2 LACP > > > > D4 | Link to prn005 - 2 10GbE-T | Trk2 LACP > > > > > > Namely: change the load balancing method to: > > > > > > > Procurve chassis(config)# trunk-load-balance L4 > > > > > > So the load balance is now based on Layer4 instead of L3. > > > > > > Besides these details, I think what you are doing should work nicely. > > > > > > MJ > > > > > > > > > > > > On 08/24/2018 12:45 PM, Uwe Sauter wrote: > > > > If using standard 802.3ad (LACP) you will always get only the > > > performance of a single link between one host and another. > > > > > > > > Using "bond-xmit-hash-policy layer3+4" might get you a better > > > performance but is not standard LACP. > > > > > > > > > > > > > > > > > > > > Am 24.08.18 um 12:01 schrieb Gilberto Nunes: > > > >> So what bond mode I suppose to use in order to get more speed? I > mean > > > how > > > >> to join the nic to get 4 GB? I will use Ceph! > > > >> I know I should use 10gb but I dont have it right now. > > > >> > > > >> Thanks > > > >> Em 24/08/2018 03:01, "Dietmar Maurer" > escreveu: > > > >> > > > >>>> This 802.3ad do no suppose to agrengate the speed of all available > > > NIC?? > > > >>> > > > >>> No, not really. One connection is limited to 1GB. If you start more > > > >>> parallel connections you can gain more speed. > > > >>> > > > >>> > > > >> _______________________________________________ > > > >> pve-user mailing list > > > >> pve-user at pve.proxmox.com > > > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > >> > > > > > > > > _______________________________________________ > > > > pve-user mailing list > > > > pve-user at pve.proxmox.com > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mark at openvs.co.uk Fri Aug 24 18:20:19 2018 From: mark at openvs.co.uk (Mark Adams) Date: Fri, 24 Aug 2018 19:20:19 +0300 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> Message-ID: also, balance-rr through a switch requires each nic to be on a seperate vlan. You probably need to remove your lacp config also but this depends on switch model and configuration. so safest idea is remove it. so I think you have 3 nodes.... for example: node1: ens0 on port 1 vlan 10 ens1 on port 4 vlan 11 ens2 on port 7 vlan 12 ens3 on port 10 vlan 13 node2: ens0 on port 2 vlan 10 ens1 on port 5 vlan 11 ens2 on port 8 vlan 12 ens3 on port 11 vlan 13 node3: ens0 on port 3 vlan 10 ens1 on port 6 vlan 11 ens2 on port 9 vlan 12 ens3 on port 12 vlan 13 then I belive your iperf test will return ~3Gbps... i seem to remember performance doesnt get much better than this but I cant remember why. Also can't say if this is a good setup for ceph performance.. Cheers On Fri, 24 Aug 2018, 18:58 Josh Knight, wrote: > I don't know your topology, I'm assuming you're going from nodeA -> > switch -> nodeB ? Make sure that entire path is using RR. You could > verify this with interface counters on the various hops. If a single hop > is not doing it correctly, it will limit the throughput. > > On Fri, Aug 24, 2018 at 11:20 AM Gilberto Nunes < > gilberto.nunes32 at gmail.com> > wrote: > > > So I try balance-rr with LAG in the switch and still get 1 GB > > > > pve-ceph02:~# iperf3 -c 10.10.10.100 > > Connecting to host 10.10.10.100, port 5201 > > [ 4] local 10.10.10.110 port 52674 connected to 10.10.10.100 port 5201 > > [ ID] Interval Transfer Bandwidth Retr Cwnd > > [ 4] 0.00-1.00 sec 116 MBytes 974 Mbits/sec 32 670 > > KBytes > > [ 4] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 3 597 > > KBytes > > [ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 3 509 > > KBytes > > [ 4] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 660 > > KBytes > > [ 4] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 6 585 > > KBytes > > [ 4] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 720 > > KBytes > > [ 4] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 3 650 > > KBytes > > [ 4] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 4 570 > > KBytes > > [ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 708 > > KBytes > > [ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 8 635 > > KBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bandwidth Retr > > [ 4] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec 59 > > sender > > [ 4] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec > > receiver > > > > iperf Done. > > > > > > > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > > > > > > > > > 2018-08-24 12:02 GMT-03:00 Josh Knight : > > > > > Depending on your topology/configuration, you could try to use bond-rr > > mode > > > in Linux instead of 802.3ad. > > > > > > Bond-rr mode is the only mode that will put pkts for the same > mac/ip/port > > > tuple across multiple interfaces. This will work well for UDP but TCP > > may > > > suffer performance issues because pkts can end up out of order and > > trigger > > > TCP retransmits. There are some examples on this page, you may need to > > do > > > some testing before deploying it to ensure it does what you want. > > > > > > > > > https://wiki.linuxfoundation.org/networking/bonding#bonding-driver-options > > > > > > As others have stated, you can adjust the hashing, but a single flow > > > (mac/ip/port combination) will still end up limited to 1Gbps without > > using > > > round robin mode. > > > > > > > > > On Fri, Aug 24, 2018 at 6:52 AM mj wrote: > > > > > > > Hi, > > > > > > > > Yes, it is our undertanding that if the hardware (switch) supports > it, > > > > "bond-xmit-hash-policy layer3+4" gives you best spread. > > > > > > > > But it will still give you 4 'lanes' of 1GB. Ceph will connect using > > > > different ports, ip's etc, en each connection should use a different > > > > lane, so altogether, you should see a network throughput that > > > > (theoretically) could be as high as 4GB. > > > > > > > > That is how we understand it. > > > > > > > > You can also try something on the switch, like we did on our > ProCurve: > > > > > > > > > Procurve chassis(config)# show trunk > > > > > > > > > > Load Balancing Method: L3-based (default) > > > > > > > > > > Port | Name Type | Group Type > > > > > ---- + -------------------------------- --------- + ------ > -------- > > > > > D1 | Link to prn004 - 1 10GbE-T | Trk1 LACP > > > > > D2 | Link to prn004 - 2 10GbE-T | Trk1 LACP > > > > > D3 | Link to prn005 - 1 10GbE-T | Trk2 LACP > > > > > D4 | Link to prn005 - 2 10GbE-T | Trk2 LACP > > > > > > > > Namely: change the load balancing method to: > > > > > > > > > Procurve chassis(config)# trunk-load-balance L4 > > > > > > > > So the load balance is now based on Layer4 instead of L3. > > > > > > > > Besides these details, I think what you are doing should work nicely. > > > > > > > > MJ > > > > > > > > > > > > > > > > On 08/24/2018 12:45 PM, Uwe Sauter wrote: > > > > > If using standard 802.3ad (LACP) you will always get only the > > > > performance of a single link between one host and another. > > > > > > > > > > Using "bond-xmit-hash-policy layer3+4" might get you a better > > > > performance but is not standard LACP. > > > > > > > > > > > > > > > > > > > > > > > > > Am 24.08.18 um 12:01 schrieb Gilberto Nunes: > > > > >> So what bond mode I suppose to use in order to get more speed? I > > mean > > > > how > > > > >> to join the nic to get 4 GB? I will use Ceph! > > > > >> I know I should use 10gb but I dont have it right now. > > > > >> > > > > >> Thanks > > > > >> Em 24/08/2018 03:01, "Dietmar Maurer" > > escreveu: > > > > >> > > > > >>>> This 802.3ad do no suppose to agrengate the speed of all > available > > > > NIC?? > > > > >>> > > > > >>> No, not really. One connection is limited to 1GB. If you start > more > > > > >>> parallel connections you can gain more speed. > > > > >>> > > > > >>> > > > > >> _______________________________________________ > > > > >> pve-user mailing list > > > > >> pve-user at pve.proxmox.com > > > > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > >> > > > > > > > > > > _______________________________________________ > > > > > pve-user mailing list > > > > > pve-user at pve.proxmox.com > > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > > > _______________________________________________ > > > > pve-user mailing list > > > > pve-user at pve.proxmox.com > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From ronny+pve-user at aasen.cx Fri Aug 24 19:06:09 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Fri, 24 Aug 2018 19:06:09 +0200 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> Message-ID: On 24.08.2018 12:01, Gilberto Nunes wrote: > So what bond mode I suppose to use in order to get more speed? I mean how > to join the nic to get 4 GB? I will use Ceph! > I know I should use 10gb but I dont have it right now. > > Thanks > Em 24/08/2018 03:01, "Dietmar Maurer" escreveu: > >>> This 802.3ad do no suppose to agrengate the speed of all available NIC?? ceph osd's with talk to different hosts, so the total traffic will spread out over the 1gbps links. using? layer3+4 will a spread this out even better. since the different osd's will use different links as well. ceph + lacp is a good fit. BUT! ceph's latency will be limited to the 1gbps link, so while you can get better thruput from 4 channels, you will not get the lower latency. and on a vm the latency impacts the "feel" of the vm performance more then the thruput.?? This is also why 25gbps is better then 40gbps (in effect 4x10gbps) for storage iops/latency kind regards Ronny Aasen From lists at merit.unu.edu Fri Aug 24 22:36:25 2018 From: lists at merit.unu.edu (mj) Date: Fri, 24 Aug 2018 22:36:25 +0200 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> Message-ID: <2c464c4a-fbed-0b2c-4ac9-ee36c1bfeced@merit.unu.edu> Hi Mark, On 08/24/2018 06:20 PM, Mark Adams wrote: > also, balance-rr through a switch requires each nic to be on a seperate > vlan. You probably need to remove your lacp config also but this depends on > switch model and configuration. so safest idea is remove it. > then I belive your iperf test will return ~3Gbps... i seem to remember > performance doesnt get much better than this but I cant remember why. > > Also can't say if this is a good setup for ceph performance.. This is really interesting info, i did not know this. Someone has tried this with ceph? Any experiences to share..? Strange that performence turns out to be ~3Gbps, instead of the expected 4... Anyone with more information on this subject? Have a nice weekend all! MJ From gilberto.nunes32 at gmail.com Fri Aug 24 22:59:49 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 24 Aug 2018 17:59:49 -0300 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: <2c464c4a-fbed-0b2c-4ac9-ee36c1bfeced@merit.unu.edu> References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> <2c464c4a-fbed-0b2c-4ac9-ee36c1bfeced@merit.unu.edu> Message-ID: I can get 3 gbps. At least 1.3 gbps. Don't know why! Em 24/08/2018 17:36, "mj" escreveu: > Hi Mark, > > On 08/24/2018 06:20 PM, Mark Adams wrote: > >> also, balance-rr through a switch requires each nic to be on a seperate >> vlan. You probably need to remove your lacp config also but this depends >> on >> switch model and configuration. so safest idea is remove it. >> > > > > then I belive your iperf test will return ~3Gbps... i seem to remember >> performance doesnt get much better than this but I cant remember why. >> >> Also can't say if this is a good setup for ceph performance.. >> > > This is really interesting info, i did not know this. Someone has tried > this with ceph? Any experiences to share..? > > Strange that performence turns out to be ~3Gbps, instead of the expected > 4... > > Anyone with more information on this subject? > > Have a nice weekend all! > > MJ > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From josh at noobbox.com Fri Aug 24 23:15:15 2018 From: josh at noobbox.com (Josh Knight) Date: Fri, 24 Aug 2018 17:15:15 -0400 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> <2c464c4a-fbed-0b2c-4ac9-ee36c1bfeced@merit.unu.edu> Message-ID: Just guessing here, if the switch doesn't support rr on its port channels, then using separate VLANs instead of bundles on the switch is essentially wiring nodeA to nodeB. That way you don't hit the port channel hashing on the switch and you keep the rr as-is from A to B. I would also try using UDP mode on iperf to see if it's TCP retransmission that's preventing you from getting closer to 4Gbps. Another useful tool is maisezahn for traffic generation, though it is more complex to run. On Fri, Aug 24, 2018, 4:59 PM Gilberto Nunes wrote: > I can get 3 gbps. At least 1.3 gbps. > Don't know why! > Em 24/08/2018 17:36, "mj" escreveu: > > > Hi Mark, > > > > On 08/24/2018 06:20 PM, Mark Adams wrote: > > > >> also, balance-rr through a switch requires each nic to be on a seperate > >> vlan. You probably need to remove your lacp config also but this depends > >> on > >> switch model and configuration. so safest idea is remove it. > >> > > > > > > > > then I belive your iperf test will return ~3Gbps... i seem to remember > >> performance doesnt get much better than this but I cant remember why. > >> > >> Also can't say if this is a good setup for ceph performance.. > >> > > > > This is really interesting info, i did not know this. Someone has tried > > this with ceph? Any experiences to share..? > > > > Strange that performence turns out to be ~3Gbps, instead of the expected > > 4... > > > > Anyone with more information on this subject? > > > > Have a nice weekend all! > > > > MJ > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From josh at noobbox.com Fri Aug 24 23:16:30 2018 From: josh at noobbox.com (Josh Knight) Date: Fri, 24 Aug 2018 17:16:30 -0400 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> <2c464c4a-fbed-0b2c-4ac9-ee36c1bfeced@merit.unu.edu> Message-ID: sorry, should say "mausezahn". It's a part of Netsniff http://netsniff-ng.org/ On Fri, Aug 24, 2018, 5:15 PM Josh Knight wrote: > Just guessing here, if the switch doesn't support rr on its port channels, > then using separate VLANs instead of bundles on the switch is essentially > wiring nodeA to nodeB. That way you don't hit the port channel hashing on > the switch and you keep the rr as-is from A to B. > > I would also try using UDP mode on iperf to see if it's TCP retransmission > that's preventing you from getting closer to 4Gbps. Another useful tool is > maisezahn for traffic generation, though it is more complex to run. > > > On Fri, Aug 24, 2018, 4:59 PM Gilberto Nunes > wrote: > >> I can get 3 gbps. At least 1.3 gbps. >> Don't know why! >> Em 24/08/2018 17:36, "mj" escreveu: >> >> > Hi Mark, >> > >> > On 08/24/2018 06:20 PM, Mark Adams wrote: >> > >> >> also, balance-rr through a switch requires each nic to be on a seperate >> >> vlan. You probably need to remove your lacp config also but this >> depends >> >> on >> >> switch model and configuration. so safest idea is remove it. >> >> >> > >> > >> > >> > then I belive your iperf test will return ~3Gbps... i seem to remember >> >> performance doesnt get much better than this but I cant remember why. >> >> >> >> Also can't say if this is a good setup for ceph performance.. >> >> >> > >> > This is really interesting info, i did not know this. Someone has tried >> > this with ceph? Any experiences to share..? >> > >> > Strange that performence turns out to be ~3Gbps, instead of the expected >> > 4... >> > >> > Anyone with more information on this subject? >> > >> > Have a nice weekend all! >> > >> > MJ >> > _______________________________________________ >> > pve-user mailing list >> > pve-user at pve.proxmox.com >> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > From mark at openvs.co.uk Fri Aug 24 23:58:36 2018 From: mark at openvs.co.uk (Mark Adams) Date: Sat, 25 Aug 2018 00:58:36 +0300 Subject: [PVE-User] Confusing about Bond 802.3ad In-Reply-To: References: <825167043.4.1535090495766@webmail.proxmox.com> <4c60a059-4f97-d48e-007a-971e33e936d1@gmail.com> <2c464c4a-fbed-0b2c-4ac9-ee36c1bfeced@merit.unu.edu> Message-ID: That is it, as I understand it Josh. you basically need to turn your switch in to X seperate switches so each nodes nic, is running on a "seperate" network. if you were to do the same thing physically without any config, with 3 nodes, you would need to have as many seperate switches as you wanted nics in the balance-rr. I understand mikrotik support balance-rr, but tbh I don't even count them as a normal switch manufacturer. Their game is routers.... I don't know any other switches which have support for balance-rr? as for the 3Gbps limit I mentioned earlier with balance-rr (no matter how many nics you have)... I don't know if that was just an issue of the day as cheap 10Gbps came along and the need evaporated for me. I would love to know if anyone has a test setup to try it though. Cheers On Sat, 25 Aug 2018, 00:15 Josh Knight, wrote: > Just guessing here, if the switch doesn't support rr on its port channels, > then using separate VLANs instead of bundles on the switch is essentially > wiring nodeA to nodeB. That way you don't hit the port channel hashing on > the switch and you keep the rr as-is from A to B. > > I would also try using UDP mode on iperf to see if it's TCP retransmission > that's preventing you from getting closer to 4Gbps. Another useful tool is > maisezahn for traffic generation, though it is more complex to run. > > On Fri, Aug 24, 2018, 4:59 PM Gilberto Nunes > wrote: > > > I can get 3 gbps. At least 1.3 gbps. > > Don't know why! > > Em 24/08/2018 17:36, "mj" escreveu: > > > > > Hi Mark, > > > > > > On 08/24/2018 06:20 PM, Mark Adams wrote: > > > > > >> also, balance-rr through a switch requires each nic to be on a > seperate > > >> vlan. You probably need to remove your lacp config also but this > depends > > >> on > > >> switch model and configuration. so safest idea is remove it. > > >> > > > > > > > > > > > > then I belive your iperf test will return ~3Gbps... i seem to remember > > >> performance doesnt get much better than this but I cant remember why. > > >> > > >> Also can't say if this is a good setup for ceph performance.. > > >> > > > > > > This is really interesting info, i did not know this. Someone has tried > > > this with ceph? Any experiences to share..? > > > > > > Strange that performence turns out to be ~3Gbps, instead of the > expected > > > 4... > > > > > > Anyone with more information on this subject? > > > > > > Have a nice weekend all! > > > > > > MJ > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Sat Aug 25 00:48:41 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 24 Aug 2018 19:48:41 -0300 Subject: [PVE-User] Ceph doesn't show all available space ! Message-ID: Ok guys I have success deploy a 3 node ceph server. In this server I have: Node1 OSD.0 = 4 TB OSD.1 = 4 TB OSD.2 = 4 TB OSD.3 = 4 TB Node2 OSD.4 = 3 TB OSD.5 = 3 TB OSD.6 = 3 TB OSD.7 = 3 TB Node3 OSD.8 = 3 TB OSD.9 = 3 TB OSD.10 = 2 TB In the Ceph cluster, the total among of data storage is: 11.17 GiB of 32.75 TiB I create a pool, with default value. But when I attach the ceph cluster storage to my proxmox, I just see 10 TB! Why I can see the 32 TB or at least 21.58 ( 32.75 - 11.17) ???? So odd! Thanks for any help --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From klaus.mailinglists at pernau.at Sat Aug 25 19:32:17 2018 From: klaus.mailinglists at pernau.at (Klaus Darilion) Date: Sat, 25 Aug 2018 19:32:17 +0200 Subject: [PVE-User] periodic Node Crash/freeze In-Reply-To: References: Message-ID: Maybe it is related to some Kernel Update regards Klaus Am 23.08.2018 um 08:57 schrieb Ml Ml: > Hello, > > i could need some hint/help since one cluster is letting me down since > 29.07.2018 . > Thats when one of my three nodes started to freeze and stop. > > In syslog the last entries are: > > Aug 21 02:33:00 node10 systemd[1]: Starting Proxmox VE replication runner... > Aug 21 02:33:01 node10 systemd[1]: Started Proxmox VE replication runner. > Aug 21 02:33:01 node10 CRON[1870491]: (root) CMD (/usr/bin/puppet > agent -vt --color false --logdest /var/log/puppet/agent.log > 1>/dev/null) > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ > > > or: > > Aug 22 16:11:12 node08 pmxcfs[5227]: [dcdb] notice: cpg_send_message > retried 1 times > Aug 22 16:11:12 node08 pmxcfs[5227]: [status] notice: members: 1/5227, 2/5058 > Aug 22 16:11:12 node08 pmxcfs[5227]: [status] notice: starting data > syncronisation > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > > I already posted it here: > https://forum.proxmox.com/threads/periodic-node-crash-freeze.46407/ > > It happened at: > 29.07.2018 node09 / pve 4.4 > 07.08.2018 node08 / pve 4.4 ( then i decided to upgrade) > 21.08.2018 node10 / pve 5.2 > 22.08.2018 node08 / pve 5.2 > > ...and i am getting nervous now since there are 60 important VMs on it. > As you can see it happened across multiple nodes with diffrent PVE Versions. > > Memtest is okay. > > As far as i googled the "^@^@^@^@^@^" appear is syslog because i can > not fully write the file to disk? > > Maybe something triggers some totem/watchdog stuff which then ends in > a disaster? > > My Ideas from here: > - disable corosync/totem and see if the problems stop > > Have you any ideas which could narrow my problem down? > > > My Setup is a 3 Node Cluster (node08, node09, node10) with ceph. > > I have 4 other 3-NodeCluster running just fine. > > Thanks a lot. > > Mario > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From ronny+pve-user at aasen.cx Sat Aug 25 21:42:13 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Sat, 25 Aug 2018 21:42:13 +0200 Subject: [PVE-User] Ceph doesn't show all available space ! In-Reply-To: References: Message-ID: On 25.08.2018 00:48, Gilberto Nunes wrote: > Ok guys > > I have success deploy a 3 node ceph server. > In this server I have: > > Node1 > > OSD.0 = 4 TB > OSD.1 = 4 TB > OSD.2 = 4 TB > OSD.3 = 4 TB > > Node2 > > OSD.4 = 3 TB > OSD.5 = 3 TB > OSD.6 = 3 TB > OSD.7 = 3 TB > > Node3 > > OSD.8 = 3 TB > OSD.9 = 3 TB > OSD.10 = 2 TB > > In the Ceph cluster, the total among of data storage is: > 11.17 GiB of 32.75 TiB > > I create a pool, with default value. > But when I attach the ceph cluster storage to my proxmox, I just see 10 TB! > Why I can see the 32 TB or at least 21.58 ( 32.75 - 11.17) ???? > > So odd! > ceph does show you the RAW total space available, since you can configure different pools. ie 3x replication Since you have 3 nodes, your data will be replicated on all 3 nodes with the default size=3 pools. so you should in theory only be able to use 3+3+2= 8TB? TB of data since once those are full you can not have 3 copies of objects any more.? IOW you get the space of the smalles node, and the rest of the space is "wasted" once you add a 4th node, or more disks on the small one.? ceph can use the remaining storage on the larger nodes as well. for small clusters, you should try to make the storage homogeneous, since it makes balancing easier.. try to make the machines more even, if you want more usable space. kind regards Ronny Aasen From gilberto.nunes32 at gmail.com Sun Aug 26 12:18:50 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Sun, 26 Aug 2018 07:18:50 -0300 Subject: [PVE-User] Migration from XenServer Message-ID: Hi anyone has a good method, 100% approved to migrate windows VM from Xen to Proxmox? Every method i tried has failed with bsod. Thanks for any help. From lists at hexis.consulting Sun Aug 26 12:23:25 2018 From: lists at hexis.consulting (Lists) Date: Sun, 26 Aug 2018 05:23:25 -0500 Subject: [PVE-User] Migration from XenServer In-Reply-To: References: Message-ID: <9c98913e-e644-a226-cab3-3116d64bb750@hexis.consulting> Yes, this method will take a long time, but it will almost always works. Boot the source into a linux live cd with ssh, give it a network address, etc. and then kick this off in a screen session: dd if=dd if=/dev/sda | pv -s 200G | dd of=/dev/zvol/pool-01/vm-106-disk-1 The last part will vary based on whether you are using zfs, ceph, lvm, etc. -Riley On 8/26/2018 5:18 AM, Gilberto Nunes wrote: > Hi anyone has a good method, 100% approved to migrate windows VM from Xen > to Proxmox? > Every method i tried has failed with bsod. > > Thanks for any help. > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From lists at hexis.consulting Sun Aug 26 12:25:53 2018 From: lists at hexis.consulting (Lists) Date: Sun, 26 Aug 2018 05:25:53 -0500 Subject: [PVE-User] Migration from XenServer In-Reply-To: <9c98913e-e644-a226-cab3-3116d64bb750@hexis.consulting> References: <9c98913e-e644-a226-cab3-3116d64bb750@hexis.consulting> Message-ID: <79657c91-86bf-e835-c1d4-625e18a1cbbf@hexis.consulting> Oh yeah, and you will need the ssh part in there, like this: dd if=/dev/sda | ssh root@ dd of=/dev/zvol//vm--disk-1 bs=1024 On 8/26/2018 5:23 AM, Lists wrote: > Yes, this method will take a long time, but it will almost always > works. Boot the source into a linux live cd with ssh, give it a > network address, etc. and then kick this off in a screen session: > > dd if=dd if=/dev/sda | pv -s 200G | dd of=/dev/zvol/pool-01/vm-106-disk-1 > > The last part will vary based on whether you are using zfs, ceph, lvm, > etc. > > -Riley > > > > On 8/26/2018 5:18 AM, Gilberto Nunes wrote: >> Hi anyone has a good method, 100% approved to migrate windows VM from >> Xen >> to Proxmox? >> Every method i tried has failed with bsod. >> >> Thanks for any help. >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gilberto.nunes32 at gmail.com Sun Aug 26 12:26:14 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Sun, 26 Aug 2018 07:26:14 -0300 Subject: [PVE-User] Migration from XenServer In-Reply-To: <9c98913e-e644-a226-cab3-3116d64bb750@hexis.consulting> References: <9c98913e-e644-a226-cab3-3116d64bb750@hexis.consulting> Message-ID: Hum I see! Low level copy. Nice trick. Thanks Em dom, 26 de ago de 2018 07:23, Lists escreveu: > Yes, this method will take a long time, but it will almost always works. > Boot the source into a linux live cd with ssh, give it a network > address, etc. and then kick this off in a screen session: > > dd if=dd if=/dev/sda | pv -s 200G | dd of=/dev/zvol/pool-01/vm-106-disk-1 > > The last part will vary based on whether you are using zfs, ceph, lvm, etc. > > -Riley > > > > On 8/26/2018 5:18 AM, Gilberto Nunes wrote: > > Hi anyone has a good method, 100% approved to migrate windows VM from Xen > > to Proxmox? > > Every method i tried has failed with bsod. > > > > Thanks for any help. > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Mon Aug 27 05:00:52 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Mon, 27 Aug 2018 00:00:52 -0300 Subject: [PVE-User] Linux improvements for ceph and general running vm's Message-ID: Hi there Can any one tell me if this options bellow can improve somethings related network: net.ipv4.tcp_low_latency = 1 net.core.rmem_default = 10000000 net.core.wmem_default = 10000000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rfc1337=1 net.ipv4.tcp_window_scaling=1 net.ipv4.tcp_workaround_signed_windows=1 net.ipv4.tcp_sack=1 net.ipv4.tcp_fack=1 net.ipv4.tcp_low_latency=1 net.ipv4.ip_no_pmtu_disc=0 net.ipv4.tcp_mtu_probing=1 net.ipv4.tcp_frto=2 net.ipv4.tcp_frto_response=2 net.ipv4.tcp_congestion_control=illinois Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From aderumier at odiso.com Mon Aug 27 08:34:07 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Mon, 27 Aug 2018 08:34:07 +0200 (CEST) Subject: [PVE-User] Linux improvements for ceph and general running vm's In-Reply-To: References: Message-ID: <1689330932.4145024.1535351647608.JavaMail.zimbra@oxygem.tv> Hi, do you want to improve latency or throughput ? They are lot of thing to tune in ceph.conf before tuning network. can you send your ceph.conf too ? (ceph cluster && proxmox nodes if ceph is outside proxmox) ssd ? hdd ? ----- Mail original ----- De: "Gilberto Nunes" ?: "proxmoxve" Envoy?: Lundi 27 Ao?t 2018 05:00:52 Objet: [PVE-User] Linux improvements for ceph and general running vm's Hi there Can any one tell me if this options bellow can improve somethings related network: net.ipv4.tcp_low_latency = 1 net.core.rmem_default = 10000000 net.core.wmem_default = 10000000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rfc1337=1 net.ipv4.tcp_window_scaling=1 net.ipv4.tcp_workaround_signed_windows=1 net.ipv4.tcp_sack=1 net.ipv4.tcp_fack=1 net.ipv4.tcp_low_latency=1 net.ipv4.ip_no_pmtu_disc=0 net.ipv4.tcp_mtu_probing=1 net.ipv4.tcp_frto=2 net.ipv4.tcp_frto_response=2 net.ipv4.tcp_congestion_control=illinois Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From lcaron at unix-scripts.info Mon Aug 27 11:49:08 2018 From: lcaron at unix-scripts.info (Laurent CARON) Date: Mon, 27 Aug 2018 11:49:08 +0200 Subject: [PVE-User] Operations (snapshot) on resource pools Message-ID: Hi, The resource pool feature of proxmox is really nice for permissions, ... on VMs, storage, ... I'd like to know if it is possible to have mass operations on a pool. Eg: Snapshot all the VM from a pool (as defined in proxmox interface) at the same time Thanks From lindsay.mathieson at gmail.com Mon Aug 27 12:15:11 2018 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Mon, 27 Aug 2018 20:15:11 +1000 Subject: [PVE-User] Operations (snapshot) on resource pools In-Reply-To: References: Message-ID: <13b88422-f340-4521-565d-f4220f01fef8@gmail.com> On 27/08/2018 7:49 PM, Laurent CARON wrote: > Eg: Snapshot all the VM from a pool (as defined in proxmox interface) > at the same time Yes, that would be nice - and things like migrate, shutdown etc. -- Lindsay From thomas.naumann at ovgu.de Mon Aug 27 12:30:24 2018 From: thomas.naumann at ovgu.de (Naumann, Thomas) Date: Mon, 27 Aug 2018 10:30:24 +0000 Subject: [PVE-User] How to mount Ganesha-NFS via NFS-Plugin in Proxmox? Message-ID: Hi, for backup testing purposes we run a ceph-cluster with radosgw (S3) and nfs-ganesha to export s3 via nfs. cluster is running on ubuntu 16.04. package details: nfs-ganesha 2.6.2-0-gb9685b8-xenial nfs-ganesha-rgw:amd64 2.6.2-0-gb9685b8-1xenial radosgw 12.2.7-1xenial ceph 12.2.7-1xenial detail of "ganesha.conf": EXPORT { Export_ID=100; Path = "/"; Pseudo = /; Access_Type = RW; NFS_Protocols = 3,4; Transport_Protocols = UDP,TCP; Sectype = sys; Squash = No_Root_Squash; FSAL { Name = RGW; User_Id = "test"; Access_Key_Id ="test"; Secret_Access_Key = "test"; } } RGW { ceph_conf = "/etc/ceph/ceph.conf"; name = "client.rgw.blackhole-eh"; cluster = "ceph"; } LOG { Facility { name = FILE; destination = "/var/log/ganesha/ganesha.log"; enable = active; } } mounts via cli (mount -t nfs -o nfsvers=4.1,noauto,soft,sync,proto=tcp x.x.x.x:/ /mnt/ganesha/) are working without issues - thats fine. but how to mount this nfs-ressource via proxmox-nfs-plugin? "pvesm nfsscan x.x.x.x" ends without an result. best regards -- Thomas Naumann Otto-von-Guericke Universit?t Magdeburg Universit?tsrechenzentrum Universit?tsplatz 2 39106 Magdeburg fon: +49 391 67-58563 email: thomas.naumann at ovgu.de From thomas.naumann at ovgu.de Mon Aug 27 13:17:42 2018 From: thomas.naumann at ovgu.de (Naumann, Thomas) Date: Mon, 27 Aug 2018 11:17:42 +0000 Subject: [PVE-User] How to mount Ganesha-NFS via NFS-Plugin in Proxmox? In-Reply-To: <14949_1535365841_5B83D2D1_14949_1410_1_d08cdcafb59d10f8a948328b739788203da6a229.camel@ovgu.de> References: <14949_1535365841_5B83D2D1_14949_1410_1_d08cdcafb59d10f8a948328b739788203da6a229.camel@ovgu.de> Message-ID: <70b386f88e7f60f7fc780d70c2bf95d0a1235b1b.camel@ovgu.de> detail of "storage.cfg": nfs: test path /mnt/pve/ganesha/ server 10.54.2.4 export / options vers=3,soft content backup maxfiles 10 result in "journalctl": ... Aug 27 13:14:34 tr-25-3 pmxcfs[15192]: [status] notice: received log Aug 27 13:14:48 tr-25-3 pvestatd[15777]: file /etc/pve/storage.cfg line 79 (section 'test') - unable to parse value of 'export': invalid format - value does not look like a valid absolute path Aug 27 13:14:48 tr-25-3 pvestatd[15777]: file /etc/pve/storage.cfg line 82 (skip section 'test'): missing value for required option 'export' ... -- Thomas Naumann Otto-von-Guericke Universit?t Magdeburg Universit?tsrechenzentrum Universit?tsplatz 2 39106 Magdeburg fon: +49 391 67-58563 email: thomas.naumann at ovgu.de On Mon, 2018-08-27 at 10:30 +0000, Naumann, Thomas wrote: > Hi, > > for backup testing purposes we run a ceph-cluster with radosgw (S3) > and > nfs-ganesha to export s3 via nfs. cluster is running on ubuntu 16.04. > package details: > nfs-ganesha 2.6.2-0-gb9685b8- > xenial > nfs-ganesha-rgw:amd64 2.6.2-0-gb9685b8-1xenial > radosgw 12.2.7-1xenial > ceph 12.2.7-1xenial > > detail of "ganesha.conf": > EXPORT > { > Export_ID=100; > Path = "/"; > Pseudo = /; > Access_Type = RW; > NFS_Protocols = 3,4; > Transport_Protocols = UDP,TCP; > Sectype = sys; > Squash = No_Root_Squash; > > FSAL { > Name = RGW; > User_Id = "test"; > Access_Key_Id ="test"; > Secret_Access_Key = "test"; > } > } > > RGW { > ceph_conf = "/etc/ceph/ceph.conf"; > name = "client.rgw.blackhole-eh"; > cluster = "ceph"; > } > > LOG { > Facility { > name = FILE; > destination = "/var/log/ganesha/ganesha.log"; > enable = active; > } > } > > mounts via cli (mount -t nfs -o > nfsvers=4.1,noauto,soft,sync,proto=tcp > x.x.x.x:/ /mnt/ganesha/) are working without issues - thats fine. > > but how to mount this nfs-ressource via proxmox-nfs-plugin? > > "pvesm nfsscan x.x.x.x" ends without an result. > > > best regards > -- > Thomas Naumann > > Otto-von-Guericke Universit?t Magdeburg > Universit?tsrechenzentrum > Universit?tsplatz 2 > 39106 Magdeburg > > fon: +49 391 67-58563 > email: thomas.naumann at ovgu.de > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gilberto.nunes32 at gmail.com Mon Aug 27 13:34:35 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Mon, 27 Aug 2018 08:34:35 -0300 Subject: [PVE-User] Linux improvements for ceph and general running vm's In-Reply-To: <1689330932.4145024.1535351647608.JavaMail.zimbra@oxygem.tv> References: <1689330932.4145024.1535351647608.JavaMail.zimbra@oxygem.tv> Message-ID: Hi Alexander 1 - Ceph.conf are untouched 2 - network is gigaethernet with bond balance-rr 4 nic each 6 servers 3 - hdd all is sata 4 - each server has a standard ssd Thanks Em seg, 27 de ago de 2018 03:34, Alexandre DERUMIER escreveu: > Hi, > > do you want to improve latency or throughput ? > > > They are lot of thing to tune in ceph.conf before tuning network. > can you send your ceph.conf too ? (ceph cluster && proxmox nodes if ceph > is outside proxmox) > > ssd ? hdd ? > > > ----- Mail original ----- > De: "Gilberto Nunes" > ?: "proxmoxve" > Envoy?: Lundi 27 Ao?t 2018 05:00:52 > Objet: [PVE-User] Linux improvements for ceph and general running vm's > > Hi there > > Can any one tell me if this options bellow can improve somethings related > network: > > net.ipv4.tcp_low_latency = 1 > net.core.rmem_default = 10000000 > net.core.wmem_default = 10000000 > net.core.rmem_max = 16777216 > net.core.wmem_max = 16777216 > > net.ipv4.tcp_rfc1337=1 > > net.ipv4.tcp_window_scaling=1 > > net.ipv4.tcp_workaround_signed_windows=1 > > net.ipv4.tcp_sack=1 > > net.ipv4.tcp_fack=1 > > net.ipv4.tcp_low_latency=1 > > net.ipv4.ip_no_pmtu_disc=0 > > net.ipv4.tcp_mtu_probing=1 > > net.ipv4.tcp_frto=2 > > net.ipv4.tcp_frto_response=2 > > net.ipv4.tcp_congestion_control=illinois > > Thanks a lot > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From Jan.Dreyer at arvato.com Mon Aug 27 15:01:21 2018 From: Jan.Dreyer at arvato.com (Dreyer, Jan, SCM-IT) Date: Mon, 27 Aug 2018 13:01:21 +0000 Subject: [PVE-User] Problems with HP Smart Array P400 and ZFS In-Reply-To: <3518a8e3-ce55-6bee-e0e7-d9aa68ab3f21@proxmox.com> References: <8B7B44EF-AE80-4F19-BEAD-7EC6F6837EC5@arvato.com> <3518a8e3-ce55-6bee-e0e7-d9aa68ab3f21@proxmox.com> Message-ID: <8264CA12-33E4-429E-BE20-73E1B01F221A@arvato.com> Hi. First thanks for your (and Stoiko Ivanov?s) answers. > Am 24.08.2018 um 12:15 schrieb Thomas Lamprecht >: > > Are your firmware versions all up to date? No, I?m pretty sure it?s not. As we don?t have an active support agreement for such old hardware anymore, we are not able to get the newest. I will try with the latest available Firmware Maintenance CD (May 2011), maybe it covers this already. If not, I think we have to either live with the error or replace / remove the hardware ? Gru? / Regards ? Jan Dreyer IT Administrator ? Operations ? A-SCM-IT IOX Expert This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorized copying, disclosure or distribution of the information in this email is strictly forbidden. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From lists at merit.unu.edu Tue Aug 28 10:06:53 2018 From: lists at merit.unu.edu (lists) Date: Tue, 28 Aug 2018 10:06:53 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi Message-ID: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> Hi, I am trying to move a physical windows 2008 enterprise uefi installation (Version 6.0.6002 Service Pack 2 Build 6002) into proxmox, and I'm getting nowhere. Tried all kinds of approaches, and this was my latest attempt: Creating a full system backup using windows backup, and then boot the windows install iso in proxmox, to perform a system restore from this backup into proxmox. But as soon as I enable uefi in my proxmox VM config, the windows iso no longer boots. However, the physical server IS this same OS in uefi mode, the combination should work, I guess. Anyone with a tip or a tric..? This is proxmox 4.4-20, so it's a bit older. I could try it on a fresh new proxmox 5.2 install, but first I wanted to ask here. Anyone? MJ From yannis.milios at gmail.com Tue Aug 28 10:21:50 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Tue, 28 Aug 2018 09:21:50 +0100 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> Message-ID: Did you interrupt the boot process on the VM by pressing ESC, in order to select the DVD drive as the boot device ? On Tue, 28 Aug 2018 at 09:09, lists wrote: > Hi, > > I am trying to move a physical windows 2008 enterprise uefi installation > (Version 6.0.6002 Service Pack 2 Build 6002) into proxmox, and I'm > getting nowhere. > > Tried all kinds of approaches, and this was my latest attempt: > > Creating a full system backup using windows backup, and then boot the > windows install iso in proxmox, to perform a system restore from this > backup into proxmox. > > But as soon as I enable uefi in my proxmox VM config, the windows iso no > longer boots. However, the physical server IS this same OS in uefi mode, > the combination should work, I guess. > > Anyone with a tip or a tric..? > > This is proxmox 4.4-20, so it's a bit older. I could try it on a fresh > new proxmox 5.2 install, but first I wanted to ask here. > > Anyone? > > MJ > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Sent from Gmail Mobile From lists at merit.unu.edu Tue Aug 28 10:38:13 2018 From: lists at merit.unu.edu (lists) Date: Tue, 28 Aug 2018 10:38:13 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> Message-ID: Hi, yes :-) ("press any key to boot the windows dvd" or something like that) The "windows is loading files..." progress bar appears, and goes full, and then it just stops progressing. (and I waited many hours) MJ On 28-8-2018 10:21, Yannis Milios wrote: > Did you interrupt the boot process on the VM by pressing ESC, in order to > select the DVD drive as the boot device ? > > > > On Tue, 28 Aug 2018 at 09:09, lists wrote: > >> Hi, >> >> I am trying to move a physical windows 2008 enterprise uefi installation >> (Version 6.0.6002 Service Pack 2 Build 6002) into proxmox, and I'm >> getting nowhere. >> >> Tried all kinds of approaches, and this was my latest attempt: >> >> Creating a full system backup using windows backup, and then boot the >> windows install iso in proxmox, to perform a system restore from this >> backup into proxmox. >> >> But as soon as I enable uefi in my proxmox VM config, the windows iso no >> longer boots. However, the physical server IS this same OS in uefi mode, >> the combination should work, I guess. >> >> Anyone with a tip or a tric..? >> >> This is proxmox 4.4-20, so it's a bit older. I could try it on a fresh >> new proxmox 5.2 install, but first I wanted to ask here. >> >> Anyone? >> >> MJ >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> From lindsay.mathieson at gmail.com Tue Aug 28 10:54:00 2018 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Tue, 28 Aug 2018 18:54:00 +1000 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> Message-ID: <791c10d4-0b0a-c92a-f82c-cc1329325a75@gmail.com> What virtual hardware are you giving it? bus, disk etc. On 28/08/2018 6:06 PM, lists wrote: > Hi, > > I am trying to move a physical windows 2008 enterprise uefi > installation (Version 6.0.6002 Service Pack 2 Build 6002) into > proxmox, and I'm getting nowhere. > > Tried all kinds of approaches, and this was my latest attempt: > > Creating a full system backup using windows backup, and then boot the > windows install iso in proxmox, to perform a system restore from this > backup into proxmox. > > But as soon as I enable uefi in my proxmox VM config, the windows iso > no longer boots. However, the physical server IS this same OS in uefi > mode, the combination should work, I guess. > > Anyone with a tip or a tric..? > > This is proxmox 4.4-20, so it's a bit older. I could try it on a fresh > new proxmox 5.2 install, but first I wanted to ask here. > > Anyone? > > MJ > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Lindsay From lists at merit.unu.edu Tue Aug 28 10:59:11 2018 From: lists at merit.unu.edu (lists) Date: Tue, 28 Aug 2018 10:59:11 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: <791c10d4-0b0a-c92a-f82c-cc1329325a75@gmail.com> References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> <791c10d4-0b0a-c92a-f82c-cc1329325a75@gmail.com> Message-ID: Here is the VM config: > balloon: 0 > bios: ovmf > boot: dcn > bootdisk: ide0 > cores: 2 > efidisk0: local-btrfs:107/vm-107-disk-2.qcow2,size=128K > ide0: local-btrfs:107/vm-107-disk-1.qcow2,size=233G > ide2: local-btrfs:iso/win2008_enterprise_x64.iso,media=cdrom > memory: 8192 > name: win8-uefi > net0: e1000=BA:7F:7B:91:9B:1C,bridge=vmbr0 > numa: 0 > ostype: win8 > scsihw: virtio-scsi-pci > smbios1: uuid=626ef0a6-0161-492f-80b7-fbc437c4cdb2 > sockets: 2 OS Type set to Windows 8.x/2012/2012r2 KVM Hardware virtualization: yes Freez cpu at startup: no Protection: no Qemu Agent: no Bus: IDE Anything else I need to tell? MJ On 28-8-2018 10:54, Lindsay Mathieson wrote: > What virtual hardware are you giving it? bus, disk etc. > > On 28/08/2018 6:06 PM, lists wrote: >> Hi, >> >> I am trying to move a physical windows 2008 enterprise uefi >> installation (Version 6.0.6002 Service Pack 2 Build 6002) into >> proxmox, and I'm getting nowhere. >> >> Tried all kinds of approaches, and this was my latest attempt: >> >> Creating a full system backup using windows backup, and then boot the >> windows install iso in proxmox, to perform a system restore from this >> backup into proxmox. >> >> But as soon as I enable uefi in my proxmox VM config, the windows iso >> no longer boots. However, the physical server IS this same OS in uefi >> mode, the combination should work, I guess. >> >> Anyone with a tip or a tric..? >> >> This is proxmox 4.4-20, so it's a bit older. I could try it on a fresh >> new proxmox 5.2 install, but first I wanted to ask here. >> >> Anyone? >> >> MJ >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From lists at merit.unu.edu Tue Aug 28 11:20:22 2018 From: lists at merit.unu.edu (lists) Date: Tue, 28 Aug 2018 11:20:22 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> <791c10d4-0b0a-c92a-f82c-cc1329325a75@gmail.com> Message-ID: If, during windows iso boot, I press F8 for advanced options, and I select safe mode, I can the last line displayed is: Loaded: \windows\system32\drivers\disk.sys Then progress stops. I changed the VM disk config from IDE to SCSI, to see if that made a difference. But alas, no further progress... :-( MJ On 28-8-2018 10:59, lists wrote: > Here is the VM config: > >> balloon: 0 >> bios: ovmf >> boot: dcn >> bootdisk: ide0 >> cores: 2 >> efidisk0: local-btrfs:107/vm-107-disk-2.qcow2,size=128K >> ide0: local-btrfs:107/vm-107-disk-1.qcow2,size=233G >> ide2: local-btrfs:iso/win2008_enterprise_x64.iso,media=cdrom >> memory: 8192 >> name: win8-uefi >> net0: e1000=BA:7F:7B:91:9B:1C,bridge=vmbr0 >> numa: 0 >> ostype: win8 >> scsihw: virtio-scsi-pci >> smbios1: uuid=626ef0a6-0161-492f-80b7-fbc437c4cdb2 >> sockets: 2 > > OS Type set to Windows 8.x/2012/2012r2 > KVM Hardware virtualization: yes > Freez cpu at startup: no > Protection: no > Qemu Agent: no > Bus: IDE > > Anything else I need to tell? > > MJ > > On 28-8-2018 10:54, Lindsay Mathieson wrote: >> What virtual hardware are you giving it? bus, disk etc. >> >> On 28/08/2018 6:06 PM, lists wrote: >>> Hi, >>> >>> I am trying to move a physical windows 2008 enterprise uefi >>> installation (Version 6.0.6002 Service Pack 2 Build 6002) into >>> proxmox, and I'm getting nowhere. >>> >>> Tried all kinds of approaches, and this was my latest attempt: >>> >>> Creating a full system backup using windows backup, and then boot the >>> windows install iso in proxmox, to perform a system restore from this >>> backup into proxmox. >>> >>> But as soon as I enable uefi in my proxmox VM config, the windows iso >>> no longer boots. However, the physical server IS this same OS in uefi >>> mode, the combination should work, I guess. >>> >>> Anyone with a tip or a tric..? >>> >>> This is proxmox 4.4-20, so it's a bit older. I could try it on a >>> fresh new proxmox 5.2 install, but first I wanted to ask here. >>> >>> Anyone? >>> >>> MJ >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From yannis.milios at gmail.com Tue Aug 28 11:41:04 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Tue, 28 Aug 2018 10:41:04 +0100 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> Message-ID: > yes :-) ("press any key to boot the windows dvd" or something like that) > > The "windows is loading files..." progress bar appears, and goes full, > and then it just stops progressing. (and I waited many hours) > Have you tried to boot from the same ISO from BIOS legacy mode ? If you have same issue there, then perhaps you should check if the ISO file is corrupted. It could be also something related to the old qemu/pve version as well, I would try with a more recent version. Keep to disk controller to IDE for best compatibility, you can change that later to SCSI or something else... Y From t.lamprecht at proxmox.com Tue Aug 28 11:42:28 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 28 Aug 2018 11:42:28 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> <791c10d4-0b0a-c92a-f82c-cc1329325a75@gmail.com> Message-ID: <635c23aa-b0e5-71fa-814e-73c46f08a4ff@proxmox.com> PVE 5.2 contains a newer version of OVMF (our used EFI implementation) with a lot of fixes, updating could help - but is certainly no guarantee - especially as your windows is already in the process of booting. On 8/28/18 11:20 AM, lists wrote: > If, during windows iso boot, I press F8 for advanced options, and I select safe mode, I can the last line displayed is: > > Loaded: \windows\system32\drivers\disk.sys > > Then progress stops. > > I changed the VM disk config from IDE to SCSI, to see if that made a difference. But alas, no further progress... :-( > Hmm, maybe try SCSI but set the scsihw (SCSI Controller in the VM's Option Tab) to LSI 53C895A? Or SATA, Windows is often a bit picky about this stuff... > On 28-8-2018 10:59, lists wrote: >> Here is the VM config: >> >>> balloon: 0 >>> bios: ovmf >>> boot: dcn >>> bootdisk: ide0 >>> cores: 2 >>> efidisk0: local-btrfs:107/vm-107-disk-2.qcow2,size=128K >>> ide0: local-btrfs:107/vm-107-disk-1.qcow2,size=233G >>> ide2: local-btrfs:iso/win2008_enterprise_x64.iso,media=cdrom >>> memory: 8192 >>> name: win8-uefi >>> net0: e1000=BA:7F:7B:91:9B:1C,bridge=vmbr0 >>> numa: 0 >>> ostype: win8 >>> scsihw: virtio-scsi-pci >>> smbios1: uuid=626ef0a6-0161-492f-80b7-fbc437c4cdb2 >>> sockets: 2 You had a 2 socket with 2 cores each system on the bare metal setup? If not, please use 4 cores, suddenly having a NUMA system could also be a problem for windows. cheers, Thomas >> >> OS Type set to Windows 8.x/2012/2012r2 >> KVM Hardware virtualization: yes >> Freez cpu at startup: no >> Protection: no >> Qemu Agent: no >> Bus: IDE >> >> Anything else I need to tell? >> >> MJ >> >> On 28-8-2018 10:54, Lindsay Mathieson wrote: >>> What virtual hardware are you giving it? bus, disk etc. >>> >>> On 28/08/2018 6:06 PM, lists wrote: >>>> Hi, >>>> >>>> I am trying to move a physical windows 2008 enterprise uefi installation (Version 6.0.6002 Service Pack 2 Build 6002) into proxmox, and I'm getting nowhere. >>>> >>>> Tried all kinds of approaches, and this was my latest attempt: >>>> >>>> Creating a full system backup using windows backup, and then boot the windows install iso in proxmox, to perform a system restore from this backup into proxmox. >>>> >>>> But as soon as I enable uefi in my proxmox VM config, the windows iso no longer boots. However, the physical server IS this same OS in uefi mode, the combination should work, I guess. >>>> >>>> Anyone with a tip or a tric..? >>>> >>>> This is proxmox 4.4-20, so it's a bit older. I could try it on a fresh new proxmox 5.2 install, but first I wanted to ask here. >>>> >>>> Anyone? >>>> >>>> MJ >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From lists at merit.unu.edu Tue Aug 28 11:46:05 2018 From: lists at merit.unu.edu (lists) Date: Tue, 28 Aug 2018 11:46:05 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> Message-ID: Hi Yannis, On 28-8-2018 11:41, Yannis Milios wrote: > Have you tried to boot from the same ISO from BIOS legacy mode ? If you > have same issue there, then perhaps you should check if the ISO file is > corrupted. Yes, in legacy it works, also verified the MD5. > It could be also something related to the old qemu/pve version as well, I > would try with a more recent version. Keep to disk controller to IDE for > best compatibility, you can change that later to SCSI or something else... Perhaps yes, I should try installing latest proxmox, and try on that... MJ From lists at merit.unu.edu Tue Aug 28 12:02:16 2018 From: lists at merit.unu.edu (lists) Date: Tue, 28 Aug 2018 12:02:16 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: <635c23aa-b0e5-71fa-814e-73c46f08a4ff@proxmox.com> References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> <791c10d4-0b0a-c92a-f82c-cc1329325a75@gmail.com> <635c23aa-b0e5-71fa-814e-73c46f08a4ff@proxmox.com> Message-ID: <085e4c6c-365a-e96d-f7d0-bf5ca8a92592@merit.unu.edu> Hi Thomas, Thanks for the additional insights! I'll give them a try, and report back! MJ On 28-8-2018 11:42, Thomas Lamprecht wrote: > PVE 5.2 contains a newer version of OVMF (our used EFI implementation) with > a lot of fixes, updating could help - but is certainly no guarantee - > especially as your windows is already in the process of booting. > > On 8/28/18 11:20 AM, lists wrote: >> If, during windows iso boot, I press F8 for advanced options, and I select safe mode, I can the last line displayed is: >> >> Loaded: \windows\system32\drivers\disk.sys >> >> Then progress stops. >> >> I changed the VM disk config from IDE to SCSI, to see if that made a difference. But alas, no further progress... :-( >> > > Hmm, maybe try SCSI but set the scsihw (SCSI Controller in the VM's Option Tab) > to LSI 53C895A? Or SATA, Windows is often a bit picky about this stuff... > >> On 28-8-2018 10:59, lists wrote: >>> Here is the VM config: >>> >>>> balloon: 0 >>>> bios: ovmf >>>> boot: dcn >>>> bootdisk: ide0 >>>> cores: 2 >>>> efidisk0: local-btrfs:107/vm-107-disk-2.qcow2,size=128K >>>> ide0: local-btrfs:107/vm-107-disk-1.qcow2,size=233G >>>> ide2: local-btrfs:iso/win2008_enterprise_x64.iso,media=cdrom >>>> memory: 8192 >>>> name: win8-uefi >>>> net0: e1000=BA:7F:7B:91:9B:1C,bridge=vmbr0 >>>> numa: 0 >>>> ostype: win8 >>>> scsihw: virtio-scsi-pci >>>> smbios1: uuid=626ef0a6-0161-492f-80b7-fbc437c4cdb2 >>>> sockets: 2 > > You had a 2 socket with 2 cores each system on the bare metal setup? > If not, please use 4 cores, suddenly having a NUMA system could also be > a problem for windows. > > cheers, > Thomas > >>> >>> OS Type set to Windows 8.x/2012/2012r2 >>> KVM Hardware virtualization: yes >>> Freez cpu at startup: no >>> Protection: no >>> Qemu Agent: no >>> Bus: IDE >>> >>> Anything else I need to tell? >>> >>> MJ >>> >>> On 28-8-2018 10:54, Lindsay Mathieson wrote: >>>> What virtual hardware are you giving it? bus, disk etc. >>>> >>>> On 28/08/2018 6:06 PM, lists wrote: >>>>> Hi, >>>>> >>>>> I am trying to move a physical windows 2008 enterprise uefi installation (Version 6.0.6002 Service Pack 2 Build 6002) into proxmox, and I'm getting nowhere. >>>>> >>>>> Tried all kinds of approaches, and this was my latest attempt: >>>>> >>>>> Creating a full system backup using windows backup, and then boot the windows install iso in proxmox, to perform a system restore from this backup into proxmox. >>>>> >>>>> But as soon as I enable uefi in my proxmox VM config, the windows iso no longer boots. However, the physical server IS this same OS in uefi mode, the combination should work, I guess. >>>>> >>>>> Anyone with a tip or a tric..? >>>>> >>>>> This is proxmox 4.4-20, so it's a bit older. I could try it on a fresh new proxmox 5.2 install, but first I wanted to ask here. >>>>> >>>>> Anyone? >>>>> >>>>> MJ >>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user at pve.proxmox.com >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From mark at tuxis.nl Tue Aug 28 16:27:42 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 28 Aug 2018 16:27:42 +0200 Subject: [PVE-User] VZDump and Ceph Snapshots Message-ID: <50f6c4cc40313a6e5e0bffc261c712248bbd6533.camel@tuxis.nl> Hi, I'm currently using vzdump with the snapshot method to periodically generate vma-files for disasterrecovery. vzdump in snapshot mode instructs Qemu to start backing up the disk to a specific location, and while doing so, VM users can suffer poor performance. We run practically all VMs on Ceph storage, which has snapshot functionality. Would it be feasable to alter VZDump to use the following flow: * Start * guest-fs-freeze * rbd snap $image at vzdump_$timstamp * guest-fs-thaw * qemu-img convert -O raw rbd:$image at vzdump_$timstamp $tmpdir * vma create * rbd snap rm $image at vzdump_$timstamp * Done Regards, -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From christian at meiring.de Tue Aug 28 16:34:06 2018 From: christian at meiring.de (Christian Meiring) Date: Tue, 28 Aug 2018 16:34:06 +0200 Subject: [PVE-User] windows 2008 enterprise with uefi In-Reply-To: References: <93f795a1-0e04-257f-9992-cd898b626b41@merit.unu.edu> Message-ID: <46535519.rRm6eYJmBc@meir-bor> Am Dienstag, 28. August 2018, 11:20:22 CEST schrieb lists: > If, during windows iso boot, I press F8 for advanced options, and I > select safe mode, I can the last line displayed is: > > Loaded: \windows\system32\drivers\disk.sys > > Then progress stops. i do not know the whole magic, but on older windows installations (xp, 2k3, maybe w7) you must apply a MergeIDE.reg entry. This must be done inside running physical machine, before the backup. > > I changed the VM disk config from IDE to SCSI, to see if that made a > difference. But alas, no further progress... :-( > > MJ > > > > On 28-8-2018 10:59, lists wrote: > > Here is the VM config: > > > >> balloon: 0 > >> bios: ovmf > >> boot: dcn > >> bootdisk: ide0 > >> cores: 2 > >> efidisk0: local-btrfs:107/vm-107-disk-2.qcow2,size=128K > >> ide0: local-btrfs:107/vm-107-disk-1.qcow2,size=233G > >> ide2: local-btrfs:iso/win2008_enterprise_x64.iso,media=cdrom > >> memory: 8192 > >> name: win8-uefi > >> net0: e1000=BA:7F:7B:91:9B:1C,bridge=vmbr0 > >> numa: 0 > >> ostype: win8 > >> scsihw: virtio-scsi-pci > >> smbios1: uuid=626ef0a6-0161-492f-80b7-fbc437c4cdb2 > >> sockets: 2 > > > > OS Type set to Windows 8.x/2012/2012r2 > > KVM Hardware virtualization: yes > > Freez cpu at startup: no > > Protection: no > > Qemu Agent: no > > Bus: IDE > > > > Anything else I need to tell? > > > > MJ > > > > On 28-8-2018 10:54, Lindsay Mathieson wrote: > >> What virtual hardware are you giving it? bus, disk etc. > >> > >> On 28/08/2018 6:06 PM, lists wrote: > >>> Hi, > >>> > >>> I am trying to move a physical windows 2008 enterprise uefi > >>> installation (Version 6.0.6002 Service Pack 2 Build 6002) into > >>> proxmox, and I'm getting nowhere. > >>> > >>> Tried all kinds of approaches, and this was my latest attempt: > >>> > >>> Creating a full system backup using windows backup, and then boot the > >>> windows install iso in proxmox, to perform a system restore from this > >>> backup into proxmox. > >>> > >>> But as soon as I enable uefi in my proxmox VM config, the windows iso > >>> no longer boots. However, the physical server IS this same OS in uefi > >>> mode, the combination should work, I guess. > >>> > >>> Anyone with a tip or a tric..? > >>> > >>> This is proxmox 4.4-20, so it's a bit older. I could try it on a > >>> fresh new proxmox 5.2 install, but first I wanted to ask here. > >>> > >>> Anyone? > >>> > >>> MJ > >>> _______________________________________________ > >>> pve-user mailing list > >>> pve-user at pve.proxmox.com > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > >> > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Christian Meiring Wilh. W?lfing GmbH & Co.KG Weseler Landstra?e 26 D - 46325 Borken Tel +49 2861 8004 193 Fax +49 2861 8004 243 http://www.wilh-wuelfing.de eMail: christian.meiring at wilh-wuelfing.de Kommanditgesellschaft, Sitz Borken, Reg.-Gericht Coesfeld, HRA 3479 Pers?nlich haftende Gesellschafterin: W?lfing-Beteiligungs-GmbH, Sitz Borken, Reg.-Gericht Coesfeld, HRB 5340 Gesch?ftsf?hrer: Johannes Dowe, Josef K?lker From aderumier at odiso.com Wed Aug 29 08:11:08 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Wed, 29 Aug 2018 08:11:08 +0200 (CEST) Subject: [PVE-User] VZDump and Ceph Snapshots In-Reply-To: <50f6c4cc40313a6e5e0bffc261c712248bbd6533.camel@tuxis.nl> References: <50f6c4cc40313a6e5e0bffc261c712248bbd6533.camel@tuxis.nl> Message-ID: <421551402.4853146.1535523068942.JavaMail.zimbra@oxygem.tv> Hi, vma backup only work on running vm (attached disk), so no, it's not possible currently. Currently, I'm doing ceph backup to another remote ceph backup cluster with custom script * Start * guest-fs-freeze * rbd snap $image at vzdump_$timstamp * guest-fs-thaw * rbd export $image at vzdump_$timstamp | rbd import .... .... works very fine. (with incremental backup) I'll try to put my code on github soon. (I have a nice gui/cli too, to restore a full vm, or some files inside the backup) ----- Mail original ----- De: "Mark Schouten" ?: "proxmoxve" Envoy?: Mardi 28 Ao?t 2018 16:27:42 Objet: [PVE-User] VZDump and Ceph Snapshots Hi, I'm currently using vzdump with the snapshot method to periodically generate vma-files for disasterrecovery. vzdump in snapshot mode instructs Qemu to start backing up the disk to a specific location, and while doing so, VM users can suffer poor performance. We run practically all VMs on Ceph storage, which has snapshot functionality. Would it be feasable to alter VZDump to use the following flow: * Start * guest-fs-freeze * rbd snap $image at vzdump_$timstamp * guest-fs-thaw * qemu-img convert -O raw rbd:$image at vzdump_$timstamp $tmpdir * vma create * rbd snap rm $image at vzdump_$timstamp * Done Regards, -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mark at tuxis.nl Wed Aug 29 08:54:38 2018 From: mark at tuxis.nl (Mark Schouten) Date: Wed, 29 Aug 2018 08:54:38 +0200 Subject: [PVE-User] VZDump and Ceph Snapshots In-Reply-To: <421551402.4853146.1535523068942.JavaMail.zimbra@oxygem.tv> References: <50f6c4cc40313a6e5e0bffc261c712248bbd6533.camel@tuxis.nl> <421551402.4853146.1535523068942.JavaMail.zimbra@oxygem.tv> Message-ID: <2ca0f86af258876be9f3202ab8344ca1993378fa.camel@tuxis.nl> On Wed, 2018-08-29 at 08:11 +0200, Alexandre DERUMIER wrote: > vma backup only work on running vm (attached disk), > so no, it's not possible currently. Doesn't vma just create an archive of configfiles and raw images in a directory? > Currently, I'm doing ceph backup to another remote ceph backup > cluster > with custom script I want my customers to be able to restore easily without Ceph-hassle. So I want to put the functionality in vzdump.. > * Start > * guest-fs-freeze > * rbd snap $image at vzdump_$timstamp > * guest-fs-thaw > * rbd export $image at vzdump_$timstamp | rbd import .... I think we are allow to do rbd export to STDOUT and into vma? Proxmox developers? -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From mark at tuxis.nl Wed Aug 29 09:22:08 2018 From: mark at tuxis.nl (Mark Schouten) Date: Wed, 29 Aug 2018 09:22:08 +0200 Subject: [PVE-User] mlock'ing VM processes Message-ID: Hi, I have issues on clusters with lots of Windows VM's, where the host decides to swap VM memory in favor of (it seems) filecache. root at proxmox05:~# free -m total used free shared buffers cach ed Mem: 386874 382772 4101 326 0 2080 23 -/+ buffers/cache: 174749 212125 Swap: 8191 1775 6416 The customers are experiencing slower VM's because of parts of their memory being swapped out. It looks like libvirt has a solution to lock a VM's memory into 'real' memory. [1] Is there a way we can make Proxmox do the same? [1]: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_tuning_and_optimization_guide/chap-virtualization_tuning_optimization_guide-memory-backing -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From elacunza at binovo.es Wed Aug 29 14:04:56 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 29 Aug 2018 14:04:56 +0200 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: Message-ID: You should change the weight of the 8TB disk, so that they have the same as the other 4TB disks. Thanks should fix the performance issue, but you'd waste half space on those 8TB disks :) El 23/08/18 a las 00:19, Brian : escribi?: > Its really not a great idea because the larger drives will tend to > get more writes so your performance won't be as good as all the same > size where the writes will be distributed more evenly. > > On Wed, Aug 22, 2018 at 8:05 PM Gilberto Nunes > wrote: >> Hi there >> >> >> It's possible create a Ceph cluster with 4 servers, which has differents >> disk sizes: >> >> Server A - 2x 4TB >> Server B, C - 2x 8TB >> Server D - 2x 4TB >> >> This is ok? >> >> Thanks >> >> --- >> Gilberto Nunes Ferreira >> >> (47) 3025-5907 >> (47) 99676-7530 - Whatsapp / Telegram >> >> Skype: gilberto.nunes36 >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From marcus.haarmann at midoco.de Wed Aug 29 17:56:51 2018 From: marcus.haarmann at midoco.de (Marcus Haarmann) Date: Wed, 29 Aug 2018 17:56:51 +0200 (CEST) Subject: [PVE-User] Snapshot rollback slow Message-ID: <1022407081.5614.1535558211407.JavaMail.zimbra@midoco.de> Hi, we have a small Proxmox cluster, on top of ceph. Version is proxmox 5.2.6, Ceph 12.2.5 luminous Hardware is 4 machines, dual Xeon E5, 128 GB RAM local SATA (raid1) for OS local SSD for OSD (2 OSD per machine, no Raid here) 4x 10GBit (copper) NICs We came upon the following situation: VM snapshot was created to perform a dangerous installation process, which should be revertable Installation was done and a rollback to snapshot was initiated (because something went wrong). However, the rollback of snapshot took > 1 hour and during this timeframe, the whole cluster was reacting veeeeery slow. We tried to find out the reason for this, and it looks like an I/O bottleneck. For some reason, the main I/O was done on two local OSD processes (on the same host where the VM was running). The iostat output said the data transmission rate was about 30MB/s per OSD disk but util was 100%. (whatever this means) The underlying SSD are not damaged and have a significant higher throughput normally. OSD is based on filestore/XFS (we encountered some problems with bluestore and decided to use filestore again) There are a lot of read/write operations in parallel at this time. Normal cluster operation is relatively fluent, only copying machines affects I/O but we can see transfer rates > 200 MB/s in iostat in this case. (this is not very fast for the SSD disks from my point of view, but it is not only sequential write) Also, I/O utilization is not near 100% when a copy action is executed. SSD and SATA disks are on separate controllers. Any ideas where to tune for better snapshot rollback performance ? I am not sure how the placement of the snapshot data is done from proxmox or ceph. Under the hood, there are rbd devices, which are snapshotted. So it should be up to the ceph logic where the snapshots are done (maybe depending on the initial layout of the original device ) ? Would the crush map influence that ? Also live backup takes snapshots as I can see. We have had very strange locks on running backups in the past (mostly gone since the disks were put on separate controllers). Could this be the same reason ? Another thing we found is the following (not on all hosts): [614673.831726] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon [614673.848249] libceph: mon2 192.168.16.34:6789 session established [614704.551754] libceph: mon2 192.168.16.34:6789 session lost, hunting for new mon [614704.552729] libceph: mon1 192.168.16.32:6789 session established [614735.271779] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon [614735.272339] libceph: mon2 192.168.16.34:6789 session established This leads to a kernel problem, which is still not solved (because not backported to 4.15). I am not sure if this is a reaction to a ceph problem or the reason for the ceph problem. Any thoughts on this ? Marcus Haarmann From yannis.milios at gmail.com Wed Aug 29 19:02:49 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Wed, 29 Aug 2018 18:02:49 +0100 Subject: [PVE-User] Snapshot rollback slow In-Reply-To: <1022407081.5614.1535558211407.JavaMail.zimbra@midoco.de> References: <1022407081.5614.1535558211407.JavaMail.zimbra@midoco.de> Message-ID: Can?t comment on the I/O issues, but in regards to the snapshot rollback, I would personally prefer to clone the snapshot instead of rolling back. It has been proven for me much faster to recover in emergencies. Then, after recovering, to release the clone from the its snapshot reference, you can flatten the clone. You can find this info in Ceph docs. On Wed, 29 Aug 2018 at 16:56, Marcus Haarmann wrote: > Hi, > > we have a small Proxmox cluster, on top of ceph. > Version is proxmox 5.2.6, Ceph 12.2.5 luminous > Hardware is 4 machines, dual Xeon E5, 128 GB RAM > local SATA (raid1) for OS > local SSD for OSD (2 OSD per machine, no Raid here) > 4x 10GBit (copper) NICs > > We came upon the following situation: > VM snapshot was created to perform a dangerous installation process, which > should be revertable > Installation was done and a rollback to snapshot was initiated (because > something went wrong). > However, the rollback of snapshot took > 1 hour and during this timeframe, > the whole cluster > was reacting veeeeery slow. > We tried to find out the reason for this, and it looks like an I/O > bottleneck. > For some reason, the main I/O was done on two local OSD processes (on the > same host where the VM was running). > The iostat output said the data transmission rate was about 30MB/s per OSD > disk but util was 100%. (whatever this means) > The underlying SSD are not damaged and have a significant higher > throughput normally. > OSD is based on filestore/XFS (we encountered some problems with bluestore > and decided to use filestore again) > There are a lot of read/write operations in parallel at this time. > > Normal cluster operation is relatively fluent, only copying machines > affects I/O but we can see > transfer rates > 200 MB/s in iostat in this case. (this is not very fast > for the SSD disks from my point of view, > but it is not only sequential write) > Also, I/O utilization is not near 100% when a copy action is executed. > > SSD and SATA disks are on separate controllers. > > Any ideas where to tune for better snapshot rollback performance ? > I am not sure how the placement of the snapshot data is done from proxmox > or ceph. > > Under the hood, there are rbd devices, which are snapshotted. So it should > be up to the ceph logic > where the snapshots are done (maybe depending on the initial layout of the > original device ) ? > Would the crush map influence that ? > > Also live backup takes snapshots as I can see. We have had very strange > locks on running backups > in the past (mostly gone since the disks were put on separate > controllers). > > Could this be the same reason ? > > Another thing we found is the following (not on all hosts): > [614673.831726] libceph: mon1 192.168.16.32:6789 session lost, hunting > for new mon > [614673.848249] libceph: mon2 192.168.16.34:6789 session established > [614704.551754] libceph: mon2 192.168.16.34:6789 session lost, hunting > for new mon > [614704.552729] libceph: mon1 192.168.16.32:6789 session established > [614735.271779] libceph: mon1 192.168.16.32:6789 session lost, hunting > for new mon > [614735.272339] libceph: mon2 192.168.16.34:6789 session established > > This leads to a kernel problem, which is still not solved (because not > backported to 4.15). > I am not sure if this is a reaction to a ceph problem or the reason for > the ceph problem. > > Any thoughts on this ? > > Marcus Haarmann > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Sent from Gmail Mobile From gaio at sv.lnf.it Wed Aug 29 19:09:35 2018 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Wed, 29 Aug 2018 19:09:35 +0200 Subject: [PVE-User] Snapshot rollback slow In-Reply-To: <1022407081.5614.1535558211407.JavaMail.zimbra@midoco.de> References: <1022407081.5614.1535558211407.JavaMail.zimbra@midoco.de> Message-ID: <20180829170935.GU3632@sv.lnf.it> Mandi! Marcus Haarmann In chel di` si favelave... > However, the rollback of snapshot took > 1 hour and during this timeframe, the whole cluster > was reacting veeeeery slow. AFAIK this is a 'feature' of ceph (again AFAIK fixed/better handled in recent ceph version and on bluestore): a deletion (of a volume or a snapshot rollback) trigger a 'write amplification'. In older ceph version there's some workarounds... Sorry for be not so precise, look at ceph mailing list archive for more precise info... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gilberto.nunes32 at gmail.com Thu Aug 30 14:16:18 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 30 Aug 2018 09:16:18 -0300 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: Message-ID: Hi there Eneko Sorry.... Can you show me how can I do that? I meant, change de weight??? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-29 9:04 GMT-03:00 Eneko Lacunza : > You should change the weight of the 8TB disk, so that they have the same > as the other 4TB disks. > > Thanks should fix the performance issue, but you'd waste half space on > those 8TB disks :) > > El 23/08/18 a las 00:19, Brian : escribi?: > > Its really not a great idea because the larger drives will tend to >> get more writes so your performance won't be as good as all the same >> size where the writes will be distributed more evenly. >> >> On Wed, Aug 22, 2018 at 8:05 PM Gilberto Nunes >> wrote: >> >>> Hi there >>> >>> >>> It's possible create a Ceph cluster with 4 servers, which has differents >>> disk sizes: >>> >>> Server A - 2x 4TB >>> Server B, C - 2x 8TB >>> Server D - 2x 4TB >>> >>> This is ok? >>> >>> Thanks >>> >>> --- >>> Gilberto Nunes Ferreira >>> >>> (47) 3025-5907 >>> (47) 99676-7530 - Whatsapp / Telegram >>> >>> Skype: gilberto.nunes36 >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > > -- > Zuzendari Teknikoa / Director T?cnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mark at tuxis.nl Thu Aug 30 14:22:02 2018 From: mark at tuxis.nl (Mark Schouten) Date: Thu, 30 Aug 2018 14:22:02 +0200 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: Message-ID: On Wed, 2018-08-29 at 14:04 +0200, Eneko Lacunza wrote: > You should change the weight of the 8TB disk, so that they have the > same > as the other 4TB disks. > > Thanks should fix the performance issue, but you'd waste half space > on > those 8TB disks :) Wouldn't it be more efficient to do just place a 4Tb and a 8Tb disk in each server? Changing weight will not cause the available space counters to drop accordingly, I think. So it's probably confusing.. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From gilberto.nunes32 at gmail.com Thu Aug 30 14:30:47 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 30 Aug 2018 09:30:47 -0300 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: Message-ID: The environmente has this configuration: CEPH-01 4x 4 TB CEPH-02 4x 3 TB CEPH-03 2x 3 TB 1x 2 TB CEPH-04 4x 2 TB CEPH-05 2x 8 TB CEPH-06 2x 3 TB 1x 2 TB 1x 1 TB Any advice to, at least, mitigate the low performance? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-30 9:22 GMT-03:00 Mark Schouten : > On Wed, 2018-08-29 at 14:04 +0200, Eneko Lacunza wrote: > > You should change the weight of the 8TB disk, so that they have the > > same > > as the other 4TB disks. > > > > Thanks should fix the performance issue, but you'd waste half space > > on > > those 8TB disks :) > > Wouldn't it be more efficient to do just place a 4Tb and a 8Tb disk in > each server? > > Changing weight will not cause the available space counters to drop > accordingly, I think. So it's probably confusing.. > > -- > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ > Mark Schouten | Tuxis Internet Engineering > KvK: 61527076 | http://www.tuxis.nl/ > T: 0318 200208 | info at tuxis.nl > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mark at tuxis.nl Thu Aug 30 14:37:48 2018 From: mark at tuxis.nl (Mark Schouten) Date: Thu, 30 Aug 2018 14:37:48 +0200 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: Message-ID: <08e79360ae02c218fe9b1aa49c4a996f2c022ffd.camel@tuxis.nl> On Thu, 2018-08-30 at 09:30 -0300, Gilberto Nunes wrote: > Any advice to, at least, mitigate the low performance? Balance the number of spinning disks and the size per server. This will probably be the safest. It's not said that not balancing degrades performance, it's said that it might potentially cause degraded performance. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From elacunza at binovo.es Thu Aug 30 14:57:53 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Thu, 30 Aug 2018 14:57:53 +0200 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: <08e79360ae02c218fe9b1aa49c4a996f2c022ffd.camel@tuxis.nl> References: <08e79360ae02c218fe9b1aa49c4a996f2c022ffd.camel@tuxis.nl> Message-ID: El 30/08/18 a las 14:37, Mark Schouten escribi?: > On Thu, 2018-08-30 at 09:30 -0300, Gilberto Nunes wrote: >> Any advice to, at least, mitigate the low performance? > Balance the number of spinning disks and the size per server. This will > probably be the safest. > > It's not said that not balancing degrades performance, it's said that > it might potentially cause degraded performance. Yes I agree, although that might probably overload the biggest disks, too. But all depends on the space and performance requirements/desires, really :) Cheers Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From gilberto.nunes32 at gmail.com Thu Aug 30 15:23:54 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 30 Aug 2018 10:23:54 -0300 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: <08e79360ae02c218fe9b1aa49c4a996f2c022ffd.camel@tuxis.nl> Message-ID: SO, what you guys think about this HDD distribuiton? CEPH-01 1x 3 TB 1x 2 TB CEPH-02 1x 4 TB 1x 3 TB CEPH-03 1x 4 TB 1x 3 TB CEPH-04 1x 4 TB 1x 3 TB 1x 2 TB CEPH-05 1x 8 TB 1x 2 TB CEPH-06 1x 3 TB 1x 1 TB 1x 8 TB --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-30 9:57 GMT-03:00 Eneko Lacunza : > El 30/08/18 a las 14:37, Mark Schouten escribi?: > >> On Thu, 2018-08-30 at 09:30 -0300, Gilberto Nunes wrote: >> >>> Any advice to, at least, mitigate the low performance? >>> >> Balance the number of spinning disks and the size per server. This will >> probably be the safest. >> >> It's not said that not balancing degrades performance, it's said that >> it might potentially cause degraded performance. >> > Yes I agree, although that might probably overload the biggest disks, too. > But all depends on the space and performance requirements/desires, really :) > > Cheers > Eneko > > -- > Zuzendari Teknikoa / Director T?cnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Thu Aug 30 15:27:56 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 30 Aug 2018 10:27:56 -0300 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: <08e79360ae02c218fe9b1aa49c4a996f2c022ffd.camel@tuxis.nl> Message-ID: Right now the ceph are very slow 343510/2089155 objects misplaced (16.443%) Status HEALTH_WARN Monitors pve-ceph01: pve-ceph02: pve-ceph03: pve-ceph04: pve-ceph05: pve-ceph06: OSDs In Out Up 21 0 Down 0 0 Total: 21 PGs active+clean: 157 active+recovery_wait+remapped: 1 active+remapped+backfill_wait: 82 active+remapped+backfilling: 2 active+undersized+degraded+remapped+backfill_wait: 8 Usage 7.68 TiB of 62.31 TiB Reads: Writes: IOPS: Reads: IOPS: Writes: () Degraded data redundancy: 21495/2089170 objects degraded (1.029%), 8 pgs degraded, 8 pgs undersized pg 21.0 is stuck undersized for 63693.346103, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,9] pg 21.2 is stuck undersized for 63693.346973, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,10] pg 21.6f is stuck undersized for 62453.277248, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,5] pg 21.8b is stuck undersized for 63693.361835, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,8] pg 21.c3 is stuck undersized for 63693.321337, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,9] pg 21.c5 is stuck undersized for 66587.797684, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,8] pg 21.d4 is stuck undersized for 62453.047415, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,6] pg 21.e1 is stuck undersized for 62453.276631, current state active+undersized+degraded+remapped+backfill_wait, last acting [2,5] --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-30 10:23 GMT-03:00 Gilberto Nunes : > SO, what you guys think about this HDD distribuiton? > > CEPH-01 > 1x 3 TB > 1x 2 TB > > CEPH-02 > 1x 4 TB > 1x 3 TB > > CEPH-03 > 1x 4 TB > 1x 3 TB > > CEPH-04 > 1x 4 TB > 1x 3 TB > 1x 2 TB > > CEPH-05 > 1x 8 TB > 1x 2 TB > > CEPH-06 > 1x 3 TB > 1x 1 TB > 1x 8 TB > > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > 2018-08-30 9:57 GMT-03:00 Eneko Lacunza : > >> El 30/08/18 a las 14:37, Mark Schouten escribi?: >> >>> On Thu, 2018-08-30 at 09:30 -0300, Gilberto Nunes wrote: >>> >>>> Any advice to, at least, mitigate the low performance? >>>> >>> Balance the number of spinning disks and the size per server. This will >>> probably be the safest. >>> >>> It's not said that not balancing degrades performance, it's said that >>> it might potentially cause degraded performance. >>> >> Yes I agree, although that might probably overload the biggest disks, >> too. But all depends on the space and performance requirements/desires, >> really :) >> >> Cheers >> Eneko >> >> -- >> Zuzendari Teknikoa / Director T?cnico >> Binovo IT Human Project, S.L. >> Telf. 943569206 >> Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) >> www.binovo.es >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > From infolist at schwarz-fr.net Thu Aug 30 16:47:44 2018 From: infolist at schwarz-fr.net (Phil Schwarz) Date: Thu, 30 Aug 2018 16:47:44 +0200 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: References: <08e79360ae02c218fe9b1aa49c4a996f2c022ffd.camel@tuxis.nl> Message-ID: <6bbcd443-9fc8-3c1f-fa06-bc606982c214@schwarz-fr.net> Hope you did change a single disk at a time ! Be warned (if not) that moving an OSD from a server to another triggers a rebalancing of almost the complete datas stored upon in order to follow crushmap. For instance exchanging two OSDs between servers result in a complete rebalance of the two OSDS,a ccording to my knowledge. 16% of misplaced datas could be acceptable or not depending on your needs of redundancy and throughput, but it's not a low value that could be underestimated. Best regards Le 30/08/2018 ? 15:27, Gilberto Nunes a ?crit?: > Right now the ceph are very slow > > 343510/2089155 objects misplaced (16.443%) > Status > > HEALTH_WARN > Monitors > pve-ceph01: > pve-ceph02: > pve-ceph03: > pve-ceph04: > pve-ceph05: > pve-ceph06: > OSDs > In Out > Up 21 0 > Down 0 0 > Total: 21 > PGs > active+clean: > 157 > > active+recovery_wait+remapped: > 1 > > active+remapped+backfill_wait: > 82 > > active+remapped+backfilling: > 2 > > active+undersized+degraded+remapped+backfill_wait: > 8 > > Usage > 7.68 TiB of 62.31 TiB > Reads: > Writes: > IOPS: Reads: > IOPS: Writes: > > () > Degraded data redundancy: 21495/2089170 objects degraded (1.029%), 8 pgs > degraded, 8 pgs undersized > > pg 21.0 is stuck undersized for 63693.346103, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,9] > pg 21.2 is stuck undersized for 63693.346973, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,10] > pg 21.6f is stuck undersized for 62453.277248, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,5] > pg 21.8b is stuck undersized for 63693.361835, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,8] > pg 21.c3 is stuck undersized for 63693.321337, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,9] > pg 21.c5 is stuck undersized for 66587.797684, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,8] > pg 21.d4 is stuck undersized for 62453.047415, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,6] > pg 21.e1 is stuck undersized for 62453.276631, current state > active+undersized+degraded+remapped+backfill_wait, last acting [2,5] > > > > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > 2018-08-30 10:23 GMT-03:00 Gilberto Nunes : > >> SO, what you guys think about this HDD distribuiton? >> >> CEPH-01 >> 1x 3 TB >> 1x 2 TB >> >> CEPH-02 >> 1x 4 TB >> 1x 3 TB >> >> CEPH-03 >> 1x 4 TB >> 1x 3 TB >> >> CEPH-04 >> 1x 4 TB >> 1x 3 TB >> 1x 2 TB >> >> CEPH-05 >> 1x 8 TB >> 1x 2 TB >> >> CEPH-06 >> 1x 3 TB >> 1x 1 TB >> 1x 8 TB >> >> >> --- >> Gilberto Nunes Ferreira >> >> (47) 3025-5907 >> (47) 99676-7530 - Whatsapp / Telegram >> >> Skype: gilberto.nunes36 >> >> From gilberto.nunes32 at gmail.com Thu Aug 30 16:47:25 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 30 Aug 2018 11:47:25 -0300 Subject: [PVE-User] PRoxmox and ceph with just 3 server. Message-ID: Hi there It's possible create a scenario with 3 PowerEdge r540, with Proxmox and Ceph. The server has this configuration: 32 GB memory SAS 2x 300 GB SSD 1x 480 GB 2 VM with SQL and Windows server. Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From martin at proxmox.com Thu Aug 30 20:40:51 2018 From: martin at proxmox.com (Martin Maurer) Date: Thu, 30 Aug 2018 20:40:51 +0200 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: References: Message-ID: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> Hello, Not really. Please read in detail the following: https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ On 30.08.2018 16:47, Gilberto Nunes wrote: > Hi there > > It's possible create a scenario with 3 PowerEdge r540, with Proxmox and > Ceph. > The server has this configuration: > > 32 GB memory > SAS 2x 300 GB > SSD 1x 480 GB > > 2 VM with SQL and Windows server. > > Thanks > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Best Regards, Martin Maurer martin at proxmox.com http://www.proxmox.com ____________________________________________________________________ Proxmox Server Solutions GmbH Br?uhausgasse 37, 1050 Vienna, Austria Commercial register no.: FN 258879 f Registration office: Handelsgericht Wien From gilberto.nunes32 at gmail.com Thu Aug 30 20:46:50 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 30 Aug 2018 15:46:50 -0300 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> Message-ID: Hi Martin. Not really worried about highest performance, but to know if it will work properly, mainly HA! I plan work with mesh network too. Tanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-30 15:40 GMT-03:00 Martin Maurer : > Hello, > > Not really. Please read in detail the following: > > https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ > > > On 30.08.2018 16:47, Gilberto Nunes wrote: > >> Hi there >> >> It's possible create a scenario with 3 PowerEdge r540, with Proxmox and >> Ceph. >> The server has this configuration: >> >> 32 GB memory >> SAS 2x 300 GB >> SSD 1x 480 GB >> >> 2 VM with SQL and Windows server. >> >> Thanks >> >> --- >> Gilberto Nunes Ferreira >> >> (47) 3025-5907 >> (47) 99676-7530 - Whatsapp / Telegram >> >> Skype: gilberto.nunes36 >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> > -- > Best Regards, > > Martin Maurer > > martin at proxmox.com > http://www.proxmox.com > > ____________________________________________________________________ > Proxmox Server Solutions GmbH > Br?uhausgasse 37, 1050 Vienna, Austria > Commercial register no.: FN 258879 f > Registration office: Handelsgericht Wien > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From ronny+pve-user at aasen.cx Thu Aug 30 23:16:20 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Thu, 30 Aug 2018 23:16:20 +0200 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> Message-ID: <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> if HA is important, you should consider having a 4th ceph osd server (does not have to also be proxmox) with ceph's default of 3 replicas, that you will want to use in? a production setup, you do not have any failure domain. IOW the loss of any one node = a degraded ceph cluster.? if you have an additional node, ceph will rebalance and return to HEALTH_OK on the failure of a node. with vm's iops are important so you must keep latency to a minimum. both of these are explained a bit more in detail in the link he posted. kind regards Ronny Aasen On 30.08.2018 20:46, Gilberto Nunes wrote: > Hi Martin. > > Not really worried about highest performance, but to know if it will work > properly, mainly HA! > I plan work with mesh network too. > > Tanks a lot > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > 2018-08-30 15:40 GMT-03:00 Martin Maurer : > >> Hello, >> >> Not really. Please read in detail the following: >> >> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ >> >> >> On 30.08.2018 16:47, Gilberto Nunes wrote: >> >>> Hi there >>> >>> It's possible create a scenario with 3 PowerEdge r540, with Proxmox and >>> Ceph. >>> The server has this configuration: >>> >>> 32 GB memory >>> SAS 2x 300 GB >>> SSD 1x 480 GB >>> >>> 2 VM with SQL and Windows server. >>> >>> Thanks >>> >>> --- >>> Gilberto Nunes Ferreira >>> >>> (47) 3025-5907 >>> (47) 99676-7530 - Whatsapp / Telegram >>> >>> Skype: gilberto.nunes36 >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >>> >> -- >> Best Regards, >> >> Martin Maurer >> >> martin at proxmox.com >> http://www.proxmox.com >> >> ____________________________________________________________________ >> Proxmox Server Solutions GmbH >> Br?uhausgasse 37, 1050 Vienna, Austria >> Commercial register no.: FN 258879 f >> Registration office: Handelsgericht Wien >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gilberto.nunes32 at gmail.com Fri Aug 31 00:21:28 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 30 Aug 2018 19:21:28 -0300 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> Message-ID: An HPE Server will remain after deploy 3 servers with proxmox and ceph. I thing I will use this HPE server as 4th node! --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-30 18:16 GMT-03:00 Ronny Aasen : > if HA is important, you should consider having a 4th ceph osd server (does > not have to also be proxmox) > > with ceph's default of 3 replicas, that you will want to use in a > production setup, you do not have any failure domain. > IOW the loss of any one node = a degraded ceph cluster. if you have an > additional node, ceph will rebalance and return to HEALTH_OK on the failure > of a node. > > with vm's iops are important so you must keep latency to a minimum. > > both of these are explained a bit more in detail in the link he posted. > > > kind regards > Ronny Aasen > > > > On 30.08.2018 20:46, Gilberto Nunes wrote: > >> Hi Martin. >> >> Not really worried about highest performance, but to know if it will work >> properly, mainly HA! >> I plan work with mesh network too. >> >> Tanks a lot >> >> --- >> Gilberto Nunes Ferreira >> >> (47) 3025-5907 >> (47) 99676-7530 - Whatsapp / Telegram >> >> Skype: gilberto.nunes36 >> >> >> >> >> 2018-08-30 15:40 GMT-03:00 Martin Maurer : >> >> Hello, >>> >>> Not really. Please read in detail the following: >>> >>> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark- >>> 2018-02.41761/ >>> >>> >>> On 30.08.2018 16:47, Gilberto Nunes wrote: >>> >>> Hi there >>>> >>>> It's possible create a scenario with 3 PowerEdge r540, with Proxmox and >>>> Ceph. >>>> The server has this configuration: >>>> >>>> 32 GB memory >>>> SAS 2x 300 GB >>>> SSD 1x 480 GB >>>> >>>> 2 VM with SQL and Windows server. >>>> >>>> Thanks >>>> >>>> --- >>>> Gilberto Nunes Ferreira >>>> >>>> (47) 3025-5907 >>>> (47) 99676-7530 - Whatsapp / Telegram >>>> >>>> Skype: gilberto.nunes36 >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>>> >>>> -- >>> Best Regards, >>> >>> Martin Maurer >>> >>> martin at proxmox.com >>> http://www.proxmox.com >>> >>> ____________________________________________________________________ >>> Proxmox Server Solutions GmbH >>> Br?uhausgasse 37, 1050 Vienna, Austria >>> Commercial register no.: FN 258879 f >>> Registration office: Handelsgericht Wien >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >>> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From elacunza at binovo.es Fri Aug 31 08:13:35 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Fri, 31 Aug 2018 08:13:35 +0200 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: References: Message-ID: <4a198f08-be02-fed7-24db-fa2192f2bcf1@binovo.es> Hi Gilberto, It's technically possible. I don't know what performance you expect for those 2 SQL servers though (don't expect much). Cheers El 30/08/18 a las 16:47, Gilberto Nunes escribi?: > Hi there > > It's possible create a scenario with 3 PowerEdge r540, with Proxmox and > Ceph. > The server has this configuration: > > 32 GB memory > SAS 2x 300 GB > SSD 1x 480 GB > > 2 VM with SQL and Windows server. > > Thanks > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From ronny+pve-user at aasen.cx Fri Aug 31 10:58:23 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Fri, 31 Aug 2018 10:58:23 +0200 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> Message-ID: when adding a older machine to your cluster, keep in mind that the slowest node with determine the overall speed of the ceph cluster (since a vm's disk will be spread all over) for RBD vm's you want low latency, so use things like nvram > ssd > hdd with osd latency significant difference here. 100Gb/25Gb > 40Gb/10Gb (1Gb is useless in this case imho) as long as you have enough cores, higher ghz is better then lower ghz. due to lower latency kind regards. Ronny Aasen On 31. aug. 2018 00:21, Gilberto Nunes wrote: > An HPE Server will remain after deploy 3 servers with proxmox and ceph. > I thing I will use this HPE server as 4th node! > > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > 2018-08-30 18:16 GMT-03:00 Ronny Aasen : > >> if HA is important, you should consider having a 4th ceph osd server (does >> not have to also be proxmox) >> >> with ceph's default of 3 replicas, that you will want to use in a >> production setup, you do not have any failure domain. >> IOW the loss of any one node = a degraded ceph cluster. if you have an >> additional node, ceph will rebalance and return to HEALTH_OK on the failure >> of a node. >> >> with vm's iops are important so you must keep latency to a minimum. >> >> both of these are explained a bit more in detail in the link he posted. >> >> >> kind regards >> Ronny Aasen >> >> >> >> On 30.08.2018 20:46, Gilberto Nunes wrote: >> >>> Hi Martin. >>> >>> Not really worried about highest performance, but to know if it will work >>> properly, mainly HA! >>> I plan work with mesh network too. >>> >>> Tanks a lot >>> >>> --- >>> Gilberto Nunes Ferreira >>> >>> (47) 3025-5907 >>> (47) 99676-7530 - Whatsapp / Telegram >>> >>> Skype: gilberto.nunes36 >>> >>> >>> >>> >>> 2018-08-30 15:40 GMT-03:00 Martin Maurer : >>> >>> Hello, >>>> >>>> Not really. Please read in detail the following: >>>> >>>> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark- >>>> 2018-02.41761/ >>>> >>>> >>>> On 30.08.2018 16:47, Gilberto Nunes wrote: >>>> >>>> Hi there >>>>> >>>>> It's possible create a scenario with 3 PowerEdge r540, with Proxmox and >>>>> Ceph. >>>>> The server has this configuration: >>>>> >>>>> 32 GB memory >>>>> SAS 2x 300 GB >>>>> SSD 1x 480 GB >>>>> >>>>> 2 VM with SQL and Windows server. >>>>> >>>>> Thanks >>>>> >>>>> --- >>>>> Gilberto Nunes Ferreira >>>>> >>>>> (47) 3025-5907 >>>>> (47) 99676-7530 - Whatsapp / Telegram >>>>> >>>>> Skype: gilberto.nunes36 >>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user at pve.proxmox.com >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>> >>>>> >>>>> -- >>>> Best Regards, >>>> >>>> Martin Maurer >>>> >>>> martin at proxmox.com >>>> http://www.proxmox.com >>>> >>>> ____________________________________________________________________ >>>> Proxmox Server Solutions GmbH >>>> Br?uhausgasse 37, 1050 Vienna, Austria >>>> Commercial register no.: FN 258879 f >>>> Registration office: Handelsgericht Wien >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Fri Aug 31 13:01:53 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 31 Aug 2018 08:01:53 -0300 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> Message-ID: Thanks a lot for all this advice guys. I still learn with Ceph. So I have a doubt regarding how to change the weight from certain hdd Is there some command to do that? Em sex, 31 de ago de 2018 05:58, Ronny Aasen escreveu: > when adding a older machine to your cluster, keep in mind that the > slowest node with determine the overall speed of the ceph cluster (since > a vm's disk will be spread all over) > > > for RBD vm's you want low latency, so use things like > nvram > ssd > hdd with osd latency significant difference here. > > 100Gb/25Gb > 40Gb/10Gb (1Gb is useless in this case imho) > > as long as you have enough cores, higher ghz is better then lower ghz. > due to lower latency > > kind regards. > Ronny Aasen > > > > On 31. aug. 2018 00:21, Gilberto Nunes wrote: > > An HPE Server will remain after deploy 3 servers with proxmox and ceph. > > I thing I will use this HPE server as 4th node! > > > > > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > > > > > > > > > 2018-08-30 18:16 GMT-03:00 Ronny Aasen : > > > >> if HA is important, you should consider having a 4th ceph osd server > (does > >> not have to also be proxmox) > >> > >> with ceph's default of 3 replicas, that you will want to use in a > >> production setup, you do not have any failure domain. > >> IOW the loss of any one node = a degraded ceph cluster. if you have an > >> additional node, ceph will rebalance and return to HEALTH_OK on the > failure > >> of a node. > >> > >> with vm's iops are important so you must keep latency to a minimum. > >> > >> both of these are explained a bit more in detail in the link he posted. > >> > >> > >> kind regards > >> Ronny Aasen > >> > >> > >> > >> On 30.08.2018 20:46, Gilberto Nunes wrote: > >> > >>> Hi Martin. > >>> > >>> Not really worried about highest performance, but to know if it will > work > >>> properly, mainly HA! > >>> I plan work with mesh network too. > >>> > >>> Tanks a lot > >>> > >>> --- > >>> Gilberto Nunes Ferreira > >>> > >>> (47) 3025-5907 > >>> (47) 99676-7530 - Whatsapp / Telegram > >>> > >>> Skype: gilberto.nunes36 > >>> > >>> > >>> > >>> > >>> 2018-08-30 15:40 GMT-03:00 Martin Maurer : > >>> > >>> Hello, > >>>> > >>>> Not really. Please read in detail the following: > >>>> > >>>> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark- > >>>> 2018-02.41761/ > >>>> > >>>> > >>>> On 30.08.2018 16:47, Gilberto Nunes wrote: > >>>> > >>>> Hi there > >>>>> > >>>>> It's possible create a scenario with 3 PowerEdge r540, with Proxmox > and > >>>>> Ceph. > >>>>> The server has this configuration: > >>>>> > >>>>> 32 GB memory > >>>>> SAS 2x 300 GB > >>>>> SSD 1x 480 GB > >>>>> > >>>>> 2 VM with SQL and Windows server. > >>>>> > >>>>> Thanks > >>>>> > >>>>> --- > >>>>> Gilberto Nunes Ferreira > >>>>> > >>>>> (47) 3025-5907 > >>>>> (47) 99676-7530 - Whatsapp / Telegram > >>>>> > >>>>> Skype: gilberto.nunes36 > >>>>> _______________________________________________ > >>>>> pve-user mailing list > >>>>> pve-user at pve.proxmox.com > >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>>>> > >>>>> > >>>>> -- > >>>> Best Regards, > >>>> > >>>> Martin Maurer > >>>> > >>>> martin at proxmox.com > >>>> http://www.proxmox.com > >>>> > >>>> ____________________________________________________________________ > >>>> Proxmox Server Solutions GmbH > >>>> Br?uhausgasse 37, 1050 Vienna, Austria > >>>> Commercial register no.: FN 258879 f > >>>> Registration office: Handelsgericht Wien > >>>> > >>>> _______________________________________________ > >>>> pve-user mailing list > >>>> pve-user at pve.proxmox.com > >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>>> > >>>> _______________________________________________ > >>> pve-user mailing list > >>> pve-user at pve.proxmox.com > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>> > >> > >> > >> _______________________________________________ > >> pve-user mailing list > >> pve-user at pve.proxmox.com > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From elacunza at binovo.es Fri Aug 31 13:10:27 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Fri, 31 Aug 2018 13:10:27 +0200 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> Message-ID: <14c8b459-2d54-fef7-bbd9-4f30f8347016@binovo.es> You can do so from CLI: ceph osd crush reweight osd.N https://ceph.com/geen-categorie/difference-between-ceph-osd-reweight-and-ceph-osd-crush-reweight/ El 31/08/18 a las 13:01, Gilberto Nunes escribi?: > Thanks a lot for all this advice guys. > I still learn with Ceph. > So I have a doubt regarding how to change the weight from certain hdd > Is there some command to do that? > > Em sex, 31 de ago de 2018 05:58, Ronny Aasen > escreveu: > >> when adding a older machine to your cluster, keep in mind that the >> slowest node with determine the overall speed of the ceph cluster (since >> a vm's disk will be spread all over) >> >> >> for RBD vm's you want low latency, so use things like >> nvram > ssd > hdd with osd latency significant difference here. >> >> 100Gb/25Gb > 40Gb/10Gb (1Gb is useless in this case imho) >> >> as long as you have enough cores, higher ghz is better then lower ghz. >> due to lower latency >> >> kind regards. >> Ronny Aasen >> >> >> >> On 31. aug. 2018 00:21, Gilberto Nunes wrote: >>> An HPE Server will remain after deploy 3 servers with proxmox and ceph. >>> I thing I will use this HPE server as 4th node! >>> >>> >>> --- >>> Gilberto Nunes Ferreira >>> >>> (47) 3025-5907 >>> (47) 99676-7530 - Whatsapp / Telegram >>> >>> Skype: gilberto.nunes36 >>> >>> >>> >>> >>> 2018-08-30 18:16 GMT-03:00 Ronny Aasen : >>> >>>> if HA is important, you should consider having a 4th ceph osd server >> (does >>>> not have to also be proxmox) >>>> >>>> with ceph's default of 3 replicas, that you will want to use in a >>>> production setup, you do not have any failure domain. >>>> IOW the loss of any one node = a degraded ceph cluster. if you have an >>>> additional node, ceph will rebalance and return to HEALTH_OK on the >> failure >>>> of a node. >>>> >>>> with vm's iops are important so you must keep latency to a minimum. >>>> >>>> both of these are explained a bit more in detail in the link he posted. >>>> >>>> >>>> kind regards >>>> Ronny Aasen >>>> >>>> >>>> >>>> On 30.08.2018 20:46, Gilberto Nunes wrote: >>>> >>>>> Hi Martin. >>>>> >>>>> Not really worried about highest performance, but to know if it will >> work >>>>> properly, mainly HA! >>>>> I plan work with mesh network too. >>>>> >>>>> Tanks a lot >>>>> >>>>> --- >>>>> Gilberto Nunes Ferreira >>>>> >>>>> (47) 3025-5907 >>>>> (47) 99676-7530 - Whatsapp / Telegram >>>>> >>>>> Skype: gilberto.nunes36 >>>>> >>>>> >>>>> >>>>> >>>>> 2018-08-30 15:40 GMT-03:00 Martin Maurer : >>>>> >>>>> Hello, >>>>>> Not really. Please read in detail the following: >>>>>> >>>>>> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark- >>>>>> 2018-02.41761/ >>>>>> >>>>>> >>>>>> On 30.08.2018 16:47, Gilberto Nunes wrote: >>>>>> >>>>>> Hi there >>>>>>> It's possible create a scenario with 3 PowerEdge r540, with Proxmox >> and >>>>>>> Ceph. >>>>>>> The server has this configuration: >>>>>>> >>>>>>> 32 GB memory >>>>>>> SAS 2x 300 GB >>>>>>> SSD 1x 480 GB >>>>>>> >>>>>>> 2 VM with SQL and Windows server. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> --- >>>>>>> Gilberto Nunes Ferreira >>>>>>> >>>>>>> (47) 3025-5907 >>>>>>> (47) 99676-7530 - Whatsapp / Telegram >>>>>>> >>>>>>> Skype: gilberto.nunes36 >>>>>>> _______________________________________________ >>>>>>> pve-user mailing list >>>>>>> pve-user at pve.proxmox.com >>>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>>>> >>>>>>> >>>>>>> -- >>>>>> Best Regards, >>>>>> >>>>>> Martin Maurer >>>>>> >>>>>> martin at proxmox.com >>>>>> http://www.proxmox.com >>>>>> >>>>>> ____________________________________________________________________ >>>>>> Proxmox Server Solutions GmbH >>>>>> Br?uhausgasse 37, 1050 Vienna, Austria >>>>>> Commercial register no.: FN 258879 f >>>>>> Registration office: Handelsgericht Wien >>>>>> >>>>>> _______________________________________________ >>>>>> pve-user mailing list >>>>>> pve-user at pve.proxmox.com >>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>>> >>>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user at pve.proxmox.com >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>> >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From yannis.milios at gmail.com Fri Aug 31 13:15:57 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Fri, 31 Aug 2018 12:15:57 +0100 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: <14c8b459-2d54-fef7-bbd9-4f30f8347016@binovo.es> References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> <14c8b459-2d54-fef7-bbd9-4f30f8347016@binovo.es> Message-ID: This seems a good reading as well... https://ceph.com/geen-categorie/ceph-osd-reweight/ On Fri, 31 Aug 2018 at 12:10, Eneko Lacunza wrote: > You can do so from CLI: > > ceph osd crush reweight osd.N > > > https://ceph.com/geen-categorie/difference-between-ceph-osd-reweight-and-ceph-osd-crush-reweight/ > > El 31/08/18 a las 13:01, Gilberto Nunes escribi?: > > Thanks a lot for all this advice guys. > > I still learn with Ceph. > > So I have a doubt regarding how to change the weight from certain hdd > > Is there some command to do that? > > > > Em sex, 31 de ago de 2018 05:58, Ronny Aasen > > escreveu: > > > >> when adding a older machine to your cluster, keep in mind that the > >> slowest node with determine the overall speed of the ceph cluster (since > >> a vm's disk will be spread all over) > >> > >> > >> for RBD vm's you want low latency, so use things like > >> nvram > ssd > hdd with osd latency significant difference here. > >> > >> 100Gb/25Gb > 40Gb/10Gb (1Gb is useless in this case imho) > >> > >> as long as you have enough cores, higher ghz is better then lower ghz. > >> due to lower latency > >> > >> kind regards. > >> Ronny Aasen > >> > >> > >> > >> On 31. aug. 2018 00:21, Gilberto Nunes wrote: > >>> An HPE Server will remain after deploy 3 servers with proxmox and ceph. > >>> I thing I will use this HPE server as 4th node! > >>> > >>> > >>> --- > >>> Gilberto Nunes Ferreira > >>> > >>> (47) 3025-5907 > >>> (47) 99676-7530 - Whatsapp / Telegram > >>> > >>> Skype: gilberto.nunes36 > >>> > >>> > >>> > >>> > >>> 2018-08-30 18:16 GMT-03:00 Ronny Aasen : > >>> > >>>> if HA is important, you should consider having a 4th ceph osd server > >> (does > >>>> not have to also be proxmox) > >>>> > >>>> with ceph's default of 3 replicas, that you will want to use in a > >>>> production setup, you do not have any failure domain. > >>>> IOW the loss of any one node = a degraded ceph cluster. if you have > an > >>>> additional node, ceph will rebalance and return to HEALTH_OK on the > >> failure > >>>> of a node. > >>>> > >>>> with vm's iops are important so you must keep latency to a minimum. > >>>> > >>>> both of these are explained a bit more in detail in the link he > posted. > >>>> > >>>> > >>>> kind regards > >>>> Ronny Aasen > >>>> > >>>> > >>>> > >>>> On 30.08.2018 20:46, Gilberto Nunes wrote: > >>>> > >>>>> Hi Martin. > >>>>> > >>>>> Not really worried about highest performance, but to know if it will > >> work > >>>>> properly, mainly HA! > >>>>> I plan work with mesh network too. > >>>>> > >>>>> Tanks a lot > >>>>> > >>>>> --- > >>>>> Gilberto Nunes Ferreira > >>>>> > >>>>> (47) 3025-5907 > >>>>> (47) 99676-7530 - Whatsapp / Telegram > >>>>> > >>>>> Skype: gilberto.nunes36 > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> 2018-08-30 15:40 GMT-03:00 Martin Maurer : > >>>>> > >>>>> Hello, > >>>>>> Not really. Please read in detail the following: > >>>>>> > >>>>>> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark- > >>>>>> 2018-02.41761/ > >>>>>> > >>>>>> > >>>>>> On 30.08.2018 16:47, Gilberto Nunes wrote: > >>>>>> > >>>>>> Hi there > >>>>>>> It's possible create a scenario with 3 PowerEdge r540, with Proxmox > >> and > >>>>>>> Ceph. > >>>>>>> The server has this configuration: > >>>>>>> > >>>>>>> 32 GB memory > >>>>>>> SAS 2x 300 GB > >>>>>>> SSD 1x 480 GB > >>>>>>> > >>>>>>> 2 VM with SQL and Windows server. > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> --- > >>>>>>> Gilberto Nunes Ferreira > >>>>>>> > >>>>>>> (47) 3025-5907 > >>>>>>> (47) 99676-7530 - Whatsapp / Telegram > >>>>>>> > >>>>>>> Skype: gilberto.nunes36 > >>>>>>> _______________________________________________ > >>>>>>> pve-user mailing list > >>>>>>> pve-user at pve.proxmox.com > >>>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>> Best Regards, > >>>>>> > >>>>>> Martin Maurer > >>>>>> > >>>>>> martin at proxmox.com > >>>>>> http://www.proxmox.com > >>>>>> > >>>>>> ____________________________________________________________________ > >>>>>> Proxmox Server Solutions GmbH > >>>>>> Br?uhausgasse 37, 1050 Vienna, Austria > >>>>>> Commercial register no.: FN 258879 f > >>>>>> Registration office: Handelsgericht Wien > >>>>>> > >>>>>> _______________________________________________ > >>>>>> pve-user mailing list > >>>>>> pve-user at pve.proxmox.com > >>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>>>>> > >>>>>> _______________________________________________ > >>>>> pve-user mailing list > >>>>> pve-user at pve.proxmox.com > >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>>>> > >>>> > >>>> _______________________________________________ > >>>> pve-user mailing list > >>>> pve-user at pve.proxmox.com > >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>>> > >>> _______________________________________________ > >>> pve-user mailing list > >>> pve-user at pve.proxmox.com > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>> > >> _______________________________________________ > >> pve-user mailing list > >> pve-user at pve.proxmox.com > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > -- > Zuzendari Teknikoa / Director T?cnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Fri Aug 31 13:25:25 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 31 Aug 2018 08:25:25 -0300 Subject: [PVE-User] PRoxmox and ceph with just 3 server. In-Reply-To: References: <41341b66-ed84-9b36-645d-240289add14d@proxmox.com> <64ccba38-68ec-e10b-e143-d3359b1ca7e6@aasen.cx> <14c8b459-2d54-fef7-bbd9-4f30f8347016@binovo.es> Message-ID: That's was nice guys. Thank you for helping... Em sex, 31 de ago de 2018 08:16, Yannis Milios escreveu: > This seems a good reading as well... > https://ceph.com/geen-categorie/ceph-osd-reweight/ > > On Fri, 31 Aug 2018 at 12:10, Eneko Lacunza wrote: > > > You can do so from CLI: > > > > ceph osd crush reweight osd.N > > > > > > > https://ceph.com/geen-categorie/difference-between-ceph-osd-reweight-and-ceph-osd-crush-reweight/ > > > > El 31/08/18 a las 13:01, Gilberto Nunes escribi?: > > > Thanks a lot for all this advice guys. > > > I still learn with Ceph. > > > So I have a doubt regarding how to change the weight from certain hdd > > > Is there some command to do that? > > > > > > Em sex, 31 de ago de 2018 05:58, Ronny Aasen > > > escreveu: > > > > > >> when adding a older machine to your cluster, keep in mind that the > > >> slowest node with determine the overall speed of the ceph cluster > (since > > >> a vm's disk will be spread all over) > > >> > > >> > > >> for RBD vm's you want low latency, so use things like > > >> nvram > ssd > hdd with osd latency significant difference here. > > >> > > >> 100Gb/25Gb > 40Gb/10Gb (1Gb is useless in this case imho) > > >> > > >> as long as you have enough cores, higher ghz is better then lower ghz. > > >> due to lower latency > > >> > > >> kind regards. > > >> Ronny Aasen > > >> > > >> > > >> > > >> On 31. aug. 2018 00:21, Gilberto Nunes wrote: > > >>> An HPE Server will remain after deploy 3 servers with proxmox and > ceph. > > >>> I thing I will use this HPE server as 4th node! > > >>> > > >>> > > >>> --- > > >>> Gilberto Nunes Ferreira > > >>> > > >>> (47) 3025-5907 > > >>> (47) 99676-7530 - Whatsapp / Telegram > > >>> > > >>> Skype: gilberto.nunes36 > > >>> > > >>> > > >>> > > >>> > > >>> 2018-08-30 18:16 GMT-03:00 Ronny Aasen : > > >>> > > >>>> if HA is important, you should consider having a 4th ceph osd server > > >> (does > > >>>> not have to also be proxmox) > > >>>> > > >>>> with ceph's default of 3 replicas, that you will want to use in a > > >>>> production setup, you do not have any failure domain. > > >>>> IOW the loss of any one node = a degraded ceph cluster. if you have > > an > > >>>> additional node, ceph will rebalance and return to HEALTH_OK on the > > >> failure > > >>>> of a node. > > >>>> > > >>>> with vm's iops are important so you must keep latency to a minimum. > > >>>> > > >>>> both of these are explained a bit more in detail in the link he > > posted. > > >>>> > > >>>> > > >>>> kind regards > > >>>> Ronny Aasen > > >>>> > > >>>> > > >>>> > > >>>> On 30.08.2018 20:46, Gilberto Nunes wrote: > > >>>> > > >>>>> Hi Martin. > > >>>>> > > >>>>> Not really worried about highest performance, but to know if it > will > > >> work > > >>>>> properly, mainly HA! > > >>>>> I plan work with mesh network too. > > >>>>> > > >>>>> Tanks a lot > > >>>>> > > >>>>> --- > > >>>>> Gilberto Nunes Ferreira > > >>>>> > > >>>>> (47) 3025-5907 > > >>>>> (47) 99676-7530 - Whatsapp / Telegram > > >>>>> > > >>>>> Skype: gilberto.nunes36 > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> 2018-08-30 15:40 GMT-03:00 Martin Maurer : > > >>>>> > > >>>>> Hello, > > >>>>>> Not really. Please read in detail the following: > > >>>>>> > > >>>>>> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark- > > >>>>>> 2018-02.41761/ > > >>>>>> > > >>>>>> > > >>>>>> On 30.08.2018 16:47, Gilberto Nunes wrote: > > >>>>>> > > >>>>>> Hi there > > >>>>>>> It's possible create a scenario with 3 PowerEdge r540, with > Proxmox > > >> and > > >>>>>>> Ceph. > > >>>>>>> The server has this configuration: > > >>>>>>> > > >>>>>>> 32 GB memory > > >>>>>>> SAS 2x 300 GB > > >>>>>>> SSD 1x 480 GB > > >>>>>>> > > >>>>>>> 2 VM with SQL and Windows server. > > >>>>>>> > > >>>>>>> Thanks > > >>>>>>> > > >>>>>>> --- > > >>>>>>> Gilberto Nunes Ferreira > > >>>>>>> > > >>>>>>> (47) 3025-5907 > > >>>>>>> (47) 99676-7530 - Whatsapp / Telegram > > >>>>>>> > > >>>>>>> Skype: gilberto.nunes36 > > >>>>>>> _______________________________________________ > > >>>>>>> pve-user mailing list > > >>>>>>> pve-user at pve.proxmox.com > > >>>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>> Best Regards, > > >>>>>> > > >>>>>> Martin Maurer > > >>>>>> > > >>>>>> martin at proxmox.com > > >>>>>> http://www.proxmox.com > > >>>>>> > > >>>>>> > ____________________________________________________________________ > > >>>>>> Proxmox Server Solutions GmbH > > >>>>>> Br?uhausgasse 37, 1050 Vienna, Austria > > >>>>>> Commercial register no.: FN 258879 f > > >>>>>> Registration office: Handelsgericht Wien > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>>> pve-user mailing list > > >>>>>> pve-user at pve.proxmox.com > > >>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>> pve-user mailing list > > >>>>> pve-user at pve.proxmox.com > > >>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >>>>> > > >>>> > > >>>> _______________________________________________ > > >>>> pve-user mailing list > > >>>> pve-user at pve.proxmox.com > > >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >>>> > > >>> _______________________________________________ > > >>> pve-user mailing list > > >>> pve-user at pve.proxmox.com > > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >>> > > >> _______________________________________________ > > >> pve-user mailing list > > >> pve-user at pve.proxmox.com > > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >> > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > -- > > Zuzendari Teknikoa / Director T?cnico > > Binovo IT Human Project, S.L. > > Telf. 943569206 > > Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > > www.binovo.es > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Fri Aug 31 16:08:33 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 31 Aug 2018 11:08:33 -0300 Subject: [PVE-User] Proxmox Ceph with differents HDD Size In-Reply-To: <6bbcd443-9fc8-3c1f-fa06-bc606982c214@schwarz-fr.net> References: <08e79360ae02c218fe9b1aa49c4a996f2c022ffd.camel@tuxis.nl> <6bbcd443-9fc8-3c1f-fa06-bc606982c214@schwarz-fr.net> Message-ID: Thanks for all buddies that replied my messages. Indeed I used ceph osd primary-affinity And we felt some performance increment. What's help here is that we have 6 proxmox ceph server: ceph01 - HDD with 5 900 rpm ceph02 - HDD with 7 200 rpm ceph03 - HDD with 7 200 rpm ceph04 - HDD with 7 200 rpm ceph05 - HDD with 5 900 rpm ceph06 - HDD with 5 900 rpm So what I do is define weight 0 to HDD's with 5 900 rpm and define weight 1 to HDD's with 7 200 rpm. ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 62.31059 root default -3 14.55438 host pve-ceph01 0 hdd 3.63860 osd.0 up 1.00000 0 1 hdd 3.63860 osd.1 up 1.00000 0 2 hdd 3.63860 osd.2 up 1.00000 0 3 hdd 3.63860 osd.3 up 1.00000 0 -5 10.91559 host pve-ceph02 4 hdd 2.72890 osd.4 up 1.00000 1.00000 5 hdd 2.72890 osd.5 up 1.00000 1.00000 6 hdd 2.72890 osd.6 up 1.00000 1.00000 7 hdd 2.72890 osd.7 up 1.00000 1.00000 -7 7.27708 host pve-ceph03 8 hdd 2.72890 osd.8 up 1.00000 1.00000 9 hdd 2.72890 osd.9 up 1.00000 1.00000 10 hdd 1.81929 osd.10 up 1.00000 1.00000 -9 7.27716 host pve-ceph04 11 hdd 1.81929 osd.11 up 1.00000 1.00000 12 hdd 1.81929 osd.12 up 1.00000 1.00000 13 hdd 1.81929 osd.13 up 1.00000 1.00000 14 hdd 1.81929 osd.14 up 1.00000 1.00000 -11 14.55460 host pve-ceph05 15 hdd 7.27730 osd.15 up 1.00000 0 16 hdd 7.27730 osd.16 up 1.00000 0 -13 7.73178 host pve-ceph06 17 hdd 0.90959 osd.17 up 1.00000 0 18 hdd 2.72890 osd.18 up 1.00000 0 19 hdd 1.36440 osd.19 up 1.00000 0 20 hdd 2.72890 osd.20 up 1.00000 0 Tha's it! Thanks again. --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 2018-08-30 11:47 GMT-03:00 Phil Schwarz : > Hope you did change a single disk at a time ! > > Be warned (if not) that moving an OSD from a server to another triggers > a rebalancing of almost the complete datas stored upon in order to > follow crushmap. > > For instance exchanging two OSDs between servers result in a complete > rebalance of the two OSDS,a ccording to my knowledge. > > 16% of misplaced datas could be acceptable or not depending on your > needs of redundancy and throughput, but it's not a low value that could > be underestimated. > > Best regards > > > > Le 30/08/2018 ? 15:27, Gilberto Nunes a ?crit : > > Right now the ceph are very slow > > > > 343510/2089155 objects misplaced (16.443%) > > Status > > > > HEALTH_WARN > > Monitors > > pve-ceph01: > > pve-ceph02: > > pve-ceph03: > > pve-ceph04: > > pve-ceph05: > > pve-ceph06: > > OSDs > > In Out > > Up 21 0 > > Down 0 0 > > Total: 21 > > PGs > > active+clean: > > 157 > > > > active+recovery_wait+remapped: > > 1 > > > > active+remapped+backfill_wait: > > 82 > > > > active+remapped+backfilling: > > 2 > > > > active+undersized+degraded+remapped+backfill_wait: > > 8 > > > > Usage > > 7.68 TiB of 62.31 TiB > > Reads: > > Writes: > > IOPS: Reads: > > IOPS: Writes: > > > > () > > Degraded data redundancy: 21495/2089170 objects degraded (1.029%), 8 pgs > > degraded, 8 pgs undersized > > > > pg 21.0 is stuck undersized for 63693.346103, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,9] > > pg 21.2 is stuck undersized for 63693.346973, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,10] > > pg 21.6f is stuck undersized for 62453.277248, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,5] > > pg 21.8b is stuck undersized for 63693.361835, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,8] > > pg 21.c3 is stuck undersized for 63693.321337, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,9] > > pg 21.c5 is stuck undersized for 66587.797684, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,8] > > pg 21.d4 is stuck undersized for 62453.047415, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,6] > > pg 21.e1 is stuck undersized for 62453.276631, current state > > active+undersized+degraded+remapped+backfill_wait, last acting [2,5] > > > > > > > > > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > > > > > > > > > 2018-08-30 10:23 GMT-03:00 Gilberto Nunes : > > > >> SO, what you guys think about this HDD distribuiton? > >> > >> CEPH-01 > >> 1x 3 TB > >> 1x 2 TB > >> > >> CEPH-02 > >> 1x 4 TB > >> 1x 3 TB > >> > >> CEPH-03 > >> 1x 4 TB > >> 1x 3 TB > >> > >> CEPH-04 > >> 1x 4 TB > >> 1x 3 TB > >> 1x 2 TB > >> > >> CEPH-05 > >> 1x 8 TB > >> 1x 2 TB > >> > >> CEPH-06 > >> 1x 3 TB > >> 1x 1 TB > >> 1x 8 TB > >> > >> > >> --- > >> Gilberto Nunes Ferreira > >> > >> (47) 3025-5907 > >> (47) 99676-7530 - Whatsapp / Telegram > >> > >> Skype: gilberto.nunes36 > >> > >> >