[PVE-User] PVE kernel (sources, build process, etc.)
Uwe Sauter
uwe.sauter.de at gmail.com
Wed Aug 22 11:45:15 CEST 2018
Hi Marcus,
no, I haven't disabled Spectre/Meltdown mitigations (yet).
For 4 of my hosts I have the Intel microcode from 2018-05-08 running which is the most up-to-date version from Ubuntu.
Using the spectre-meltdown-checker.sh script from https://github.com/speed47/spectre-meltdown-checker gives below output which
let's me believe that performace should be goot with mitigations enabled.
#### output for sandybridge based hosts ####
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch: NO
* Kernel has mask_nospec64 (arm64): NO
> STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization)
CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW)
* Mitigation 1
* Kernel is compiled with IBRS support: YES
* IBRS enabled and active: YES (for kernel and firmware code)
* Kernel is compiled with IBPB support: YES
* IBPB enabled and active: YES
* Mitigation 2
* Kernel has branch predictor hardening (arm): NO
* Kernel compiled with retpoline option: YES
* Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation)
> STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability)
CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface: YES (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI): YES
* PTI enabled and active: YES
* Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced)
* Running as a Xen PV DomU: NO
> STATUS: NOT VULNERABLE (Mitigation: PTI)
CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
* CPU microcode mitigates the vulnerability: YES
> STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability)
CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
* Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
* Kernel supports speculation store bypass: YES (found in /proc/self/status)
> STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
* Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)
> STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)
##### end #####
The other 2 hosts are base on Westmere architecture and won't get microcode updates as fas as I know. Disabling mitigations on
these hosts might be worth a try. Same script from above gives:
#### output of westmere based hosts ####
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch: NO
* Kernel has mask_nospec64 (arm64): NO
> STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization)
CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline)
* Mitigation 1
* Kernel is compiled with IBRS support: YES
* IBRS enabled and active: NO
* Kernel is compiled with IBPB support: YES
* IBPB enabled and active: NO
* Mitigation 2
* Kernel has branch predictor hardening (arm): NO
* Kernel compiled with retpoline option: YES
* Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation)
> STATUS: NOT VULNERABLE (Full retpoline is mitigating the vulnerability)
IBPB is considered as a good addition to retpoline for Variant 2 mitigation, but your CPU microcode doesn't support it
CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface: YES (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI): YES
* PTI enabled and active: YES
* Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced)
* Running as a Xen PV DomU: NO
> STATUS: NOT VULNERABLE (Mitigation: PTI)
CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
* CPU microcode mitigates the vulnerability: NO
> STATUS: VULNERABLE (an up-to-date CPU microcode is needed to mitigate this vulnerability)
CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
* Mitigated according to the /sys interface: NO (Vulnerable)
* Kernel supports speculation store bypass: YES (found in /proc/self/status)
> STATUS: VULNERABLE (Your CPU doesn't support SSBD)
CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
* Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)
> STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)
##### end #####
Am 22.08.18 um 11:08 schrieb Marcus Haarmann:
> Hi,
> did you try this:
> (taken from the ceph list)
>
> This is PTI, I think. Try to add "noibrs noibpb nopti nospectre_v2" to
> kernel cmdline and reboot.
>
>
> Did this make a difference ?
> We are struggeling with proxmox/ceph too and we have the suspect that it is kernel related or network related,
> but could not narrow it down to a specific reason.
> But the effects are different... We encountered stuck I/O on rdb devices.
> And kernel says it is losing a mon connection and hunting for a new mon all the time (when backup takes
> place and heavy I/O is done).
>
> Marcus Haarmann
>
> ----------------------------------------------------------------------------------------------------------------------------------
> *Von: *"uwe sauter de" <uwe.sauter.de at gmail.com>
> *An: *"Thomas Lamprecht" <t.lamprecht at proxmox.com>, "pve-user" <pve-user at pve.proxmox.com>
> *Gesendet: *Mittwoch, 22. August 2018 10:50:19
> *Betreff: *Re: [PVE-User] PVE kernel (sources, build process, etc.)
>
>>>>
>>>>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ?
>>>>>
>>>>
>>>> Yes. (Note that this may not get much updates anymore)
>>>>
>>>>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ?
>>>>>
>>>>
>>>> Yes. We're normally on the latest stable release tagged on the master branch.
>>>>
>>>
>>> I'll checkout both and compare the myri10ge drivers…
>>>
>>
>> What's your exact issue, if I may ask?
>>
>> cheers,
>> Thomas
>>
>>
>>
>
> Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those
> are gone.
>
> Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs.
>
>
> Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different
> settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well.
>
>
>
> Long story is here:
>
> https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html
>
> https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html
>
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html
>
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html
>
> https://marc.info/?l=ceph-devel&m=153449419830984&w=2
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
More information about the pve-user
mailing list