[PVE-User] Ceph Cache Tiering

[Adam Thompson]

Writeback clearly defines what is and isn't protected, so yes, there are some guarantees.  There are also some guaranteed data-loss scenarios.

Even "Writeback (unsafe)" provides guarantees of a sort, but there is a big difference between it and simple "Writeback".

Choosing the appropriate caching policy is an exercise in risk management, not just blind adherence to  "my data must be safe at all times". 
With journaling filesystems on VM and on host and/or storage server, Writeback is often an acceptable choice for general purpose use.  No, it's not the absolute safest option, but it does provide *some* well-defined data integrity guarantees.

Your typical Linux VM operates in writeback mode inside the VM anyway; when it calls flush(2) or sync(2) (or whatever it's called nowadays), the block driver issues a SCSI flush command, which is interpreted by QEMU which issues a SCSI flush command, which is interpreted by the block storage layer which does a SCSI flush command, which is interpreted by the disks, which flush their own buffers and finally signal write-complete, which propagates back up the chain.
A VM in "writeback" mode operates identically to "writethrough" mode until QEMU determines that the client OS knows how to manage writeback caching properly through the use of barriers, flush commands, etc., at which point it enables writeback caching that is no more volatile than the write cache on your disk.  (And, of course, this assumes that everything is battery-backed, and - ha, ha - that all the software and firmware and hardware involved is bug-free.)

Essentially, "writeback" provides under most circumstances, almost exactly the same data guarantees a standard Linux ext4 filesystem running with "defaults" mount options running against an HDD with write caching enabled gives you.  Turning off write cache altogether usually results in a massive performance hit, and nearly every filesystem (and databases, independently) write data in such a way that they can recover from unexpected shutdown even in a "writeback" environment.

Having said all that - don't use "Writeback (unsafe)" for production data unless you want to lose a month of your life rebuilding a key database like I did :-(.

Ultimately, "Always Use DirectIO" is a religious belief, not a technically sound belief.  There are situations where it makes sense, and other situations where it doesn't.  The same can be said of *every single* setting - if there were only One True Correct Setting, it wouldn't be available as a user-settable option in the first place!
-Adam


> -----Original Message-----
> From: pve-user [mailto:pve-user-bounces at pve.proxmox.com] On
> Behalf Of Lindsay Mathieson
> Sent: October 11, 2016 23:12
> To: PVE User List <pve-user at pve.proxmox.com>
> Subject: Re: [PVE-User] Ceph Cache Tiering
> 
> On 12 October 2016 at 13:28, Adam Thompson
> <athompso at athompso.net> wrote:
> > Not a bloody chance... WriteBack is the only thing that gives both
> acceptable performance characteristics and data guarantees.
> 
> Eh? I didn't think writeback gave data guarantees, quite the opposite.
> 
> 
> --
> Lindsay
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user