[PVE-User] Spillover issue

[Alwin Antreich]

On Tue, Mar 24, 2020 at 01:12:03PM +0100, Eneko Lacunza wrote:
> Hi Allwin,
> 
> El 24/3/20 a las 12:24, Alwin Antreich escribió:
> > On Tue, Mar 24, 2020 at 10:34:15AM +0100, Eneko Lacunza wrote:
> > > We're seeing a spillover issue with Ceph, using 14.2.8:
> [...]
> > > 3. ceph health detail
> > >     HEALTH_WARN BlueFS spillover detected on 3 OSD
> > >     BLUEFS_SPILLOVER BlueFS spillover detected on 3 OSD
> > >     osd.3 spilled over 5 MiB metadata from 'db' device (556 MiB used of
> > >     6.0 GiB) to slow device
> > >     osd.4 spilled over 5 MiB metadata from 'db' device (552 MiB used of
> > >     6.0 GiB) to slow device
> > >     osd.5 spilled over 5 MiB metadata from 'db' device (551 MiB used of
> > >     6.0 GiB) to slow device
> > > 
> > > I may be overlooking something, any idea? Just found also the following ceph
> > > issue:
> > > 
> > > https://tracker.ceph.com/issues/38745
> > > 
> > > 5MiB of metadata in slow isn't a big problem, but cluster is permanently in
> > > health Warning state... :)
> > The DB/WAL device is to small and all the new metadata has to be written
> > to the slow device. This will destroy performance.
> > 
> > I think the size changes, as the DB gets compacted.
> Yes. But it isn't too small... it's 6 GiB and there's only ~560MiB of data.
Yes true. I meant the used of size. But the message is oddly.

You should find the compaction stats in the OSD log files. It could be,
as in the bug tracker reasoned, that the compaction needs to much space
and spills over to the slow device. Addionally, if no set extra, the WAL
will take up 512 MB on the DB device.

If the above doesn't give any information then you may need to export
the bluefs (RocksDB). Then you can run the kvstore-tool on it.

> > 
> > The easiest way ist to destroy and re-create the OSD with a bigger
> > DB/WAL. The guideline from Facebook for RocksDB is 3/30/300 GB.
> 
> It's well below the 3GiB limit in the guideline ;)
For now. ;)

--
Cheers,
Alwin