[PVE-User] Ceph bluestore OSD Journal/DB disk size

[Eneko Lacunza]

Hi all,

I have noticed that our office Proxmox cluster has a Bluestore OSD with 
a very small db partition. This OSD was created from GUI on 12th march 
this year:

This node has 4 OSDs:
- osd.12: bluestore, all SSD
- osd.3: bluestore, SSD db + spinning
- osd.2: filestore, SSD journal + spinning
- osd.4: filestore, SSD journal + spinning

We have two pools in the cluster (SSD and HDD).

I see that for osd.3 block.db points to /dev/sdb8, which is 1G in size:

# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0   1,8T  0 disk
├─sda1         8:1    0   100M  0 part /var/lib/ceph/osd/ceph-12
└─sda2         8:2    0   1,8T  0 part
sdb            8:16   0 931,5G  0 disk
├─sdb1         8:17   0   100M  0 part /var/lib/ceph/osd/ceph-3
└─sdb2         8:18   0 931,4G  0 part
sdc            8:32   0 931,5G  0 disk
└─sdc1         8:33   0 931,5G  0 part /var/lib/ceph/osd/ceph-4
sdd            8:48   0 931,5G  0 disk
└─sdd1         8:49   0 931,5G  0 part /var/lib/ceph/osd/ceph-2
sde            8:64   0 186,3G  0 disk
├─sde1         8:65   0  1007K  0 part
├─sde2         8:66   0   127M  0 part /boot/efi
├─sde3         8:67   0  59,9G  0 part
│ ├─pve-root 253:0    0  18,6G  0 lvm  /
│ └─pve-swap 253:1    0   952M  0 lvm  [SWAP]
├─sde5         8:69   0     5G  0 part
├─sde6         8:70   0     5G  0 part
└─sde8         8:72   0     1G  0 part

This was created from GUI. I see that currently GUI doesn't allow to 
specify journal/DB partition size... (I can't test all the process until 
creation...)

I think 1GB may be too small for a default value, and that it could be 
preventing the full db to be placed in that partition, as per ceph-users 
mailing list messages:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030740.html
https://www.spinics.net/lists/ceph-devel/msg39315.html

Maybe 3GB would be a better default? Also it seems that for not very 
dense OSD nodes 30GB (or whatever is the next level) would be feasible too.

I see the following in a perf dump of osd.3
     "bluefs": {
         "gift_bytes": 0,
         "reclaim_bytes": 0,
         "db_total_bytes": 1073733632,
         "db_used_bytes": 664797184,
         "wal_total_bytes": 0,
         "wal_used_bytes": 0,
         "slow_total_bytes": 40004222976,
         "slow_used_bytes": 1228931072,
         "num_files": 19,
         "log_bytes": 1318912,
         "log_compactions": 1,
         "logged_bytes": 164077568,
         "files_written_wal": 2,
         "files_written_sst": 17,
         "bytes_written_wal": 1599916960,
         "bytes_written_sst": 752941742
     },

So, 665MB used of db partition, and 1.2GB of additional data in slow 
storage...

Thanks a lot
Eneko

-- 
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943569206
Astigarraga bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
www.binovo.es