View source for Proxmox Cluster file system (pmxcfs)

{{Note|Article about Proxmox VE 2.0 beta}}
=Introduction=

Proxmox Cluster file system (pmxcfs) is a database-driven file system for storing configuration files, replicated in real time on all nodes using corosync. We use this to store all PVE related configuration files. Although the file system stores all data inside a persistent database on disk, a copy of the data resides in RAM. That imposes restriction on the maximal size, which is currently 30MB. This is still enough to store the configuration of several thousand virtual machines.

====Advantages====

* seamless replication of all configuration to all nodes in real time
* provides strong consistency checks to avoid duplicate VM IDs
* read-only when a node looses quorum
* automatic updates of the corosync cluster configuration to all nodes.

====POSIX Compatibility====

The file system is based on fuse, so the behavior is POSIX like. But
many feature are simply not implemented, because we do not need them:

* you can just generate normal files and directories, but no symbolic links, ...
* you can't rename non-empty directories (because this makes it easier to guarantee that VMIDs are unique).
* you can't change file permissions (permissions are based on path)
* O_EXCL creates were not atomic (like old NFS)
* O_TRUNC creates are not atomic (fuse restriction)
* ...

====File access rights====

All files/dirs are owned by user 'root' and have group
'www-data'. Only root has write permissions, but group 'www-data' can
read most files. Files below the following paths:

 /etc/pve/priv/
 /etc/pve/nodes/${NAME}/priv/

are only accessible by root.

=Technologie=

We use the [http://www.corosync.org Corosync Cluster Engine] for cluster communication, and [http://www.sqlite.org SQlite] for the database file. The filesystem is implemented in user space using [http://fuse.sourceforge.net/ FUSE].


=File system layout=

The file system is mounted at:

 /etc/pve

====Files====

 cluster.conf  => corosync/cman cluster configuration file
 storage.cfg   => PVE storage configuration
 user.cfg      => PVE access control configuration (users/groups/...)
 domains.cfg   => PVE Authentication domains 
 authkey.pub   => public key used by ticket system

 priv/shadow.cfg  => shadow password file
 priv/authkey.key => private key used by ticket system

 nodes/${NAME}/pve-ssl.pem                 => public ssl key fo apache
 nodes/${NAME}/priv/pve-ssl.key            => private ssl key
 nodes/${NAME}/qemu-server/${VMID}.conf    => VM configuration data for KVM VMs
 nodes/${NAME}/openvz/${VMID}.conf         => VM configuratin data for OpenVZ containers

====Symbolic links====

 local => nodes/${LOCALNAME}
 qemu-server => nodes/${LOCALNAME}/qemu-server/
 openvz => nodes/${LOCALNAME}/openvz/

====Special status files for debugging (JSON)====

 .version    => file versions (to detect file modifications)
 .members    => Info about cluster members
 .vmlist     => List of all VMs
 .clusterlog => Cluster log (last 50 entries)
 .rrd        => RRD data (most recent entries)


====Enable/Disable debugging====

 # enable verbose syslog messages
 echo "1" >/etc/pve/.debug 

 # disable verbose syslog messages
 echo "0" >/etc/pve/.debug 

=Source code=

The source code is available [https://git.proxmox.com/?p=pve-cluster.git;a=summary here].

[[Category: Proxmox VE 2.0]]