prior to the introduction of snapshot as volume chains, volume names of
almost arbitrary form were accepted. only forbid filenames which are
part of the newly introduced namespace for snapshot files, while
deprecating other names not following our usual naming scheme, instead
of forbidding them outright.
Fixes: b63147f5df "plugin: fix volname parsing"
Co-authored-by: Shannon Sterz <s.sterz@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Tested-by: Shannon Sterz <s.sterz@proxmox.com>
Link: https://lore.proxmox.com/20250731111519.931104-2-f.gruenbichler@proxmox.com
Without untainting, offline-deleting a volume-chain snapshot on a
directory storage via the GUI fails with an "Insecure dependecy in
exec [...]" error, because volume_snapshot_delete uses the filename
its qemu-img invocation.
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
The LVM plugin can only use qcow2 format when the
'snapshot-as-volume-chain' configuration option is set. The format
information is currently only recorded statically in the plugin data.
This causes issues, for example, restoring a guest volume that uses
qcow2 as a format hint on an LVM storage without the option set will
fail, because the plugin data indicates that qcow2 is supported.
Introduce a dedicated method, so that plugins can indicate what
actually is supported according to the storage configuration.
The implementation for LVM is done in a separate commit.
Remove the now unused default_format() function from Plugin.pm.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
[WB: docs: add missing params, drop =pod line, use !! for bools]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Not perfect but now it's still easy to rename and the new variant fits
a bit better to the actual design and implementation.
Add best-effort migration for storage.cfg, this has been never
publicly released after all.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This also changes the return values, since their meanings are rather
weird from the storage point of view. For instance, "internal" meant
it is *not* the storage which does the snapshot, while "external"
meant a mixture of storage and qemu-server side actions. `undef` meant
the storage does it all...
┌────────────┬───────────┐
│ previous │ new │
├────────────┼───────────┤
│ "internal" │ "qemu" │
│ "external" │ "mixed" │
│ undef │ "storage" │
└────────────┴───────────┘
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Just to safe, as this is already check higher in the stack.
Technically, it's possible to implement snapshot file renaming,
and update backing_file info with "qemu-img rebase -u".
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
by moving the preallocation handling to the call site, and preparing
them for taking further options like cluster size in the future.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
add a snapext option to enable the feature
When a snapshot is taken, the current volume is renamed to snap volname
and a current image is created with the snap volume as backing file
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Returns if the volume is supporting qemu snapshot:
'internal' : do the snapshot with qemu internal snapshot
'external' : do the snapshot with qemu external snapshot
undef : does not support qemu snapshot
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
This add a $running param to volume_snapshot,
it can be used if some extra actions need to be done at the storage
layer when the snapshot has already be done at qemu level.
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
This compute the whole size of a qcow2 volume with datas + metadatas.
Needed for qcow2 over lvm volume.
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
and use it for plugin linked clone
This also enable extended_l2=on, as it's mandatory for backing file
preallocation.
Preallocation was missing previously, so it should increase performance
for linked clone now (around x5 in randwrite 4k)
cluster_size is set to 128k, as it reduce qcow2 overhead (reduce disk,
but also memory needed to cache metadatas)
l2_extended is not enabled yet on base image, but it could help too
to reduce overhead without impacting performance
bench on 100G qcow2 file:
fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --iodepth=32 --ioengine=libaio --name=test
fio --filename=/dev/sdb --direct=1 --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --name=test
base image:
randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 20215
randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 22219
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20217
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 21742
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21599
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 22037
clone image with backing file:
randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 3912
randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 21476
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20563
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 22265
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 18016
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21611
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
for better backwards compatibility. This also means using path()
rather than filesystem_path() as the latter does not return protocol
paths.
Some protocol paths are not implemented (considered all that are
listed by grepping for '\.protocol_name' in QEMU):
- ftp(s)/http(s), which would access web servers via curl. This one
could be added if there is enough interest.
- nvme://XXXX:XX:XX.X/X, which would access a host NVME device.
- null-{aio,co}, which are mainly useful for debugging.
- pbs, because path-based access is not used anymore for PBS,
live-restore in qemu-server already defines a driver-based device.
- nfs and ssh, because the QEMU build script used by Proxmox VE does
not enable them.
- blk{debug,verify}, because they are for debugging.
- the ones used by blkio, i.e. io_uring, nvme-io_uring,
virtio-blk-vfio-pci, virtio-blk-vhost-user and
virtio-blk-vhost-vdpa, because the QEMU build script used by Proxmox
VE does not enable blkio.
- backup-dump and zeroinit, because they should not be used by the
storage layer directly.
- gluster, because support is dropped in Proxmox VE 9.
- host_cdrom, because the storage layer should not access host CD-ROM
devices.
- fat, because it hopefully isn't used by any third-party plugin here.
Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Everything the default plugin method implementation can return is
allowed, so there is no breakage introduced by this patch.
By far the most common drivers will be 'file' and 'host_device', which
the default implementation of the plugin method currently uses. Other
quite common ones will be 'iscsi' and 'nbd'. There might also be
plugins with 'rbd' and it is planned to support QEMU protocol-paths in
the default plugin method implementation, where the 'rbd:' protocol
will also be supported.
Plugin authors are encouraged to request additional drivers and
options based on their needs on the pve-devel mailing list. The list
just starts out more restrictive, but everything where there is no
good reason to not allow could be allowed in the future upon request.
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Plugins can guard based on the machine version to be able to switch
drivers or options in a safe way without the risk of breaking older
versions.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This is mostly in preparation for external qcow2 snapshot support.
For internal qcow2 snapshots, which currently are the only supported
variant, it is not possible to attach the snapshot only. If access to
that is required it will need to be handled differently, e.g. via a
FUSE/NBD export.
Such accesses are currently not done for running VMs via '-drive'
either, so there still is feature parity.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
For '-drive', qemu-server sets special cache options for EFI disk
using RBD. In preparation to seamlessly switch to the new '-blockdev'
interface, do the same here. Note that the issue from bug #3329, which
is solved by these cache options, still affects current versions.
With -blockdev, the cache options are split up. While cache.direct and
cache.no-flush can be set in the -blockdev options, cache.writeback is
a front-end property and was intentionally removed from the -blockdev
options by QEMU commit aaa436f998 ("block: Remove cache.writeback from
blockdev-add"). It needs to be configured as the 'write-cache'
property for the ide-hd/scsi-hd/virtio-blk device.
The default is already 'writeback' and no cache mode can be set for an
EFI drive configuration in Proxmox VE currently, so there will not be
a clash.
┌─────────────┬─────────────────┬──────────────┬────────────────┐
│ │ cache.writeback │ cache.direct │ cache.no-flush │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│writeback │ on │ off │ off │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│none │ on │ on │ off │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│writethrough │ off │ off │ off │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│directsync │ off │ on │ off │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│unsafe │ on │ off │ on │
└─────────────┴─────────────────┴──────────────┴────────────────┘
Table from 'man kvm'.
Alternatively, the option could only be set once when allocating the
RBD volume. However, then we would need to detect all cases were a
volume could potentially be used as an EFI disk later. Having a custom
disk type would help a lot there. The approach here was chosen as it
is catch-all and should not be too costly either.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This is in preparation to switch qemu-server from using '-drive' to
the modern '-blockdev' in the QEMU commandline options as well as for
the qemu-storage-daemon, which only supports '-blockdev'. The plugins
know best what driver and options are needed to access an image, so
a dedicated plugin method returning the necessary parameters for
'-blockdev' is the most straight-forward.
There intentionally is only handling for absolute paths in the default
plugin implementation. Any plugin requiring more needs to implement
the method itself. With PVE 9 being a major release and most popular
plugins not using special protocols like 'rbd://', this seems
acceptable.
For NBD, etc. qemu-server should construct the blockdev object.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
As the GlusterFS project is unmaintained since a while and other
projects like QEMU also drop support for using it natively.
One can still use the gluster tools to mount an instance manually and
then use it as directory storage; the better (long term) option will
be to replace the storage server with something maintained though, as
PVE 8 will be supported until the middle of 2026 users have some time
before they need to decide what way they will go.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
using the new top-level `make tidy` target, which calls perltidy via
our wrapper to enforce the desired style as closely as possible.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The result from the file_size_info() call is not used by
volume_export_formats() and most failure scenarios of file_size_info()
lead to an undefined return value rather than a failure. This includes
the case for a non-existent file. The default path() implementation
doesn't do any existence check either.
An interesting scenario where file_size_info() does fail, is when the
volume is corrupted or not in the queried format. But this is a rare
edge case, so an early check doesn't seem worth it. It will be caught
by volume_export() itself, or in case of VM migration, also when
querying the size during scanning of local volumes.
While checking for the definedness of $size could serve as an early
sanity check, it is not currently done and other plugins don't do such
early checks in their implementation of volume_export_formats()
either. Keep the implementation abstract in Plugin.pm too and avoid
doing IO. Callers that want to do early existence checks or similar
can do so themselves explicitly, covering all plugins.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Link: https://lore.proxmox.com/20250303092445.13873-4-f.ebner@proxmox.com
In list context, the file_size_info() function in Plugin.pm would
return 0 for the size of a subvol directory, but in scalar context 1.
As reported in the community forum [0], the change in commit e50dde0
("volume export: rely on storage plugin's format"), changing the
caller in volume_export() to scalar context exposed the inconsistency
in the return value for the size. This led to breakage of migration
with unsized btrfs subvolumes.
Align the returned values to avoid such surprises. Caller that would
treat 0 as an error should use list context and check if it is a
subvol, if they are able to handle them.
NOTE: it is not possible to attach either a subvol, or a volume with
zero size to a VM.
Existing callers of file_size_info() do not break with this change:
GuestImport/OVF.pm - parse_ovf() +
API/Qemu.pm - create_disks():
Doesn't support subvol directories, dies if size is 0.
Storage/Plugin.pm - create_base() +
Storage/{BTRFS,}Plugin.pm - list_images():
Checks for definedness of $size.
Storage/{BTRFS,ESXi,}Plugin.pm - volume_size_info():
Transitive, see below.
Storage/Plugin.pm - volume_export():
Regressed by commit e50dde0, this change will restore previous
behavior.
Storage/Plugin.pm - volume_export_formats() +
GuestImport.pm - extract_disk_from_import_file() +
Storage.pm - assert_iso_content():
Doesn't use the result.
CLI/qm.pm - importdisk:
Dies early if source is a directory.
CLI/qm.pm - importovf:
Calls parse_ovf() earlier.
Existing callers of volume_size_info() do not break with this change:
API2/Storage/Content.pm - info:
Uses list context, not affected by change.
QemuMigrate.pm - scan_local_volumes() +
API2/Qemu.pm - create_disks(), first call:
Uses list context, not affected by change, does not support subvol
directories.
API2/Qemu.pm - create_disks(), second call +
API2/Qemu.pm - import_from_volid():
Doesn't support subvol directories, dies if size is 0.
API2/Qemu.pm - resize_vm +
VZDump/QemuServer.pm - prepare() +
QemuServer.pm - clone_disk() +
QemuServer.pm - create_efidisk():
Doesn't support subvol directories (see NOTE above), but would not
die directly if size is 0. (In case of create_efidisk(), the size of
the just created disk is queried.)
API2/LXC.pm - resize_vm:
For directory plugins, the subsequent call to resize_volume() fails
with "can't resize this image format".
For BTRFS, quotas are currently not supported and the call to
resize_volume() fails with "failed to get btrfs subvolume ID from:".
This is because the btrfs 'subvol show' command is invoked with
'-q', so there is no output. Even if it would work, it would be more
correct to use 0 as the current size to add to the new quota rather
than 1.
LXC/Config.pm - rescan_volume():
This will happily use size=1 from before this change, but that is
not correct. A subsequent 'pct rescan' will correct the size to 0.
It is only used when hotplugging an existing subvol directory.
LXC.pm - copy_volume():
This will happily use size=1 from before this change, but that is
not correct and will lead to failure "mkfs.ext4: Device size
reported to be zero.". That is because with non-zero size, the
allocation of the volume for the clone will be done with 'raw'
format by the alloc_disk() helper in LXC.pm rather than 'subvol'.
This change will make cloning containers with unsized subvol
directories possible.
QemuServer/Cloudinit.pm - commit_cloudinit_disk():
Doesn't support subvol directories (see NOTE above), will allocate
a new volume when size is 0.
[0]: https://forum.proxmox.com/threads/162943/
Fixes: e50dde0 ("volume export: rely on storage plugin's format")
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Link: https://lore.proxmox.com/20250303092445.13873-2-f.ebner@proxmox.com
Hard-coding a list of sensitive properties means that custom plugins
cannot define their own sensitive properties for the on_add/on_update
hooks.
Have plugins declare the list of their sensitive properties in the
plugin data. For backwards compatibility, return the previously
hard-coded list if no such declaration is present.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Link: https://lore.proxmox.com/20250404133204.239783-6-f.ebner@proxmox.com
The new_backup_provider() method can be used by storage plugins for
external backup providers. If the method returns a provider, Proxmox
VE will use callbacks to that provider for backups and restore instead
of using its usual backup/restore mechanisms.
The backup provider API is split into two parts, both of which again
need different implementations for VM and LXC guests:
1. Backup API
In Proxmox VE, a backup job consists of backup tasks for individual
guests. There are methods for initialization and cleanup of the job,
i.e. job_init() and job_cleanup() and for each guest backup, i.e.
backup_init() and backup_cleanup().
The backup_get_mechanism() method is used to decide on the backup
mechanism. Currently, 'file-handle' or 'nbd' for VMs, and 'directory'
for containers is possible. The method also let's the plugin indicate
whether to use a bitmap for incremental VM backup or not. It is enough
to implement one mechanism for VMs and one mechanism for containers.
Next, there are methods for backing up the guest's configuration and
data, backup_vm() for VM backup and backup_container() for container
backup, with the latter running
Finally, some helpers like getting the provider name or volume ID for
the backup target, as well as for handling the backup log.
The backup transaction looks as follows:
First, job_init() is called that can be used to check backup server
availability and prepare the connection. Then for each guest
backup_init() followed by backup_vm() or backup_container() and finally
backup_cleanup(). Afterwards job_cleanup() is called. For containers,
there is an additional backup_container_prepare() call while still
privileged. The actual backup_container() call happens as the
(unprivileged) container root user, so that the file owner and group IDs
match the container's perspective.
1.1 Backup Mechanisms
VM:
Access to the data on the VM's disk from the time the backup started
is made available via a so-called "snapshot access". This is either
the full image, or in case a bitmap is used, the dirty parts of the
image since the last time the bitmap was used for a successful backup.
Reading outside of the dirty parts will result in an error. After
backing up each part of the disk, it should be discarded in the export
to avoid unnecessary space usage on the Proxmox VE side (there is an
associated fleecing image).
VM mechanism 'file-handle':
The snapshot access is exposed via a file descriptor. A subroutine to
read the dirty regions for incremental backup is provided as well.
VM mechanism 'nbd':
The snapshot access and, if used, bitmap are exported via NBD.
Container mechanism 'directory':
A copy or snapshot of the container's filesystem state is made
available as a directory. The method is executed inside the user
namespace associated to the container.
2. Restore API
The restore_get_mechanism() method is used to decide on the restore
mechanism. Currently, 'qemu-img' for VMs, and 'directory' or 'tar' for
containers are possible. It is enough to implement one mechanism for
VMs and one mechanism for containers.
Next, methods for extracting the guest and firewall configuration and
the implementations of the restore mechanism via a pair of methods: an
init method, for making the data available to Proxmox VE and a cleanup
method that is called after restore.
2.1. Restore Mechanisms
VM mechanism 'qemu-img':
The backup provider gives a path to the disk image that will be
restored. The path needs to be something 'qemu-img' can deal with,
e.g. can also be an NBD URI or similar.
Container mechanism 'directory':
The backup provider gives the path to a directory with the full
filesystem structure of the container.
Container mechanism 'tar':
The backup provider gives the path to a (potentially compressed) tar
archive with the full filesystem structure of the container.
See the PVE::BackupProvider::Plugin module for the full API
documentation.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
[WB: replace backup_vm_available_bitmaps with
backup_vm_query_incremental, which instead of a bitmap name provides
a bitmap mode that is 'new' (create or *recreate* a bitmap) or 'use'
(use an existing bitmap, or create one if none exists)]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Tested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Link: https://lore.proxmox.com/20250404133204.239783-5-f.ebner@proxmox.com
The new 'pve-storage-image-format' standard option uses a simple enum
instead of a subroutine verifier. Since the 'pve-storage-format'
format that is replaced by it was used in pve-guest-common's
StorageTunnel, the format cannot be removed without a versioned
breaks.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Acked-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
The format was dropped in QEMU binary version 2.2 with commit
550830f935 ("block: delete cow block driver").
This follows qemu-server commit "drive: remove ancient 'cow' from
formats".
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Acked-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
