Hard-coding a list of sensitive properties means that custom plugins
cannot define their own sensitive properties for the on_add/on_update
hooks.
Have plugins declare the list of their sensitive properties in the
plugin data. For backwards compatibility, return the previously
hard-coded list if no such declaration is present.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Link: https://lore.proxmox.com/20250404133204.239783-6-f.ebner@proxmox.com
The new_backup_provider() method can be used by storage plugins for
external backup providers. If the method returns a provider, Proxmox
VE will use callbacks to that provider for backups and restore instead
of using its usual backup/restore mechanisms.
The backup provider API is split into two parts, both of which again
need different implementations for VM and LXC guests:
1. Backup API
In Proxmox VE, a backup job consists of backup tasks for individual
guests. There are methods for initialization and cleanup of the job,
i.e. job_init() and job_cleanup() and for each guest backup, i.e.
backup_init() and backup_cleanup().
The backup_get_mechanism() method is used to decide on the backup
mechanism. Currently, 'file-handle' or 'nbd' for VMs, and 'directory'
for containers is possible. The method also let's the plugin indicate
whether to use a bitmap for incremental VM backup or not. It is enough
to implement one mechanism for VMs and one mechanism for containers.
Next, there are methods for backing up the guest's configuration and
data, backup_vm() for VM backup and backup_container() for container
backup, with the latter running
Finally, some helpers like getting the provider name or volume ID for
the backup target, as well as for handling the backup log.
The backup transaction looks as follows:
First, job_init() is called that can be used to check backup server
availability and prepare the connection. Then for each guest
backup_init() followed by backup_vm() or backup_container() and finally
backup_cleanup(). Afterwards job_cleanup() is called. For containers,
there is an additional backup_container_prepare() call while still
privileged. The actual backup_container() call happens as the
(unprivileged) container root user, so that the file owner and group IDs
match the container's perspective.
1.1 Backup Mechanisms
VM:
Access to the data on the VM's disk from the time the backup started
is made available via a so-called "snapshot access". This is either
the full image, or in case a bitmap is used, the dirty parts of the
image since the last time the bitmap was used for a successful backup.
Reading outside of the dirty parts will result in an error. After
backing up each part of the disk, it should be discarded in the export
to avoid unnecessary space usage on the Proxmox VE side (there is an
associated fleecing image).
VM mechanism 'file-handle':
The snapshot access is exposed via a file descriptor. A subroutine to
read the dirty regions for incremental backup is provided as well.
VM mechanism 'nbd':
The snapshot access and, if used, bitmap are exported via NBD.
Container mechanism 'directory':
A copy or snapshot of the container's filesystem state is made
available as a directory. The method is executed inside the user
namespace associated to the container.
2. Restore API
The restore_get_mechanism() method is used to decide on the restore
mechanism. Currently, 'qemu-img' for VMs, and 'directory' or 'tar' for
containers are possible. It is enough to implement one mechanism for
VMs and one mechanism for containers.
Next, methods for extracting the guest and firewall configuration and
the implementations of the restore mechanism via a pair of methods: an
init method, for making the data available to Proxmox VE and a cleanup
method that is called after restore.
2.1. Restore Mechanisms
VM mechanism 'qemu-img':
The backup provider gives a path to the disk image that will be
restored. The path needs to be something 'qemu-img' can deal with,
e.g. can also be an NBD URI or similar.
Container mechanism 'directory':
The backup provider gives the path to a directory with the full
filesystem structure of the container.
Container mechanism 'tar':
The backup provider gives the path to a (potentially compressed) tar
archive with the full filesystem structure of the container.
See the PVE::BackupProvider::Plugin module for the full API
documentation.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
[WB: replace backup_vm_available_bitmaps with
backup_vm_query_incremental, which instead of a bitmap name provides
a bitmap mode that is 'new' (create or *recreate* a bitmap) or 'use'
(use an existing bitmap, or create one if none exists)]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Tested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Link: https://lore.proxmox.com/20250404133204.239783-5-f.ebner@proxmox.com
The web UI uses the download-url endpoint for downloading an ISO, VZ
template, or OVA file via wget. In a setup where this request has to
go over a proxy (configured in the http_proxy datacenter option), the
download only works for http:// URLs, not https:// URLs. The reason is
that the download-url handler does not pass the https_proxy option to
the download_file_from_url helper, hence the helper only sets the
http_proxy environment variable for wget, not the https_proxy one.
Fix this by also passing the https_proxy option to the
download_file_from_url helper.
This will break setups that rely on http_proxy not being respected for
https:// URLs. For example, setups that have a proxy for external
connections, but download e.g. ISO files (only) via https from an
internal repository that the proxy doesn't serve.
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
Reviewed-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Link: https://lore.kernel.org/r/20250326105108.34911-2-f.weber@proxmox.com
The new 'pve-storage-image-format' standard option uses a simple enum
instead of a subroutine verifier. Since the 'pve-storage-format'
format that is replaced by it was used in pve-guest-common's
StorageTunnel, the format cannot be removed without a versioned
breaks.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Acked-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
The API endpoint will automatically detect the format from the
extension for raw, qcow2 and vmdk, but it was not yet possible to
specify the format explicitly via the parameter. This could be
annoying/surprising to users. There also might be third-party plugins
that want to use vmdk, but not require a suffix in the name. Add
'vmdk' as an allowed format to avoid these issues and for consistency
by using the 'pve-storage-format' format.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Acked-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
The format was dropped in QEMU binary version 2.2 with commit
550830f935 ("block: delete cow block driver").
This follows qemu-server commit "drive: remove ancient 'cow' from
formats".
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Acked-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Check if there is already a logged session present and fall back to
previous TCP check port connection.
pvestatd is calling check_connection every 10 seconds. This check
produces a lot of noise at the iscsi server logging.
Signed-off-by: Victor Seva <linuxmaniac@torreviejawireless.org>
Tested-by: Mira Limbeck <m.limbeck@proxmox.com>
Reviewed-by: Mira Limbeck <m.limbeck@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
Reviewed-by: Friedrich Weber <f.weber@proxmox.com>
The helper iterates with the BTRFS_SNAPSHOT_REGEX regular expression,
so there will always be a snapshot name.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
It was originally introduced as $snapshot to match the argument name of
foreach_subvol, we rename it here to make it clear that it only contains
the snapshot name.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
When a subvolume is transferred via btrfs send/receive the resulting
image contains the received_uuid property set. This property is required
to do incremental snapshots.
A downside though is that once the received_uuid property is set, it is
not possible to make the image readwrite again without the force (-f)
flag, and in such case the received_uuid property is lost. Since we know
the images are only set to rw for the duration of the move, it is safe
to set the flag forcefully and then in a future commit add the
received_uuid property by force.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Replaces the current use without changes. The `$dir` variable is not
used anymore at that moment so it is defined later.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Suppose we are taking a snapshot of VM 100's disk-0. The
dir_glob_foreach runs over $path=/subvolume/images/100, lists all
snapshot names and appends their names to the path of the disk, e.g.
/subvolume/images/vm-100-disk-0@SNAP_NAME, but the original directory
$path might contain a second disk `vm-100-disk-1` which is also listed
by the dir_glib_foreach.
By using the helper we only iterate over the snapshots of the guest.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
In this context a subvolume means a BTRFS subvolume.
`$volume\@$snap_name` would be for example
`btrfs_volume/images/102/vm-102-disk-0@snap_name`.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Minor cleanup to reduce the amount of `$param->{...}` to variables in
the upload and download url API handler.
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
A static cache key 'directiscsi' was used to cache storeid-specific
information. This was causing issues in case of multiple instances.
Drop $cache usage instead of fixing the issue as there is no caller
using it and the same portal/storeid multiple times.
Remove $storeid from iscsi_ls() and its callers.
Signed-off-by: Dmitry Petrov <dpetrov67@gmail.com>
A static cache key 'rbd' was used to cache pool/storeid-specific
information. This was causing issues in case of multiple RBD pools.
Drop $cache usage instead of fixing the issue as there is no caller
using it and the same pool/storeid multiple times.
Remove $pool from rbd_ls() and get_rbd_path($scfg) from list_images().
Signed-off-by: Dmitry Petrov <dpetrov67@gmail.com>
This became outdated after Ceph commit ac547a5b7dc ("rbd: return 0 and
an empty list when pool is entirely empty") 11 years ago. See also:
https://tracker.ceph.com/issues/6693
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Previously, the size was rounded down which, in case of an image with
non-1KiB-aligned sze (only possible for external plugins or manually
created images) would lead to errors when attempting to write beyond
the end of the too small allocated target image.
For image allocation, the size is already rounded up to the
granularity of the storage. Do the same for import.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This module's purpose is to provide shared functions, constants, etc.
for storage plugins and storage-related operations.
It starts out with a align_size_up() function, that will (initially)
be used for volume import.
[FE: start out with a different function for my use case
fixup Makefile]
Originally-by: Max Carrara <m.carrara@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Currently, the helper would not distinguish between different kinds
of errors. Instead of relying on an error, list the images and check
there.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
For now, only 'raw+size' is supported and it's not possible to
export/import with snapshots. The volume or snapshot is exported or
imported via the corresponding 'rbd' commands.
Introducing an 'rbd' transport format might be feasible for more
complete (i.e. with snapshots, incremental) transfer between two RBD
storages.
Use the '--dest-pool' switch rather than '-p' for import, because the
latter is deprecated.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Current export implementations luckily seems to not run into this
issue yet. However, for the upcoming implementation for RBD, mapping a
volume would print the device path to STDOUT, thus messing up the
export stream.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
The plugin API does not require call context detection for the
returned value of the path() method. See other plugins like
ISCSIDirect/ZFS that do not implement it. So do not expect it for
external plugins either.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
in case this gets called with an explicit format that is none of:
- 'auto-detect'
- 'subvol'
- a member of the list of known "qemu" formats
this should only affect third-party storage plugins that either call this
directly with a format, or via inherited code that gets a format from
parse_volname and passes it to file_size_info.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
in case of an array context, it should also return the format, else a caller
might assume it failed.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
since `volume_size_info` passes the parsed format to `file_size_info`, which
prints a warning if the format is undef before falling back to auto-detection,
and these should always be treated as raw files anyway.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
otherwise, `volname.vmdk.raw` would be picked up as a `vmdk` formatted
image, which may not be desired.
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
The only (transitive) caller that seems to be interested in the format
is the API endpoint for content listing.
The warning about not being able to query in the expected format might
not be seen by consumers that only use the API result, so this helps
admins detect such images. It is also for future-proofing, should any
new callers want to use only images of certain formats to error out
early.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
In almost all cases, there already is an expected format. Allow
setting a special value 'auto-detect' to opt-in to automatic format
detection by 'qemu-img' for the exceptions.
Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This avoids auto-detection by qemu-img and so the information will be
correct with respect to the actual image format on the storage layer.
Should the image not be in the correct format, warn and try again
querying as raw, so the image is still listed. The image is present,
so it is better if it is listed and for some backwards compatibility.
The format is still returned as the matched format in such a case,
because that is how the image is treated, even if corrupt.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This avoids auto-detection by qemu-img and so the information will be
correct with respect to the actual image format on the storage layer.
Should the image not be in the correct format, warn and try again
querying as raw, so the image is still listed. The image is present,
so it is better if it is listed and for some backwards compatibility.
The format is still returned as the matched format in such a case,
because that is how the image is treated, even if corrupt.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
By passing in the format, qemu-img will also check that the file is a
valid instance of the expected format instead of auto-detecting. The
size will also be correct when there previously was a wrong
auto-detection, but it is already only used for checking that the call
was successful.
It looks like this could also avoid wrongly failing checks with the
parent information, but a raw image cannot have a base volume and raw
images are those where the auto-detection might lead to a different
format.
Remove the unused variable $used while at it.
The check for $format in the result became obsolete with 35533c6 ("add
subvol support for directory storage").
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This avoids auto-detection by qemu-img and so the information will be
correct with respect to the actual image format on the storage layer.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This avoids auto-detection by qemu-img and so the information will be
correct with respect to the actual image format on the storage layer.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
