Commit Graph

156 Commits

Author SHA1 Message Date
9177cc2eda clone image: specify base format option with qemu-img
and avoid a warning. It is deprecated to auto-detect the format of the base
volume. See commit d9f059aa6cfccefaffa3532556e966df4a99ece2 in qemu for more
information.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-05-03 13:07:02 +02:00
cba2b7c1d4 prune backups: improve internal errors messages slightly
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 12:49:55 +02:00
4b702ac361 plugin: get_subdir_files: add comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 12:46:46 +02:00
68ce0b31e8 prune backups: make vmid filtering more robust
by relying on archive_info's vmid first. archive_info is already used to
determine if it's a standard name, and in that case the vmid is certainly set.

Also add asserts to make sure we got what we expected.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-09 12:41:34 +02:00
edb92f707a plugin: subdir files: backup: don't match for vmid against the full path
Only match against the file name to avoid false positives with
directory names containing "-$vmid-".

Found while trying to debug/reproduce a forum thread[0], but the path
there should not be affected by this...

[0]: https://forum.proxmox.com/threads/vzdump-removing-too-many-backups.87072/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-09 12:41:10 +02:00
68f1fc2783 mark PBS storages as shared
Like this, the property will get added when parsing the storage configuration
and PBS storages will correctly show up as shared storages in API results.

AFAICT the only affected PBS operation is free_image via vdisk_free, which will
now be protected by a cluster-wide lock, and that shouldn't hurt.

Another issue this fixes, which is the reason this patch exists, was reported
in the forum[0]. The free space from PBS storages was counted once for each node
that had access to the storage.

[0]: https://forum.proxmox.com/threads/pve-6-3-the-storage-size-was-displayed-incorrectly.83136/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-01-28 16:43:20 +01:00
e9991d2694 Storage/Plugin: add get/update_volume_comment and implement for dir
and add the appropriate api call to set and get the comment
we need to bump APIVER for this and can bump APIAGE, since
we only use it at this new call that can work with the default
implementation

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-24 10:23:25 +01:00
6fef456c8d rename comment to notes
so that we are more consistent with pbs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-24 10:23:25 +01:00
1b87f01388 prune: introduce keep-all option
useful to have an alternative to the old maxfiles = 0. There has to
be a way for vzdump to distinguish between:
1. use the /etc/vzdump.conf default (when no options are configured for the storage)
2. use no limit (when keep-all=1)

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-23 15:27:17 +01:00
afeda18256 plugin: hooks: avoid that method param count gets returned
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-23 15:12:58 +01:00
343ca2570c prune: allow having all prune options zero/missing
This is basically necessary for the GUI's prune widget, because we want to
pass along all options equal to zero when all the number fields are cleared.
And it's more similar to how it's done in PBS now.

Bumped the APIAGE and APIVER, in case some external plugin needs to adapt to
the now less restrictive schema for 'prune-backups'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-16 10:14:11 +01:00
1d95f21615 avoid unecessary try for over optimization
That was lots of code and hash map touching for the case where one
avoided a extra stat, which result probably was in the page cache
anyway, for the case that a backup has a comment.

A case which is rather  be unlikely - comments are normally done for
the occasional explicit backup (e.g., before major upgrade, before a
configuration change in that guest, ...), at least not worth some
relatively complicated effort making that sub harder to read and
maintain.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-12 17:31:26 +01:00
d260b9f357 storage: get subdirectory files: read .comment files for comments
we have no way of setting them yet via the API, but we can read them
now

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-12 17:17:44 +01:00
2c036838ed add check for fsfreeze before snapshot
In order to take a snapshot of a container volume, which can be mounted
read-only with RBD, the volume needs to be frozen (fsfreeze (8)) before taking
the snapshot.

This commit adds helpers to determine if the FIFREEZE ioctl needs to be called
for the volume.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-11-10 18:58:45 +01:00
d1f4700063 file_size_info: handle dangling symlinks
and other stat failure modes.

this method returns undef if 'qemu-img info ...' fails to return
information, so callers must handle this already.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-05 12:51:44 +02:00
48d0cd02c1 fix indentation of $prune_backups_format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-03 18:21:09 +02:00
a0933d7e16 prune-backups CLI: use keep-options directly
Makes the interface cleaner; e.g. --keep-daily=2 instead of
--prune-backups=keep-daily=2

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-09-03 18:20:06 +02:00
8f26b3910d Add prune_backups to storage API
Implement it for generic storages supporting backups
(i.e. directory-based storages) and add a wrapper for PBS.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-07-24 15:44:53 +02:00
3353698f45 Introduce prune-backups property for directory-based storages
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-07-24 15:44:53 +02:00
e9667b3418 base plugin: fall back to empty hash to avoid checking archive info definedness
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-06 19:40:10 +02:00
fb821c1828 Expand archive_info to include ctime, vmid and is_std_name
where 'is_std_name' shows whether the backup name uses the standard naming
schema and most likely was created by our tools.

Also adds a '^' to the existing filename matching regex, which
should be fine since basename() is used beforehand.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-06-06 19:38:02 +02:00
3d10acf89e Fix #2737: Can't call method "mode"
on an undefined value at /usr/share/perl5/PVE/Storage/Plugin.pm line 928

This error message crops up when a file is deleted after getting the
file list and before the loop passed the file entry.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-05-15 18:12:01 +02:00
014d36dbbb Fix: #2124 storage: add zstd support
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
277cafc0ff backup: compact regex for backup file filter
the more compact form of the regex should allow easier addition of new
file extensions.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
40c795e7df Fix: backup: ctime was from stat not file name
The vzdump file was passed with the full path to the regex. That regex
captures the time from the file name, to calculate the epoch.

As the regex didn't match, the ctime from stat was taken instead. This
resulted in the ctime shown when the file was changed, not when the
backup was made.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
c48801b52a test: parse_volname
Test to reduce the potential for accidental breakage on regex changes.
And to make sure that all vtype_subdirs are parsed.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
92ae59df9e storage: replace build-in stat occurrences
with File::stat::stat to minimize variable declarations. And allow to
mock this method in tests instead of the perl build-in stat.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
a97d3ee49f Introduce allow_rename parameter for pvesm import and storage_migrate
and also return the ID of the allocated volume. This option
allows plugins to choose a new name if there is a collision.

In storage_migrate, the API version of the receiving side is checked.

In Storage.pm's volume_import, when a plugin returns 'undef',
it can be assumed that the import with the requested volid was
successful (it should've died otherwise) and so volid is returned.
This is done for backwards compatibility with foreign plugins.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-09 09:41:01 +02:00
e6f4eed435 Allow passing options to volume_has_feature
With the option valid_target_formats it's possible
to let the caller specify possible formats for the target
of an operation.
[0]: If the option is not set, assume that every format is valid.

In most cases the format of the the target and the format
of the source will agree (and therefore assumption [0] is
not actually assuming very much and ensures backwards
compatability). But when cloning a volume on a storage
using Plugin.pm's implementation (e.g. directory based
storages), the result is always a qcow2 image.

When cloning containers, the new option can be used to detect
that qcow2 is not valid and hence the clone feature is not
available.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-27 08:50:23 +01:00
6c25dbd495 base plugin: get_subdir_files: split stat variables into single lines
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-03-06 19:27:24 +01:00
51eee96d31 base plugin: return ctime for vm images
Changed file_size_info() to additionally return ctime to avoid
another stat() call.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2020-03-06 19:26:45 +01:00
ff9c5451a5 base plugin: add ctime for all files
Creation time makes sense for other file types also.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2020-03-06 19:26:45 +01:00
9c629b3e76 base plugin: add ctime for backup files
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2020-03-06 19:26:45 +01:00
93afc379a3 followup: fix VMID regex, use same as JSONSchema does
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-21 16:02:23 +01:00
1b39642528 list_volumes: try to return vmid also for backups
this way the content listing api also returns the vmid on content
listings which, among other things, is useful for the gui for
filtering

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-02-21 16:00:58 +01:00
0ff4cfead1 PVE/Storage/Plugin.pm: introduce on_update_hook
We need this to correctly update the password file.
2020-02-20 12:39:44 +01:00
a420842d01 don't (accidentally) modify PVE::Cluster::vmlist
when listing volumes, otherwise an empty hash can be persisted into the
current worker's $vmlist, which could cause issues at various other API
endpoints.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-01-08 12:01:10 +01:00
481f6177a5 grammar fix: s/does not exists/does not exist/g
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-13 12:11:26 +01:00
01e872db98 base find_free_diskname: refactor mapping from disks to volid array
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-12 12:59:57 +01:00
a44c0147bc Use a common interface for find_free_diskname
We can use 'list_images' to get the desired volume IDs in
'find_free_diskname' for most plugins. For the two LVM plugins, 'list_images'
potentially skips untagged volumes, so we keep the custom version. For the
RBD plugin, 'list_images' is much more costly than the custom version, so we
keep the custom version.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2019-12-12 12:52:43 +01:00
b5eff97d66 pvesm import: make error messages consistent
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-11-27 14:36:19 +01:00
f105c1766e pvesm import: improve handling of interrupted export
since 'pvesm export' and 'pvesm import' are connected via a pipe and
SSH, a fatal error in the former can lead to no valid header being
written to the pipe. handle this more gracefully by printing an easier
to understand error message, instead of uninitialized warnings with no
context.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-11-27 14:36:19 +01:00
97cf933fe5 fix #2309: allow LVM Volume Group names starting with a digit
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-26 14:17:19 +01:00
0877df047c cleanuo: no need to negate we have not-equal
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-21 12:43:55 +01:00
a4e41b0bbf change var name ct to type to prevent confusion with container
Signed-off-by: Tim Marx <t.marx@proxmox.com>
2019-11-21 12:43:55 +01:00
5dae1a319b fix #2467: avoid duplicate volumes & tag with correct content type
The bugfix for #2317 introduced a kind of odd API behavior, where
each volume was returned twice from our API if a storage has both
'rootdir' & 'images' content types enabled. To give the content type
of the volume an actual meaning, it is now inferred from the
associated guest, if there's no guest or we don't have an owner for
that volume we default to 'images'.

At the volume level, there is no option to list volumes based on
content types, since the volumes do not know what type they are
actually used for.

Signed-off-by: Tim Marx <t.marx@proxmox.com>
2019-11-21 12:43:55 +01:00
80699b1da5 followup: add JSON use and cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-09-20 13:32:18 +02:00
af0335e82f change file_size_info sub to use qemu-img info json decoding
Using the json output, as suggested by Thomas, we now die if the decoding
fails and, if not, all return values are set to the corresponding decoded
values. That should prevent any unforeseen null size values, except if
qemu-img info reports it, which we then consider as valid.

Signed-off-by: Tim Marx <t.marx@proxmox.com>
2019-09-20 13:01:00 +02:00
ce8b24a937 fix vmid filter for backup listing
$1 and $2 get set to undef from the vmid filter regex, so we have to do
the name/format regex after, else we get errors like:

'use of unitiialized value $1[...]'

and the listing is empty

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-09-09 16:38:37 +02:00
1dbbd5abd6 file_size_info: move parser to own variable
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-09-09 15:36:14 +02:00