Commit Graph

212 Commits

Author SHA1 Message Date
014d36dbbb Fix: #2124 storage: add zstd support
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
277cafc0ff backup: compact regex for backup file filter
the more compact form of the regex should allow easier addition of new
file extensions.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
73fcb7bfd1 Fix: add missing snippets subdir
since it is a valid content type and adapt the path_to_volume_id_test.
Also adds an extra check if all vtype_subdirs are returned.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
892dc99249 Fix: path_to_volume_id returned wrong content
type for backup files. Patch includes changes of the test as well.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
cd554b79d1 storage: test: split archive format/compressor
detection into separate functions so they are reusable and easier
modifiable. This patch also adds the test for archive_info.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
2668adcec2 Add comment for volume_has_feature
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-09 12:05:11 +02:00
3881e68025 ZFS: use -p flag and remove zfs_parse_size
ZFS supports the -p flag in the list command since a few years now.
Let us use the real byte values and avoid the error prone calculation
from human readable numbers that can lead to incorrect numbers if the
reported human readable value is a rounded number.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-04-09 10:20:06 +02:00
683a3f463f storage_migrate: add volname_for_storage helper
to guess a valid volname for a targetstorage of a different type.
This makes it possible to migrate raw volumes between 'dir' and 'lvm'
storages.

It is only used when the storage type for the source storage X
and target storage Y differ and should work as long as Y uses
the standard naming scheme (VMID/vm-VMID-name.fmt respectively vm-VMID-name).
If it doesn't, we get an invalid name and fail, which is the old
behavior (except if X and Y have different types but the same
non-standard naming-scheme, where the old behavior did work).

The original name is preserved, except for a possible extension
and it's also checked if the format is valid for the target storage.
Example: mylvm:vm-123-disk-4 <-> mydir:123/vm-123-disk-4.raw

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-09 09:41:01 +02:00
a97d3ee49f Introduce allow_rename parameter for pvesm import and storage_migrate
and also return the ID of the allocated volume. This option
allows plugins to choose a new name if there is a collision.

In storage_migrate, the API version of the receiving side is checked.

In Storage.pm's volume_import, when a plugin returns 'undef',
it can be assumed that the import with the requested volid was
successful (it should've died otherwise) and so volid is returned.
This is done for backwards compatibility with foreign plugins.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-09 09:41:01 +02:00
dc3655a1a5 Collect optional parameters for storage_migrate into $opts
Sanitizing $with_snapshots is done on extraction to save a line.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-09 09:41:01 +02:00
95015dbbf2 storage_migrate: check if target storage supports content type
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-27 14:04:09 +01:00
e6f4eed435 Allow passing options to volume_has_feature
With the option valid_target_formats it's possible
to let the caller specify possible formats for the target
of an operation.
[0]: If the option is not set, assume that every format is valid.

In most cases the format of the the target and the format
of the source will agree (and therefore assumption [0] is
not actually assuming very much and ensures backwards
compatability). But when cloning a volume on a storage
using Plugin.pm's implementation (e.g. directory based
storages), the result is always a qcow2 image.

When cloning containers, the new option can be used to detect
that qcow2 is not valid and hence the clone feature is not
available.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-27 08:50:23 +01:00
c72155735d volume_resize: align size to 1 KiB
1. Avoids the error
qemu-img: The new size must be a multiple of 512
for qcow2 disks.
2. Because volume_import expects disk sizes to be a multiple of 1 KiB.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-02-19 15:24:09 +01:00
c855ac150c implement extract_vzdump_config for PBSPlugin 2020-02-19 14:00:04 +01:00
271fe39460 PVE/Storage/PBSPlugin.pm: start new proxmox backup server plugin 2020-02-19 14:00:04 +01:00
8e55b4f288 storage_migrate: only set errfunc for send stream
since we redirect the output to our (insecure) socket, logfunc is only
used for STDERR anyway, so we might as well make it explicit on the
caller side.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-02-05 09:37:09 +01:00
aca83310dd storage_migrate: also log with an insecure connection if there is a log function
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-02-05 08:50:14 +01:00
94dd8b0cfa Remove unused string
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-02-05 08:50:14 +01:00
481f6177a5 grammar fix: s/does not exists/does not exist/g
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-13 12:11:26 +01:00
65bb98596c use PVE::SSHInfo
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-11-18 12:03:00 +01:00
2f08fb4bac use PVE::DataCenterConfig
to make sure that the corresponding cfs_read_file works() works.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-11-18 12:03:00 +01:00
6ed43d81dc cleanup: omit module prefix for call to methods in same module
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-10-31 07:36:27 +01:00
cfdffd8a20 Actually use target_volid
Migration with --targetstorage was broken because of this.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-17 09:28:52 +02:00
b295e05ebe followup: just use case sensitive flag, no manual hack
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-09-03 09:42:15 +02:00
fd9ca08c70 fixup: use our to make $iso_extension_re value available for other modules
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-09-03 09:41:50 +02:00
4c69349140 fix #2216: Allow .img files in 'iso' type storages
To maintain full (backwards) compatibility, leave the type name as
'iso' - this makes this patch work without changing every consumer of
storage APIs.

Note that currently these files can only be attached as a CDROM/DVD
drive, so USB-only images can be uploaded but might not work in VMs.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-09-03 07:09:28 +02:00
be78543909 fix 2317: list images on 'rootdir' only storages
plugins can still override list_volumes if they want separate methods to
list rootdir and images content.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-08-06 13:39:43 +02:00
c2fc9fbe1d storage plugin: new list_volumes plugin method
This cleanup improve code reuse, and allows plugins to override list_volumes.
2019-07-19 13:49:29 +02:00
228e5be920 pvesm import: allow to pass a tcp://<IP> as file
To allow getting closer to finally drop "pvecm mtunnel".
Code parts taken from pipe_socket_to_command

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
[regex fixup]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-11 10:13:37 +02:00
5f73686fd3 fix #2226: ignore broken symlinks in *_list
Broken symlinks (and other files without a size) will now show up as 0
byte instead of causing a format validation error in the API.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-06-17 11:49:33 +02:00
11efb06659 drop un-maintained sheepdog plugin
as already announced over two months ago[0], remove the unofficial
SheepDog plugin now completely. Besides that it was never fully
supported in Proxmox VE one of its main developer and ex-maintainer
declared it as abandoned[1], and thus just let's remove it, git
allows to resurrect it any time if a wonder happens anyway.

[0]: https://pve.proxmox.com/pipermail/pve-user/2019-March/170497.html
[1]: http://lists.wpkg.org/pipermail/sheepdog/2019-March/068449.html

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-06-04 17:02:47 +02:00
396aedff95 get_bandwidt_limits: ignore 'undef' as storage
If one of the storages passed in $storage_list was not defined
get_bandwidth_limit died (see [0], of an occurence of this).
This patch changes the behavior to ignore undef as storage instead.

[0] https://pve.proxmox.com/pipermail/pve-devel/2019-April/036515.html

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2019-04-05 18:15:01 +02:00
074bdd354f Storage::get_bandwidth_limit: fix if condition
Passing 'undef' as '$storage_list' led to a warning about using an
uninitialized value as array_ref.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2019-03-29 09:04:48 +01:00
775fdc697d followup: improve comment
while the commit message tells it nicely a comment should add
additional info for people just giving this a quick look

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-02-05 17:42:42 +01:00
061b9ca666 check_volume_access: tighten checks for iso/tmpl
(custom) templates might contain sensitive data, so require at least
read access on the underlying storage to access ISO and template files.

the same permissions are already needed for listing them, so this is
unlikely to cause fallout.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-02-05 17:14:50 +01:00
7c7ae12f43 add new content type 'snippets'
will be used to contain files which can be executed as hookscripts or
contain custom cloud-init configs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-01-31 11:04:29 +01:00
98bf79f78b remove usb scan code
this is now in PVE::SysFSTools

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-11-19 13:20:04 +01:00
a0908caa99 APIAGE followup: fix typo and print versions in error message
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-19 12:18:03 +01:00
042dd4be1f plugin loader: add an APIAGE
With the addition of the map/unmap_volume() methods we made
an (actually unnecessary) API version bump.
All current users of these methods fall back to path() when
they return undef, so plugins implementing version 1 are
in fact compatible currently. (In fact, the default
Plugin::map_volume() could fall back to it on its own, but
doesn't currently).

For now let's just allow plugins older plugins to also be
loaded by introducing an API age variable. With it, if we
have a reason to break older plugins, we can have a
deprecation period during which older plugins cause a
warning instead of refusing to load altogether.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-11-19 10:47:46 +01:00
40d698932e implement map_volume and unmap_volume
This allows to request a mapped device/path explicitly, regardles of
the storage option, eg. krbd option in the RBDplugin.

Bump of the storage ABI => 2

Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2018-11-09 17:25:51 +01:00
9280153e10 rename check_available to assert_sid_unused
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-10-03 14:49:14 +02:00
76c1e57be7 refactor disk/storage checks for Disk API
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-10-03 14:30:22 +02:00
e34ce14443 Cephfs storage plugin
- ability to mount through kernel and fuse client
 - allow mount options
 - get MONs from ceph config if not in storage.cfg
 - allow the use of ceph config with fuse client
 - Delete secret on cephfs storage creation

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:18:19 +02:00
41aacc6cde Fix #1542: show storage utilization per pool
- get storage utilization per pool
 - use librados2-perl for pool status
 - add librados2-perl as build-depends and depends in debian/control

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-04-18 10:43:14 +02:00
4cab0acd27 Add cifsscan. 2018-03-16 12:06:37 +01:00
d787523984 Register CIFSPlugin in the storage plugin system. 2018-03-16 12:06:14 +01:00
77445e9b80 bwlimit: apply limits when there's no override
Before, 'undef' was equivalent to unlimited, but '0' is the
"explicitly unlimited" value, so if the user doesn't request
an override, apply limits as if the user was unprivileged
(otherwise there's no way for privileged users to explicitly
ask to not override the configured limits).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-03-07 14:19:07 +01:00
9edb99a5a7 add Storage::get_bandwidth_limit helper
Takes an operation, an optional requested bandwidth
limit override, and a list of storages involved in the
operation and lowers the requested bandwidth against global
and storage-specific limits unless the user has permissions
to change those.
This means:
 * Global limits apply to all users without Sys.Modify on /
   (as they can change datacenter.cfg options via the API).
 * Storage specific limits apply to users without
   Datastore.Allocate access on /storage/X for any involved
   storage X.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-01-31 12:25:32 +01:00
856c54bd75 add 'format' parameter to storage list call
we will use this for the gui to figure out if we have to show
a size selector, a file selector, which formats are there, etc.

we have to include this data even for not active storages, else
we cannot show the correct fields

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-11-14 13:23:52 +01:00
8fe00d9944 Add logfung in storage_migrate
The log function is used by storage replication to redirect the output in the correct log.
2017-10-16 14:59:57 +02:00