Commit Graph

476 Commits

Author SHA1 Message Date
4050fcc16b move Storage/CephTools to CephConfig
it is not a storage plugin, and it makes more sense to have it
top-level, but there we cannot name it CephTools because of the
existing ones in pve-manager

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:26:11 +01:00
d35a0b4b62 Fix #2019: CephFS storage misses maxfiles
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-12-07 13:47:05 +01:00
54e0b0034b cephfs: tell systemd that mount mount requires network
As we mount this manually and thus systemd doesn't know about any
dependency for cephFS mounts, this got umounted only at the last
stage of shutdown, where network wasn't active anymore.

But, CephFS needs to be connected to an active MDS for a clean
unmount so without network this mount would delay shutdown for quite
a bit, until after some minutes systemd gave up and forced unmount.

So tell systemd that this mount requires network, which can be done
with the '_netdev'[0] mount option, that lucky for us can be also
passed to a mount call and isn't only available for fstab.

with this a mount gets, among others:
> Wants=network-online.target
> Before=umount.target remote-fs.target
> After=remote-fs-pre.target system.slice network.target network-online.target -.mount

Which does the trick for us.

[0]: https://www.freedesktop.org/software/systemd/man/systemd.mount.html#_netdev

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-27 12:10:36 +01:00
49cc7802f7 LVM: lock on volume_resize
This is important for shared LVM storages. As with deletes and
creates of images, as else we may have not the up-to-date metadata
and extents may get reused if another node created an image during
the same time, for example.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-15 10:13:57 +01:00
40d698932e implement map_volume and unmap_volume
This allows to request a mapped device/path explicitly, regardles of
the storage option, eg. krbd option in the RBDplugin.

Bump of the storage ABI => 2

Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2018-11-09 17:25:51 +01:00
dd9e97ed14 find_free_diskname: fixup regex match operator
Co-developed-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-19 11:21:37 +02:00
0057171085 Fix #1925: untaint rbd JSON output
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Reviewed-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2018-09-19 11:21:37 +02:00
dd1fa860d0 get_vm_disk_number: follow up cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-14 09:19:08 +02:00
f4cc2c4afe Fix #1913: get_vm_disk_number: clone plugindata to avoid side effects
Accessing a non-existing 'format' key in plugindata (e.g., in LvmThinPlugin),
created it by autovivication, thus breaking the fallback to the default value
'raw' upon any following access.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-14 09:17:40 +02:00
e5b2206f8a rbd: krbd_feature_disable was not disabling features
$features is actually an array reference, so use it as one.
This broke creation and migration of disks on rbd storages

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-12 14:56:36 +02:00
83a40b8d3b refactor is_valid_vm_diskname to private simpler sub
This was newly introduced and is only used once, so having a
wantarray return mechanism, without ever using it or knowing for sure
if this may help with reuse of the method is not ideal.

Make the sub a module private one just returning the vm disk number
or explicit undef. Pass it the $suffix variable, to avoid recomputing
it every time called by out caller's loop.

If there's re-use potential in the future we can actually decide what
makes sense to return.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-11 07:52:43 +02:00
c4a29df483 refactor finding next diskname for all plugins
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-10 12:21:10 +02:00
59fa9fd6a3 next diskname: start ids with 0 to honor MAX_VOLUMES_PER_GUEST
else we can only have MAX_VOLUMES_PER_GUEST-1 disk per VMID,
not tragic but possible confusing

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-10 11:21:12 +02:00
c0535aa72f make max number of disks a constant
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-07 15:28:15 +02:00
345f898108 add vm_diskname helpers (get_next, is_valid)
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-07 15:28:15 +02:00
aa14def420 Addition to fix #1895, skip image if no owner
Non conforming image names are not ignored anymore by the new rbd_ls
implementation, this patch adds the old behaviour.

This fix is a temporary workaround and should be removed, once the new
image name parser is ready.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-09-07 13:51:18 +02:00
1be93fe2fd rdb: followup cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-06 15:19:16 +02:00
c093e93b28 rbd: remove unused size conversion function
since the json output gives the sizes in bytes, we do not
need to convert anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-09-06 15:18:23 +02:00
89a8800bc3 fix #1895: use json for 'rbd ls -l' and 'rbd info'
since ceph changed the plain output format for 12.2.8
we have to change the code anyway, and when were at it,
we can change it to the (hopefully) more robust json output

Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-09-06 15:11:26 +02:00
ca552c7639 Fix #1858: lvm_find_free_diskname check for base
lvm_find_free_diskname only checked for existing volumes starting with 'vm-',
and not with 'base-'.

Unify implementation with other Plugins.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-08-07 12:14:11 +02:00
f15ac9b5a8 LIO: followup: various small cleanups
move two loop bodies from

if (condition) {
    ...
}

too
next if !condition;

...

to save an indentation level

rename variables to a bit shorter version, i.e.:
s/oneTarget/target/
s/oneTpg/tpg/

and a comment rewording

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-08-02 14:57:07 +02:00
ff69c66022 LIO: followup: shorter stderr/out logging
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-08-02 14:56:20 +02:00
ccdf8ddbda LIO: followup: fix indentation 2018-08-02 14:56:20 +02:00
d9254744a8 LIO: followup: remove trailing whitespaces
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-08-02 14:56:20 +02:00
46c6107eb1 Linux LIO/targetcli support
Introducing LIO/targetcli support allowing to use recent linux
distributions as iSCSI targets for ZFS volumes.

In order for this to work, two preconditions have to be met:

1. the portal has to be set up correctly using targetcli
2. the initiator has to be authorized to connect to the target
   based on the initiator's InitiatorName

When adding a LIO iSCSI target, a new "LIO target portal group" field needs
to be correctly populated in the "Add: ZFS over iSCSI" popup, containing the
fitting "LIO target portal group" name (typically something like 'tpg1').

Signed-Off-By: Udo Rader <udo.rader@bestsolution.at>
Tested-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-08-02 14:56:20 +02:00
2c2fd98b87 add metadata_size and _used to lv list
so that we can show it in the webinterface and the user can check
how full it is

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 11:16:05 +02:00
49d149e638 extend list_thinpools for multiple vgs and more information
if no vg is given, give back all thinpools from all vgs
if verbose is 1, then give back the information about the thinpools
(like size and free)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 11:10:33 +02:00
8cccb3447b add an option to include pvs in lvm_vgs
this will be used for the lvm part of the disk management

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 09:38:27 +02:00
074b2cb4fa remove unused Data::Dumper usages
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-11 12:23:44 +02:00
5527c824dd cephtools: simplify ceph_check_keyfile 2018-07-04 16:56:24 +02:00
5402cea50d cephfs plugin: followup with some code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-04 15:07:33 +02:00
fe0d606f5e Use keyfile create/remove from CephTools
in the RBDPlugin, that is also shared by the CephFSPlugin

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:18:32 +02:00
e34ce14443 Cephfs storage plugin
- ability to mount through kernel and fuse client
 - allow mount options
 - get MONs from ceph config if not in storage.cfg
 - allow the use of ceph config with fuse client
 - Delete secret on cephfs storage creation

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:18:19 +02:00
3e47917203 Add simple keyring check for cephfs/rbd
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:13:21 +02:00
9b7ba1db42 Merge RBD and CephFS code into a helper module
Some methods for connecting to a ceph cluster are the same for RBD and
CephFS, these are merged into the helper modules.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:13:21 +02:00
f9602323c6 lvm: use add hook
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
2e109b4bc2 rbd: use add/delete hooks
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
ab5e32bb1d cifs: use add/delete hooks
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
3932ca0d1b add on_add and on_delete hooks
on_add_hook allows to encapsulate storage specific add steps, like
copying a keyring (RDB) or creating a volume group (LVM) in a clean
manner.
The same for deletion with on_delete_hook, here all should be cleaned
up, as much as possible.

Until now, this was done directly in the api config CREATE and DELETE
code, respectively, with a series of

if ($storage_type eq 'foo) {
    ...
} elsif ($storage_type eq 'bar') {
    ...
}

which isn't really that nice...

Another nice result of this approach is that also external plugins
can use those hooks and to their setup/cleanup steps sanely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
3160dbf139 CIFS: add mkdir option allowing to switch off path creation
see commit: c7616abcb2
2018-06-14 07:49:43 +02:00
5bc3edb243 followup: fix typo 2018-06-06 15:38:22 +02:00
c2f12dc648 Use enums for smbvers to prevent incorrect input 2018-06-06 15:36:25 +02:00
ff6fa67fb6 Make check_connection less costly.
This command will only check the needed share
and do not query the hole server shares.

This reduce the answer time and also has the benefit we check the
credentials on this share and not on the server.
2018-05-22 13:18:12 +02:00
9a75947b49 fix #1770: allow ex/import linked clones.
We encode the base-volume-name in the volname what is not needed.
On ex/import we need the correct dataset name.
2018-05-22 11:36:36 +02:00
840e3797a5 cifs: fix uninitialized warning in check_connection
otherwise this spams the log when check_connection runs into a timeout.
2018-05-16 10:06:00 +02:00
5fc02afb7d fix #1754: rbd: fix handling of external clusters
with the recent refactoring, external clusters were not handled
correctly with librd if a pveceph or storage specific ceph config
exists.

change the behaviour to include the pveceph config file only for pveceph
managed clusters, and a storage specific one only for external ones.

set mon_host correctly using the values from storage.cfg for external
librbd clusters.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2018-05-08 10:09:58 +02:00
fc05c9a0d8 fixup error message typo: s/sould/should/
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-05-08 09:34:37 +02:00
cf8ded6a96 fix #1691: increase timeout in worker
A ZFS storage under heavy load can take more time.
2018-05-07 12:01:20 +02:00
894b9ecc1e fix #1691: replace udev check
`zfs create` add the creation job in a worker queue,
which should normally execute instantly. But there are circumstances
where the job will take a while to get processed.
If this is the case udev settle will see no dev in the queue and the program
will continue without an allocated dev.

The busy waiting is not best practice but the only way to be sure,
that the block device exists.
2018-05-07 12:01:20 +02:00
0b45fb3eb5 Fix #1750: set monhost split to old behavior
The path method of the RBDPlugin got a list with comma separated monhosts,
but it needs the list with semi-colon for qemu.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-05-04 14:18:04 +02:00