neither vmdk images with multiple children, nor ones with multiple extents
(that might in turn be backed by multiple files) are allowed when an image is
untrusted.
Reported-by: Friedrich Weber <f.weber@proxmox.com>
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Add a SAFE_CHAR_WITH_WHITESPACE_CLASS_RE sister variant of the shared
SAFE_CHAR_CLASS_RE shared regex to the base storage module, this use
case exist is a generic one after all, and use swap the untaint method
that parses the file a disk references to it.
Note that this is only the disk file name from inside the archive and
thus during the extraction to a staging/working directory, from there
it will be imported as volume allocated by the common storage system,
and thus follow our ordinary volume name scheme.
Improves disk detection when importing, e.g., the from upstream
provided GNS3 OVA.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
If the base image (parent) of an image contains e.g. whitespace in it's
path, the current untainting would not match and it would seem there was
no parent.
Since untrusted files are not allowed to have backing parts, just warn,
when encountering this case to keep backwards compatibility.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since we want to handle ova files (which are only ovf+images bundled in
a tar file) for import, add code that handles that.
we introduce a valid volname for files contained in ovas like this:
storage:import/archive.ova/disk-1.vmdk
by basically treating the last part of the path as the name for the
contained disk we want.
in that case we return 'import' as type with 'vmdk/qcow2/raw' as format
(we cannot use something like 'ova+vmdk' without extending the 'format'
parsing to that for all storages/formats. This is because it runs
though a verify format check at least once)
we then provide a function to use for that:
* extract_disk_from_import_file: this actually extracts the file from
the archive. Currently only ova is supported, so the extraction with
'tar' is hardcoded, but again we can easily extend/modify that should
we need to.
we currently extract into the either the import storage or a given
target storage in the images directory so if the cleanup does not
happen, the user can still see and interact with the image via
api/cli/gui
we have to modify the `parse_ovf` a bit to handle the missing disk
images, and we parse the size out of the ovf part (since this is
informal only, it should be no problem if we cannot parse it sometimes)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
in DirPlugin and not Plugin (because of cyclic dependency of
Plugin -> OVF -> Storage -> Plugin otherwise)
only ovf is currently supported (though ova will be shown in import
listing), expects the files to not be in a subdir, and adjacent to the
ovf file.
listed will be all ovf/qcow2/raw/vmdk files.
ovf because it can be imported, and the rest because they can be used
in the 'import-from' part of qemu-server.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
this allows checking some extra attributes for images which come from
a potentially malicious source.
since file_size_info is not part of the plugin API, no API bump is
needed. if desired, a similar check could also be implemented in
volume_size_info, which would entail bumping both APIVER and APIAGE
(since the additional parameter would make checking untrusted volumes
opt-in for external plugins).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
A popular ISO compressed exclusively with bz2 is OPNsense [2].
Since this requires adding `bz2` to the list of known compression
formats we add decompression methods for vmz and tar.
[2] https://opnsense.org/download/
Suggested-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Tested-By: Aaron Lauterer <a.lauterer@proxmox.com>
If the json was empty, for example if the qemu-img command times out, a
message
warn "could not parse qemu-img info command output for '$filename' - $err\n";
would have been printed.
This message could lead one to think the issue lies in the contents of
the json, even if the previous warning said that there was a timeout.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Commit 7020491 ("esxi: add 'port' config parameter") started using
the 'port' option in a second plugin, but the definition stayed in the
PBS plugin. Avoid the hidden dependency and move the definition to the
base plugin instead.
It is necessary to mark it as optional or it would be required always.
Clarify that the option is not used by NFS and CIFS.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
these can just as well be `die` statements right there, there is no complicated
cleanup that would warrant a goto statement..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
This prevents importing from vmdks with whitespaces in file names.
Further, some operations that include file sizes (like listing disks)
would potentially fail entirely if a custom disk with a badly name
backing device exists in a VM images directory since they don't expect
this. Specifically, since we don't necessarily know the actual naming
scheme of the current storage in the plain Plugin.pm version, we don't
check the full name anyway, so why bother with whitespaces...
See-also: https://forum.proxmox.com/threads/new-import-wizard-available-for-migrating-vmware-esxi-based-virtual-machines.144023/page-16#post-658697
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Instead of just using it as a warning and then trying to parse an
empty string as json.
For example, trying to parse unsupported vmdks, previously we'd see
something like this:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
could not parse qemu-img info command output for
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk'
- malformed JSON string, neither tag, array, object, number, string
or atom, at character offset 0 (before "(end of string)") at
src/PVE/Storage/Plugin.pm line 962, <DATA> line 960.
Now it simply shows:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Add a new 'import' content type which will be the corner stone for a
better API and UI integrated way to import virtual guests into Proxmox
VE.
For starters this will be used to implement a ESXi adapter, so that
those VMs can get imported nicely.
Later we want to integrate the OVF/OVA import skeletons we got in
qemu-server to something more usable here.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: add more commit message with some background ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The default description doesn't make it clear what the property is
used for in the context of storages.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
dd supports a 'status' flag, which enables it to show the copied bytes,
duration, and the transfer rate, which then get printed to stderr.
Signed-off-by: Leo Nunner <l.nunner@proxmox.com>
adds information for how to decompress isos.
generates the compressor regex from a list of comression formats (to
avoid redundancy)
extends the download_url wtih the functionality to handley compression
for images
Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
Move the warning over to creating and updating storage configs, which
is much less noisy as the constantly called activate storage (e.g.,
pvestatd).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since commit 8e623a2930, the inequality
check for content-dirs prints a warning if a content directory path
could not be resolved, i.e., if `abs_path` returns undef. Among other
things, `abs_path` returns undef if the path has an inner (= any but
last) component that does not exist. This can happen for a storage
with content type `iso,vztmpl` and `create-subdirs` set to 0, in case
`template/` does not exist. In this case, the warnings printed by
pvestatd are quite noisy.
As missing content directories are not a problem per se, remove the
warning and just ignore the directory during the inequality check.
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
This is slightly confusing due to both options, the legacy convoluted
one and the new targeted one, exist, but before the rework we skip if
either of those sub-expressions was true, so doing it needs both to
be true.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
checking the content dirs for clashes via abs_path must be done after
the logic for creating them ran, as abs_path is working on actual
filesystem level, so it will return undf if the directory does not
exist, in which case we then set a hash entry for "undef", and the
next for loop round then resolved again to "undef", resulting in a
false-positive of the check.
Avoid the dangerous "return if" stanzas and reverse them to an actual
if block, which is much safer to adapt. Then move the check for
duplicate content-dir usage after that.
best viewed with white space change ignored: git show -w
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This makes it possible to add all mount options offered by mount.cifs.
NFS & CIFS now share the options parameter since they use it for the
same purpose.
Signed-off-by: Stefan Hrdlicka <s.hrdlicka@proxmox.com>
[FE: rebase + style fixes]
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
[T: fix merge conflict ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This prevents strange interactions in case the same content directory
is used for multiple content types.
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
In the previous code, if `create-base-path` was explicitly
set to false, it would be treated the same as if it was
undef, falling through to whatever 'mkdir' was.
Instead, the new options should always be preferred, and the
logic can be simplified to a single line.
Here's the table showing the difference, 'u' being 'undef':
config: mkdir: u 0 1 u 0 1 u 0 1
create: u u u 0 0 0 1 1 1
=========================
mkpath: old: 1 0 1 0 0 1 1 1 1
new: 1 0 1 0 0 0 1 1 1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The `mkdir` option has two meanings[0][1] which are split up in `create-path`
and `create-sub-dirs`.
The `create-base-path` option decides if the path to the storage is
automatically created or not.
The `create-subdirs` options decides if the default directory
structure (dump, images, ...) at the storage location is created.
The `mkdir` option is still working but will trigger a warning in the
logs.
As a side effect, this also fixes#3214 because the `create-base-path` option
is now run after the `is_mountpoint` check in the `activate_storage`
method in DirPlugin.pm.
The 'mkpath' command has been moved into a new helper function that
first determines if the conditions to create the path is true, called
'config_aware_base_mkdir'.
[0] https://lists.proxmox.com/pipermail/pve-devel/2020-December/046575.html
[1] https://lists.proxmox.com/pipermail/pve-devel/2020-December/046576.html
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
