A python module for configuration of block devices

Support resize of encrypted (LUKS) block devices

Motivation

Many people resize devices and many people encrypt devices. Since it is possible to resize encrypted block devices it makes sense for blivet to be able to do so.

Requirements

1. be able to resize any encrypted block device that contains formatting considered resizeable by blivet

Scope

This should amount to a resize method in blivet.formats.LUKS, some expansion of the logic in Blivet.resizeDevice, and adaptation of any anaconda code that schedules resize actions directly.

Notes

Since we do not currently support resize of LVM PVs (and therefore VGs) there is no need to be concerned with this use case.

Currently blivet expects pesize parm of the vgcreate function to be in MiB. However it does not accept any floating values. This limits passing any values which are KiBs for pesize to create a vg. This patch fixes the issue by accepting floating values also.

Bug url: https://bugzilla.redhat.com/show_bug.cgi?id=1198568

Signed-off-by: Timothy Asir Jeyasingh [email protected]

libbytesize provides an implementation of what we need for manipulations and calculations with sizes in bytes. It's a little bit faster and more importantly it can be shared with other projects because it's written in C. Thus we should use it.

At some point it would be nice if we could drop our own Size class and the blivet/size.py module altogether because they don't bring any extra value, but for now we need this thin wrapper to keep the API (almost*) stable. The only real changes are:

1. Strings like "100 kibibytes" are no longer accepted, I doubt anybody has ever used this functionality. If the opposite turns out to be true such support will be added to libbytesize and no change will be needed in Blivet.

2. humanReadable() now ignores the 'skip' and 'min_value' attributes. The former one is always replaced by 'True' (i.e. trailing zeroes are always stripped), the latter one is ignored because it made things really cryptic and weird.

• see the commit, mainly the part modifying/removing the tests for more details

Note: tests are expected to be failing at this phase -- libbytesize is not available as a package yet. I am of course not going to merge this before that changes.

These commits change how we represent LVs. Instead of having special classes for various types of LVs, we now have a single class with the type of LV being more a dynamic property than a once-given thing. This will allow us to implement various LVM-related features in the future that have so far been limited by our multi-class representation.

I'm not adding any new functionality or features so no tests are added. Just the existing ones are modified to continue working.

I'll add a section to the release_notes.rst file tomorrow as an extra commit to this PR.

Implements setting UUIDs via passing new_uuid="..." to the format.

This is similar to setting labels, but while labels are set by just passing the label=something argument, this uses new_uuid=something because first of all, an UUID is not optional and we also have an uuid argument that's already used by the device tree populator to pass the UUID from the udev attributes.

File systems that support setting UUIDs are now ext2/3/4, FAT, NTFS, XFS, JFS, ReiserFS, HFS+ and swap. Not supported is HFS, because it doesn't seem to have a notion for setting serial or any kind of volume ID.

Where a class is listed that means instantiating that class is supported. Where is it not, we are specifying the public methods and attributes. There are some issues with undocumented members and singletons, but they are minor.

Since blivet is supposed to be compatible with both python 2 and 3 IMO it makes sense to run the test suite with both versions. (not to mention build the entire package with 2 or 3) At present only python2 is used to execute the test suite while python3 is used for pylint (due to pocketlint being python3 only).

This PR introduces the following changes:

• adds Makefile targets to execute the test suite under python 2 and 3 separately
• make test now runs the test suite with both python 2 and 3
• adds 2 separate targets to execute test suite with coverage report for python 2 and 3
• adds new check-requires target which checks if Requires:/BuildRequires and a few manual added ones are present
• replaces /usr/bin/python with /usr/bin/env python inside test cases. NOTE: I don't see any executable tests/*.py files but maybe this is a good measure. Otherwise this line doesn't have any effect and can be removed.
• updates the test cases to take care of several deprecation warnings coming from unittest. I've introduces a BlivetTestCase class which adds some aliases in case we're running with python2.

TODO:

There are still a few places which use hard-coded python version:

• Makefile uses PYTHON=python2 which is used in a few places calling setup.py
• some scripts/ use /usr/bin/python explicitly
• runpylint.py uses /usr/bin/python3 explicitly

make check uses pocketlint which is Python3 only. I have not looked into how hard it will be to build it for Python2. Will do it tomorrow.

Test results:

Running the test suite with python2 gives me 4 errors:

======================================================================
ERROR: testAvailabilityMDRAIDplugin (devices_test.dependencies_test.MockingDeviceDependenciesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/blivet/tests/devices_test/dependencies_test.py", line 64, in testAvailabilityMDRAIDplugin
    self.assertIsNotNone(ActionCreateFormat(self.luks, fmt=getFormat("macefi")))
  File "blivet/deviceaction.py", line 530, in __init__
    raise ValueError("resource to create this format %s is unavailable" % fmt)
ValueError: resource to create this format non-existent macefi filesystem is unavailable

======================================================================
ERROR: testGetLabelArgs (formats_test.labeling_test.MethodsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/blivet/tests/formats_test/labeling_test.py", line 73, in testGetLabelArgs
    reiserfs = self.fs["reiserfs"]
KeyError: 'reiserfs'

======================================================================
ERROR: testActionDependencies (action_test.DeviceActionTestCase)
Verify correct functioning of action dependencies.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/blivet/tests/action_test.py", line 944, in testActionDependencies
    format_lv = ActionCreateFormat(testlv, fmt)
  File "blivet/deviceaction.py", line 530, in __init__
    raise ValueError("resource to create this format %s is unavailable" % fmt)
ValueError: resource to create this format non-existent None is unavailable

======================================================================
ERROR: testContainerActions (action_test.DeviceActionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/blivet/tests/action_test.py", line 1122, in testContainerActions
    fmt=new_lv_format)
  File "tests/storagetestcase.py", line 245, in scheduleCreateFormat
    action = blivet.deviceaction.ActionCreateFormat(device, fmt)
  File "blivet/deviceaction.py", line 530, in __init__
    raise ValueError("resource to create this format %s is unavailable" % fmt)
ValueError: resource to create this format non-existent xfs filesystem is unavailable

----------------------------------------------------------------------
Ran 467 tests in 704.377s

FAILED (errors=4, skipped=319)

Running with python3 gives me much more of the type:

======================================================================
ERROR: testMountingXFS (formats_test.selinux_test.SELinuxContextTestCase)
XFS does not have a lost+found directory.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/blivet/tests/formats_test/selinux_test.py", line 67, in testMountingXFS
    an_fs = fs.XFS(device=self.loopDevices[0], label="test")
  File "/root/blivet/blivet/formats/fs.py", line 101, in __init__
    DeviceFormat.__init__(self, **kwargs)
  File "/root/blivet/blivet/formats/__init__.py", line 186, in __init__
    self.device = kwargs.get("device")
  File "/root/blivet/blivet/formats/__init__.py", line 315, in <lambda>
    lambda f,d: f._setDevice(d),
  File "/root/blivet/blivet/formats/__init__.py", line 306, in _setDevice
    error_msg = self._deviceCheck(devspec)
  File "/root/blivet/blivet/formats/__init__.py", line 301, in _deviceCheck
    if devspec and not devspec.startswith("/"):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

----------------------------------------------------------------------


I'm running on a recent F22 system upgraded to packages from the Rawhide repo. Let me know how would you like to proceed. 

The discoverable partitions specification defines a way to identify partitions used for various well known mount points:

https://uapi-group.org/specifications/specs/discoverable_partitions_specification/

This relies on setting well known GPT partition GUIDs, instead of parted's generic "Linux Data" GUID which more or less everything gets set to today (swap being a notable exception).

My goal is to make it possible for applications like Anaconda to "do the right thing(tm)" with little to no effort when doing disk partitioning. This series aims to make that possible, while still retaining the current behaviour by default for the sake of backwards compatibility.

With this series, a simple app usage might not need anything more than to opt-in to discoverable partitions with:

from blivet.git import gpt_set_discoverable_partitions

gpt_set_discoverable_partitions(True)

Assuming the disk has 'gpt' disk label and the app is already passing 'mountpoint' to the blivet.new_partition() method then it will "just work" and pick the right GUID for the given mount point. The mountpoint parameter would historically only be needed if a format is requested with new_partition, but now it can be passed regardless so it can be forwarded onto the PartitionDevice constructor.

The examples/partitioning.py script illustrates this, as does a comprehensive test at tests/vmtests/gpt_test.py

If the app is more advanced and creating PartitionDevice instances directly instead of via the convenient blivet.py API entrypoints, then it will need to be changed to pass mountpoint to this object constructor too, but this is still hopefully fairly straightforward.

Note that this functionality relies on pypartd exposing the type_uuid attribute on the Partition object. This is new functionality added to pyparted last month in https://github.com/dcantrell/pyparted/pull/95 Note pyparted only exposes this if compiled against a new enough parted. As a result some code additions in this series have 'hasattr' changes to attempt to gracefully degrade when the functionality is missing.

Since this is my first time looking at Blivet code in any detail, its taken me a while to get my head around the design and expected usage patterns. I'm still not entirely confident in my understanding, but the examples/partitioning.py script is at least doing the right thing with this series of changes.

Happy to have feedback on things I've overlooked / approached in the wrong/suboptimal way.

The problem is that we try to activate an lv in order to remove the formatting before destroying the lv, but the activation fails because the user set the lv to ignore activation commands.

It is possible that we should be more cautious and possibly only pass ignore_skip=True when we're about to wipe the formatting. That would require setting up a generic mechanism for passing additional arguments, which seems quite disruptive given the problem.

Opinions?

This is my first attempt to bring some test documentation to blivet before actually trying to write any more tests. Summary of changes:

• modify existing Makefile target to include test documentation in the main document
• add dependency on Graphviz which is used to create inheritance graphs for test cases (used to outline which are the base test classes in Blivet). WARNING: may break Red The Docs if they don't have graphviz installed. Possibly needs a BuildRequires change for the spec file as well.
• update .gitignore
• add tests/README.rst with basic description of blivet's test suite.

The UEFI spec supports legacy MBR (msdos) in section 5.2 of the UEFI 2.7 spec [1]. There's a number of use cases where we might boot on a UEFI device but want to support, but not by default, the creation of images using legacy MBR partitions. One of these use cases is cloud images where we can produce a single image tha will run on UEFI supported, including secure-boot, clouds but also support AWS which only supports MBR partitioning schemes [2] and in the current config anaconda fails the install where this is a perfectly valid and widely supported deployment mechanism.

So enable the ability to use legacy msdos partitioning schemes, but don't use it for default installs, so people may consume this mechanism using kickstarts rather than having the install fail for a valid use case.

[1] https://uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf [2] https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html

Signed-off-by: Peter Robinson [email protected]

Hello! Please add multi-drive support for f2fs It is impossible to select manually partitioned disk without reformating by blivet (at least in Fedora) It does not support f2fs multidrive

https://lkml.iu.edu/hypermail/linux/kernel/1611.1/01814.html

This is indirectly related to the bugzilla issue Hardware without AES-NI: use xchacha12/Adiantum instead of AES-XTS.

If the availability of AES-NI is checked to choose the most appropriate disk encryption (see the issue ticket), it might make sense to also check if the system is a 64 bit architecture.

If the system is a 64 bit architecture, the performance and consistency of BTRFS partitions can be improved by choosing xxhash instead of CRC32c by default for BTRFS. On one hand, xxhash is usually faster than CRC32c on 64 bit architectures (I expect most installations are already on 64 bit hardware). On the other hand, xxhash's 64 bit are much more resistant to collisions when compared to CRC32c's 32 bit.

Implementing the bugzilla ticket and the 64 bit-check together may save some work time, e.g. one function to get both information.

Looks like it is some issue with pep17 based build. Content of the generated .whl archive

[[email protected] dist]$ unzip -l blivet-3.4.3-py3-none-any.whl
Archive:  blivet-3.4.3-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     4986  02-01-2022 13:45   blivet/__init__.py
    14043  02-01-2022 13:45   blivet/actionlist.py
     9563  02-01-2022 13:45   blivet/arch.py
    46624  02-01-2022 13:45   blivet/blivet.py
     9454  02-01-2022 13:45   blivet/callbacks.py
    41757  04-29-2022 11:44   blivet/deviceaction.py
    79792  02-01-2022 13:45   blivet/devicefactory.py
    41238  02-01-2022 13:45   blivet/devicetree.py
     6303  02-01-2022 13:45   blivet/errors.py
     7768  02-01-2022 13:45   blivet/fcoe.py
     3716  02-01-2022 13:45   blivet/flags.py
     2436  02-01-2022 13:45   blivet/i18n.py
    22000  02-01-2022 13:45   blivet/iscsi.py
     6935  02-01-2022 13:45   blivet/mounts.py
    84871  02-01-2022 13:45   blivet/partitioning.py
     7233  02-01-2022 13:45   blivet/safe_dbus.py
     7154  02-01-2022 13:45   blivet/size.py
     2687  02-01-2022 13:45   blivet/storage_log.py
     3751  02-01-2022 13:45   blivet/threads.py
     3459  02-01-2022 13:45   blivet/tsort.py
    31981  02-01-2022 13:45   blivet/udev.py
    34721  02-01-2022 13:45   blivet/util.py
    14451  02-01-2022 13:45   blivet/zfcp.py
     1014  02-01-2022 13:45   blivet/dbus/__init__.py
     1982  02-01-2022 13:45   blivet/dbus/action.py
    10379  02-01-2022 13:45   blivet/dbus/blivet.py
     1732  02-01-2022 13:45   blivet/dbus/constants.py
     3154  02-01-2022 13:45   blivet/dbus/device.py
     2451  02-01-2022 13:45   blivet/dbus/format.py
     2971  02-01-2022 13:45   blivet/dbus/manager.py
     4683  02-01-2022 13:45   blivet/dbus/object.py
        0  02-01-2022 13:45   blivet/devicelibs/__init__.py
     1429  02-01-2022 13:45   blivet/devicelibs/btrfs.py
     2202  02-01-2022 13:45   blivet/devicelibs/crypto.py
     3521  02-01-2022 13:45   blivet/devicelibs/disk.py
    32101  02-01-2022 13:45   blivet/devicelibs/edd.py
     7685  02-01-2022 13:45   blivet/devicelibs/lvm.py
     1576  02-01-2022 13:45   blivet/devicelibs/mdraid.py
    24513  02-01-2022 13:45   blivet/devicelibs/raid.py
     1992  02-01-2022 13:45   blivet/devices/__init__.py
    23692  02-01-2022 13:45   blivet/devices/btrfs.py
     3726  02-01-2022 13:45   blivet/devices/cache.py
     7659  02-01-2022 13:45   blivet/devices/container.py
    12301  02-01-2022 13:45   blivet/devices/device.py
    25634  02-01-2022 13:45   blivet/devices/disk.py
    10118  02-01-2022 13:45   blivet/devices/dm.py
     5287  02-01-2022 13:45   blivet/devices/file.py
     6618  02-01-2022 13:45   blivet/devices/lib.py
     4458  02-01-2022 13:45   blivet/devices/loop.py
     8403  02-01-2022 13:45   blivet/devices/luks.py
   104781  02-01-2022 13:45   blivet/devices/lvm.py
    29633  02-01-2022 13:45   blivet/devices/md.py
     2087  02-01-2022 13:45   blivet/devices/network.py
     3031  02-01-2022 13:45   blivet/devices/nfs.py
     3743  02-01-2022 13:45   blivet/devices/nodev.py
     2871  02-01-2022 13:45   blivet/devices/optical.py
    38653  02-01-2022 13:45   blivet/devices/partition.py
     4820  02-01-2022 13:45   blivet/devices/raid.py
    30792  02-01-2022 13:45   blivet/devices/storage.py
        0  02-01-2022 13:45   blivet/events/__init__.py
     4285  02-01-2022 13:45   blivet/events/changes.py
    11201  02-01-2022 13:45   blivet/events/handler.py
    11418  02-01-2022 13:45   blivet/events/manager.py
    25419  02-01-2022 13:45   blivet/formats/__init__.py
     2225  02-01-2022 13:45   blivet/formats/biosboot.py
    22268  02-01-2022 13:45   blivet/formats/disklabel.py
     4031  02-01-2022 13:45   blivet/formats/dmraid.py
    48225  02-01-2022 13:45   blivet/formats/fs.py
     1573  02-01-2022 13:45   blivet/formats/fslib.py
    17959  02-01-2022 13:45   blivet/formats/luks.py
     7171  02-01-2022 13:45   blivet/formats/lvmpv.py
     4047  02-01-2022 13:45   blivet/formats/mdraid.py
     2903  02-01-2022 13:45   blivet/formats/multipath.py
     3619  02-01-2022 13:45   blivet/formats/prepboot.py
     7795  02-01-2022 13:45   blivet/formats/swap.py
       38  02-01-2022 13:45   blivet/populator/__init__.py
    20284  02-01-2022 13:45   blivet/populator/populator.py
     2493  02-01-2022 13:45   blivet/populator/helpers/__init__.py
     2712  02-01-2022 13:45   blivet/populator/helpers/boot.py
     4453  02-01-2022 13:45   blivet/populator/helpers/btrfs.py
     2082  02-01-2022 13:45   blivet/populator/helpers/devicepopulator.py
     9348  02-01-2022 13:45   blivet/populator/helpers/disk.py
     6349  02-01-2022 13:45   blivet/populator/helpers/disklabel.py
     2468  02-01-2022 13:45   blivet/populator/helpers/dm.py
     3992  02-01-2022 13:45   blivet/populator/helpers/dmraid.py
     4103  02-01-2022 13:45   blivet/populator/helpers/formatpopulator.py
     2590  02-01-2022 13:45   blivet/populator/helpers/loop.py
     6019  02-01-2022 13:45   blivet/populator/helpers/luks.py
    19727  04-29-2022 11:44   blivet/populator/helpers/lvm.py
     8471  02-01-2022 13:45   blivet/populator/helpers/mdraid.py
     2566  02-01-2022 13:45   blivet/populator/helpers/multipath.py
     2064  02-01-2022 13:45   blivet/populator/helpers/optical.py
     4678  02-01-2022 13:45   blivet/populator/helpers/partition.py
     2869  02-01-2022 13:45   blivet/populator/helpers/populatorhelper.py
      149  02-01-2022 13:45   blivet/static_data/__init__.py
     3523  02-01-2022 13:45   blivet/static_data/luks_data.py
     4047  02-01-2022 13:45   blivet/static_data/lvm_info.py
     2431  02-01-2022 13:45   blivet/static_data/mpath_info.py
     5359  02-01-2022 13:45   blivet/static_data/nvdimm.py
        0  02-01-2022 13:45   blivet/tasks/__init__.py
    20368  02-01-2022 13:45   blivet/tasks/availability.py
     1598  02-01-2022 13:45   blivet/tasks/dfresize.py
     5413  02-01-2022 13:45   blivet/tasks/fsck.py
     3085  02-01-2022 13:45   blivet/tasks/fsinfo.py
     3186  02-01-2022 13:45   blivet/tasks/fslabeling.py
     5790  02-01-2022 13:45   blivet/tasks/fsminsize.py
    10426  02-01-2022 13:45   blivet/tasks/fsmkfs.py
     6033  02-01-2022 13:45   blivet/tasks/fsmount.py
     3723  02-01-2022 13:45   blivet/tasks/fsreadlabel.py
     6128  02-01-2022 13:45   blivet/tasks/fsresize.py
     4838  02-01-2022 13:45   blivet/tasks/fssize.py
     2961  02-01-2022 13:45   blivet/tasks/fssync.py
     1784  02-01-2022 13:45   blivet/tasks/fstask.py
     2127  02-01-2022 13:45   blivet/tasks/fsuuid.py
     3362  02-01-2022 13:45   blivet/tasks/fswritelabel.py
     1949  02-01-2022 13:45   blivet/tasks/fswriteuuid.py
     3355  02-01-2022 13:45   blivet/tasks/lukstasks.py
     2885  02-01-2022 13:45   blivet/tasks/pvtask.py
     3333  02-01-2022 13:45   blivet/tasks/task.py
      400  02-01-2022 13:45   etc/dbus-1/system.d/blivet.conf
      113  02-01-2022 13:45   usr/lib/systemd/system/blivet.service
      364  02-01-2022 13:45   usr/libexec/blivetd
      106  02-01-2022 13:45   usr/share/dbus-1/system-services/com.redhat.Blivet0.service
    18092  04-29-2022 11:44   blivet-3.4.3.dist-info/COPYING
     3973  04-29-2022 11:44   blivet-3.4.3.dist-info/METADATA
       92  04-29-2022 11:44   blivet-3.4.3.dist-info/WHEEL
        7  04-29-2022 11:44   blivet-3.4.3.dist-info/top_level.txt
    10459  04-29-2022 11:44   blivet-3.4.3.dist-info/RECORD
---------                     -------
  1341519                     128 files

When using uninitialized disks the factory fails with not enough free space for new device because without a partition table we fail to find a free space for a new partition. We should report that the disk has no partition table and cannot be used because of this.

See #725 for RFE for supporting unpartitioned disks in the factory.