BIOS Implementation Test Suite - news - bits-1081: ctypes. ctypes run. run types run

bits-1081 released with these changes:

The "ctypes. ctypes run. run types run" release.

This release introduces preliminary support for the Python ctypes module in EFI builds. This includes the initial infrastructure to compile libffi as part of BITS.

ctypes data types, ctypes structures, and read/write access to memory work. This replaces many uses of bits.memory, struct.pack, struct.unpack, and the BITS unpack module. In particular, the use of ctypes data types and structures allows round-trip conversions from structures in memory to Python data types and back, generally without making copies.

This initial implementation provides enough support for all the data structures in the efi module, which has now completely switched over to ctypes-based data types and structures.

Notable limitations:

  • No support for non-EFI builds yet, because GRUB uses stdcall and regparm=3 for PC BIOS builds, which libffi does not support on non-Windows systems.
  • Function calls to EFI functions still require efi.call; the ctypes-based FFI does not yet work, as it does not understand the EFI calling convention.
  • dlopen does not work, so there's no way to obtain pointers to existing C functions to call.

Note that to support this change, the version of Python built into BITS now internally uses UTF-16 (2-byte characters) rather than UCS-4 (4-byte characters), to match EFI. This allows the use of the ctypes functions for the C wchar_t type, rather than hand-rolled functions for two-byte Unicode. Python still has full support for all of Unicode, including characters outside the Basic Multilingual Plane; this just changes the internal representation. However, attempting to use characters outside the BMP in EFI calls may or may not work, depending on your firmware; your mileage may vary.

This release also introduces support for reading and writing files using EFI. On EFI systems, BITS now has full support for writing arbitrary files to a FAT filesystem. The new efi.get_boot_fs() function will retrieve the filesystem BITS booted from, as an efi_file, a new Python file-like object. In addition to the usual file methods, an efi_file provides methods to open and create files and directories. For example:

import efi
root = efi.get_boot_fs()
newdir = root.mkdir("newdir")
newfile = newdir.create("newfile")
newfile.write("Hello world!\n")

Or, chaining the calls together (convenient for command-line usage):

efi.get_boot_fs().mkdir("newdir").create("newfile").write(data)

The new functions acpi.efi_save_tables() and efi.save_tables() use this new filesystem write capability to save copies of ACPI and EFI tables, for later inspection.

Note that EFI firmware typically only supports FAT filesystems, not iso9660 (the filesystem used on optical media). The BITS .iso images, even when imaged to a USB disk, keep most of their files in an iso9660 filesystem, with the exception of a small FAT filesystem used to store GRUB itself as an EFI binary. Thus, get_boot_fs() will not produce the expected results when booting from an optical disc or from a disk created from a BITS .iso; for full read/write support, create a FAT-based BITS disk, following the procedure in INSTALL.txt.

This release adds support for many new EFI protocols:

  • efi: Add read/write file support using FileProtocol, including a file-like object

    The efi_file wrapper includes the standard Python file-like methods (read, write, seek, tell, close, flush), EFI-specific properties (file_info, file_system_info, volume_label) and methods (delete), and methods to create or open another efi_file relative to a directory (open, create, mkdir).

    Also add a definition of the EFI SimpleFileSystemProtocol to open a block device, with a root property to get the root directory as an efi_file.

    Add a get_boot_fs() function to return the filesystem BITS booted from, obtained via the DeviceHandle of the LoadedImageProtocol on the image handle. Useful as the root for writing files.

  • efi: Add DevicePathProtocol and DevicePathToTextProtocol

    DevicePathToTextProtocol includes helpers to transform the returned paths into Python unicode strings, and then free the original memory.

  • efi: Add LoadedImageProtocol

Other changes in this release:

  • python: Support the zlib module

  • bits.present, mkpresent: Use zlib to compress and decompress slide images

    Provides an order of magnitude improvement to disk usage and load time.

  • acpi: Add an efi_save_tables utility function to save ACPI tables to files

    As the name suggests, this function only works on EFI, since it uses the EFI file write support.

    In addition to the binary dumps, this also includes text decodes of selected structures, and enough address information to recreate the tables in memory.

  • efi: Add a function efi.save_tables() to save the core EFI tables to files

  • efi: As a workaround for limitations in the internal EFI function call interface, add a compatibility layer for 64-bit arguments on 32-bit platforms. Wrap 64-bit arguments in efi.split64(), and they will automatically be split into pairs of 32-bit arguments on 32-bit platforms, for compatibility with the current efi.call interface.

  • Add EFI tests. The first round of tests verifies the CRCs of the core EFI tables.

  • efi: Add a base class for EFI protocols

    Protocols are ctypes structures with an associated GUID; they provide a .from_handle classmethod to get the protocol from an EFI handle (via OpenProtocol) and wrap it in the protocol class.

  • efi: Add more known UUIDs

  • efi: When printing structures, format pointers and unsigned types as hex, and list out the contents of arrays.

  • efi: Various new helper functions:

    • Add a helper function check_status to throw an exception for != EFI_SUCCESS
    • Add a helper check_error_value to handle non-status return values. Some EFI functions return a non-status return value, but still use the EFI_ERROR bit to indicate an error. Add a helper that checks only that bit, and otherwise returns the value for subsequent use.
    • Add a locate_handles helper to call LocateHandle with a given GUID. This helper handles the two-pass memory allocation, and returns a ctypes array of handles.
    • Add helper functions to compute table CRC32 values. These compute the CRC32 as if the table's CRC32 field is 0, and then compare the result to the table's actual CRC32 field.
    • Add a helper function efi.to_bytes to convert a ctypes structure to raw bytes

  • README.Developers.txt: Document Python patches for ctypes and libffi