BIOS Implementation Test Suite - news - bits-945; now, with 100% more EFI

bits-945 released with these changes:

Add initial support for 32-bit EFI.

BITS now builds GRUB and the BITS-specific GRUB modules for Python, ACPICA, SMP, and other support modules for both 32-bit BIOS (i386_pc) and 32-bit EFI (i386_efi), and constructs a .iso which boots both ways.

The resulting .iso successfully boots as a CD using KVM with OVMF, and still boots as either a CD or hard disk on BIOS-based systems. Note that the .iso does not support booting as a hard disk on EFI-based systems.

On EFI platforms, Python's sys.platform now returns "BITS-EFI"; BIOS platforms will continue to return "BITS". (Future 64-bit EFI support will also use "BITS-EFI"; use bits.ptrsize to find the native size of a pointer, and use the "P" type with the struct or unpack modules to pack or unpack a native-sized pointer.)

On EFI, ACPICA finds the RSDP as an EFI configuration table rather than via memory search.

BITS now provides an "efi" module with some preliminary interfaces to access EFI data structures and call EFI functions. This module provides all the functionality needed to find EFI interfaces by UUID and call arbitrary EFI functions entirely from Python, without writing any new C code. For example, to print an iconic greeting:

import bits, efi
greeting = efi.encode_UCS2_mem("Hello world!\r\n")
efi.call(efi.system_table.ConOut.OutputString,
         efi.system_table.ConOut._addr,
         bits.memory_addr(greeting))

The new efi module includes:

  • The efi.call function seen above
  • Helpers for buffer and UCS-2 Unicode string manipulation
  • efi.system_table, a decoded version of the EFI system table and all the structures and interfaces nested within it, including BootServices, RuntimeServices, and ConIn/ConOut/StdErr.
  • efi.print_variable_names, a larger code sample which demonstrates how to retrieve, decode, and print the list of EFI variables using GetNextVariableName
  • efi.log_efi_info, a function now called at BITS init time to log basic information about the EFI firmware, including the firmware vendor, the firmware version, the UUIDs of supported configuration tables, and the corresponding names for any recognized UUIDs.

Expect the efi module interface to change in future versions as the support for calling EFI functions and decoding EFI data structures becomes simpler and easier to use.

Note that EFI's default text input API doesn't support returning Ctrl-characters, making many common line-editing keys unavailable, and making the essential Ctrl-D keystroke to exit Python impossible; BITS now accepts Escape as an alternative to Ctrl-D.

Other supporting infrastructure leading to EFI support:

  • bits: Add bits.ptrsize, which contains the machine's native pointer size (4 or 8)

  • bits: Add bits.malloc to create a new buffer of a specified size

    Through the wonders of garbage collection, there is no corresponding bits.free.

  • bits: Add a memmove function to manipulate memory directly

    Needed to write buffer manipulation functions in Python rather than in

  • Add GRUB patch fixing EFI memory allocator page rounding

    The EFI memory allocator rounded allocation sizes up to 1k boundaries rather than 4k page boundaries.

  • Add GRUB patch to support allocation below 1M in the EFI memory allocator

    Needed to allocate memory for an IPI handler.

  • Add GRUB patch for EFI memory allocator memory type mapping

    EFI's AllocatePages call will refuse attempts to allocate memory and leave it marked as EfiConventionalMemory (free memory available for subsequent allocations), so map requests for GRUB_MEMORY_AVAILABLE into EfiLoaderCode rather than EfiConventionalMemory.

  • smp: After attempting to allocate a page below 1M, verify its address

  • smp: Allocate working memory with grub_memalign, not grub_mmap_malign_and_register

    On BIOS builds, the difference is harmless, but on EFI builds the latter calls the firmware's AllocatePages rather than using GRUB's internal memory allocator. Since we don't need to reserve the working memory after OS boot, we don't need to use the firmware's memory allocator.

  • smp: Handle systems where the BSP already has an IDTR

    On EFI, GRUB sets up a non-zero IDTR, rather than using the real-mode one IDTR at address 0. Fix that code path to not attempt to "restore" to a real-mode IDTR. Instead, save and restore the GPF handler in the existing IDTR around the function call in smp_function.

  • smbios: Gracefully handle systems without SMBIOS tables

  • mptable: Handle systems that don't have an EBDA

    On systems without an EBDA, the EBDA address in the BDA contains a 0; handle that rather than searching at 0.

  • smp: Find the MADT using acpica rather than searching for the RSDP in memory

    In addition to eliminating duplicate code, this allows smp to work on EFI, which finds the RSDP as an EFI system table rather than through a memory search.

  • readline: Add Escape as an alternate EOF character, for when Ctrl-D doesn't work

    EFI's default text input API doesn't support returning Ctrl-characters, making the essential Ctrl-D keystroke impossible; add Escape as an alternative. Also improves compatibility with GRUB's own command line, which uses Escape to return to the menu.

Other improvements:

  • README.Developers.txt: Document that the lnxboot partition number patch got merged upstream

    GRUB merged the patch in bzr revno 5185.

  • mptable: Check for the MP Floating Pointer Structure in a non-spec-compliant location

    Some BIOSes put the MP Floating Pointer Structure in the 0xE0000 block, rather than a spec-compliant location like the 0xF0000 block. Search that block to find the table in that block, and add a test that fails on such BIOSes.

  • python: Split pyfs into a separate C module, _pyfs

  • smbios: Add a blank line before printing the SMBIOS information to the log

  • smbios: Wrap each SMBIOS structure when printing

Build system improvements:

  • build: Build from copies of the source directories, not the originals

    The build procedures for GRUB and Python potentially leave byproducts in the source directories. Rather than attempting to clean those up afterwards, build from copies of the source and leave the originals pristine. This then removes the need to clean the source trees both before and after the builds.

    The build procedure now verifies at the beginning that the source directories appear pristine (no generated Makefile).

  • Support out-of-tree GRUB builds: Use $(srcdir) or $(top_srcdir) as appropriate when constructing include paths.

  • Configuring and building the same GRUB source tree for two different platforms fails without doing a clean in-between, so instead, do an out-of-tree configure and build for each platform. (See below for many other build system improvements.)

  • build: Set CCACHE_BASEDIR so ccache works on the temporary source copies

    Normally, ccache will never hit if the path to the source or include files changes; thus, copying all the source trees to a temporary directory makes ccache always miss. Set CCACHE_BASEDIR to the root of that temporary directory, so that ccache can still hit if the remainder of the paths match.

  • build: Copy grub modules manually rather than running grub-install

    Rather than running grub-install to copy grub modules, and mocking up all the other things it wants to do as no-ops, just copy the modules manually.

Bugfixes:

  • acpi: Fix improper access to number_system_localities field which caused an exception.

  • In generated menuentries, stop deleting modules after import.

  • smp: Initialize working memory to 0

    smp_init added a magic signature to its working memory, and checked for that signature to see if it had already run; however, smp_init did not clear its working memory after allocating it. On systems that don't clear RAM on reboot, smp_init would allocate its working memory at the same address each time, and would thus think it had initialized when it hadn't, causing Bad Things to happen.

    Fix by initializing working memory after allocation.

  • acpica: Make acpica_init_state static

    Nothing outside of acpica.c accesses acpica_init_state, so make it static and drop it from acpica.h.

  • bits: Make cpuid_result print all eight digits of its registers

    Format using #010x, since the 0x counts as part of the length.

  • testacpi: Fix typo in _PSD test