Troubleshooting

Server is unreachable

Follow these steps to identify the cause:

Boot order incorrect

As our servers are run in UEFI mode, the boot order is not fixed and can be misconfigured.

We generally set the following boot order:

1. PXE (Network boot)
2. USB (e.g. Virtual CD-ROM of the Virtual Console im RMS)
3. Operating systems

These is set automatically on start of a Rescue oder bei Reinstall via CloudHub. Both of these work regardless of the configured boot order.

The boot order can differ from our specification due to following reasons:

• The operating system was installed via CD/ISO
  • Installation processes create their entry at the front of the boot order
• At least one boot was done directly from the storage layer (Disk, RAID controllers)
  • Usually an operating system installation sets up a fallback bootloader, that may be used by UEFI. If booted, this creates a new boot entry for all found operating system EFI bootloaders at the front of the boot order. There is no deduplication being done, so that duplicates may be created.
  • Due to this reason we disable direct boot from RAID controllers. However with software RAIDs this issue can still arise when booting a single disk directly.
• The boot order has been changed manually via efibootmgr

Manual fixing

By using efibootmgr -v the boot order can be investigated.

Following an example with a boot order (BootOrder) that prevents automatic boot of other entries than the operating system, such as “Generic USB Boot” (Virtual CD-ROM):

BootCurrent: 0012
Timeout: 0 seconds
BootOrder: 0012,0011,0010,000F
Boot000F* Generic USB Boot	UsbClass(ffff,ffff,255,255)
Boot0010* Slot 1 Port 1 : HP Ethernet 10Gb 2-port 560SFP+ Adapter - NIC (PXE IPv4)	PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(48df37xxxxxx,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0011* Slot 1 Port 1 : HP Ethernet 10Gb 2-port 560SFP+ Adapter - NIC (PXE IPv6)	PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(48df37xxxxxx,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0012* debian-13 (YAMP)	HD(1,GPT,380b4edf-92c7-46b8-b4f9-795c4c2a1279,0x800,0x5f000)/File(\EFI\debian\shimx64.efi)

Through BootCurrent the currently booted entry is visible (in this case the operating system).

BootOrder lists the configured boot order from left to right.

The command efibootmgr -n 0011,0010,000F,0012 corrects above boot order, so that PXE and USB are tried before the operating system.

Boot entry incorrect

If the entry in UEFI for a given operating system is incorrect, our Rescue can be used to rewrite it.

First the EFI partition has to be mounted, e.g. via:

mkdir /mnt/efi
mount "$(fdisk -l | grep EFI | awk '{print $1}')" /mnt/efi/

Following commands display the UUID of the EFI partition and available bootloaders:

blkid "$(fdisk -l | grep EFI | awk '{print $1}')"
find /mnt/efi/ -type f -iname '*.efi'

Example output:

/dev/sda1: UUID="CD79-BCE2" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6c57f5db-fc23-477b-8179-2d947ab21aa2"
/mnt/efi/EFI/debian/shimx64.efi
/mnt/efi/EFI/debian/grubx64.efi
/mnt/efi/EFI/debian/mmx64.efi
/mnt/efi/EFI/debian/fbx64.efi
/mnt/efi/EFI/BOOT/BOOTX64.EFI
/mnt/efi/EFI/BOOT/grubx64.efi
/mnt/efi/EFI/BOOT/mmx64.efi
/mnt/efi/EFI/BOOT/fbx64.efi

/dev/sda is the blockdevice in this case and 6c57f5db-fc23-477b-8179-2d947ab21aa2 the UUID of the EFI partition.

The operating system privides as choosable bootloaders: EFI/debian/shimx64.efi and EFI/debian/grubx64.efi.

BOOTX64.EFI and fbx64.efi/fallback.efi lead to issues with the boot order and should not be used.

The output of efibootmgr -v | grep 'HD(' has to be compared to potentially existing entries, e.g.:

Boot0012* debian-13 HD(1,GPT,fdff4d61-2c9c-442d-8b4c-354dff375f44,0x800,0x5f000)/File(\EFI\debian\shimx64.efi)

In this example the UUID of the entry is wrong - it should point to the EFI partition.

As a result a new entry has to be created before deleting the old one, e.g.:

efibootmgr --create-only -d /dev/sda -p 1 -L "debian-13" -l '\\EFI\\debian\\shimx64.efi'

• -d the blockdevice of the EFI partition
• -p the partiton number of the EFI partition (derived from /dev/sda1)
• -L is the display name of the entry
• -l refers to chosen bootloader in the EFI partition using EFI pathspec (hence the use of backslashes)

Next the old entry can be deleted, e.g.:

efibootmgr -B -b 0012

And last but not least, the new entry must be added to the boot order, see boot order fixing.