Introduction
With PXE, an edge device installs EVE-OS over the network instead of from a USB drive. This is useful for testing EVE-OS versions, where devices are reinstalled regularly, and for deploying EVE-OS to many edge devices.
This article describes how to build a PXE server with Docker Compose, prepare the edge device, and take it from installation to onboarding in Edge Infrastructure Services. It covers an HTTP setup and a TFTP-only setup for EVE-OS 13.4.0 and greater on 64-bit x86 (amd64) edge devices with UEFI. For supported versions prior to 13.4.0, see Install EVE-OS 6.3.0 to 13.3.0 over the Network with PXE.
Overview
The PXE server needs:
- Three services: DHCP (Kea), TFTP (tftp-hpa), and HTTP (NGINX), running as Docker containers.
- Five files from the EVE-OS netboot archive, copied to the folders of the TFTP server and the web server.
- One change: a single line added to
ipxe.efi.cfg.
The edge device requests the files in this order:
Edge device │ ├─ PXE DHCP ──► IP address, next-server, boot file ipxe.efi │ TFTP ──► ipxe.efi │ ├─ iPXE DHCP ──► IP address, next-server, boot file ipxe.efi │ TFTP ──► ipxe.efi.cfg │ HTTP ──► EFI/BOOT/BOOTX64.EFI │ ├─ GRUB HTTP ──► EFI/BOOT/grub.cfg │ HTTP ──► installer.iso ▼ Installer runs
GRUB reads its configuration file from the same location that delivered the GRUB binary. Because ipxe.efi.cfg points at the web server, GRUB retrieves both EFI/BOOT/grub.cfg and installer.iso over HTTP. Place grub.cfg next to BOOTX64.EFI in the web root, not in the TFTP root.
Important: All installation settings come from
installer.iso. Installation settings inipxe.efi.cfghave no effect.
The installation starts in one of two ways:
-
From the network: the PXE client in the device firmware loads
ipxe.efifrom the PXE server. -
From the hard disk: the device firmware starts
ipxe.efidirectly from the hard disk, and the PXE block is skipped.
From iPXE onward, both ways are identical and lead to the same installation result.
Prerequisites
- A PXE server host running Linux, with a static IP address on the imaging network. The containers use
network_mode: host, which gives them direct access to the host network. DHCP and TFTP require this. Docker Desktop on macOS and Windows does not provide host networking in the same way, so it is not suitable for the PXE server. - Docker Engine 23.0 or greater on the PXE server host. That release and later include Docker Compose v2, which provides the
docker composecommand used in this article. The olderdocker-composecommand with a hyphen is Compose v1 and is not supported. - Network access from the PXE server host to Docker Hub, to download the EVE-OS netboot archive and the container images.
- The EVE-OS version to install, 13.4.0 or greater.
- An edge device with 64-bit x86 (amd64) architecture and UEFI firmware, able to boot from the network or from a hard disk that you can write to.
Values Used in the Examples
All examples in this article use the following values. Replace them with the values of your network.
| Value | Example | Used in |
|---|---|---|
| IP address of the PXE server | 192.168.104.2 |
ipxe.efi.cfg, kea-dhcp4.conf
|
| Network interface of the PXE server | ens19 |
kea-dhcp4.conf |
| Subnet | 192.168.104.0/24 |
kea-dhcp4.conf |
| Address pool for edge devices | 192.168.104.100 - 192.168.104.200 |
kea-dhcp4.conf |
| Default gateway | 192.168.104.1 |
kea-dhcp4.conf |
| DNS server | 8.8.8.8 |
kea-dhcp4.conf |
| EVE-OS version | 17.0.0-lts-kvm-amd64 |
Download command |
Decisions Before the Setup
Boot order: testing or fleet deployment
Testing and lab use: network boot first
In the BIOS/UEFI setup of the edge device, set the boot order to network boot (PXE) first and hard disk second.
Every boot │ ├─ PXE server answers ──────────► Reinstall EVE-OS │ └─ PXE server does not answer ──► Boot EVE-OS from hard disk
To install a different EVE-OS version, replace the files on the PXE server and restart the edge device.
Warning: Every edge device that starts from the network (PXE) while the PXE server answers is reinstalled. Its hard disk is overwritten.
Fleet deployment: iPXE on the hard disk, BIOS/UEFI locked
Use this approach when you deploy EVE-OS to many edge devices and each device must be installed one time only.
With this approach:
- Installation happens one time only. The edge device starts from the hard disk. iPXE on the hard disk performs the network installation, and EVE-OS replaces iPXE. After installation, nothing on the device can start a network installation again.
- No work on the device after installation. The BIOS/UEFI is set and locked before installation, so nobody has to return to the device to change the boot order.
- Measured boot stays valid. UEFI measures the list of boot options into TPM register PCR-1, and EVE-OS uses PCR-1 to seal the vault key. Because the BIOS/UEFI settings do not change after installation, the recorded value stays the same and the vault continues to unseal.
- Write
ipxe.efito the hard disk of the edge device. - In the BIOS/UEFI setup of the edge device, set the boot order to hard disk first.
- Lock the BIOS/UEFI setup, for example with an administrator password.
First start Every later start
┌──────────────────┐ ┌──────────────────┐
│ Hard disk: │ │ Hard disk: │
│ - ipxe.efi │ │ - EVE-OS │
└────────┬─────────┘ └────────┬─────────┘
▼ ▼
iPXE installs EVE-OS EVE-OS starts from
and replaces itself the hard disk
Warning: Changing the boot order after the edge node is onboarded changes the PCR-1 value. EVE-OS is then unable to unseal the vault key, and the edge node logs a policy check failure naming PCR index 1. Attaching any device that carries a bootable partition, most often a USB drive, has the same effect.
Network: single network or staged
Single network: installation and onboarding on one network
┌──────────────────┐ ┌──────────────────┐
│ PXE server │ │ Edge Infra- │
│ DHCP, TFTP, HTTP │ │ structure Svcs │
└────────┬─────────┘ └────────┬─────────┘
│ ┌───────┴────────┐
│ │ Router / WAN │
│ └───────┬────────┘
│ ┌──────────────┐ │
└──────┤ Switch ├───────────┘
└──────┬───────┘
│
┌───────┴────────┐
│ Edge device │
└────────────────┘
In this case the DHCP server (Kea) must provide a default gateway and DNS servers. Refer to the "DHCP: Kea configuration" section.
Staged: separate imaging and onboarding networks
The edge device installs on an isolated imaging network and onboards on a separate onboarding network.
┌──────────────────┐
│ PXE server │
│ DHCP, TFTP, HTTP │
└────────┬─────────┘
│
┌──────┴───────┐
│ Switch │
└──────┬───────┘
│
┌────────┴─────────┐
│ Edge device │
└──────────────────┘
The onboarding network must provide a DHCP server with a default gateway and DNS servers, and a route to your Edge Infrastructure Services cluster.
HTTP Setup
Services: Docker Compose with Kea, tftp-hpa, and NGINX
- Create the following directory structure.
pxe/
├── docker-compose.yml
├── artifacts/ extracted netboot archive
├── kea/
│ ├── etc/kea-dhcp4.conf Kea configuration
│ ├── lib/ Kea lease database
│ └── log/ Kea logs
├── nginx/
│ ├── conf.d/custom.conf NGINX configuration
│ ├── www/ files delivered over HTTP
│ └── log/ NGINX logs
└── tftp/
├── tftpboot/ files delivered over TFTP
└── log/ TFTP logs
- Create
docker-compose.ymlin thepxefolder.
services:
kea:
image: docker.cloudsmith.io/isc/docker/kea-dhcp4
network_mode: host
volumes:
- ./kea/etc:/etc/kea
- ./kea/lib:/var/lib/kea
- ./kea/log:/var/log/kea
restart: unless-stopped
tftp:
image: alpine:3.22
network_mode: host
command: >
sh -c "apk add --no-cache tftp-hpa &&
{ syslogd -n -O /var/log/tftp/tftpd.log & } &&
exec in.tftpd -L -vvv --secure --user root /tftpboot"
volumes:
- ./tftp/tftpboot:/tftpboot:ro
- ./tftp/log:/var/log/tftp
restart: unless-stopped
nginx:
image: nginx:alpine
network_mode: host
command: ["nginx-debug", "-g", "daemon off;"]
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d:ro
- ./nginx/www:/usr/share/nginx/html:ro
- ./nginx/log:/var/log/nginx
restart: unless-stopped
The file deliberately has no version key. Compose v2 treats that key as obsolete and prints a warning if it is present.
Note: The tftp container installs
tftp-hpafrom the Alpine package repository at every start. The PXE server host needs internet access whenever this container starts.To avoid that, build a container image with the package already installed.
FROM alpine:3.22 RUN apk add --no-cache tftp-hpa CMD ["sh", "-c", "syslogd -O /var/log/tftp/tftpd.log && exec in.tftpd -L -vvv --secure --user root /tftpboot"]Build it, for example with
docker build -t local/tftp:3.22 ., then reference it indocker-compose.yml.tftp: image: local/tftp:3.22 network_mode: host volumes: - ./tftp/tftpboot:/tftpboot:ro - ./tftp/log:/var/log/tftp restart: unless-stopped
- Create
nginx/conf.d/custom.conf.
server {
listen 80;
root /usr/share/nginx/html;
autoindex on;
}
Files: netboot archive, placement, and changes
- Change to the
artifactsfolder, then download and extract the netboot archive.
cd pxe/artifacts docker run --rm lfedge/eve:17.0.0-lts-kvm-amd64 installer_net | tar -xf -
- Optional: to use a customized
installer.iso, refer to Build a custom EVE-OS image. - Copy the files to the following folders.
Archive content Copy to ────────────────────── ────────────────────────────────────────── ./ipxe.efi ───► ./tftp/tftpboot/ipxe.efi ./ipxe.efi.cfg ───► ./tftp/tftpboot/ipxe.efi.cfg (one line added) ./EFI/BOOT/BOOTX64.EFI ─► ./nginx/www/EFI/BOOT/BOOTX64.EFI ./EFI/BOOT/grub.cfg ───► ./nginx/www/EFI/BOOT/grub.cfg ./installer.iso ───► ./nginx/www/installer.iso
grub.cfg belongs in the web root next to BOOTX64.EFI, because GRUB reads its configuration file from the same location that delivered the GRUB binary. A copy in the TFTP root is never read.
If you use a customized installer.iso, copy it to ./nginx/www/installer.iso instead.
- Add one line to
./tftp/tftpboot/ipxe.efi.cfg. Find the following commented line near the top of the file.
#set url https://foo.bar/
Add a set url line beneath it that points to your web server. The trailing slash is required, because iPXE appends EFI/BOOT/BOOTX64.EFI directly to this value.
set url http://192.168.104.2/
Leave the rest of the file unchanged. The set console, set eve_args, and set installer_args lines have no effect in EVE-OS 13.4.0 and greater, but the iseq ${buildarch} lines that follow them select the bootloader for your architecture and must stay in place.
EFI/BOOT/grub.cfg needs no changes. GRUB was chainloaded over HTTP, so it already reads installer.iso over HTTP from the web root.
DHCP: Kea configuration
Create kea/etc/kea-dhcp4.conf.
{
"Dhcp4": {
"interfaces-config": {
"interfaces": ["ens19"]
},
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/var/lib/kea/kea-leases4.csv"
},
"subnet4": [
{
"id": 1,
"subnet": "192.168.104.0/24",
"pools": [
{ "pool": "192.168.104.100 - 192.168.104.200" }
],
"next-server": "192.168.104.2",
"boot-file-name": "ipxe.efi",
"option-data": [
{ "name": "routers", "data": "192.168.104.1" },
{ "name": "domain-name-servers", "data": "8.8.8.8" }
]
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "DEBUG",
"debuglevel": 99,
"output_options": [
{ "output": "/var/log/kea/kea-dhcp4.log", "flush": true }
]
}
]
}
}
| Setting | Effect |
|---|---|
next-server, boot-file-name
|
The edge device starts the network installation. |
routers |
The edge device has a default gateway to Edge Infrastructure Services. |
domain-name-servers |
The edge device can resolve the hostname of your Edge Infrastructure Services cluster. |
Serve ipxe.efi as the boot file name on every DHCP request, including the request that iPXE itself sends after it starts. iPXE appends .cfg to this name to build the path to its own configuration file.
To install only specific edge devices, remove next-server and boot-file-name from the subnet and add a reservation per edge device:
"reservations": [
{
"hw-address": "bc:24:11:52:78:98",
"ip-address": "192.168.104.71",
"next-server": "192.168.104.2",
"boot-file-name": "ipxe.efi"
}
]
Start and install
- In the
pxefolder, start the containers.
docker compose up -d docker compose ps
- Start the edge device. The console shows output similar to the following:
>>Start PXE over IPv4. Station IP address is 192.168.104.71 Server IP address is 192.168.104.2 NBP filename is ipxe.efi Downloading NBP file... NBP file downloaded successfully. iPXE 1.0.0+ -- Open Source Network Boot Firmware -- https://ipxe.org Configuring (net0 bc:24:11:52:78:98)...... ok tftp://192.168.104.2/ipxe.efi.cfg... ok http://192.168.104.2/EFI/BOOT/BOOTX64.EFI... ok Downloading installer. This may take some time. Please wait patiently. *Boot 17.0.0-lts-kvm-amd64-installer
Note: On EVE-OS 13.4, 14.5, 16.0, and 17.0 LTS releases, including 17.0.0-lts, the edge device downloads
installer.isotwice. This roughly doubles the installation time and the network traffic. The installation still succeeds. The issue is fixed in EVE-OS 17.4.0.
Troubleshooting
| Symptom | Cause |
|---|---|
The boot stops after the BOOTX64.EFI line |
GRUB cannot find EFI/BOOT/grub.cfg. Confirm the file is in ./nginx/www/EFI/BOOT/, not only in the TFTP root. |
The boot stops after ipxe.efi.cfg loads |
The set url line is missing or has no trailing slash. |
| The device never requests a boot file |
next-server or boot-file-name is missing from the Kea configuration, or the reservation does not match the device MAC address. |
For more diagnostics, refer to Troubleshoot EVE-OS Installation.
Onboarding
- Single network: after the installation, EVE-OS starts from the hard disk and connects through the router to Edge Infrastructure Services.
- Staged: after the installation, switch off the edge device, connect it to the onboarding network, and switch it on. EVE-OS starts from the hard disk and connects to Edge Infrastructure Services.
To register the edge node, refer to Onboard an Edge Node.
TFTP-Only Setup
The TFTP-only setup uses the HTTP setup without NGINX. Every file is delivered over TFTP, and no file needs to be edited.
Warning: ZEDEDA does not recommend this setup. GRUB's TFTP transport cannot seek within a file, so every backward seek restarts the transfer from the first byte. For an installer ISO image of approximately 500 MB, that means several complete transfers before the kernel starts. GRUB's HTTP transport requests only the byte ranges it reads. Use the HTTP setup unless you cannot run a web server on the imaging network.
- Remove the
nginxservice fromdocker-compose.yml.
services:
kea:
image: docker.cloudsmith.io/isc/docker/kea-dhcp4
network_mode: host
volumes:
- ./kea/etc:/etc/kea
- ./kea/lib:/var/lib/kea
- ./kea/log:/var/log/kea
restart: unless-stopped
tftp:
image: alpine:3.22
network_mode: host
command: >
sh -c "apk add --no-cache tftp-hpa &&
{ syslogd -n -O /var/log/tftp/tftpd.log & } &&
exec in.tftpd -L -vvv --secure --user root /tftpboot"
volumes:
- ./tftp/tftpboot:/tftpboot:ro
- ./tftp/log:/var/log/tftp
restart: unless-stopped
Note: The tftp container installs
tftp-hpaat every start, as described in the "Services" section. The Dockerfile workaround described there applies here too.
- Copy all files unchanged to
./tftp/tftpboot/.
Archive content Copy to ────────────────────── ───────────────────────────────────── ./ipxe.efi ───► ./tftp/tftpboot/ipxe.efi ./ipxe.efi.cfg ───► ./tftp/tftpboot/ipxe.efi.cfg ./EFI/BOOT/BOOTX64.EFI ─► ./tftp/tftpboot/EFI/BOOT/BOOTX64.EFI ./EFI/BOOT/grub.cfg ───► ./tftp/tftpboot/EFI/BOOT/grub.cfg ./installer.iso ───► ./tftp/tftpboot/installer.iso
No set url line is needed. With that value empty, iPXE resolves EFI/BOOT/BOOTX64.EFI relative to the TFTP server it loaded its configuration file from, and GRUB then reads grub.cfg and installer.iso from the same TFTP root.
If you use a customized installer.iso, copy it to ./tftp/tftpboot/installer.iso instead.