Network Connectivity Details

Introduction

EVE-OS needs to always be able to connect to the controller, yet the configuration of the network ports might be both complex and changing over time.

In addition, some network ports might be designated for application usage, or the underlying I/O bus controllers (for example, PCI controllers) be designated for assignment to applications. That part of the configuration can also change at run-time.

Any network port configuration changes that might affect the connectivity to the controller require care to avoid permanently losing connectivity to the controller and become unmanageable as a result. Thus, it is required to perform testing as part of a configuration change and have a way to fall back to a working port configuration. This is accomplished by logic to test connectivity to the controller, and persistently maintaining a list of network configurations, including the latest and possibly one or more fallback configurations.

Network Implementation

EVE-OS networking is built on top of the Linux network stack, further integrated with some open-source tools. EVE-OS combines standard Linux network features for the data-plane, such as IP routes, IP rules, Linux bridge, iptables, and more. However, the way these tools are assembled to implement the EVE-OS network model and to address unique challenges of the edge, is in some instances somewhat unconventional, separating EVE-OS from the more traditional Linux distributions, and could be confusing even for experienced Linux users, for example:

• Main IP routing table (ip route show table main) is not used, instead every (IP-enabled) network adapter and every local network instance has its own routing table, which is selected using IP rules by the source IP address of a given packet (see ip rule list).
• (L3-enabled) Network adapter for ethernet port is implemented as a bridge that captures the port and takes its interface name and MAC address - this way we can then use the bridge as an IP endpoint and at the same time put application interfaces under this bridge to implement (single-port) switch network instance
• If flow logging is enabled, ACLs are implemented by iptables mangle table for the most part (not filter). Traffic flow that should not be allowed is marked with a "DROP mark" and routed into a dummy blackhole interface. The purpose of this is to ensure that conntrack entries are created even for dropped flows, allowing metadata collection for all application network flows, including those denied by ACLs.
• By default, DHCP servers of local network instances grant leases to applications with /32 IP addresses, even if the actual subnet is larger and with more than just one IP. This is accompanied by a link-local route for the actual subnet and is used to force routing even in situations where traffic could be just forwarded. EVE-OS does this for the ACL and flow logging purposes.
• dhcpcd is not only used to get IP address for a network adapter via DHCP, but also to assign user-configured static IP
• dnsmasq is used not only as a DNS server for local network instances, but also as a DHCP server

The process of configuring the data-plane and all its components is split between 4 EVE-OS microservices:

• NIM takes care of everything related to device connectivity (network adapters, dhcpcd, network config testing & fallback procedure)
• zedrouter deploys network instances and ensures connectivity for applications
• wwan runs ModemManager and EVE's mmagent wrapper to establish and manage cellular connectivity
• wlan runs wpa_supplicant to perform the WPA authentication for WiFi connectivity

The following diagram depicts all the components of the data-plane and how they all fit together. Also shown is the separation of the management plane between EVE-OS microservices.

Physical network ports

EVE-OS supports ethernet NICs, WiFi modules and cellular modems.

Physical network IO devices are listed as part of the device model. This should include all devices that will be used by EVE-OS (for management) or by applications (directly or shared). IO device is referenced either using a physical address that it was given on the bus through which it is connected (PCI or USB), or via the name assigned by the Linux kernel to the network interface representing the device on the data-plane. The latter approach is potentially not stable and therefore not recommended. Interface name can change after a reboot if multiple devices of the same type are initialized in a different order. This is quite common with cellular modems or with ethernet NICs operated by different drivers loaded at random order.

The prerequisite for using a given network IO device inside the EVE-OS domain (i.e. not directly assigned to an application), is that the Linux kernel used by EVE-OS is already shipped with a suitable (open-source) driver. It may be necessary to adjust the kernel options to enable building of the driver. For cellular modems this is not required, however. Cellular modems are fairly complex self-contained units controlled through AT commands and/or API calls, for which drivers and libraries are already shipped with EVE-OS and automatically loaded on boot. For more detailed information on cellular connectivity, please refer to WIRELESS.md

HW-virtualized IO

EVE-OS supports Single Root I/O Virtualization (SR-IOV) of a network interface card. SR-IOV is a virtualization technology that allows one physical device, in this case NIC, to be used by multiple virtual machines simultaneously without the mediation from the hypervisor. The actual physical device is known as a Physical Function (PF) while the others are known as Virtual Functions (VF). With this feature, the hypervisor can attach one or more VFs to a Virtual Machine so that the guest can then use the device as if it was directly assigned.

To use this feature with EVE, first you need to check that (PF) driver for your SR-IOV-enabled NIC is already shipped with EVE-OS OS. With proprietary drivers this is going to be a problem. Until we add a feature to deploy proprietary drivers for the host inside a VM just like applications (feature proposal known as "driver domain"), EVE-OS users are limited to open-source drivers already available with the Linux kernel. Please beware that so far only Intel I350 NIC with igb/igbvf drivers was tested and verified to work with EVE.

As a next step, you need to decide on the number of VFs to be created for a given SR-IOV NIC. This is a part of the device model and cannot be changed in runtime.

Finally, to use a VF with an application, use the option of the direct assignment. VF configuration allows you to either specify a custom MAC address or let the NIC generate one for you. Additionally, you may configure VF to be used as an access port for a given VLAN.

SystemAdapter

SystemAdapter (term used in EVE-OS API) assigns network config (DHCP, IP, DNS and NTP config) together with some logical attributes such as "usage", cost and shared labels to a physical network port or to a logical lower-layer network adapter (VLAN, LAG).

SystemAdapter can be used to elevate physical/L2-only adapter to an IP-ready network endpoint. SystemAdapter may either instruct EVE-OS to run DHCP client for the network interface or to assign IP config statically. However, even L2-only interface (to be used with switch NI), still needs SystemAdapter attached for it to be referenceable by higher-level objects like network instances.

SystemAdapter determines the usage of the network port, which is one of:

• for management and application traffic
• for application traffic only
• unused

Network port directly assigned to an application is without SystemAdapter.

Network Adapter MTU

The user can adjust the Maximum Transmission Unit (MTU) size of a network adapter. MTU determines the largest IP packet that the underlying link can and is allowed to carry. A smaller MTU value is often used to avoid packet fragmentation when some form of packet encapsulation is being applied, while a larger MTU reduces the overhead associated with packet headers, improves network efficiency, and increases throughput by allowing more data to be transmitted in each packet (known as a jumbo frame).

EVE-OS uses the L3 MTU, meaning the value does not include the L2 header size (e.g., Ethernet header or VLAN tag size). The value is a 16-bit unsigned integer, representing the MTU size in bytes. The minimum accepted value for the MTU is 1280, which is the minimum link MTU needed to carry an IPv6 packet (see RFC 8200, "IPv6 minimum link MTU"). If the MTU for a network adapter is not defined (zero value), EVE-OS will set the default MTU size, which depends on the network adapter type. Ethernet and WiFi adapters default to 1500 bytes, while cellular modems typically receive their MTU value from the network provider, which EVE-OS will use unless the user overrides the MTU value.

Load Spreading and Failover

Load spreading means that there are two or more similar uplink networks, for instance two Ethernet ports for redundancy, the EVE-OS will send different requests over different connections in a round-robin fashion.

Failover means that a device can have different types of uplink networks, for example Ethernet, LTE, and/or satellite connectivity, and switch from one to another to maintain controller connectivity. In such case, a cost (field cost of the SystemAdapter message) can be assigned to each uplink port so that e.g., LTE (the wwan0 port) is only used if EVE-OS can not connect via any lower cost (e.g., one or more Ethernet) ports.

The cost is a number between 0 and 255, with zero (the default) implying free, and less preferred ports being assigned a higher cost. Multiple ports can be assigned the same cost, in which case once failover has happened to cost N, then all uplink ports with cost N will be used for load spreading of the management traffic.

Furthermore, one can set network.download.max.cost to a number N if it is acceptable for EVE-OS to perform (potentially large) downloads of content and images when having failed over to uplink ports with cost N (or less).

Network Proxies

Organizations frequently utilize network proxies to regulate and supervise internet access, apply content filtering, and enforce security policies. Depending on the proxy type, adjustments to the edge-node network configuration may be necessary to ensure connectivity with the cloud.

First of all, before configuring a device, the user must find out if there is a network proxy in the network and understand how it is configured. Configuration will then depend on the type of the proxy, see below.

Transparent network proxy

A network proxy may operate transparently, meaning that clients (EVE-OS devices, user apps) making HTTP/HTTPS/… requests are not aware of its presence. When a client makes an HTTP(S) request, it addresses it directly to the destination (e.g. EVE-OS controller), but a transparent proxy will capture it (without the client knowing about it) and apply a proxying action. The action may depend on the destination URL. For example, the proxy may be configured to drop requests heading to some specific domains, but allow others.

Additionally, the proxy may break the TLS tunnel into two (client-proxy, proxy-destination) and act as a Man-in-the-Middle (MITM), reading the otherwise encrypted traffic. This is done by proxy generating server certificate for the destination on the fly and presenting it to the client during the TLS handshake. Normally, the client would detect this because it would not trust such a certificate, after all it is signed by proxy CA which is not in the default set of trusted certificates installed in EVE/app. However, if MITM proxying is desired (e.g. when company policy requires that all traffic, including encrypted, is inspected), then the device has to be told to trust the proxy CA certificate. This is why EVE-OS allows the user to insert a proxy CA certificate as part of the transparent proxy config. Since everything else is transparent, there are no additional config options.

Please note that EVE-OS logs info about the certificate presented by "the other side", which in case of MITM proxy would be the proxy and not the controller:

// In client.go:

log.Noticef("Peer certificate:(%d) Issuer: %s, Subject: %s, NotAfter: %v",

    i, cert.Issuer, cert.Subject, cert.NotAfter)

Information about TLS tunnels opened and peer certificates presented is also part of the network tracing.

Explicit network proxy

Presence of explicit proxy is known to the client making HTTP/HTTPS/… requests. The client will explicitly ask the proxy using the standard HTTP CONNECT method to make an HTTP(S) request on its behalf to a specified URL. In this case, the client is configured with a proxy address for every protocol that should be proxied. For EVE-OS management traffic, there should be a proxy address at least for HTTPS proxying.

The proxy address is entered either manually in the form [<proto>://]<host>:[<port>] (default proto is HTTP, default port is 80), or a proxy auto-discovery protocol is enabled to determine the proxy address automatically.

Beware not to confuse "protocol that is being proxied" with "protocol on which proxy listens". The latter is typically HTTP (even for HTTPS proxying).

Explicit network proxy may be configured to proxy only certain destinations, while others should be accessed directly (otherwise the request would fail). EVE-OS allows to configure the list of domains to be accessed without proxying.

Additionally to proxy addresses, it may be necessary to insert a proxy CA certificate. Just like the transparent proxy, an explicit proxy may also break the TLS tunnel and act as MITM.

Network proxy Auto-discovery

Auto-discovered proxy is effectively an explicit proxy, but instead of user configuring the proxy address for every client in the network, DHCP or DNS in combination with WPAD and PAC protocols are used by the client to discover the proxy address (for a given request URL).

Depending on how explicit vs. automatic this process is, EVE-OS offer 3 subtypes:

1. Auto-proxy discovery is fully automatic: DHCP/DNS is used to find address of WPAD server, which returns PAC file, which contains JavaScript function that returns proxy address for the given request URL
2. URL: the WPAD server address is configured by the user, the rest works as in the first case
3. PAC: user writes a PAC file by himself. Device will use the JavaScript function from there to get the proxy address, meaning there is no communication on the network to get the proxy address.

More info on how this auto-discovery works can be found on wiki:

• WPAD
• PAC

Steps to configure network proxy

To summarize, EVE-OS user should follow these steps:

1. Get information about their network proxy type/configuration (e.g. from network admin)
2. Determine if proxy operates explicitly or transparently
3. Determine if proxy auto-discovery is being used and in what scope
4. For explicit proxy, determine if there are directly accessible destinations that should not be proxied
5. Determine if proxy is expected to inspect encrypted traffic inside TLS tunnels
6. Configure the proxy accordingly for the EVE-OS management traffic (inside the network config attached to a given network port)

Note that proxying of application traffic has to be configured on the application side.

Sources of configuration

There are several sources from which nim gets the potential port configurations. Those all use the DevicePortConfig type (Go struct marshalled with JSON). There are examples of such configurations in legacy EVE-OS configuration. The value of the Key field inside DevicePortConfig reflects the source of the configuration. Different sources — such as bootstrap files, controller updates, or manual overrides — use distinct key values to identify their origin.

Fresh install

When EVE-OS is installed using a generic EVE-OS installer without device bootstrap configuration included (see CONFIG.md, section "Bootstrap configuration"), it only has the last-resort configuration available (see "Last resort" section below). This is sufficient to connect to the controller if

• DHCP is enabled on one of the uplink Ethernet ports
• WiFi is not assumed (since WiFi needs credentials)
• Cellular connectivity is not assumed (without cellular config every discovered modem will have RF function disabled)
• No enterprise proxy configuration is required to be able to connect to the controller.

If any of those assumptions is not satisfied, then it is necessary to either install EVE-OS using a single-use EVE-OS installer, shipped with the initial "bootstrap" configuration prepared for the target device (the preferred method) or to use one of the legacy mechanisms for off-line configuration management.

Bootstrap configuration

The idea behind bootstrap config is to generalize the issue of configuration bootstrapping beyond just network configuration and also to reuse the existing proto models which are already being used between the device and the controller in the on-line mode. This means that instead of users directly editing an internal structure DevicePortConfig with network configuration, it is much safer and robust to simply have the controller exporting the prepared device configuration (possibly not only for networking) into a file and baking it into the EVE-OS installation medium, avoiding any error-prone manual edits from the user in the process.

In terms of configuration handling, much of the code-path is reused between the bootstrap and a config coming (on-line) from a controller. Just like in the latter case, the networking portion of the device configuration is parsed and published to nim by zedagent using DevicePortConfig type, with key bootstrap.

This feature is described in much more detail in CONFIG.md. Even though it has some limitations in the present state, it is already the preferred method to use. Below we describe a legacy "override" method for the completeness sake, which is still supported by EVE-OS for backward-compatibility reasons.

(Legacy) Override the configuration using a USB stick

If the deployment site requires use of HTTP enterprise proxies and/or static IP configuration, then a file containing JSON-formatted content of DevicePortConfig, aka "override network config", can be placed on either the EVE-installer or onto a separate USB stick (typically identified with the key override, but this is not enforced). The latter case allows to inject network config even in run-time. However, for an already onboarded device it is required that the USB controller is enabled using debug.enable.usb as specified in configuration properties

There are examples of such configurations in legacy EVE-OS configuration

Such a file will be used by nim until it can connect to the controller and receive the configuration (either the same, or subsequent updates). Thus, for a new device using enterprise proxies and/or static IP it is imperative that the configuration first be set in the controller, then a USB stick be created with that configuration, and the device booted with that USB stick in place. That ensures that the configuration doesn't revert back once the device has connected to the controller.

From the controller

The SystemAdapter in the API specifies the intended port configuration. This is fed into the logic in nim by zedagent publishing a DevicePortConfig item with key zedagent.

At least one port must be set to be a management port, and that port needs to refer to a network with IP configuration for the device to even try to use the SystemAdapter configuration.

Last resort

If network.fallback.any.eth configuration property is set to enabled (by default it is disabled), then there is an additional lowest priority item in the list of DevicePortConfigs, called "Last resort" DPC (key lastresort), based on finding all Ethernet interfaces (i.e. excluding wireless connectivity options) which are not used exclusively by applications. The last resort configuration assumes DHCP and no enterprise proxies.

However, independent of the above property setting, if the device has no source of network configuration available (no bootstrap, override or persisted config), then EVE-OS will use the Last resort forcefully. This, for example, happens when device boots for the very first time, without bootstrap config or the (legacy) network config override being provided. In that case, device may remain stuck with no connectivity indefinitely (unless the user plugs in a USB stick with a network config later), therefore EVE-OS will try to use Last resort to see if it can provide connectivity with the controller. Once device obtains a proper network config (from the controller or a USB stick), EVE-OS will stop using Last resort forcefully and will keep this network config inside DevicePortConfigList only if and as long as it is enabled explicitly by the config (network.fallback.any.eth is enabled).

List of persisted network configurations

EVE-OS must persist device network configuration, referred to as DevicePortConfig (DPC), to ensure it can regain controller connectivity after a reboot. To support fallback scenarios, EVE-OS may retain not only the latest configuration but also some older configurations that were known to work.

The full list of persisted network configurations is maintained in: /persist/status/nim/DevicePortConfigList/global.json

This list is referred to as the DevicePortConfigList (DPCL). DevicePortConfigList includes a field CurrentIndex, pointing to the currently applied DPC.

Initially, the DevicePortConfigList is empty, and CurrentIndex is set to -1, indicating that no configuration is currently selected.CurrentIndex can also be -1 when none of the available configurations are testable. This can happen if a configuration lacks a valid management port, was very recently tested and failed, or is otherwise unusable.

Network config priority and order

Network configs are sorted in priority-decreasing order, i.e. the highest-prio config is at the index 0. Priority is assigned to DPC using a TimePriority timestamp field — newer timestamps indicate higher priority (basically newer network config is preferred over an older one).

Depending on the source of the configuration, the timestamp is assigned differently:

• Configuration from the controller: Uses the EdgeDevConfig.config_timestamp specified by the controller. Older EVE-OS versions that do not support this field simply set DPC timestamp to the time when the config was received.
• Bootstrap config: the timestamp should also be provided via config_timestamp by the controller.
• Legacy override config file: must contain a manually set TimePriority (prone to user error).
• Last-resort config: assigned the lowest possible timestamp (Jan 1, 1970), effectively the lowest priority.

Network config retention policy

EVE-OS does not retain all network configurations indefinitely in the DevicePortConfigList (DPCL). This is by design, for the following reasons:

• To avoid retrying obsolete configurations: Old and repeatedly failing configurations should not be retried endlessly, as they are unlikely to restore controller connectivity. Moreover, retrying such obsolete configs adds unnecessary delay to the network testing process, during which the device may remain unreachable on the network.
• To stay within size limits: The DPCL is persisted and published via PubSub, which has a maximum message size limit of 64 KB. Retaining too many entries can cause this limit to be exceeded, leading to fatal errors.

To balance reliability with resource constraints, the following entries are retained in the DPCL:

• The latest received network configuration (always at index 0)
• The currently used DPC, as identified by the CurrentIndex
• The most recent DPC with working controller connectivity (if any)
• The most recent DPC from the controller (key zedagent)
• If there is no DPC from the controller (key zedagent), or if none of the controller-provided DPCs have ever succeeded in a connectivity test, retain the most recent DPC from each source to ensure fallback options remain available
• A manually injected DPC (via TUI) — identified by the key manual
• The last resort configuration (key lastresort), if network.fallback.any.eth is enabled

All other entries are pruned during a compression process, triggered on any relevant DPCL change.

Testing device connectivity

The Network Interface Manager (specifically its component DPCManager) performs two types of testing:

• Validate that a new configuration is working before replacing the current configuration
• Periodically check that the existing configuration is working
• If an older "fallback" config is being used, periodically retest the latest network config to see if the issue that triggered fallback was fixed and device can therefore apply the latest config

Testing a new configuration from the controller

The testing is triggered by receiving a new configuration from the controller (or from a different source) and completes when at least one of the management ports can be used to reach the controller. If there are multiple management ports in the configuration, there might be an error reported for ports which are not working (depending on the order in which the ports are tried).

[TBD Should we verify that the new configuration is usable for some minimum time, e.g. 10 minutes, before discarding the previous/fallback configuration?]

If no management port can be used to reach the controller, then nim switches to using the next configuration in the DevicePortConfigList, which is normally the previously used configuration. In that case a failure is reported in the SystemAdapterInfo by setting lastError in DevicePortConfig and the currentIndex is set to the currently used DevicePortConfig in the list (> 0 when fallback is triggered). Note that lastFailed and lastSucceeded can be used to see if a configuration has succeeded in the past or always failed.

Periodic testing

The default timer for this is 5 minutes and can be set with timer.port.testinterval. At those intervals the device verifies that it can still reach the controller using one of the management ports.

Each attempt it starts with a different management port, which ensures that all management ports are eventually tested for connectivity. Any management port which sees a failure gets an error in the SystemAdapterInfo inside the field ErrorInfo for the particular DevicePort.

Note that if a port was tested and succeeded, the ErrorInfo.timestamp is updated and the ErrorInfo.description is set empty. This indicates the most recent successful test.

If after two attempts (spaced 5 minutes apart, see above) the device can not connect to the controller on any of the management ports in the current configuration, then it will try the fallback configuration (and if that succeeds the fact that it is using a fallback is reported with a non-zero currentIndex as above).

Trying a better configuration

If for some reason the most recent (aka the highest priority) configuration is not used (and currentIndex is reported as non-zero per above), the device will re-try the highest priority configuration. The default timer for this is 10 minutes and can be set with timer.port.testbetterinterval. That timer can be set to zero to disable this re-try.

If the re-try succeeds then SystemAdapterInfo is updated with the previously reported error cleared. The fact that it has failed in the past can be seen from the reported lastFailed timestamp in SystemAdapterInfo being non-zero.

Handling remote (temporary) failures

There is a set of failures which can be attributed to the controller having issues which does not warrant any churn or fallback on the device. The current cases are:

• the server certificate having expired (or not yet being valid)
• the server responding with a TCP Reset/Connection refused (and proxy is not in the path)

In those cases nim proceeds with the current configuration and assumes that the server will at some point in time be corrected.

Failure reporting

The device reports the status of all of the device connectivity using SystemAdapterInfo. There are two levels of errors:

• A new SystemAdapter configuration was tested, but none of the management ports could be used to connect to the controller. In that case a failure is reported by setting lastError in DevicePortStatus and the currentIndex is set to the currently usedDevicePortStatus in the list. Note that lastFailed and lastSucceeded can be used to see if a configuration has succeeded in the past or always failed.
• A particular management port could not be used to reach the controller. In that case the ErrorInfo for the particular DevicePort is set to indicate the error and timestamp.

Air-Gap Mode

Air-Gap mode allows a device to operate without connectivity to the main controller, supporting scenarios where local-only management is required. In this mode, the Local Operator Console (LOC) is used to manage the device locally.

When Air-Gap mode is enabled, the connectivity tester reports the device as connected if either the main controller or the LOC is reachable. This ensures that when network configuration changes are made, EVE-OS does not incorrectly mark the configuration as non-working or revert it, as long as LOC connectivity is maintained.

Error and warning logs about lost controller connectivity are either suppressed or downgraded to trace-level to avoid log clutter, since loss of controller connectivity is expected in this mode. However, the device still attempts to contact the controller, so it can resume normal operation if connectivity is restored.

Air-Gap mode is controlled through the configuration property airgap.mode. Setting this property to enable enables Air-Gap mode, while setting it to disable disables it. By default, Air-Gap mode is disabled.

Air-Gap mode activation is indicated in the console output by the diagnostic microservice. If Air-Gap mode is enabled and Local Operator Console (LOC) is configured, it will report:

INFO : Air-gap mode enabled, using LOC: <loc-url>

But if LOC is not configured while Air-Gap mode is enabled, the user will be warned:

WARNING: Air-gap mode enabled without LOC configuration

Troubleshooting

There are multiple sources of information that provide a picture of the current or a past state of device connectivity and when combined and investigated together can help to determine or narrow down the root cause of any connectivity issue.

LED Indication

Being physically present next to a device, the current state of the device wrt. controller connectivity and onboarding procedure can be determined based on the blinking pattern of an LED. To indicate a given state, LED blinks one or more times quickly in a row, then pauses for 1200ms and repeats continuously. By counting the number of successive blinks one may determine the current state of the edge device.

For example, if a device does not have any usable IP addresses it will blink once per iteration, twice if it has IP address(es) but still no cloud connectivity, three times if device has connected to the controller but it is not yet onboarded.

Which LED is used to indicate state progression depends on the device model. For a full list of blinking (and color) patterns and to learn which LED is used for indication in a particular device model, please refer to LED-INDICATION.md.

For a remotely accessed device (via iLO, ssh, edgeview, etc.), the blinking pattern can be extracted from the shell using:

cat /run/global/LedBlinkCounter/ledconfig.json

# Returns for example:

{"BlinkCounter":4}

Diag microservice

Diag microservice is responsible for managing diagnostic services on the device, and reporting to console for debugging if there are any issues. It informs about which network configuration (aka DevicePortConfig) is currently being used, what is the status of connectivity between the device and the controller and whether the device has already onboarded. Diag also prints the list of physical network adapters visible to EVE-OS (i.e. not assigned to applications) with their assigned IP addresses, associated DNS servers and proxy config (if any). Finally, it also outputs results of some connectivity checks that it performs, including /ping request to the controller and GET google.com request, both performed via each management interface. It does not send this information to the Controller or to the log, however. One must access the device console to have this information printed on the stdout. With ssh access, you can view this output in /run/diag.out. Should this diag output interfere with other work performed via the console, disable the printing of diagnostics with EVE-OS verbose off.

Connectivity-Related Logs

The progression and outcome of network configuration testing is logged with messages prefixed with DPC verify: (note that DPC is abbreviation for DevicePortConfig, which is a Go structure holding configuration for all management and app-shared interfaces). These messages can explain why a particular device is not using the latest configuration but instead has fallen back to a previous one. These logs are quite concise yet pack enough information to tell when a device performed testing, what was the reason to trigger it (was it periodic or config/state changed?), how the testing progressed (e.g. how long did the device waited for IP addresses?) and what was the outcome (did the latest network configuration passed the test and if not what error did the connectivity check returned). It is not necessary to enable debug logs - the most important messages related to DPC testing are logged with log levels info, warning or error.

For concrete examples of DPC verify logs, please refer to nim.md, section "Logs". That document also explains the topic of DPC reconciliation, i.e. synchronization between the intended and the current network config and how it is presented in the logs (spoiler alert: search for DPC Reconciler in the logs). This can be useful to look into if there is a suspicion that the root cause of a connectivity issue is device not being able to apply some configuration items (e.g. failed to start DHCP client).

The onboarding procedure is from the device side executed by the client microservice. Hence, to learn the state and the progression of the device onboarding, search for logs with a source field set to client.

Connectivity-Related Files

EVE-OS uses pubsub for communication between its microservices. Currently published messages are stored in files either persistently under the /persist partition or non-persistently under the /run directory. When connected to a device via console, ssh or edgeview, one may read these files to learn the current config/state/metrics as published by EVE-OS microservices.

For device connectivity, the particularly interesting topics are those published by Network Interface Manager (NIM for short):

• /persist/status/nim/DevicePortConfigList/global.json: contains the set of DevicePortConfig-s (configurations for mgmt and app-shared ports) which are currently known to the device (the latest highest-priority DPC plus some persisted previous configs). These configs are sorted in priority-decreasing order, i.e. the latest config is at the index 0. Additionally, there is a field CurrentIndex, pointing to the currently applied DPC. Each DPC entry summarizes the outcome of testing with fields: LastFailed (time of the last failed test), LastSucceeded (time of the last successful test) and State (connectivity state determined by the last test). State is an enum value but in this file presented with its integer representation. To learn what a given State number means, look for the DPCState enum in zedroutertypes.go.
• /run/nim/DeviceNetworkStatus/global.json: contains state information for the currently applied DPC. For every port it shows the currently assigned IP addresses, DNS servers, IP-based geolocation info, MAC address, administrative status (up/down), WiFi/cellular-specific details for wireless interfaces and more.

Additionally, NIM outputs the current and the intended state of the configuration into /run/nim-current-state.dot and /run/nim-intended-state.dot, respectively. This is updated on every change. The content of the files is a DOT description of the dependency graph modeling the respective state (config items have dependencies between them which decide the order at which they can be applied). Copy the content of one of the files and use an online service https://dreampuf.github.io/GraphvizOnline to plot the graph. Alternatively, generate an SVG image locally with: dot -Tsvg ./nim-current-state.dot -o nim-current-state.svg (similarly for the intended state)

Some more pubsub topics from other microservices worth looking at during device connectivity troubleshooting:

• /run/zedagent/DevicePortConfig/zedagent.json: currently published DPC from zedagent (either coming from the controller or from a bootstrap config, see CONFIG.md).
• /run/zedagent/PhysicalIOAdapterList/zedagent.json: list of physical IO adapters, published from zedagent but ultimately coming from the device model. This includes network adapters with their specification (PCI addresses, kernel interface names, etc.).
• /run/domainmgr/AssignableAdapters/global.json: the spec and state of physical IO adapters after being processed by domainmgr microservice. This means after each was assigned to the corresponding domain and available to be used either by NIM (for mgmt or as app-shared) or by an app (as app-direct). NIM waits for AssignableAdapters before applying DPC config. Change in AssignableAdapters can also trigger DPC re-testing or unblock NIM to apply a given DPC (e.g. a particular port is finally available in dom0).
• /persist/status/zedclient/OnboardingStatus/global.json: the status of the onboarding procedure. Once device is onboarded, DeviceUUID field will be non-empty and contain the assigned device UUID (also printed to /persist/status/uuid).

For WWAN connectivity info, refer to these pubsub topics:

• /run/nim/WwanConfig/global.json is published by NIM and contains configuration for cellular modems.
• /run/wwan/WwanStatus/global.json is published by wwan microservice to inform zedagent, NIM and some other microservices about the current cellular connectivity status.
• /run/wwan/WwanMetrics/global.json is published by wwan microservice and contains packet/byte counters reported by cellular modems (i.e. not from the Linux network stack)

Separately to pubsub topics, file /run/wwan/resolv.conf/<interface-name>.dhcp is updated by the wwan microservice and contain the list of nameservers that should be used with the given wwan interface. This is similar to how dhcpcd reports DNS servers for ethernet interfaces (for both DHCP and static IP config). In NIM, where this info is processed and further published via DeviceNetworkStatuspubsub topic, we can therefore process nameservers for ethernet and wwan interfaces in a very similar way and reuse some code.

Netdump and Nettrace

EVE-OS performs all communication with the controller over the HTTP protocol using the http.Client for Go. Additionally, the same protocol and the client are typically also used to download EVE/app images from data stores. Given all that, it is useful to be able to obtain diagnostics from HTTP requests processing as done by http.Client and use it for connectivity troubleshooting. For exactly this purpose we implemented nettrace package, which internally wraps and injects hooks into http.Client to monitor and record a summary of all important network operations that happen behind the scenes during request processing at different layers of the network stacks, denoted as "network traces" (or just collectively denoted as "network trace" to avoid overusing the plural form). This would include all successful and also failed TCP and TLS handshakes, DNS queries, reused connections, HTTP redirects, socked read/write operations, etc. All of this is typically hidden behind the http.Client implementation and only a single error value with limited information is available when a request fails. Moreover, nettrace allows to capture conntrack entries as well as all packets of traced TCP connections and UDP "exchanges" for more in-depth analysis. For example, with a conntrack entry attached, it is possible to tell for a given TCP/UDP traffic which ACL rule (configured by EVE-OS as iptables rule) was applied.

Network tracing comes with an additional overhead and the output can be quite large. Therefore, we must be careful about how often are network traces obtained and how do we publish them. For example, logging network trace as a single message is not an option. Instead, EVE-OS publishes network traces inside Tar/GZip archives, labeled as "netdumps" and stored persistently under /persist/netdump directory. This is done by the netdump package, which additionally adds some more files into each archive to capture the config/state of the device connectivity at the moment of the publication. All this information combined allows to troubleshoot a connectivity issue (between device and the controller or a data-store) even after it is no longer reproducible.

Every netdump package contains:

• EVE-OS directory with snapshots of pubsub publications related to device connectivity (such as DevicePortConfigList, DeviceNetworkStatus, AssignableAdapters), DOT files projecting the intended and the current network config as dependency graphs, wwan config/status/metrics and also files with basic info like the EVE-OS and Golang versions. For more info on these files, please see the section above,
• linux directory with the output from networking-related commands like ip, arp, iptables, netstat, etc.
• requests directory with attached network traces and packet captures (if enabled) collected for HTTP requests run by EVE. This can be for example request to get device config or just to ping the controller, or to download an image. Note that EVE-OS can make multiple attempts for every request, trying every port and local IP address. Every attempt will have separate trace under its own subdirectory named <request-name>-<interface>-<attempt-index>

Every netdump is published into a topic, represented by its name and by default limited in size to 10 netdumps at most (configurable by netdump.topic.maxcount). The oldest netdump of a topic is unpublished (removed from /persist/netdump) should a new netdump exceed the limit. Topics are used to separate different microservices and even to split successful and failed requests from each other. Topic name is therefore typically <microservice>-<ok|fail>. For troubleshooting purposes, netdumps of failed requests are obviously more useful, but having a trace of a "good run" can be useful to compare and find differences. Published netdump filename is a concatenation of the topic name with a publication timestamp plus the .tgz extension, for example: downloader-fail-2023-01-03T14-25-04.tgz, nim-ok-2023-01-03T13-30-36, etc.

Not all microservices that communicate over the network are traced and contribute with netdumps. Currently, traced HTTP requests are:

• /ping request done by NIM to verify connectivity for the latest DPC (testing of older DPCs is never traced). Packet capture is also enabled and the obtained pcap files are included in the published netdumps. To limit the overhead associated with tracing and packet capture, NIM is only allowed to enable them and produce netdump at most once per hour before onboarding and once per day after onboarding (configurable by netdump.topic.preonboard.interval and netdump.topic.postonboard.interval, respectively). The pre-onboard interval is intentionally lower (by default) to get more frequent diagnostics for initial connectivity troubleshooting.
• /config and /info requests done by zedagent to obtain device configuration and publish info messages, respectively. Packet capture is not enabled in this case. Netdump is produced at the interval as given by netdump.topic.postonboard.interval(/config and /info requests are not run before onboarding). For /info requests, tracing only covers publication of the ZInfoDevice message. Moreover, tracing is enabled only if the highest priority DPC is currently being applied and is reported by NIM as working. Otherwise, we will eventually get nim-fail* netdump which should be sufficient for connectivity troubleshooting. The purpose of zedagent netdumps is to debug issues specific to /config an /info requests (which are essential to keep the device remotely manageable). Netdumps are published separately into topics zedagent-config-<ok|fail> and zedagent-info-<ok|fail>.
• every download request performed by downloader using the HTTP protocol is traced and netdump is published into the topic downloader-ok or downloader-fail, depending on the outcome of the download process. By default, this does not include any packet captures. However, a limited PCAP can be enabled with config option netdump.downloader.with.pcap. Limited in the sense that it will not include TCP segments carrying non-empty payload (i.e. packets with the downloaded data). On the other hand, included will be all UDP traffic (DNS requests), TCP SYN, RST and FIN packets as well as ACK packets without data piggybacking (sufficient to tell where a download process got "broken"). The total PCAP size is limited to 64MB (packets past this limit will not be included).

In order to troubleshoot a present or a past connectivity issue, it is necessary to locate and obtain the appropriate netdump from the affected device (locate by microservice aka topic name and look for the closest timestamp). Without a remote connectivity to the device, it is possible to dump all diagnostics to a USB stick. See CONFIG.md, section "Creating USB sticks". With this method, the entire /persist/netdump directory will be copied over. If device is remotely accessible, published netdumps can be listed and copied over ssh (if enabled by config), edgeview (ls + cp commands; to download all netdumps at once use: tar//persist/netdump) or using a remote console if available.

Next steps

This is a series of articles. You will likely follow them in this order.

1. Network Overview
2. Network Connectivity Details - You are here!
3. Create a Network
4. Manage a Network
5. Use the ZEDEDA CLI to manage a network