Edge Node Debugging Tips

Prerequisites

  • ZEDEDA enterprise user account with privileges to edit edge nodes
  • Docker installed on your computer (to run zcli container)
  • Public/private key pair to use for ssh authentication
  • Your edge node is online and connected to ZEDEDA

Note that it is recommended to use the ZEDEDA Remote console to debug applications running on EVE. To access and debug both application instances and EVE itself, another useful tool that enables remote debugging will be available soon: EdgeView.

Configure zcli

Run the zcli container and log in to your ZEDEDA enterprise. Be sure to configure the correct ZEDEDA “server” URL and the correct username and password, matching your ZEDEDA login (where your username is typically your email address).

docker run -v $HOME:/h -it --rm zededa/zcli:latest
zcli> zcli configure
Server [zedcontrol.zededa.net]: zedcontrol.gmwtus.zededa.net
Login with token?(y/n) [n]: n
Username: your@email.com
Password:
Output format [text]:
zcli> zcli login
User your@email.com logged into enterprise: your_enterprise.

ZCLI to Enable USB Ports Locally

Being “secure by design”, the EVE operating system will lock down local access to physical ports and interfaces of your edge node after it has registered with ZEDEDA (as its controller). To unlock and enable local access to your edge device’s keyboard, mouse, and (sometimes) console, log in via zcli as described above, then run the following command.

zcli edge-node update <edge-node-name> --config=debug.enable.usb:true

ZCLI to Enable ssh

Identify Public/Private Key Pair

First, you will need to identify or create a public/private key pair. The private key will remain on your computer and you will need to know the exact path to that file. The public key will be configured on the edge node using ZCLI.

To look for existing keys:

ls -l ~/.ssh
-rw-------  1 kathy  staff  3389 May 21  2021 id_rsa
-rw-r--r--  1 kathy  staff   747 May 21  2021 id_rsa.pub

Search the internet for your specific operating system if you need to generate a key pair. For example, you can follow the DigitalOcean tutorial which recommends this command:

ssh-keygen

(bypass the passphrase by just hitting enter)

For the ssh-keygen command which would lead to the result of the ls command shown above it, the default path of your private key would be:

/home/USERNAME/.ssh/id_rsa

And the public key would be found in the same directory, with the suffix .pub:

/home/USERNAME/.ssh/id_rsa.pub

Depending on your OS, these types of shortcuts may also work:

~/.ssh/id_rsa.pub
$home/.ssh/id_rsa.pub

For the next section (configuring zcli), you will need to copy your public key and paste it into a command. You can copy it by writing it out to the terminal:

cat ~/.ssh/id_rsa.pub

(and then copy the entire result)

Enable ssh on the Edge Node

Be prepared to copy the entire output of the “cat” command in the previous section and paste it into the italics text portion of the command below (i.e., paste inside the quotes "...").

zcli edge-node update <edge-node-name> --config=debug.enable.ssh:"ssh-rsa YOUR_PUBLIC_KEY_HERE_ABC... your@email.com"

The <edge-node-name> must match the name as shown in your ZEDEDA GUI.

 

Test ssh Access

From the same local network as the edge node, you should now be able to access the device using ssh, by including a pointer to the path of the private key (~/.ssh/id_rsa) of the key pair. For example:

ssh -i ~/.ssh/id_rsa root@<edge_node_IP_addr>

An actual ssh login might look like this:

.ssh # ssh -i id_rsa root@192.168.1.191 
EVE is Edge Virtualization Engine
Take a look around and don't forget to use eve(1).
#

Example Commands for EVE-OS

EVE-OS is NOT a typical Linux distribution, even though it leverages the Linux kernel. At the command line of EVE-OS, you can view various things for debugging purposes, but operationally an EVE-OS device is intended to be securely managed by an EVE controller, such as the ZEDEDA solution. It is not possible to “control” very much about EVE from the command line. Example commands follow.

# eve -h
Welcome to EVE!
 commands: enter [qube (assumed pillar)] [command (assumed sh)]
         enter-user-app <qube>
         exec qube command
         list
         status
         start <qube> (requires a qube to be in a destroyed state)
         pause <qube>
         resume <qube>
         destroy <qube>
         persist list
         persist attach <disk>
         firewall drop
         verbose on|off
version

For example:

# eve list
# eve status
# ls -l /config
# cat /config/server
# cat /config/uuid

For more information about EVE, visit the source code repository on GitHub.

SSH Key clear

After rebooting the edge node, ssh access and the public shared key are removed from the edge node. To enable ssh access again, the above procedure will need to be repeated.   

In the $home/.ssh/ directory, the file known_hosts will need to be edited, and remove the shared key for the IP address of the edge node before trying to access the node again.  

Example of Error:

.ssh# ssh -i id_rsa root@192.168.1.191
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the id_rsa key sent by the remote host is
SHA256:L8SY61hfkUymPkja/6Rs133dsbITOv2l+G+7HotWmjs.
Please contact your system administrator.
Add correct host key in /Users/USERNAME/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/USERNAME/.ssh/known_hosts:12
Host key for 192.168.1.191 has changed and you have requested strict checking.
Host key verification failed.
Was this article helpful?
5 out of 5 found this helpful