Software/IoT Workflow for Raspberry Pi

A Raspberry Pi is like a micro computer, and many people use it as such. It’s tempting to utilize those HDMI and USB ports to attach a screen, a keyboard, or a mouse. However, for many others, a Raspberry Pi can be used as a development sever, a printing server, a smart home hub, or a network server. Those are just the examples I can think of, but the possible applications are countless.

So you’re a software developer or IoT developer, and you just got your first Raspberry Pi; I’m happy to share some thoughts on how to best benefit from your Raspberry Pi. Though this post is meant for Raspberry Pi, many of the ideas shared here apply to other devices.

Choosing Operating System

The microSD card is sold separately, in many cases, and you’re going to need it; the microSD is the main storage unit for your Pi. First order of business is to burn an operating system image to the microSD card. You can’t go wrong with whatever OS you choose depending on your preference, but when choosing an OS for development or server use, use a “Headless” OS. A headless OS is an operating system without all the graphical modules. No Graphical User Interface, GUI, or what’s commonly refereed to as Desktop Environment among Linux users. A good choice for a headless OS is the Raspberry Pi OS Lite. An easy way to burn the OS image to the microSD card is by using an imaging utility, and there’s an official one, the Raspberry Pi Imager.

Dropping graphic modules will free up a lot of resources; you can put those resources to a better use. Also, there will be less cables connected to your Pi.

Setting Up Connection

Your Raspberry Pi comes packed with modules including both Ethernet and WiFi modules. We will communicate with the Raspberry Pi over SSH, but which module should we use is dependent on what kind of application you have in mind for the Pi. In many cases, WiFi will do; however, in a case such as using it as a live streaming server or screen sharing between devices, Ethernet is the way to go¹. Raspberry Pi 4 model B for example has IEEE 802.11ac WiFi module which will average a latency of 30 milliseconds; on the other hand, it has a gigabit ethernet, and if you use CAT5e or CAT6 cable, your latency should be less than a millisecond. Make a choice and remember which one you will be using the most.

1. Configure WiFi and Enable SSH

I will assume you will be using both WiFi and Ethernet modules, so let’s configure your WiFi. Fire up a terminal window, and find the microSD card location on your computer. On Linux, this most likely will be /media/"USER_NAME"/"microSD_NAME"/. If you’re on Windows OS, you can use WSL. Type the following command to edit the WLAN config.

1
2

cd /media/"USER_NAME"/"microSD_NAME"/     #OR wherever your microSD is located.
nano boot/wpa_supplicant.conf

Add the following lines to the end of the file.

1
2
3
4
5
6
7

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
country=US
update_config=1
network={
    ssid="network-name"
    psk="network-password"
}

Replace network-name with your WiFi network name. Note that the name is case sensitive. Replace network-password with your network password. Use CTRL + o to save the file and CTRL + x to exit.

When the Raspberry Pi boots up, it checks if a file called ssh exits in the boot directory to decide whether to start the SSH server or not. We want the Raspberry Pi to start the SSH server on start up, so we will create the file. Run the following command.

1

touch boot/ssh

This will create an empty file. The Pi will detect the file, add SSH service to startup, and eventually delete the file.

2. Reserve The IP Address of your Pi

Eject the microSD card from your PC, and insert it into your Raspberry Pi. Power up the Raspberry Pi. Make sure your computer is on the same network as the Raspberry Pi. First, we will find out more info about the network we’re using. We’re looking for your computer’s IP and the network mask. Most home networks have net mask of “255.255.255.0”. Also, most routers use the IP range of “192.168.1.x” for their subnet except some of Comcast routers which use “10.0.0.x”. To know for sure what your router uses, run the command ifconfig on Unix/Linux and ipconfig on Windows CMD. On Linux, I get:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

enp4s0: ....

lo: ....

virbr0: ....

wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.11  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::a212:adef:2a6e:ca6d  prefixlen 64  scopeid 0x20<link>
        ....

If you’re Linux like me, your IP will be listed beside the module you’re using. I’m on Wireless-LAN, wl for short, and my IP is “192.168.1.11” If you’re on Windows, your IP should be listed in a similar manner labeled as IPv4 instead of inet.

Now, we will try to find the IP address for the Pi. We can use ARP, Address Resolution Protocol, but this is used mostly if you know the MAC address of the device; thus, we will use NMap. If you don’t have it installed, follow the directions on NMAP’s download page to install it on your system. Run the following command to discover hosts on the network.

1

sudo nmap -sn 192.168.1.0/24

Substitute the IP address with one relevant to your network. The last integer of the IP doesn’t matter, because we’re using an offset of 24 bits². This tells nmap to scan the network for hosts with IP addresses between “192.168.1.0” and “192.168.1.255”. When the scan is done, you should see the Raspberry Pi host info.

1
2
3

Nmap scan report for 192.168.1.12
Host is up (0.10s latency).
MAC Address: ff:ff:ff:ff:ff:ff (Raspberry Pi Trading)

This shows that the Raspberry Pi has the IP, “192.168.1.12” Now that we know the IP, we can SSH into the Raspberry Pi. Run the command

1

ssh pi@192.168.1.12

Replace the IP address with the one you found. You will be asked for password, the default password is raspberry.

Most home networks have the router as their DHCP server. The DHCP server is responsible for assigning IP addresses to all hosts on the network. Because IPs are dynamically assigned, the IP of the Raspberry Pi can change between boots. We want to avoid this, so we don’t have to find the IP each time we want to use the Raspberry Pi. Luckily, most -if not all- routers allow for reserving IP addresses. To do this we need the MAC addresses for the Pi. We have already seen one of them when we found the IP address of the Raspberry Pi. The one was shown then is for the WiFi module. If you’re only going to use it over WiFi, you don’t need the following step. To find both MAC addresses, SSH into the Raspberry Pi and run ifconfig, you should see the following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether ff:ff:ff:ff:ff:ff txqueuelen 1000  (Ethernet)
        ....

lo: ....

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.12  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::8229:7e0f:330d:4798  prefixlen 64  scopeid 0x20<link>
        ether ff:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        ....

You can see both eth0 for the Ethernet module and wlan0 for the WiFi module. Both has a field called ether and formatted as ff:ff:ff:ff:ff:ff and that’s the MAC address for each. On your computer, open your favorite web browser, and type your gateway/router IP address. It should be “192.168.1.1” or “10.0.0.1” or something similar depending on your router. The following are instructions for assigning static IP to a LAN device for the most popular brands.

• Netgear
• Xfinity
• Linksys
• TP-Link

For any brand you will be asked for the MAC address, possibly a device name, and the IP address you want to assign to said device. You should keep the IP address of the WiFi module as it’s. You can assign the Ethernet module any available IP address. Take note of the IP address of each module.

3. Configure SSH

We want to be able to SSH into the raspberry pi without having to enter the password each time; also, it’s not wise to use a password when you can use keys. If you’re software developer, you most likely have your SSH key setup; to check if you generated an SSH key before, look at the .ssh directory on your machine to see if you have the files id_rsa and id_rsa.pub.

1
2
3
4
5
6
7
8
9

ls -al ~/.ssh/
############ OUTPUT ##################################
total 52                                  
drwx------  2 user group 4096 Nov 11 13:16 .
drwxr-xr-x 79 user group 4096 Dec  1 18:35 ..
-rw-r--r--  1 user group  222 Nov 2 13:21 config
-rw-------  1 user group 2222 Nov 2 13:12 id_rsa
-rw-r--r--  1 user group  222 Nov 2 13:12 id_rsa.pub
########### END of OUTPUT ############################

If you don’t have your key setup, you can generate a new one using ssh-keygen. You simply run the command and it will guide you through the process. Leave the passphrase empty though; it doesn’t play well with automation. Your new key should be in the default location ~/.ssh/ as shown above. Now, we need to copy the public key to the Raspberry Pi; I usually use ssh-copy-id to do this. The general format is:

1

ssh-copy-id <USER>@<HOST_IP>

Your default user is pi, you can add new user if you like. The host IP address would be the one for WiFi as we still using it to do the initial configuration. If you don’t have ssh-copy-id, you would have to copy your public key to the Raspberry Pi manually. The Raspberry Pi documentation suggests this method:

1

cat ~/.ssh/id_rsa.pub | ssh <USERNAME>@<IP-ADDRESS> 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys'

This pipelines the output of cat as input to the ssh command. When you add quoted string to the end of ssh, you’re asking for what’s inside the string to be executed on the remote host on the fly without starting a session.

Now, we will add a ssh config file.

1

nano ~/.ssh/config

We will add two hosts; one for each module. If your WiFi module is your most used one, name the host mypi for example. The other module can take the same name with a suffix of your choosing such as mypi-ethernet. This will work nicely when you try to use auto complete. You can use any editor you like; I’m just using nano for demo purposes. We will add the following lines.

1
2
3
4
5
6
7
8

Host mypi
     HostName 192.168.1.12
     User pi
     IdentityFile ~/.ssh/id_rsa
Host mypi-ethernet
     HostName 192.168.1.13
     User pi
     IdentityFile ~/.ssh/id_rsa

Let’s discuss what we have here. The first line is the host identity; you can choose any alias you’re comfortable with. The following lines are indented because they’re configuration for Host mypi. The HostName config is the IP address or the url of the host. The User is the user used to log in, and finally the IdentityFile is the location of your key. We’re using the same user and identity file for both hosts; they’re really the same host but identified by different IP address and subsequently name. Now, you can simply SSH into the Raspberry Pi using a very simple command such as ssh mypi or ssh mypi-ethernet.

I’m not blind to how inconvenient having two hosts for one device is, but this way is the most efficient. It’s 100% efficient if you’re only using one module. But what happens in these cases:

• Your router doesn’t support IP reservation.
• You use other networks quite often.
• You really don’t like using two host names

Well, in those cases; you’re going to have to resolve the IP address each time you use ssh. We can automate that too, but it comes with high cost and that’s a time delay. If you use nmap to resolve the IP, you will waste a lot of time. Your best chance is using ARP; though, it will take a few seconds at least to resolve the IP. If this is the case, you can change the ssh config file to something like:

1
2

Host mypi
        ProxyCommand ssh -i ~/.ssh/id_rsa pi@"$(arp -a | grep 'ff:ff:ff:ff:ff:ff\|ff:ff:ff:ff:ff:ff' | grep -oE '((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])')"

In this case, we’re using a proxy command to ssh into the host. Let’s take a quick look at the commands between the quotation marks. We first use arp -a to find all hosts on the network; we pipe the output into grep 'ff:ff:ff:ff:ff:ff\|ff:ff:ff:ff:ff:ff'. Each ff:ff:ff:ff:ff:ff should be the MAC address of one of your network modules. This command will produce the line containing the IP address of the Raspberry Pi regardless of using WiFi or Ethernet. The last step is to eliminate any text other than the IP address, and we do this using grep with a regular expression.

If you’re okay with a semi-automated solution, there’s room for improvement. DHCP servers typically lease IP addresses to their clients; each client renews the lease before it expires. The DHCP server won’t mind renewing the lease except in very few situations such as IP address conflict for example. Thus, your Raspberry Pi will mostly likely only change IP if it’s powered off or loses connectivity long enough for the lease to expire. So instead of resolving the IP with every SSH command, we can resolve the IP once only per session. Start by adding this function to your .bashrc file.

1
2
3
4
5
6
7
8
9

function setPiIP {
        export MY_PI_IP="$(arp -a | grep 'ff:ff:ff:ff:ff:ff\|ff:ff:ff:ff:ff:ff' | grep -oE '((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])')"
        if [ ! -z "$MY_PI_IP" ]
        then
                echo "Found the Pi IP:$MY_PI_IP"
        else
                echo "Couldn't find the Pi IP"
        fi
}

Replace the ff:ff:ff:ff:ff:ff fields with the MAC addresses of your modules. This function will resolve the IP and export it to an environment variable. We can then use the variable for SSH. Reload the .bashrc file by either closing and reopening the terminal window or running the command source ~/.bashrc. Then change the ProxyCommand in the ssh config file to be:

1
2

Host mypi
        ProxyCommand ssh -i ~/.ssh/id_rsa pi@"$MY_PI_IP"

You will need to run the function setPiIP once before starting to use the PI for each development session. Calling the function is the same as using a terminal command. You can also rename the function to your preference.

Automating Deployment

Our next step is to try automating code deployment on the Raspberry Pi. Whether you’re using the Pi as a development or a production environment, it will save you a great deal of time. We will be using Git for version control. There’s three viable options we can use:

1. Raspberry Pi as a Git Server
   • Pros: As simple as adding a remote url to your git repo.
   • Cons: You will use more resources than necessary from your Raspberry Pi. Requires configuring post-receive hooks to issue commands after updates.
2. Repository-Specific Deploy Script
   • Pros: No extra code on the Raspberry Pi, and you won’t need to run any services. You can specify commands to execute after deployment.
   • Cons: Requires more work in the initial configuration and repo-specific config file.
3. Using PM2
   • Pros: We can use PM2 to manage the running processes on the Raspberry Pi while offering deployment support.
   • Cons: Requires NodeJS installation on your Pi.

Other options may utilize GitHub Webhooks to actively listen for push events and trigger post-deploy script; however, Github Webhooks require payload url meaning the Raspberry Pi will need a static public IP address.

It’s great to have a process manager to manage production applications; thus, I will use PM2 for automating deployment. First, SSH into your Raspberry Pi. Then you need to do the following:

1. Install NodeJS. The latest LTS version as of writing this post is 14.15.3.

1
2

curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs

2. Install PM2 globally.

1

sudo npm install pm2 -g

3. Add PM2 to startup processes and save

1
2

pm2 startup
pm2 save

4. Ensure your pi has access to your remote repository. You can do this by generating a public key on your Raspberry Pi using ssh-keygen command, and adding it to your GitHub repository as a Deploy Key. Deploy Keys are also available on GitLab and BitBucket.
5. On your development machine, you need PM2 installed as well.
6. On your local machine, go to the parent directory of your repository, and run the command PM2 init. This will generate ecosystem.config.js with the following content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

module.exports = {
  apps : [{
    script: 'index.js',
    watch: '.'
  }, {
    script: './service-worker/',
    watch: ['./service-worker']
  }],

  deploy : {
    production : {
      user : 'SSH_USERNAME',
      host : 'SSH_HOSTMACHINE',
      ref  : 'origin/master',
      repo : 'GIT_REPOSITORY',
      path : 'DESTINATION_PATH',
      'pre-deploy-local': '',
      'post-deploy' : 'npm install && pm2 reload ecosystem.config.js --env production',
      'pre-setup': ''
    }
  }
};

Visit the embedded link for more info about the subject. We briefly discuss the deploy configurations. First, you can have multiple environments such as development, staging, and production. I will assume you’re using the PI as production env. Some of these config are the same as our SSH configs such as user, host, and key which we can add. If you’re using environment variable to track your Pi, you can use process.env.MY_PI_IP for the host parameter. Here’s how my deploy config would look like:

1
2
3
4
5
6
7
8
9

    production : {
      key : '~/.ssh/id_rsa',
      user : 'pi',
      host : '192.168.1.12',   //OR host : process.env.MY_PI_IP,
      ref  : 'origin/main',   //New GitHub repos as of late 2020 uses `main` instead of `master`
      repo : 'git@github.com:repo.git', //Ensure you're using SSH format not HTTPS
      path : '/var/www/production',  //The installation path on the RaspPi
      'post-deploy' : 'pm2 startOrRestart ecosystem.config.js --env production' //Prepend `npm install &&` when working with nodejs project
    }

That’s it. You can now issue deploy commands without leaving you development environment. For the first time, you want to setup the remote for deployment; after that, you simply do updates.

1
2
3
4
5

# Setup deployment at remote location
pm2 deploy production setup

# Update remote version
pm2 deploy production update

If you’re using ESM and having errors regarding the use of requires() rather than import. You can avoid this by using a JSON file. I used a JS file in the example above to be able to access env variables which you don’t need if you know your Pi IP address. Otherwise,You will need to make the following changes.

• Rename the ecosystem.config.js file to ecosystem.config.cjs
• Change the file name where it’s used such as 'post-deploy' : 'pm2 startOrRestart ecosystem.config.js --env production' to ecosystem.config.cjs
• Specify the config in your PM2 deploy commands to signal PM2 to use specified file. Your deploy command becomes pm2 deploy ecosystem.config.cjs production update

Our initial setup with SSH configurations gives you other options to automate deployment. If you don’t like PM2, you can pursue one of the options discussed above.

Thank you for reading, please let me know if you have any questions or comments.