Updated on 2020-11-05
Explore an Arduino based IoT web server and UDP multicaster for monitoring a remote water pump
Disclaimer: This process will overwrite the default firmware that ships with the WiFi module on this board. Once you overwrite, it is possible, but not easy to put back to factory, and the default library for communicating with this module "WiFiEsp" will no longer work with it. I've also read, but have not verified that there's a very limited number of times you can flash some of the more cheaply made ESP-01s so there is some risk but I don't think that's necessarily true for these Arduino and derivative boards.
The pump monitor is a grand endeavor exploring some advanced features of the Arduino Mega 2560+WiFi R3 with emphasis on its integrated ESP8266 module. We navigate creating an asynchronous templating web server, an embedded flash filesystem which we read and write, UDP multicasting, multicast DNS publishing, slaving the ATmega2560 to the ESP8266 for performance and some basic digital I/O using the Mega. The end result is a nearly configureless IoT device that monitors a pump remotely and provides a couple of different ways to access it.
My in-laws live in nice house in the middle of the woods. They draw water from a creek, but the pump requires some love and babysitting sometimes. It's mostly due to mud and debris creating problems for it. It also can't run constantly because if it siphons too much water, the creek level lowers and you end up sucking up mud.
Years ago, my father in-law built a simple 12vdc logic board that uses water level sensors to determine what the pump was doing. The status is displayed on a series of 110vac lights. Rather than install a whole new system in the tanks, we decided to tap the circuits for the lights in the existing system, so all we're doing is hooking relays with 110vac coils into the light circuits and using the switches on them to set pins (2-4) on the Arduino high when the corresponding light goes on. The wiring is so straightforward as to be boring. The good stuff is the software that drives it. We're covering that here.
Update: Added WPS support
You'll need the following:
You can find the Arduino IDE here.
To add the board manager, you'll need to go to File|Preferences and add the following URL in the box: http://arduino.esp8266.com/stable/package_esp8266com_index.json
http://arduino.esp8266.com/stable/package_esp8266com_index.json
You can download the ESPAsyncWebServer library here. Once you download it, unzip it and you should get ESPAsyncWebServer-master folder. Extract that and rename it to ESPAsyncWebServer. Finally, in your application folder (under Program Files (x86) in Windows or under your (~) home directory in Linux) under libraries, copy the folder you extracted and renamed here.
You can download the ESPAsyncTCP library here. Similarly as before, unzip what you download and extract the folder. Take the folder and rename it ESPAsyncTCP and copy it under your libraries folder as above.
The ESP8266 Filesystem Uploader can be downloaded here. Extract the ESP8266FS folder and put it in your tools folder under your application directory. Your application directory will be under Program Files (x86) somewhere if you're in Windows or under your (~) home directory in Linux. You'll have to restart your Arduino IDE.
We have a pump that sits out in the woods and draws water from a creek. The pump can have several states, such as "pumping", "requesting water" or "ready". There are WiFi repeaters to extend the signal all the way to the pump. We want those states to be broadcast in close to real time or otherwise accessible to other networked devices so that we don't have to go down to the pump house to get the status of the pump. We don't want to have to configure machines unless absolutely unavoidable - we want this as configureless as possible.
We want this device to have no user interface - no buttons, no screen, nothing. It's in the middle of the woods in a pumphouse. It spews through a USB hosted COM port at best. The configuration should be completely automatic.
We are going to use WiFi to send a one byte multicast UDP packet that contains the pump status every quarter of a second. We will have software on the other end to repeat it. We will use the ATmega2560 CPU to handle the I/O with the pump, and the WiFi module's XDS 160Micro CPU to handle everything else. The two CPUs will communicate with each other over the 4th serial UART on the board.
The UDP WiFi packet will be picked up by either a console app or the Windows GUI app running on any machine on that network. The apps each report a status in their own way, the former by writing it to the console, the latter by placing an icon in the system tray.
In addition to that, the device will also expose an HTTP server that can be used to monitor the status without installing the software. Furthermore, the exposed site will allow you to change the SSID and network password it uses in case you need to because you're about to change it on the router. The device will store the SSID and password in non-volatile flash storage.
In order to make the HTTP server easy to find, we publish the device under a transient domain called pump.local so that it can be accessed using http://pump.local. Note that the server does not support HTTPS. We'll be using simple templated pages stored in the flash memory to source the content.
The domain will be published using Multicast DNS (mDNS) so that the IP doesn't need to be known by client machines.
Topping it all off, we will support WPS so all you need to do to get the box working is provide power and then hit the WPS button on the router.
Most web server libraries for the ESP8266 are not asynchronous but since we need to be sending out UDP packets while we're waiting for HTTP requests, we have to wait asynchronously. This is why we installed that library earlier. Using this we can set up our page handlers and then in loop() we just do our UDP thing.
We use several digital I/O pins on the Arduino board to check the pump status. To access the digital I/O pins, we must do so from the ATmega2560 CPU but our main code is running on the other CPU! Enter the serial port to the rescue. Since the two CPUs are connected by serial, we have a simple serial protocol that looks for a byte of 0xFF/255 from the serial port and then writes the pump status back to the serial port in response. In order to facilitate debug and status messages, every other thing is simply forwarded to the main serial port that's exposed over USB.
We use some of our extra flash memory for storing our html templates and for storing our SSID and network password settings. The SPIFFS filesystem and library provides facilities to read and write the files we need to enable all of this.
We don't want a bunch of a configuration settings to mess with so all you need is WPS. The first time you use the device, you must push the WPS button on the router. Once it is on a network, you can change the settings via http://pump.local/settings. We don't want to have to manage IP addresses on the network so the device uses DHCP as a matter of course. In addition, it exposes a domain via Multicast DNS (mDNS) so that the embedded web server can be accessed by a known moniker (http://pump.local). In order to make the device push status over UDP, we do multicast to reach anyone on the network who is listening rather than a specific computer.
http://pump.local/settings http://pump.local
The software includes two client applications to access the pump status. One is a Windows application that sits in the system tray and one is a console application that logs and should run on Linux as well as Windows as long as Mono is available. Of course, the status can also be reached by HTTP.
There's quite a bit here, so I'm going to break it up.
In order to flash this chip, you must set the onboard dip switch bank such that 1-4 are ON and 5-8 are OFF. You must also set the Board setting under Tools to "Arduino Mega or Mega 2560".
All this CPU will be responsible for is forwarding all serial communication from port 4 to port 1 except if a byte value of 0xFF/255 comes in. If that comes in, it's a special case where the pump status pins are queried and a status byte is returned over the serial port accordingly. This code is pretty simple. Mostly it just goes through I/O pins 2-4 and sees if they're high. Note that we check 3 before 2. The priority of those is deliberately reversed because of how the existing pump logic works.
void setup() {
// initialize the pins
pinMode(2,INPUT);
pinMode(3,INPUT);
pinMode(4,INPUT);
// initialize the serials
Serial.begin(115200);
Serial3.begin(115200);
}
void loop() {
// wait until there's data
while(0==Serial3.available());
// read a byte
int b = (byte)Serial3.read();
// if it's an escape return
// the status
if(255==b) { // read
if(digitalRead(2))
Serial3.write((byte)1);
else if(digitalRead(4))
Serial3.write((byte)3);
else if(digitalRead(3))
Serial3.write((byte)2);
else
Serial3.write((byte)0);
}
else // otherwise just forward
Serial.write((byte)b);
return;
}In order to flash this chip you must set the onboard dip switch bank such that 1-4 are OFF and 5-7 are ON, and 8 is OFF. You must also set the Board setting under Tools to "Generic ESP82666 Module" in the Arduino IDE.
This code is much more involved because it's responsible for all of the magic of this device. It runs the webserver, exposes a domain, works with the filesystem both to serve files and store settings, and communicates with the other CPU to get pump information.
I should note that I adapted this code from some example code for using ESPAsyncWebServer. That code was created and is copyrighted by Rui Santos. The copyright notice is in the .ino file below. It should be noted that the current code bears very little resemblance to the original code. I had to modify almost every aspect of it, but it nevertheless provided me with a good starting point for working with this web server library.
There's quite a bit of code here. In the setup, we do several things:
I'd like to cover one of the handlers below. Specifically, it's the handler that deals with POST /settings.
This actually opens the settings file and rewrites it based on the posted values. The act of rewriting it updates the internal flash memory so it stays when the device is switched off. After it does that, it disconnects. The next time loop() is called, it will try to reconnect with the new credentials. Unfortunately, if they are invalid, the only way to fix it at that point is to upload a new settings file to the device. That means flipping dip switches, and reflashing files which we haven't even covered yet.
/*
Rui Santos
Complete project details at https://RandomNerdTutorials.com
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
*/
// modified extensively by honey the codewitch
// Import required libraries
#include <ESP8266WiFi.h>
#include <ESPAsyncTCP.h>
#include <ESPAsyncWebServer.h>
#include <ESP8266mDNS.h>
#include <WiFiUdp.h>
#include <FS.h>
#ifndef STASSID
// BEGIN configuration properties
#define STASSID "ssid"
#define STAPSK "password"
#define STAHOSTNAME "pump"
#define STAMULTICASTIP IPAddress(239, 0, 0, 10)
#define STAPORT 11000
// END configuration properties
#endif
String process(const String& str);
void saveWiFiConfig(const String& newssid, const String& newpassword);
const char* ssid = STASSID;
const char* password = STAPSK;
char cfgssid[256];
char cfgpassword[256];
WiFiUDP Udp;
// Create AsyncWebServer object on port 80
AsyncWebServer server(80);
void setup() {
// Serial port for debugging purposes
Serial.begin(115200);
// Initialize SPIFFS
if (!SPIFFS.begin()) {
Serial.println("An Error has occurred while mounting SPIFFS");
return;
}
//
// read the settings
//
File file = SPIFFS.open("/settings", "r");
if (!file) {
Serial.println("Failed to open settings file for reading");
return;
}
if (file.available()) {
int l = file.readBytesUntil('\n', cfgssid, 255);
cfgssid[l] = 0;
ssid = cfgssid;
Serial.print("Read SSID: ");
Serial.println(ssid);
if (file.available()) {
l = file.readBytesUntil('\n', cfgpassword, 255);
cfgpassword[l] = 0;
password = cfgpassword;
Serial.println("Read Password: <omitted>");
}
}
//
// initialize the WiFi and connect
//
WiFi.mode(WIFI_STA);
bool done = false;
while (!done) {
// Connect to Wi-Fi
WiFi.begin(ssid, password);
Serial.print("Connecting to WiFi.");
// try this for 10 seconds, then check for WPS
for (int i = 0; i < 20 && WL_CONNECTED != WiFi.status(); ++i) {
Serial.print(".");
delay(500);
}
Serial.println("");
// If we're not connected, wait for a WPS signal
if (WL_CONNECTED != WiFi.status()) {
Serial.println("Connection failed. Entering auto-config mode");
Serial.println("Press the WPS button on your router");
bool ret = WiFi.beginWPSConfig();
if (ret) {
String newSSID = WiFi.SSID();
if (0 < newSSID.length()) {
Serial.println("Auto-configuration successful. Saving.");
saveWiFiConfig(newSSID, WiFi.psk());
strcpy(cfgssid,newSSID.c_str());
strcpy(cfgpassword,WiFi.psk().c_str());
Serial.println("Restarting...");
ESP.restart();
} else {
ret = false;
}
}
} else
done = true;
// if we didn't get connected, loop
}
Serial.println("");
Serial.print("Connected to ");
Serial.println(ssid);
Serial.print("Host name: ");
Serial.print(STAHOSTNAME);
Serial.println(".local");
Serial.print("IP address: ");
Serial.println(WiFi.localIP());
//
// start the multicast DNS publishing
//
if (MDNS.begin(STAHOSTNAME)) {
Serial.println("MDNS responder started");
}
//
// initialize the UDP
//
Udp.begin(STAPORT);
//
// create the HTTP handlers
//
// respond to GET requests on URL /
server.on("/", HTTP_GET, [](AsyncWebServerRequest * request) {
Serial.print("Processing request...");
request->send(SPIFFS, "/index.html", String(), false, process);
Serial.println("Done!");
});
// respond to GET requests on URL /settings
server.on("/settings", HTTP_GET, [](AsyncWebServerRequest * request) {
Serial.print("Processing request...");
request->send(SPIFFS, "/settings.html", String(), false, process);
Serial.println("Done!");
});
// respond to POST requests on URL /settings
server.on("/settings", HTTP_POST, [](AsyncWebServerRequest * request) {
// here we get the ssid and password from the args
const char* newssid = request->arg("ssid").c_str();
const char* newpassword = request->arg("password").c_str();
saveWiFiConfig(request->arg("ssid"), request->arg("password"));
// update the ssid and password
// with the new ones
strcpy(cfgssid, newssid);
Serial.print("SSID: ");
Serial.println(cfgssid);
strcpy(cfgpassword, newpassword);
Serial.print("Password: ");
Serial.println(cfgpassword);
ssid = cfgssid;
password = cfgpassword;
// now disconnect so
// we can reconnect
// with the new
// credentials
WiFi.disconnect();
// turns out unless we do this
// it won't ever reconnect
ESP.restart();
});
//
// start the www server
//
server.begin();
Serial.println("Web server started.");
}In the loop() method, we do a few things, but it's nothing as involved as setup().
void loop() {
// reconnect to the WiFi if we
// got disconnected
if (WL_CONNECTED != WiFi.status()) {
// Connect to Wi-Fi
WiFi.begin(ssid, password);
Serial.print("Connecting to WiFi");
while (WiFi.status() != WL_CONNECTED) {
Serial.print(".");
delay(500);
}
}
// here we use the serial escape 0xFF/255
// to request the status of the pump from
// the ATmega2560
// next we build a UDP multicast packet
// from that, with a lone status byte
// for a payload.
char ba[1];
Serial.write(255);
while (!Serial.available());
ba[0] = (byte)Serial.read();
Udp.beginPacketMulticast(STAMULTICASTIP, STAPORT, WiFi.localIP());
Udp.write(ba, 1);
Udp.endPacket();
// update the DNS information
MDNS.update();
// we only want to do this every
// quarter second
delay(250);
}Now we come to the method we use for processing our page templates. If it sees %PUMP_STATUS% in the file, it will send an escape (0xFF/255) to the serial port and read a status byte back. Depending on that value, we convert it to a friendly name, and return that. Otherwise, we return the SSID or network password as requested. For anything else, we return an empty string:
// this method replaces %TEMPLATE%
// values in an otherwise static
// file. There are a few different
// aliases
String process(const String& str) {
if (str == "PUMP_STATUS") {
Serial.write(255);
while (!Serial.available());
byte s = (byte)Serial.read();
char* sz;
switch (s)
{
case 0:
sz = "No power";
break;
case 1:
sz = "Ready";
break;
case 2:
sz = "Requesting water";
break;
case 3:
sz = "Pumping";
break;
default:
sz = "Error";
break;
}
return String(sz);
} else if(str=="SSID")
return String(ssid);
else if(str=="PASSWORD")
return String(password);
return String();
}Speaking of templates, we should visit our templates we use to render content. We'll start with the main landing page/status page. This is very simple. You can see the %PUMP_STATUS% alias we covered and you may have noticed that we tell the page to refresh once every second. This isn't ideal. Ideally, we'd use something like jsonp and just refresh an element in the page but it wasn't really worth the effort considering this is the least used way to access the device in my particular scenario. We do need an ongoing update for it to be useful under every circumstance. The auto-refresh does the job, if somewhat awkwardly.
<!doctype html>
<html lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8" />
<meta http-equiv="refresh" content="1" />
<title>Pump Monitor</title>
</head>
<body>
<p>Status: %PUMP_STATUS%</p>
</body>
</html>Next, we have the settings page template. There is no security on this. The security is in the 20 acres of woods which the house is in the middle of. If someone really wants to sit there with the bears and try to hack WPA2 frankly they need a hobby. If your situation is different you could set up a password and make it so it can be changed. I didn't bother in this project, as it would just annoy the client (my in laws) anyway.
<!doctype html>
<html>
<head>
<title>WiFi Settings</title>
</head>
<body>
<form action="" method="POST">
<p>SSID:</p>
<input name="ssid" type="text" value="%SSID%" /><br />
<p>Password:</p>
<input name="password" type="password" value="%PASSWORD%" /><br />
<input type="submit" value="Save" />
</form>
</body>
</html>You may be wondering at this point how to get those files onto the device. What you need to do is under the pumpMonitorEsp sketch folder, put a data folder, and then put index.html, settings.html, and settings under it. Then you need to go into Tools and select ESP Sketch Data Upload. Make sure before you do that that you put your SSID and password in the settings file! Also remember to set things up to program your ESP8266 as outlined earlier. If the serial monitor is open, it will interfere with the upload, so close it.
The pumpmon tool continuously reports pump status changes as they happen. Each change has a timestamp suitable for logging. It only expects packets once a second despite status changes being every quarter second. I've found that this is the best ratio to avoid hiccups in the status reporting. Every time a status packet comes in, a timer is set for one second. If that timer triggers, the status is changed to "Connecting". This essentially works as a timeout. The C# code is relatively brief.
const int TimeoutMS = 1000;
const int Port = 11000;
public static void Main(string[] args)
{
Console.Error.WriteLine("Press any key to exit.");
var dt = DateTime.Now;
// build a timestamp
var ts = "[" + dt.ToShortDateString() + " " + dt.ToShortTimeString() + "] ";
// report the initial status as connecting
Console.WriteLine(ts + "Connecting");
// we're going to do this in the background:
ThreadPool.QueueUserWorkItem((state) => {
var status = -1; // connecting
UdpClient uc = new UdpClient(Port);
// we need to join to make sure we recieve the packets
uc.JoinMulticastGroup(new IPAddress(new byte[] { 239, 0, 0, 10 }));
// this timer routine gets called if a packet hasn't
// been seen for at least one second:
var timer = new Timer((state2) => {
if (-1 != status)
{
dt = DateTime.Now;
// build a timestamp
ts = "[" + dt.ToShortDateString() + " " + dt.ToShortTimeString() + "] ";
Console.WriteLine(ts + "Connecting");
}
status = -1;
}, null, TimeoutMS, Timeout.Infinite);
// keep the thread alive and looking for packets:
while (true)
{
var remoteEP = new IPEndPoint(IPAddress.Any, Port);
var data = uc.Receive(ref remoteEP);
dt = DateTime.Now;
// build a timestamp
ts = "[" + dt.ToShortDateString() + " " + dt.ToShortTimeString() + "] ";
// translate the status code to text
// and reset the timer:
if (1 == data.Length)
{
switch (data[0])
{
case 0:
timer.Change(TimeoutMS, Timeout.Infinite);
if (0 != status)
Console.WriteLine(ts + "No power");
status = 0;
break;
case 1:
timer.Change(TimeoutMS, Timeout.Infinite);
if (1 != status)
Console.WriteLine(ts + "Ready");
status = 1;
break;
case 2:
timer.Change(TimeoutMS, Timeout.Infinite);
if (2 != status)
Console.WriteLine(ts + "Requesting water");
status = 2;
break;
case 3:
timer.Change(TimeoutMS, Timeout.Infinite);
if (3 != status)
Console.WriteLine(ts + "Pumping");
status = 3;
break;
}
}
}
});
// wait for a keypress to exit
Console.ReadKey();
}This application sits in the windows system tray and displays different colored icons depending on the state of the pump. It also logs, like pumpmon does. In fact, aside from GUI glue, the application is nearly identical to pumpmon so I'm not even going to cover it here. The only weird thing going on is the dynamic creation of icons out of bitmap images that get drawn on the fly. The reason I did that is my father in law is colorblind and I didn't want to risk making icons he couldn't see that well, so I made circles of colors he signed off on and I left it at that.