Updated on 2020-11-10
An advanced network connected clock that puts the Arduino through its paces
Sometimes, there's more to a device than meets the eye. We're using a clock as an excuse to explore some advanced techniques for IoT devices, even the humble and ironically named Mega. We'll be putting together a clock based on some tutorials, and then writing some witchy code to give it some magic. I'll also be providing you with some libraries that ease the automatic network configuration portion in your own apps.
You'll need an Arduino Mega 2560+WiFi R3.
You'll need a DS1307 or similar Real-time Clock.
You'll need a DHT11 temperature and humidity sensor.
You'll need a 16x2 LCD display with a hitachi interface.
You'll need the usual wires and prototype board.
You'll need the Arduino IDE. You must go to File|Preferences and add the following to the Additional Board Managers text box:
If there is already a URL present, delimit them with commas.
Now get yourself the latest ESP8266FS zip file here.
If you're on Linux, you'll want to get the ESP8266FS folder in the zip, and find your Arduino application directory. That should be off your home directory. Mine is ~/arduino-1.8.13. Do not get it mixed up with ~/Arduino. Under there, there is a tools folder and that's where you want your ESP8266FS folder to go.
I've never done it on Windows, but this is how you do it: You'll need the ESP8266FS folder from the zip. Find your program directory. It is probably something like C:\Program Files (x86)\arduino-1.8.13. Inside, there is a tools folder. That is where the ESP8266FS folder needs to go.
Either way, you'll need to restart the Arduino IDE. You'll know it took if you now have a Tools|ESP8266 Sketch Data Upload option.
You'll also need to install all of the libraries in ESP8266AutoConfLibs.zip. After unzipping this, you can go to Sketch|Include Library|Add Zip Library... and select each of the zips to install them.
This is the most involved part of the project, and the main reason I created this project and article, so we're going to spend some time with this subtopic in particular.
My IoT devices typically autoconfigure their WiFi connections and support WPS. What I mean is my devices don't require anything to be done to them other than to be switched on. Once the WPS button on the router is pressed, they will connect. This is relatively straightforward for devices where a network connection is required because we can stall the rest of the application until the connection and configuration takes place, but what about an application like a clock that needs to run continuously while ideally looking for Internet connections or WPS signals in the background? With the default ESP8266WiFi library, this is pretty much impossible.
Here are the steps we take to connect to a network:
In the foreground where we block until the process completes, this is relatively straightforward as I said. It's considerably more complicated when it must be done in the background. The ESP8266WiFi library's beginWPSconfig() is synchronous, so I struggled for awhile before making an asynchronous version like begin() is. My library is called ESP8266AsyncWiFi and the aforementioned method is the only behavior that I changed. When I first tried this, I tried to do it without forking the WiFi library but the result ended up randomly crashing because I couldn't call some private methods that made it work.
That wasn't the end of the mess however. In order to make all the timeouts work and everything without using delay(), and also to manage the different transitions between the steps above, I built a state machine that is called from loop(). The state machine moves from one state to the next as the connection and WPS negotiation process is navigated. There are two timeouts involved - one for the connection and one for the WPS search. Basically, we just go back and forth between trying to connect and trying to find a WPS signal but we do so in a way that doesn't block. It's not pretty - state machines rarely are, but at least it's not more complicated than it needs to be.
I've turned all of my autoconfiguration stuff into several libraries which I've included with this distribution. You can install each of the zips using the Arduino IDE by going to Sketch|Include Library|Add Zip Library...
How to pick which of the AutoConf libraries is right for your project:
That's just a general guide to using them. We'll be selecting #3 for our project since we don't require a network connection in order to function and because we need a filesystem. This way, our clock can serve a small website and webservice while automatically searching for a WPS signal or a usable WiFi connection in the background.
I've basically cobbled together the hardware from several example projects, here, here, and here. I'll expect you can follow them, and combine them onto one prototype board. Remember on the Mega when you go to attach the clock, you'll want to set it to the second I2C interface (SDA20/SCL21), not the first set. You'll also want the DHT sensor's S line to be plugged into A0 and the corresponding code will need to be updated to change the pin to A0 in the code. If your clock isn't a DS1307, you'll need to adjust your code and wiring accordingly.
Once you have it wired up and tested with some throwaway code, we can get to the meat. Here's some throwaway code for testing:
#include <LiquidCrystal.h>
#include <dht.h>
#include <RTClib.h>
RTC_DS1307 RTC;
dht DHT;
float _temperature;
float _humidity;
#define DHT11_PIN A0
// initialize the library by providing the nuber of pins to it
LiquidCrystal LCD(8, 9, 4, 5, 6, 7);
void setup() {
Serial.begin(115200);
LCD.begin(16, 2);
pinMode(DHT11_PIN, INPUT);
if (! RTC.begin()) {
Serial.println(F("Couldn't find RTC"));
while (true);
}
RTC.adjust(DateTime(__DATE__, __TIME__));
}
void loop()
{
int chk = DHT.read11(DHT11_PIN);
float f = DHT.temperature;
if (-278 < f) {
_temperature = f;
_humidity = DHT.humidity;
}
DateTime now = RTC.now();
char sz[16];
memset(sz,' ',16);
sprintf(sz, "%d:%02d:%02d",
now.hour(),
now.minute(),
now.second());
int c = strlen(sz);
if(c<16)
sz[c]=' ';
LCD.setCursor(0,0);
LCD.print(sz);
LCD.setCursor(0, 1);
memset(sz,' ',16);
if (1==(millis() / 1000L) % 2) {
sprintf(sz,"Temp: %dC/%dF",(int)_temperature,(int)((int)_temperature * 1.8) + 32);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
} else {
sprintf(sz,"Humidity: %d",(int)_humidity);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
}
LCD.print(sz);
}
That should display the time, temperature, and humidity on the display if everything is working.
This processor will be responsible for managing the LCD output and communicating the clock and sensor readings with the XDS 160Micro processor on the WiFi module. We'll be using that processor to do almost all the hard work because it's much more capable and it's got WiFi basically built right in instead of needing to be accessed over serial. We will be using a serial port though but just to get sensor data and clock information back and forth. For the most part, aside from updating the clock's display, this simply listens on a serial port for an incoming 255 value. If it gets that, it reads an opcode next, and then command bytes which depend on the opcode. Any other value gets forwarded to the serial port that's exposed via USB.
This processor will be responsible for negotiating the WiFi network, running the webserver, communicating with an NTP server, and setting the clock when necessary. Any time it needs sensor or clock information it must query the ATMega2560 over a serial line. It does this by sending the byte 255, followed by 0 and then receives all of the clock and sensor data. If it sends a 255 followed by a 1 and then a 32-but number representing the Unix time, it will set the clock. It runs a web server at http://clock.local which will present the time, temperature and humidity. It runs a JSON service at http://clock.local/time.json.
http://clock.local http://clock.local/time.json
This facility is what automatically scans for WiFi and WPS, and manages the SSID and password stored in the device.
Before we dive into what makes it work, let's look at how to use it:
#include <ESP8266AsyncAutoConfFS.h> // header chosen from #3 above
In our setup() method:
ESPAsyncAutoConf.begin();
If you weren't using the async version, you'd invoke ESPAutoConf.begin() instead.
In the loop() method:
ESPAsyncAutoConf.update();
// your loop code here.
Similarly as above, if you weren't using the async versions, you'd refer to ESPAutoConf instead.
You can see if you're connected to the network as normal:
if(WL_CONNECTED == WiFi.status()) Serial.println("Connected!");
With the synchronous versions after the update() call, you will always be connected. For the asynchronous versions, they very well may not be connected after update() is called.
Remember to always check whether you're connected before you do a network operation when using the asynchronous libraries!
That's about it for using it. Let's dive into how it was made.
I've already covered the mechanics of the synchronous versions of these libraries here. The code in the libraries is updated a bit since the article but the concept hasn't changed. I will be focusing on the asynchronous configuration process in this article.
Most of the meat of these asynchronous libraries is in update() which is typically called from loop():
#include "ESP8266AsyncAutoConfFS.h"
_ESPAsyncAutoConf ESPAsyncAutoConf;
void _ESPAsyncAutoConf::begin() {
SPIFFS.begin();
int i = 0;
// Read the settings
_cfgssid[0] = 0;
_cfgpassword[0] = 0;
File file = SPIFFS.open("/wifi_settings", "r");
if (file) {
if (file.available()) {
int l = file.readBytesUntil('\n', _cfgssid, 255);
_cfgssid[l] = 0;
if (file.available()) {
l = file.readBytesUntil('\n', _cfgpassword, 255);
_cfgpassword[l] = 0;
}
}
file.close();
}
// Initialize the WiFi
WiFi.mode(WIFI_STA);
_wifi_timestamp = 0;
_wifi_conn_state = 0;
}
void _ESPAsyncAutoConf::update() {
// connect, reconnect or discover the WiFi
switch (_wifi_conn_state) {
case 0: // connect
if (WL_CONNECTED != WiFi.status())
{
if (0 < strlen(_cfgssid)) {
Serial.print(F("Connecting to "));
Serial.println(_cfgssid);
if (WiFi.begin(_cfgssid, _cfgpassword)) {
// set the state to connect
// in progress and reset
// the timestamp
_wifi_conn_state = 1;
_wifi_timestamp = 0;
}
} else {
// set the state to begin
// WPS and reset the
// timestamp
_wifi_conn_state = 2;
_wifi_timestamp = 0;
}
}
break;
case 1: // connect in progress
if (WL_CONNECTED != WiFi.status()) {
if (!_wifi_timestamp)
_wifi_timestamp = millis();
else if (20000 <= (millis() - _wifi_timestamp)) {
Serial.println(F("Connect attempt timed out"));
// set the state to begin
// WPS and reset the
// timestamp
_wifi_conn_state = 2;
_wifi_timestamp = 0;
}
} else {
Serial.print(F("Connected to "));
// store the WiFi configuration
Serial.println(WiFi.SSID());
strcpy(_cfgssid,WiFi.SSID().c_str());
strcpy(_cfgpassword,WiFi.psk().c_str());
File file = SPIFFS.open("/wifi_settings", "w");
if (file) {
file.print(_cfgssid);
file.print("\n");
file.print(_cfgpassword);
file.print("\n");
file.close();
}
// set the state to connected
_wifi_conn_state = 4;
_wifi_timestamp = 0;
}
break;
case 2: // begin wps
Serial.println(F("Begin WPS search"));
if (WL_CONNECTED != WiFi.status()) {
WiFi.beginWPSConfig();
// set the state to WPS in
// progress
_wifi_conn_state = 3;
_wifi_timestamp = 0;
}
break;
case 3: // wps in progress
if (WL_CONNECTED != WiFi.status()) {
if (!_wifi_timestamp)
_wifi_timestamp = millis();
else if (30000 <= (millis() - _wifi_timestamp)) {
Serial.println(F("WPS search timed out"));
// set the state to connecting
_wifi_conn_state = 0;
_wifi_timestamp=0;
}
} else {
// eventually goes to 4:
_wifi_conn_state = 1;
_wifi_timestamp = 0;
}
break;
case 4:
if (WL_CONNECTED != WiFi.status()) {
// set the state to connecting
_wifi_conn_state = 0;
_wifi_timestamp = 0;
}
break;
}
}
You may notice that this is a state machine. We've covered what it does in the conceptualization section prior. The logic is a bit messy but necessarily so in order to handle all the cases. I love state machines in concept but not so much in practice because they can be difficult to read. However, sometimes they are just the right tool for the job.
The whole idea of this routine is to break up the process of connecting, scanning WPS, connecting, scanning WPS ad nauseum into a coroutine - a "restartable method" - that's where the state machine comes in. Each time loop() is called, we pick up where we left off last time because we're tracking the state with _wifi_conn_state. It's not really the most self explanatory code but if you pay close attention, you can follow it.
coroutine - a "restartable method"
Here is the processing code for the ATmega2560 CPU which we've already covered prior, so let's get to the code:
#include <dht.h>
#include <RTClib.h>
#include <LiquidCrystal.h>
// make sure S is on analog 0
#define DHT11_PIN A0
// these unions make it
// easy to convert
// numbers to bytes
typedef union {
float fp;
byte bin[4];
} binaryFloat;
typedef union {
uint32_t ui;
byte bin[4];
} binaryUInt;
float _temperature;
float _humidity;
dht DHT;
RTC_DS1307 RTC;
LiquidCrystal LCD(8, 9, 4, 5, 6, 7);
void setup() {
// initialize everything
Serial.begin(115200);
Serial3.begin(115200);
pinMode(DHT11_PIN,INPUT);
if (! RTC.begin()) {
Serial.println(F("Couldn't find the clock hardware"));
while (1);
}
if (! RTC.isrunning())
Serial.println(F("The clock is not running!"));
LCD.begin(16, 2);
}
void loop() {
// update the temp and humidity
// note that sometimes the DHT11
// will return -999s for the
// values. We check for that.
int chk = DHT.read11(DHT11_PIN);
float tmp = DHT.temperature;
if (-278 < tmp) {
_temperature = tmp;
_humidity = DHT.humidity;
}
DateTime now = RTC.now();
// format our time and other
// info to send to the LCD
char sz[16];
memset(sz,' ',16);
sprintf(sz, "%d:%02d:%02d",
now.hour(),
now.minute(),
now.second());
int c = strlen(sz);
if(c<16)
sz[c]=' ';
LCD.setCursor(0,0);
LCD.print(sz);
LCD.setCursor(0, 1);
memset(sz,' ',16);
if (1==(millis() / 2000L) % 2) {
sprintf(sz,"Temp: %dC/%dF",(int)_temperature,(int)((int)_temperature * 1.8) + 32);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
} else {
sprintf(sz,"Humidity: %d",(int)_humidity);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
}
LCD.print(sz);
// now wait for incoming serial data
if (Serial3.available()) {
byte b = Serial3.read();
// if it's not our escape byte
// of 255, just forward it
if (255 != b)
{
Serial.write(b);
return;
}
b = Serial3.read();
switch (b) {
case 0: // get unixtime, temp and humidity
// read one int32 and two floats from the
// serial line
binaryUInt data;
data.ui = RTC.now().unixtime();
Serial3.write(data.bin, 4);
binaryFloat data2;
data2.fp = _temperature;
Serial3.write(data2.bin, 4);
data2.fp = _humidity;
Serial3.write(data2.bin, 4);
break;
case 1: // set clock
// read an int32 and set the
// clock
Serial3.readBytes((char*)data.bin,4);
RTC.adjust(DateTime(data.ui));
break;
}
}
}
Remember to set your dips to 1-4 ON and 5-8 OFF. Also select the "Arduino Mega or Mega 2560" from the boards menu before flashing the above code.
Here's the code for the XDS Micro160. As you can see, this is a lot more involved. This is due to all the features of the clock more than anything.
#include <ESP8266AsyncAutoConfFS.h>
#include <ESPAsyncTCP.h>
#include <ESPAsyncWebServer.h>
#include <ESP8266mDNS.h>
#include <WiFiUdp.h>
// make it easier to convert
// between numbers and bytes:
typedef union {
float fp;
byte bin[4];
} binaryFloat;
typedef union {
uint32_t ui;
byte bin[4];
} binaryUInt;
// the host name for the webserver
#define HOSTNAME "clock"
// local port to listen for UDP packets
unsigned int localPort = 2390;
// time.nist.gov NTP server address
IPAddress timeServerIP;
const char* ntpServerName = "time.nist.gov";
// NTP time stamp is in the first 48 bytes of the message
const int NTP_PACKET_SIZE = 48;
//buffer to hold incoming and outgoing packets
byte packetBuffer[ NTP_PACKET_SIZE];
// A UDP instance to let us send and receive packets over UDP
WiFiUDP udp;
unsigned long _ntp_timestamp;
unsigned long _mdns_timestamp;
bool _net_begin;
AsyncWebServer server(80);
void sendNTPpacket(IPAddress& address);
String process(const String& arg);
void setup() {
_ntp_timestamp = 0;
_mdns_timestamp = 0;
_net_begin = false;
Serial.begin(115200);
ESPAsyncAutoConf.begin();
// web handlers
server.on("/", HTTP_GET, [](AsyncWebServerRequest * request) {
request->send(SPIFFS, F("/index.html"), String(), false, process);
});
server.on("/time.json", HTTP_GET, [](AsyncWebServerRequest * request) {
request->send(SPIFFS, F("/time.json"), String(), false, process);
});
}
void loop() {
if (WL_CONNECTED != WiFi.status()) {
_mdns_timestamp = 0;
_net_begin = false;
}
// give the clock a chance to connect
// to the network:
ESPAsyncAutoConf.update();
// check if we're connected
if (WL_CONNECTED == WiFi.status()) {
// the first time we connect,
// start the services
if (!_net_begin) {
_net_begin = true;
MDNS.begin(F(HOSTNAME));
server.begin();
udp.begin(localPort);
Serial.print(F("Started http://"));
Serial.print(F(HOSTNAME));
Serial.println(F(".local"));
}
// now send an NTP packet every 5 minutes:
if (!_ntp_timestamp)
_ntp_timestamp = millis();
else if (300000 <= millis() - _ntp_timestamp) {
WiFi.hostByName(ntpServerName, timeServerIP);
sendNTPpacket(timeServerIP); // send an NTP packet to a time server
_ntp_timestamp = 0;
}
// update the MDNS responder every second
if (!_mdns_timestamp)
_mdns_timestamp = millis();
else if (1000 <= millis() - _mdns_timestamp) {
MDNS.update();
_mdns_timestamp = 0;
}
// if we got a packet from NTP, read it
if (0 < udp.parsePacket()) {
udp.read(packetBuffer, NTP_PACKET_SIZE); // read the packet into the buffer
//the timestamp starts at byte 40 of the received packet and is four bytes,
// or two words, long. First, esxtract the two words:
unsigned long highWord = word(packetBuffer[40], packetBuffer[41]);
unsigned long lowWord = word(packetBuffer[42], packetBuffer[43]);
// combine the four bytes (two words) into a long integer
// this is NTP time (seconds since Jan 1 1900):
unsigned long secsSince1900 = highWord << 16 | lowWord;
// Unix time starts on Jan 1 1970. In seconds, that's 2208988800:
const unsigned long seventyYears = 2208988800UL;
// subtract seventy years:
unsigned long epoch = secsSince1900 - seventyYears;
// use the data to set the clock
Serial.write((byte)255);
Serial.write((byte)1);
binaryUInt data;
data.ui = epoch;
Serial.write((char*)data.bin, 4);
}
}
}
String process(const String& arg)
{
// replace template parameters in
// the web page with actual values
Serial.write((byte)255);
Serial.write((byte)0);
binaryUInt data;
Serial.read((char*)data.bin, 4);
unsigned long epoch = data.ui;
binaryFloat dataf;
Serial.read((char*)dataf.bin, 4);
float tmp = dataf.fp;
Serial.read((char*)dataf.bin, 4);
float hum = dataf.fp;
if (arg == "TIMESTAMP") {
char sz[256];
// print the hour, minute and second:
sprintf(sz, "%d:%02d:%02d", (epoch % 86400L) / 3600,
(epoch % 3600) / 60,
epoch % 60
);
return String(sz);
} else if (arg == "TEMPERATURE") {
char sz[256];
sprintf(sz, "%f",tmp);
return String(sz);
} else if(arg=="HUMIDITY") {
char sz[256];
sprintf(sz, "%f",hum);
return String(sz);
}
return String();
}
// send an NTP request to the time server at the given address
void sendNTPpacket(IPAddress& address) {
Serial.println("sending NTP packet...");
// set all bytes in the buffer to 0
memset(packetBuffer, 0, NTP_PACKET_SIZE);
// Initialize values needed to form NTP request
// (see URL above for details on the packets)
packetBuffer[0] = 0b11100011; // LI, Version, Mode
packetBuffer[1] = 0; // Stratum, or type of clock
packetBuffer[2] = 6; // Polling Interval
packetBuffer[3] = 0xEC; // Peer Clock Precision
// 8 bytes of zero for Root Delay & Root Dispersion
packetBuffer[12] = 49;
packetBuffer[13] = 0x4E;
packetBuffer[14] = 49;
packetBuffer[15] = 52;
// all NTP fields have been given values, now
// you can send a packet requesting a timestamp:
udp.beginPacket(address, 123); //NTP requests are to port 123
udp.write(packetBuffer, NTP_PACKET_SIZE);
udp.endPacket();
}
Remember to set your dips such that 1-4 are OFF, 5-7 are ON, and 8 is OFF. Select the "Generic ESP8266" from the boards menu before flashing the code. After flashing the code, make sure you upload the data directory to your flash memory as well. When you're done flashing, set the dips back such that 1-4 are ON and 5-8 are OFF.
Note that our "webservice" above is just a templated JSON file:
{
"time": "%TIMESTAMP%",
"temperature": %TEMPERATURE%,
"humidity": %HUMIDITY%
}
Those % delimited values are resolved using the process() routine from above. It's a shameless way to make a dynamic webservice, but it works, and keeps our poor little CPU from having to do a bunch of JSON string concatenations.
Those get fetched using the following HTML and JavaScript. Please excuse my mess:
<!doctype html>
<html>
<head>
<title>Clock</title>
</head>
<body>
<span>Time:</span><span id="TIME">Loading</span><br />
<span>Temp:</span><span id="TEMPERATURE">Loading</span><br />
<span>Humidity:</span><span id="HUMIDITY">Loading</span>
<script>
function askTime() {
var xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function () {
if (
this.readyState == 4 &&
(this.status == 0 || this.status == 200)
) {
var obj = JSON.parse(this.responseText);
document.getElementById('TIME').innerHTML = obj['time'];
var far = obj['temperature'] * 1.8 + 32;
document.getElementById('TEMPERATURE').innerHTML =
Math.round(obj['temperature']) + 'C/' + Math.round(far) + 'F';
document.getElementById('HUMIDITY').innerHTML = Math.round(
obj['humidity'],
);
}
};
xmlhttp.open('GET', 'time.json', true);
xmlhttp.send();
setTimeout(function () {
askTime();
}, 1000);
}
askTime();
</script>
</body>
</html>
Nothing to see here, it's just the classic JSON based AJAX technique stripped of every fancy JS framework and down to the metal.