Updated on 2020-11-26
Efficiently send large amounts of data with a tiny amount of memory
Recently, for a project I am developing, I had to log a significant amount of data to an IoT device's flash and then batch send the data to a bulk uploader for a JSON based REST server hosted by thingspeak.com. The data would often be far larger than the available RAM which is why I was storing it in a log in flash in the first place. JSON only compounds the issue because while compact, it's much larger than the binary log's footprint for the same data. A solution was needed that would allow me to upload this large set of data over HTTP without running out of RAM and crashing my device.
For all their power, ESP32 devices have just 520kB of RAM to work with. Its predecessor, the ESP8266 has a miserly 80kB for user use. This doesn't leave a lot of room for sending data. Any dynamic content of a non-trivial size must be streamed rather than loaded into memory before it is sent. This creates something of a problem for traditional HTTP transfers because of the need for a Content-Length header, itself necessary because of how TCP works. In order to work around this problem, chunked transfer encoding was devised. It's a layer of transport protocol over HTTP that sends data in small units or "chunks", each of a known length. This avoids the need for the Content-Length header and allows for streaming dynamically generated content over HTTP.
Unfortunately, the core libraries for the ESP32 and ESP8266 do not include chunked transfer support. We must roll our own if we need it. In this article, we do exactly that.
It should be noted that while this code is primarily for the ESP family, it will work with some slight modification with pretty much any Arduino compliant SoC device.
As I said, chunked transfer encoding is essentially a protocol layer on top of HTTP. Instead of a Content-Length header, you need a Transfer-Encoding: chunked header. This will signal to the receiver that they should expect the data in chunks.
Each chunk meanwhile is simply a value in hex that specifies the length of the chunk followed by a carriage return and line feed, and then followed by the data of the specified value's length and another carriage return/line feed combo. The final chunk is simply a zero length chunk. The chunk size value does not include the carriage returns or line feeds - only the payload itself. Here's an example:
5
Hello
7
World!
0This sends "Hello World!" to the receiver. This is how we reduce memory requirements and allow for streaming of dynamically generated content. Now all we need is enough room for a chunk at a time in RAM instead of the whole document.
First of all, before we get to the code, in C++ I will use classes, but I often favor procedural code for simple things, including this project. I will leave making this OO as an exercise for you dear reader, if you so desire to do so.
We'll cover the code for emitting a chunk as described previously:
// write an HTTP chunked fragment to the network client
void httpWriteChunked(const char* sz) {
int cl = (sz) ? strlen(sz) : 0;
if (0 < cl) {
Serial.print(sz);
char szt[1024];
sprintf(szt, "%x\r\n%s\r\n", cl, sz);
_client.print(szt);
} else {
Serial.println(F("<chunk terminator>"));
_client.print("0\r\n\r\n");
}
}This is pretty straightforward as long as you're familiar with printf/sprintf formatting strings. Basically, we check for a null string and then get the string length (or 0 if null) and write it out, followed by a CRLF followed by the data followed by yet another CRLF. If it's a final chunk (null or empty string was passed), we indicate that by writing 0 followed by two CRLFs.
Now we get serious. The setup() routine basically does everything else and there's quite a bit to it:
void setup() {
Serial.begin(115200);
WiFi.begin(SSID,PASSWORD);
for(int i = 0;i<30 && WL_CONNECTED!=WiFi.status();++i) {
delay(500);
}
if(WL_CONNECTED!=WiFi.status()) {
Serial.println(F("Could not connect to WiFi network"));
while(true);
}
// get the current time from the NTP server
DateTime.setServer(NTP_SERVER);
DateTime.begin();
if (!DateTime.isTimeValid()) {
Serial.println("Could not fetch Internet time");
while(true);
}
long int dnow = DateTime.utcTime();
if (!_client.connect(REST_SERVER, 80))
{
Serial.println(F("Could not connect to server"));
while(true);
}
char sz[1536]; // 1.5kb
// build the request
sprintf_P(sz, PSTR("POST %S HTTP/1.1\r\nHost: %S\r\n"), REST_PATH, REST_SERVER);
strcat_P(sz, PSTR("Accept: application/json\r\n"));
strcat_P(sz, PSTR("Content-Type: application/json\r\n"));
strcat_P(sz, PSTR("Transfer-Encoding: chunked\r\nConnection: close\r\n\r\n"));
_client.print(sz);
httpWriteChunked("{\"write_api_key\":\"YCKKPCFMQTDQKGHK\",\"updates\":[");
for(int i = 0; i < 5; ++i) {
String str;
// back "date" each response by 1 second for testing
time_t tts = (time_t)(dnow);
DateTimeClass dt(tts-(5-i));
if (0<i)
str = dt.format(",{\"created_at\":\"%Y-%m-%d %H:%M:%S +0000\",\"field1\":\"");
else {
str = dt.format("{\"created_at\":\"%Y-%m-%d %H:%M:%S +0000\",\"field1\":\"");
}
strcpy(sz,str.c_str());
char szn[32];
strcat(sz,itoa(i,szn,10));
strcat(sz,"\"}");
httpWriteChunked(sz);
}
httpWriteChunked("]}");
httpWriteChunked(NULL); // terminator
// now read the response
String line = _client.readStringUntil('\n');
if (0 != strncmp("HTTP/1.1 202 ", line.c_str(), 13))
{
Serial.println();
Serial.println(F("HTTP request failed:"));
Serial.println(line);
}
while (_client.connected()) {
line = _client.readStringUntil('\n');
if (line == "\r") {
// once we read the headers, terminate
// we don't need the rest
Serial.println(F("Success! Visit https://thingspeak.com/channels/1243886 for data"));
}
}
_client.stop();
}The first thing we do after initializing the serial port is begin connecting to the WiFi network.
Once that succeeds, we need a timestamp to send to Thingspeak but there is no clock to draw from so we go to an Internet time (NTP) server to get the current time.
Next, we connect to the Thingspeak server and begin sending data, starting with the request line and some headers. This shouldn't take more than 1.5kB, so that is our buffer size. We end up using a bit more than that all told because of the date formatting mostly but it's marginal.
As soon as we write the headers. we send our first chunk which is the very beginning of the JSON data set we're sending to the server. It's basically a preamble of sorts that primarily exists to carry our API key tagged to the main payload.
Next, we write 5 entries in a loop, using chunked transfer encoding to send them 1 entry at a time. This way, we do not need more than our 1.5kB, which is key to this demonstration. 1.5kB is kind of arbitrary. You should select a value that balances chunk size (bigger is better) with memory requirements.
Note that in each entry, we're manufacturing a timestamp. We create timestamps for the current time and the previous 4 seconds. This is because Thingspeak displays one datapoint per field per second at maximum so we just make the server believe that we made these entries starting 5 seconds ago.
Now we send our final JSON termination sequence and the null chunk terminator.
After that, we can begin reading our response. Honestly, we don't care about the response beyond the HTTP status line, because that tells us all we need to know in this case - whether the operation succeeded or not. Note that there's a bug I haven't run down where sometimes it will claim the request failed when it actually succeeded because it reads a blank line instead of the status line. I'm not sure why that is yet but it's not that important for this demonstration.
That's all there is to it. You can use the same method to send data if you're running an HTTP server instead of a client. Happy coding!
Currently, the code will sometimes read a blank line where it expected the HTTP status line and so it will report a failure when the request may have succeeded. I haven't run this down because it didn't have a lot to do with the chunked transfer encoding itself.