Updated on 2020-12-06
A rundown of some common tricks and pitfalls when working with SD readers for IoT gadgets
These days, micro SD cards are a popular way to store data. They are relatively durable, and store a large amount of data in a small form factor. Many smartphones and most cameras accept micro SD cards. So can your little device. However, using them isn't always straightforward. In this article, I will present you with a few common problems and solutions for using these card readers.
I'm not covering a great deal about wiring here. The idea is to wire the SD module into your board's SPI bus, and then wire up the power. Just make sure you wire your correct power input as some modules take 3.3VDC like the card itself and some take 5VDC and have an onboard voltage regulator. Check your module's datasheet before you blow it up. Honestly, they're so cheap you could just try it on 3.3VDC and if it doesn't work, move it to 5. The worst you do is blow it and/or the card.
For the second half of this, I'm covering an ESP32 specific feature. To follow along there, make the following connections:
SDM CS ➜ ESP32 GPIO4 SDM SCK ➜ ESP32 GPIO13 SDM MOSI ➜ ESP32 GPIO17 SDM MISO ➜ ESP32 GPIO16 SDM VCC ➜ ESP32 VIN (+5vdc) ** other boards may use 3.3vdc! SDM GND ➜ ESP32 GND
The left side (SDM) is your SD reader module. The right side is your ESP32 development board. Actual pinouts vary so make sure you use your board's pinout map. The GPIO numbers are always the same regardless of the board, as are VIN and GND. The biggest variable, again, is what power your SD module takes.
Several of the pitfalls you run into with these devices start with lack of card change detection. That is, there's just no way of detecting when a card has been inserted and removed. The fundamental problem is that the card itself acts as an SPI device but SPI has no facility to support device change detection. While it's possible, and even cheap and easy to add such detection to such a reader using an additional pin, I have yet to see an Arduino compatible module that supports it out of the box. You'd have to add your own microswitch somehow. That's difficult. Fortunately, there are software solutions to the bigger problems that one can encounter because of this.
Say it with me: "The card is the device" - when you remove a card, you must totally reset your library. Since we don't know when a card is removed, we must assume every time that the device is a new one.
The process for reading and writing therefore, looks like this:
Here's what it looks like - char* path is the path, and char *sz is the text:
// initialize SD
if(!SD.begin()){
Serial.println(F("MicroSD Card Mount Failed"));
return -1;
}
// write the file
File file = SD.open(path,FILE_APPEND);
if(!file) {
Serial.println("Open file failed!");
return -1;
}
size_t c = strlen(sz);
size_t result = file.write((uint8_t*)sz,c);
if(c!=result)
Serial.println("Write failed!");
file.close();
// make sure all files are closed here
SD.end();
return result;Here, we're appending the passed in string to the file. We can use this code as an example of a logging function. Every time it is called, the new string is appended to the file. If the card is removed and reinserted it will not break. If the card is changed for another card, a new log will be created from there. If no SD is inserted, it will not write anything. This is fairly robust. In the next section, we make it even more so.
With IoT devices, there isn't a whole lot of RAM so if you're taking in data at intervals, you're probably going to be logging it to a file as you go as suggested above. The problem with this is that someone can remove the SD card at any time, leaving you with a half finished file, or possibly filesystem. Also, if we're using the technique where we reinitialize before we write every time that creates a performance issue, because we're reinitializing the library and the device every time we want to write a miniscule amount of data.
The solution to this involves staging writes to an intermediary like flash. We can use the SPIFFs library to do this. Basically, the idea is we create a working file in flash and write to that. When we wish to "save", we take that file and copy it in one go to the SD card. While it doesn't completely eliminate the possibility that someone removes the card in the middle of the save, it can drastically cut it down versus writing to a card over a long term.
We must initialize SPIFFS once in setup:
// we can init spiffs once
if(!SPIFFS.begin()) {
Serial.println(F("Unable to mount SPIFFS filesystem"));
while(true); // halt
}Next, whenever we want to append a file - here char *sz is the text:
if(!stagingFile) {
// create the staging file
stagingFile = SPIFFS.open("/staging",FILE_WRITE);
if(!stagingFile) {
Serial.println(F("Unable to open staging file."));
return -1;
}
}
// write the file
size_t c = strlen(sz);
size_t result = stagingFile.write((uint8_t*)sz,c);
if(c!=result)
Serial.println(F("Write failed!"));
return result;Notice how we're checking to see if the staging file is open and if not, we are opening it. That's so that commits and appends can happen at any time even though commits close the file. Speaking of commits, here's the commit routine, where char *path is the file to write to the SD:
size_t result = 0;
if(!stagingFile) return -1;
stagingFile.close();
File src = SPIFFS.open("/staging",FILE_READ);
if(!src) return -1;
// initialize SD
if(!SD.begin()){
Serial.println(F("MicroSD Card Mount Failed"));
return -1;
}
File dst = SD.open(path,FILE_WRITE);
if(!dst) return -1;
// copy
while(src.available()) {
if(-1!=dst.write(src.read()))
++result;
}
dst.close();
src.close();
return result;What we're doing here is first closing the staging file so we can reopen it for reading. Then we initialize the SD, and open the destination file for writing. Finally, we copy one file to the other before closing both and returning the number of bytes copied.
To use it, we just keep appending until we're ready to commit it. When we commit it, it writes to the SD in one go. Unlike the previous method, this will only stage for the duration of a boot. You can't append to the stage, reboot, and then append more. This is by design. You could change that simply by opening the staging file with FILE_APPEND instead of FILE_WRITE and then calling SPIFFS.remove("/staging"); to remove it once you've copied it to the SD.
Note: Due to hardware considerations, this section is largely specific to the ESP32, aside from some general concepts.
Some SPI devices do not play well with others. You'll find that SPI devices like the RA8875 which uses SPI mode 3 will not work on the same bus as an SD reader, forcing you to attach your reader to a second bus. With an ESP32, this is easy once you know how. Other devices might leave you out in the cold if they only have one bus. Using a dedicated bus can also ensure your card operates at peak performance. Note however that the SD libraries that ship with your board manager are not always optimized for speed, favoring compatibility. You may need to use a recent version of SdFat if you want full speed I/O. Using SdFat is outside the scope here, but the same principles outlined here previously at least apply.
The ESP32 based development boards have multiple SPI buses. However, the second SPI is by default mapped to the same pins as the integrated flash, making them off limits in the default configuration. To overcome that, these boards also support pin muxing, allowing you to remap the second SPI bus (among other things) to different pins. The idea here - at least with the ESP32, is to route the second SPI bus to some free pins and wire the SD up to that. The reason we favor wiring the SD up to the second bus as opposed to wiring the other device to the second bus instead is that not all device libraries support using an alternate bus out of the box but the SD libraries typically do.
What we need to do is initialize an instance of the SDI driver class using the 2nd bus and then feed it our desired pins. We accomplish this first with the following global:
// configure this instance to use the
// 2nd SPI bus called "HSPI" instead
// of the first, default bus "VSPI"
SPIClass sdSPI(HSPI);Espressif, for whatever reason, calls their first SPI bus "VSPI" and their second bus "HSPI". These are random designators. The buses are equivelent except for the pins they are mapped to by default and their named designations.
Now, we're not quite done. Every time we initialize the SD device - that is, every time before calling SD.begin() we must call sdSPI.begin() on the SPI class we made and pass it our pins. We must then pass that to the SD.begin() call, like so:
// initialized the alternate SPI port and pin mappings
sdSPI.begin(SDM_CLK,SDM_MISO,SDM_MOSI,SDM_CS); //CLK,MISO,MOSI,CS
// initialize SD
if(!SD.begin(SDM_CS,sdSPI)){
Serial.println(F("MicroSD Card Mount Failed"));
return -1;
}Note how we're not checking the result of sdSPI.begin(). This is because it could have been initialized by a prior call so we don't care if it fails. If it actually fails, then we'll report the failure once the SD portion doesn't initialize.
So now, we've encountered and successfully avoided some of the major traps of using an SD reader from an IoT device. Particularly, the lack of card change detection makes writing more challenging and we've explored a couple of ways to overcome some of that. We've also looked at using an SD reader from the second SPI bus of an ESP32 based development board. I hope this helps you in your future projects.