Prang Once Again
Updated on 2022-08-25
Dusting off an old MIDI toy project and making it less of a toy.
Introduction
Update: I've ported Prang to the Teensy 4.1, and I've published the project on GitHub. It doesn't really warrant its own article, but I thought it certainly deserved an update for those following this. The nice thing is the USB hosting is built into that device - no soldering other than the teensy's headers themselves - which you'd have to do anyway. The SD reader is built in too, and is orders of magnitude faster than those little 1-bit SPI breakouts you so often find. The end result is a faster product, that's also easier to put together, and roughly the same cost, if not less expensive after the USB host device and ESP32S3 devkit are accounted for.
Disclaimers: This is a fairly involved project, and it will require a little soldering. It is also licensed GPL due to me having to modify a GPL licensed library that is included with the project.
Prang allows you to take tracks out of a MIDI file and trigger them with keys on your musical keyboard. Any keys left over are simply passed through from in to out. The upshot is you can use pieces of MIDI compositions to back your performances, triggering them with the strike of a note.
I can't play to save my life, so it's a good thing my life isn't on the line. Here, though, hopefully this illustrates what Prang does:
It works by sitting between your USB of your keyboard and your PC. It intercepts all the USB MIDI traffic and alters and injects it with MIDI data from MIDI files.
Here's the heart of it. Note that it's not the entire circuit. The user interface is not shown. There's a screen, an encoder and some buttons not shown here.
We're gonna build it.
A Segue Into Licensing
I want to be clear that only the code at the actual github branch (or in this zip) is GPL licensed. My usual license scheme is MIT, and all the dependencies other than in /lib are MIT. If anyone wants to tell me I can't use MIT and GPL licenses together in the same project, bring lawyers, guns, and money and we'll sort this out.
For any of you out there that don't appreciate this license scheme, I don't either, but my hand was forced due to the USB Host Library needing to be modified by me. I also decided that it was less onerous to make it GPL3 than it was to make it available for non-commercial use only, which I had considered. A lot of work went into this, and unlike most of my projects, I don't feel right about someone being able to take it and simply sell it without giving back, particularly since this is an end to end product rather than some library - which will only become even more true after I get the PCBs designed for fabrication. Those will be released GPL3 as well. I've actually spent over $200 just on incidentals to get this working and tested, and countless hours of labor, so I'm invested, and if someone wants to use this that's great, but I expect you to invest as well - back into the community.
Hardware
An ESP32S3 DevkitC or similar, like this one. I don't know why these are so pricey, given the MCU itself is only $1.80 on Mouser. If you can find them cheaper, go for it.
USB2.0 Host Shield like this one. They're Broken As Designed and must be modified, but they are the best solution for this. See here for the pinouts, since it isn't labelled.
Some sort of SD reader module, like this one. These ones are 5v I believe, and they are what I use, but I'd prefer 3.3v without the regulator to be honest. These however, were cheap.
An ILI9341 display, like this. Actually, mine is an ILI9342C because it's what I had. You may need to change line 75 in ./src/main.cpp from ili9342c to ili9341 or ili9341v depending on your variant. If the display doesn't show correctly, change that. Note that this project is actually designed for a much cheaper but more elusive 1.18inch 240x135 ST7789VW display like on the TTGO, but I'm not going to force you to source one of those, so I've written the code to use the much more common 320x240 ILI9341. You probably have 3. It doesn't need touch or anything like that but it's actually hard to find ones without unless you're willing to wait for shipping from China. You can use any TFT/LCD/OLED display that's GFX compatible as long as it's at least 240x135, but you'll have to change a little bit of the source.
You'll need a rotary encoder, like these.
You'll need two momentary buttons, like these.
You're going to need some USB cables, a soldering iron, and probably some tweezers.
Kludge Time
In their infinite wisdom, the designers of the USB Host Shield thought it fine to try to power the USB host port with 3.3 volts. USB takes about 5. Some devices will work with 3.3 but many will not. To fix this, we're going to sever the 3.3v supply to the USB "VBUS" line, and run a 5v wire to it ourselves.
First, find the little black resistor that is just to the right of the term "2k2". Your mission is to get rid of it. Take a hot soldering iron and loosen one of the leads, and then prize up the end with the tweezers once its flowy. do the other side as needed. Try not to swear.
Now, for the hard part. It's easier to do this before the headers are soldered on, so don't make the mistake I did, or you'll have to get your better half to finish it, like I did. Anyway, on the underside, you'll see a hole just inside where the headers go, near the USB port itself, that says VBUS near it. It will be the hole closest to the USB port that is not a header pin hole. Solder a long, preferably red wire onto that hole. The other end goes to 5 volts.
Don't worry about this potentially wrecking the device for other projects. It won't. In fact, it will fix it. This simply brings the USB host port to spec.
Wiring
I'm assuming you know your way around circuits such that I've given you the entire guide to wiring the project within some #defines in main.cpp. It's not that complicated. It's just that there's a lot of it that's not that complicated. Have fun!
Extensibility
I intend to have this circuit fabbed and to be able to source components to construct these boxes for a variety of MIDI applications, of which Prang is only one. Same circuit, different firmware + different color of enclosure = new piece of equipment. I can think of a dozen MIDI boxes I could make with this same setup.
Software
This basically requires Platform IO due to the number of dependencies, and the fact that they're all fetched from the PIO repository. It is not possible to build this project with the Arduino IDE as far as I know, because it does not support GNU C++17.
It also assumes all your platforms and such are up to date, as it uses a pretty new board.
MIDI-OX is also helpful to route MIDI on your PC, but you may already have it or other tools you prefer.
What It Does
When you first turn this mess on, it will demand you insert an SD card if you haven't already. The SD card should have at least one MIDI file on it.
If you have more than one MIDI file, after loading the list of MIDI files on the root directory of the card, it will ask you which one to select. If there's only one, it will select that one automatically.
After that, you must tell it the base or root octave to map tracks to. By default, mapping starts at MIDI C4, which is the left hand octave of most MIDI keyboards.
Once you do that, it will ask you what level of quantization you want, if any. This defaults to 4 beats, which is a nice mix of error correction and control. Quantization only effects the timing of the MIDI file track playback, not keys that are not mapped to MIDI file tracks.
Finally, it will ask if you'd like to save. If you choose yes, it will not ask you these two things the next time you use this same SD card. If you want to override the settings on the SD card hold one of the buttons down as you turn the machine on. The settings are written to the root of the SD card as prang.csv.
Now you can start playing. If you turn the encoder at this point, the tempo multiplier is adjusted. Each track keeps its own tempo, and it may change throughout the track. The multiplier is applied to any tempo value in any of the tracks, so a track at 100bpm with a tempo multiplier of x1.5 will play at 150bpm.
Keys are mapped starting at MIDI file track 0 at the base octave and working to the right as the track index increases. Only channel 0 is mapped. All other keys and channels are mapped through as is.
How It Works
Prang acts as a USB Host to a MIDI device plugged into its USB-A port. It listens for MIDI messages, and anything it doesn't want to intercept it passes to its other USB port (miniUSB in this, though you can wire up a USB-B port to GPIO 20 (D+) and 19 (D-) if you want to be more audio standard.
Anything it does intercept is intercepted and sent to the quantizer.
The quantizer is kind of complicated in terms of what it does. I should also note, again, quantization only applies to these intercepted keys. When no key is playing, and a key is pressed, that key becomes the "master" key to follow. Other keys are considered early or late relative to this "master" key. So when the next key is played, it checks the time relative to that, and passes the offset to the "MIDI sampler", which we'll cover. When a key is depressed, if it was the master, a new master is hunted for, by searching for the next playing key. If one is found, that is the new master, otherwise there is no sound and no master, until a key is pressed again.
The sampler's job is to mix MIDI tracks as they are requested. Basically, you can start and stop MIDI file tracks at any given point. When you start a track, you can even give it a positive or negative "advance" for quantization so that it can be adjusted to start early or late as needed. It's easier in concept to the quantizer but the actual implementation is pretty complicated, since it needs to juggle multiple MIDI clocks and cursors.
Now, everything in Prang is written such that it can be cooperatively threaded. You generally call update() on something to give it timeslices. However, to make sure that the MIDI remains uninterrupted, Prang runs the MIDI processing on the secondary core on a very high priority thread. This works because we're not doing anything with Bluetooth or Wifi that would otherwise need that core.
To facilitate communication between the MIDI task and the main application task two queues are used: One queue to the MIDI thread, and one queue to the main thread. This facilitates bidirectional communication. This is important since the user interface, like the knob, buttons, and screen are controlled by the main thread, while the MIDI thread handles the actual MIDI input. The UI needs to tell the MIDI thread when the tempo multiplier has changed, and the MIDI thread needs to tell the UI when a key came in early, late or on time.
It should be pointed out that the main app code is pretty long, but the UI is sequential, and pretty much runs from top to bottom through setup before playback finally starts, so despite its relative unwieldiness it's not impossible to navigate.
Coding this Mess
For a more detailed view of the project at a prior state of development, see this article. That article also covers the MIDI sampler and setup() so we'll be largely skipping those here.
Prang uses my SFX library to do much of the MIDI processing. I didn't break out the input processing through SFX because I didn't need much abstraction at that level and I wanted to keep things tight, but the midi_sampler that sits at the heart of the app makes heavy use of it.
It uses my GFX library to handle the graphical user interface components.
It uses my FreeRTOS thread pack to handle the threading and message queues.
Also well, pretty much all of the libraries are my code except for the actual MIDI drivers (both in and out) and the encoder library. All the code under /lib is 3rd party, even if modified by me. Otherwise all the code is my code, even if it's pulled in via platformio.ini lib_deps.
This is all run without using any PSRAM, and it uses TrueType off of SPIFFS, which requires loading a font into RAM before it can be drawn without making you want to get out and push. In order to make that work, we load and unload our font as needed.
We have a smaller TrueType font embedded as a header (telegrama.hpp) which is used as a fall back and system font. It never gets unloaded, and requires no SRAM since it is in flash.
There is also a MIDI.jpg that gets displayed, also from SPIFFS.
Remember to use Project Tasks|Upload Filesystem Image before you can run the first time.
I think it might actually be simpler to cover the midi_task() in main.cpp first, so let's start there. This routine is the high priority thread that handles all of our MIDI in and out, and drives the MIDI sampler.
void midi_task(void* state) {
uint8_t buffer[MIDI_EVENT_PACKET_SIZE];
uint16_t rcvd;
int i, j;
while (true) {
queue_info qi;
// handle incoming queue messages
if (queue_to_thread.receive(&qi, false)) {
switch (qi.cmd) {
case 1:
sampler.tempo_multiplier(qi.value);
break;
default:
break;
}
}
Usb.Task();
if (midi_in) {
if (midi_in.RecvData(&rcvd, buffer) == 0) {
const uint8_t* p = buffer + 1;
last_status = *(p++);
int base_note = base_octave * 12;
bool note_on = false;
int note;
int vel;
unsigned long long next_off_elapsed = 0;
int next_off_track = -1;
queue_info qi;
int s = last_status;
if (s < 0xF0) {
s &= 0xF0;
}
switch ((midi_message_type)s) {
case midi_message_type::note_on:
note_on = true;
case midi_message_type::note_off:
note = *(p++);
vel = *(p++);
// is the note within our captured notes?
if ((last_status & 0x0F) == 0 &&
note >= base_note &&
note < base_note + sampler.tracks_count()) {
if (note_on && vel > 0) {
quantizer.start(note - base_note);
qi.cmd = 1;
qi.value = (int)quantizer.last_timing();
queue_to_main.send(qi, false);
} else {
quantizer.stop(note - base_note);
}
} else {
// just forward it
tud_midi_stream_write(0, buffer + 1, 3);
}
break;
case midi_message_type::polyphonic_pressure:
case midi_message_type::control_change:
case midi_message_type::pitch_wheel_change:
case midi_message_type::song_position:
// length 3 message - forward it
tud_midi_stream_write(0, buffer + 1, 3);
break;
case midi_message_type::program_change:
case midi_message_type::channel_pressure:
case midi_message_type::song_select:
// length 2 message - forward it
tud_midi_stream_write(0, buffer + 1, 2);
break;
case midi_message_type::system_exclusive:
for (j = 2; j < sizeof(buffer); ++j) {
if (buffer[j] == 0xF7) {
break;
}
}
// sysex message - forward it
tud_midi_stream_write(0, buffer + 1, j);
break;
case midi_message_type::reset:
case midi_message_type::end_system_exclusive:
case midi_message_type::active_sensing:
case midi_message_type::start_playback:
case midi_message_type::stop_playback:
case midi_message_type::tune_request:
case midi_message_type::timing_clock:
// length 1 message - forward it
tud_midi_stream_write(0, buffer + 1, 1);
break;
}
}
}
// send any pending sampler MIDI data
sampler.update();
// release a timeslice to the OS
vTaskDelay(1);
}
}Considering we have to do some parsing of the binary stream, this code really isn't that complicated.
Before we do any MIDI processing, we check if there is a waiting message in the queue. If so, we process it, in this case changing the tempo multiplier.
After that, we process MIDI input. The input is basically MIDI wire protocol prefixed by a byte whose purpose I do not know. Weirdly, a MIDI message always takes 64 bytes over USB, even if the actual message is usually between 1 and 3 bytes. Also as far as I can tell, the MIDI status byte is always present in these messages. Otherwise, it's vanilla MIDI. Everything but note on and note off messages gets passed straight through. The only reason we have to examine them at all otherwise is to see how many bytes they are. Only note on and note off messages that fall between the actively mapped range on channel zero get intercepted. When we find one, we send it to the quantizer, itself attached to the sampler. The other thing we do is we check the timing of the note that was just played, and send an indicator back to the main thread about that, so that it can display it on the screen.
The next major routine is scan_file() which reads an entire MIDI file looking at each track and searching for tempo change messages. The results can be none, one, or many. None defaults to 120bpm which is the MIDI standard. One means the track is in the specified tempo. If there's more than one tempo, it's indicated as varied.
sfx_result scan_file(File& file, midi_file_info* out_info) {
midi_file mf;
file_stream fs(file);
sfx_result r = midi_file::read(fs, &mf);
if (r != sfx_result::success) {
return r;
}
out_info->tracks = (int)mf.tracks_size;
int32_t file_mt = 500000;
for (size_t i = 0; i < mf.tracks_size; ++i) {
if (mf.tracks[i].offset != fs.seek(mf.tracks[i].offset)) {
return sfx_result::end_of_stream;
}
bool found_tempo = false;
int32_t mt = 500000;
midi_event_ex me;
me.absolute = 0;
me.delta = 0;
while (fs.seek(0, seek_origin::current) < mf.tracks[i].size) {
size_t sz = midi_stream::decode_event(true, fs, &me);
if (sz == 0) {
return sfx_result::unknown_error;
}
if (me.message.status == 0xFF && me.message.meta.type == 0x51) {
int32_t mt2 = (me.message.meta.data[0] << 16) |
(me.message.meta.data[1] << 8) |
me.message.meta.data[2];
if (!found_tempo) {
found_tempo = true;
mt = mt2;
file_mt = mt;
} else {
if (mt != file_mt) {
mt = 0;
file_mt = 0;
break;
}
if (mt != mt2) {
mt = 0;
file_mt = 0;
break;
}
}
}
}
}
out_info->microtempo = file_mt;
out_info->type = mf.type;
return sfx_result::success;
}Most of the rest of the file is setup() and UI support code. It's incredibly long, boring and ugly so I'm not presenting it here. See the Prang article I linked to earlier for an overview of it, as it's pretty much the same now, but with more screens and features, like saving.
Now all we have left in the main portion of the application is loop():
void loop() {
queue_info qi;
if (queue_to_main.receive(&qi, false)) {
if (qi.cmd == 1) {
off_ts = millis() + 1000;
auto px = color_t::white;
switch ((int)qi.value) {
case 0:
px = color_t::green;
break;
case -1:
px = color_t::blue;
break;
case 1:
px = color_t::red;
break;
default:
break;
}
draw::filled_ellipse(lcd, rect16(point16(20, 20), 10), px);
}
}
if (off_ts != 0 && millis() >= off_ts) {
off_ts = 0;
draw::filled_ellipse(lcd, rect16(point16(20, 20), 10), color_t::white);
}
bool inc;
int64_t ec = (encoder.getCount() / 4);
if (encoder_old_count != ec) {
inc = (encoder_old_count > ec);
encoder_old_count = ec;
if (inc) {
if (tempo_multiplier < 4.99) {
tempo_multiplier += .01;
update_tempo_mult();
}
} else {
if (tempo_multiplier > .01) {
tempo_multiplier -= .01;
update_tempo_mult();
}
}
}
}loop() does just a few things.
First, it listens for incoming messages from the message queue. These indicate a note was pressed and the timing of it. We indicate that by displaying a circle that's either red, green, or blue, depending if the note was late, early, or on time, respectively.
Next, it checks if a second has passed since the last time it displayed a note on (from above). If it did, it draws a white circle over the previous circle, erasing it.
Finally, it checks the encoder knob to see if it has changed, and if it has, it updates the tempo multiplier.
One thing that's different about Prang since the last time we covered it is improved quantization which has been refactored into midi_quantizer. This class takes start() and stop() commands for individual MIDI sampler tracks and computes the appropriate quantization advance (negative or positive) for each start(). The meat of the class is that method:
sfx_result midi_quantizer::start(size_t index) {
if(m_sampler==nullptr ||
index<0||
index>=m_sampler->tracks_count()) {
return sfx_result::invalid_argument;
}
m_last_key_ticks = m_sampler->elapsed(index);
if(!m_quantize_beats || m_follow_key==-1) {
m_sampler->start(index);
m_key_advance[index]=0;
m_follow_key = index;
m_last_timing = midi_quantizer_timing::exact;
return sfx_result::success;
}
unsigned long long smp_elapsed;
unsigned long long adv=0;
int tb = m_sampler->timebase(m_follow_key)
* m_quantize_beats;
smp_elapsed=m_sampler->elapsed(m_follow_key)
- m_key_advance[m_follow_key];
adv= smp_elapsed % tb;
unsigned long long adv2=adv-tb;
if(adv>-adv2) {
adv=adv2;
m_last_timing = midi_quantizer_timing::early;
} else if(adv!=0) {
m_last_timing = midi_quantizer_timing::late;
}
sfx_result r = m_sampler->start(index,adv);
if(r!=sfx_result::success) {
return r;
}
m_key_advance[index]=adv;
return sfx_result::success;
}Here, we're tracking the last elapsed tick count the most recent key was pressed. Then if we're not quantizing yet we simply start the track, set the follow key and last timing, and return. Otherwise, we use the current follow key and the elapsed time for that key to compute where the next or previous quantization mark falls. Whichever is closest, we use to decide our advance, before passing that to the sampler and updating our stored key advance.
Conclusion
With a universal hardware design for potentially limitless MIDI processing boxes and a full featured MIDI library like SFX, you can design new gear to your heart's content, or simply modify Prang to suit your exact requirements. Happy building!
History
- 8th August, 2022 - Initial submission