GFX Part 1: A Device Agnostic Pixel

Updated on 2021-04-11

A device independent graphics library for IoT devices. Part 1 of a series.

C++graphicsArduino2DC++17

Introduction

Update: Today a new version of this code is available as part of the next article in the series. It's recommended that you use that code instead, as there are bug fixes.

the next article

One of the fun or terrible things (depending on your perspective and mood) about IoT is everything is bare metal.

This makes implementing a graphics library pretty complicated, as every wheel must be essentially reinvented, and we must be able to handle a myriad of different screen mode configurations.

Did you think we would use DirectX? No such luck. You have a frame buffer over an SPI or I2C bus attached to a display with a completely arbitrary display mode instead. This device will probably support blting and reading and writing pixels. That's pretty much all you can count on, although some devices may support other operations, DMA transfers, and other fancy things. Keep in mind that the binary format for these screen modes is completely arbitrary and device dependent.

We'll be covering the humble pixel, and only the pixel in this article. The reason is that everything in a graphics library builds on this, and also just covering the pixel is a lengthy topic, as it's one of the more complicated parts of a device independent graphics library.

Building this Mess

This code was tested with clang 11 and gcc 10 using -std=c++17

The latest version of MSVC does not want to compile this code. Frankly, I don't care, as nothing I've encountered cross compiles with MSVC anyway, and the primary purpose of this code is for IoT devices that can't be readily targeted with MSVC. There's nothing Linux specific in this code, and I believe it's simply standards compliant C++, but it's possible that I am using some features specific to gcc and clang without realizing it.

It may be possible to target the C++14 standard with some modifications to the source, since much of it is compliant with 14. I just haven't run down all the errors/C++17 specific features.

Conceptualizing this Mess

In the introduction, I said that the display mode is arbitrary. Here I'm talking about not only the resolution and the color depth, but the binary layout of the frame buffer in memory. For example, some display modes may expect an 8 bit value for the blue channel, followed by another for green and finally one for red. Another may expect a 5 bit value for the red channel, a 6 bit value for the green channel, and a 5 bit value for the blue channel. Yet another might be grayscale, and still another may be monochrome. There may be others as well, especially when you're dealing with image files as well as frame buffers. For example, the JPEG format uses a color model that isn't even RGB.

You can see how this might get complicated. What we need is the ability to declare things like this:

// the first described example above
// 24-bit color in BGR
using bgr888 = pixel<
    channel_traits<channel_name::B,8>,
    channel_traits<channel_name::G,8>,
    channel_traits<channel_name::R,8>
>;
// the second example above
// 16-bit color in RGB
using rgb565 = pixel<
    channel_traits<channel_name::R,5>,
    channel_traits<channel_name::G,6>,
    channel_traits<channel_name::B,5>
>;
// 8-bit grayscale
using gsc8 = pixel<
    channel_traits<channel_name::L,8>
>;
// 1-bit mono
using mono1 = pixel<
    channel_traits<channel_name::L,1>
>;

Here, we've declared four different pixel<> formats, each with one or more channel_traits<>, and each channel with a channel_name and bit depth.

After that, we need a way to initialize them, access individual channel values, and preferably convert between formats.

Since we're targetting devices with extremely limited CPU and RAM, we should punt as much as we can to compile time. Consequently, the code may sometimes have to go through elaborate lengths to prevent a runtime computation when a compile time one would suffice. Also, we need to make sure we've designed the library so that a rich compile time API is exposed in order to facilitate creating complicated pixel operations at compile time.

To do all of this involves some metaprogramming. The code cajoles the compiler into doing pretty complicated computations in order to facilitate this, but the tradeoff is efficient and flexible code, especially in scenarios where values are known at compile time.

What we want is to be able to initialize pixels, query them for channel data, set channel data, and even convert between pixel formats such that if all values are known at compile time no code is generated for these operations. Instead, the request for these operations is simply replaced with a constant representing the pixel's intrinsic word value, as though pixel itself was simply a C++ intrinsic integral value. With something like that, we can take the color purple's RGB model definition and convert it to a different color model - even monochrome or grayscale without having to do the conversion at run time. The compiler computes the final value for us. We can also convert to different binary layouts and bit depths as well.

We also need the run time code to perform well. We need to be able to do the above where not all values are known at compile time, and generate efficient code for that. In many cases, we can do this by doing as much as we can at compile time, and then doing the remaining tasks at run time.

It would also be helpful to provide a rich set of features for querying various properties of a pixel at compile time. For example, we should be able to retrieve channels by name or index, and get their bit depth, their minimum and maximum values, a channel mask, and perhaps some other things as well.

Finally, it should run on major platforms. To facilitate this, pixels are internally stored in native endian format, but accessible also as big endian, since that's useful for representing binary pixels in a frame buffer, and it will do byte order conversion if necessary when getting and setting channel values. It should be noted that with ARM processors, switching the endian mode at run time is possible, but this library will not support that feature, and it should not be used with this code.

Disclaimer: I have not tested this code on a big endian processor, since I do not currently own one. I'm waiting for one to be shipped to me. That said, I'm reasonably confident this will work on a big endian machine.

Using this Mess

The major things you can do with a pixel include defining them, initializing them, getting or setting individual channel values, converting them to other pixel formats and querying them for metadata about the pixel definition.

For all of the examples below, we'll be using the pixel definitions from above.

Initialization

There are three ways to initialize a pixel:

rgb565 pixel;                // initializes to 0
// note our green channel is 0-63 not 0-31
rgb565 pixel2(0,31,15);      // dark cyan - one int per channel
rgb565 pixel3(true,0,.5,.5); // same as above, but reals

Note that there is no way to generically initialize a pixel of any arbitrary configuration, because a pixel may have as few as 1 and as many as 64 channels. However, you can do things, like set it to a known color, like color::purple which will work with any pixel format that can be converted from RGB color models. Otherwise, your constructor invocation is going to be specific to the type of pixel you instantiated, depending on the number of channels, and each channel's int_type or real_type.

Accessing Channel Values

There are several ways to access channel values. You can retrieve and set them as real numbers or integers and either by channel index or channel name. Getting and setting happens at compile time, if possible. The index or name must be known at compile time, since it is a template argument:

auto r = pixel.channel<0>();                  // get the red value
auto rf = pixel.channelf<0>();                // get the red value
auto r2 = pixel.channel<channel_name::R>();   // get the red value
auto rf2 = pixel.channelf<channel_name::R>(); // get the red value

Setting them is similar:

pixel.channel<0>(16);               // set red to 16
pixel.channelf<0>(.5);              // set red to .5 (~16)
pixel.channel<color_name::R>(16);   // set red to 16
pixel.channelf<color_name::R>(.5);  // set red to .5 (~16);

Unchecked Versions

There are also channel_unchecked<>() template methods which do not check the index for validity. This allows you to bypass compiler checks on the passed in index. Sometimes, this is necessary because the compiler can't determine that the index you are passing is guaranteed to be valid, even if it is. In these cases, the compiler will error when using the standard channel accessor methods. When that happens, use the unchecked methods, but beware that passing in an invalid index yields an "empty channel" where getting and setting do nothing and all of the channel metadata is zeroed. Due to that, you really should check the index you pass in for validity beforehand.

Converting Between Pixel Formats

The pixel<> exposes a template method called convert<>() which allows you to convert from one pixel format to another. Currently, it will convert grayscale or monochrome to an RGB color model or vice versa, it will convert between bit depths, and it will reorder channel data as necessary. If you want to add support for other color models, like HUV, you'd add code to this routine.

There are two versions of this method. One passes the result as an out value and returns a bool. The second one returns the converted value. The former is safer, since it will report when a conversion could not be performed, whereas the latter will simply return a zeroed pixel on failure. If you're sure the conversion will succeed, you can simply use the latter one to simplify your code. The only time it will fail is if a particular color model is not supported:

// safer convert call
gsc8 pixel;
// todo: check the return value for success
rgb888(true,1,0,.3333).convert(&pixel);

// easier call
pixel = rgb888(true,1,0,.3333).convert<gsc8>();

If the pixel being converted from was initialized with values known at compile time then there is no run time overhead for this method. The call is eliminated from the code.

Predefined Colors

Under the color "enum" (actually a template struct) there are several predefined colors like black, white, dark_green, cyan, etc. You can initialize a pixel of any type convertible from RGB:

mono1 m = color<mono1>::green;    // will resolve to white/1
rgb565 c = color<rgb565>::yellow;

Accessing the Raw Pixel Data

It is also possible to read and write the pixel data as an integer word, but this data is stored internally in native endian format. If big endian, it would be left to right in channel order, and the unused bits in the word are padded to the right. For example, a 24-bit pixel's machine word would be 32-bits with the 8 remaining bits to the right of the pixel data when using big endian byte order or to the left with little endian. This representation is accessible as big endian like this:

pixel.value(0xFFFFAA00);    // set the pixel to pale yellow

You can use the native_value field to get the pixel data in the machine's native endian mode.

Querying Pixel and Channel Metadata

Pixels provide a rich collection of metadata that describes everything from the binary layout to the valid range of values and scaling for each channel, to the name of each channel. Furthermore, they provide powerful comparison mechanisms to evaluate two pixel types for similarities and differences. None of this generates code or causes execution of any code at run time.

Pixel Data

  • type - the declared type of the pixel, itself
  • int_type - the integer type that holds the pixel data
  • size_t channels - the count of declared channels
  • size_t bit_depth - the sum of the bit depths of each declared channel
  • int_type mask - a mask of the pixel value
  • size_t packed_size - the minimum size in bytes required to hold the pixel data
  • bool byte_aligned - true if the pixel is a whole number of bytes
  • size_t total_size_bits - the total size in bits of the pixel. Based on int_type
  • size_t packed_size_bits - the packed_size in bits
  • size_t pad_right_bits - the number of unused bits on the right

Determining the Color Model

It can be useful to know the color model of the pixel you're working with. That is to say, is the pixel an RGB/BGR style pixel? Is it grayscale or monochrome? Is it something else, like HUV, or YCbCr?

Since you can define your own arbitrary channel types, you might think it would be challenging to determine if a pixel is an arbitrary color model, and normally you'd be correct. However, through the magic of type lists and metaprogramming I've exposed a helper "method" called has_channel_names<> that makes it easy:

// check for an RGB color model
bool isRgb = bgr888::has_channel_names<
        channel_name::R,
        channel_name::G,
        channel_name::B>::value;

Sometimes, it can be slightly more complicated to determine the color model. One example is for monochrome or grayscale. In this case, we use channel_name::L for luminosity. However, we'd also want to make sure we only have the one channel. We can determine that by checking the number of channels:

bool isBW = mono1::has_channel_names<
        channel_name::L>::value &&
        mono1::channels==1;

Equality, Supersets and Subsets

There may be cases where you need to determine whether one pixel type have the same channels (identified by name) as another pixel type, or whether one pixel is a subset or superset of another in terms of the channels it has.

// see if the two pixel types have the same
// channel names in the same order
printf("bgr888::equals<rgb565> = %s\r\n\r\n",
    bgr888::equals<rgb565>::value?"true":"false");

// see if the two pixel types have the same
// channel names in any order
printf("bgr888::unordered_equals<rgb565> = %s\r\n\r\n",
    bgr888::unordered_equals<rgb565>::value?"true":"false");

Above the first line is false, and the second evaluates to true.

is_superset_of<> and is_subset_of<> work similarly. They will return true in the case of (unordered) equality as well.

Retrieving Channel Data

You can retrieve a channel's metadata by index or name using channel_by_index<> or channel_by_name<>, respectively. You can also translate a name to an index using channel_index_by_name<>. channel_by_index_unchecked<> allows you to bypass compiler errors which may come up if the compiler can't determine that the passed in index will always be a valid channel. If the index is invalid, an "empty channel" will be returned. The metadata for such a channel is zeroed. It's best to ensure the index is valid beforehand.

Once you retrieve a channel, you can get its metadata.

The channel<> has a number of static fields and type aliases on it which return various pieces of information:

  • type - an alias for the declared type of this channel, itself
  • pixel_type - the declaring pixel type
  • name_type - the type that represents the channel name
  • int_type - the integer type that holds the channel value
  • real_type - the real type that holds the channel value.
  • pixel_int_type - a shortcut to pixel_type::int_type
  • size_t bits_to_left - the number of bits to the left of this channel data
  • size_t total_bits_to_right - the number of bits to the right, including padding
  • size_t bits_to_right - the number of bits to the right, excluding padding
  • char* name() - the name of the channel
  • size_t bit_depth - the channel bit depth
  • int_type value_mask - a mask of the channel value
  • pixel_int_type channel_mask - a mask of the channel, as part of the pixel data
  • int_type min - the minimum value. Can be set in the channel_traits<>
  • int_type max - the maximum value. Can be set in the channel_traits<>
  • int_type scale - the integer scale - the denominator
  • real_type scalef - the real scale - the reciprocal of scale.

You don't need to worry about any of this, unless you need it down the road. It's purposefully laden with information because its better to have it and not need it than need it and not have it in pretty much all cases due to the fact that there's no run time overhead for it. At worst, retrieving the name puts a string in your executable's .text section. Everything else is zero impact.

Extending this Mess

It may be desirable to expand the number of color models available. Most of the time, you can simply declare a pixel type with the desired channels you want, but there's still the issue of conversion to and from things like RGB, as well as the possibility that you'll need more channel_name entries. Modification of pixels is done in gfx_pixel.hpp.

Adding Channel Names

You can use the GFX_CHANNEL_NAME(x) macro to declare a new channel name. The name must be a valid C identifier. You do not have to put new channel names under channel_name, but you can, and you might find it preferable to keep them all in one place.

Adding Conversions to Other Color Models

The second way to extend this is to modify the pixel<>'s main convert<>() routine so that it can support your new color model. Doing so may seem complicated at first, but most of the code is boilerplate. Look for the // TODO:comments for where to add additional if/else conditions for additional source and destination formats. It may be desirable to do chains of conversions. For example, instead of writing code to handle conversion from Y'UV to/from RGB and Y'UV to/from grayscale, you can implement the latter case by recursively converting your value to RGB and then converting from that to grayscale. Remember how to determine color models. You'll want to add code for that at the top of the routine. Currently, you'll see RGB and BW (black and white). Remember to always look up channels by name because pixels may have the same channels but in different orders. Also use helpers::convert_channel_depth() to convert to the destination bit depth. See the existing code.

Optimization Notes

This code has not been optimized for compile times. There are ways to speed up the compile times if you find it is taking too long to compile. Most of the time taken will be in things like testing for equality or retrieving by channel name or index, but you can pretty much assume all template calls are compiler intensive. Particularly under the gfx::helpers, you'll find there are many optimization opportunities.

You may have noticed this code is aggressively inlined. The reason for that is twofold:

First, it is possible to "deinline" a method by wrapping it with another method, but you cannot inline a method that is not. Therefore I inlined for flexibility.

Second, I've noticed that when you optimize for size with "gcc -Os", which I recommend, inlined methods are deinlined if the code is duplicated. In other words, if you call an inlined method twice it will deinline it, meaning no extra code bloat, while at the same time giving you the advantage of inlined methods if you only call it once.

What's Next

We can't do a whole lot with pixels yet. We need to be able to draw them somewhere, and that sort of thing. In the next installment, we will be using pixels to implement efficient bitmap operations.

History

  • 11th April, 2021 - Initial submission