GUI Architecture with Graphics APIs

Is performance still an issue in this era of cheap 486 computers and super-fast Pentium computers? You bet. How many programs that you use really run so fast that you wouldn't be happier if they ran faster? We're so used to slow software that when a compile-and-link sequence that took 2 minutes on PC takes just 10 seconds on a 486 computer, we're ecstatic - when in truth we should be settling for nothing less than instantaneous response. ~ Michael Abrash's Graphics Programming Black Book, featured in Bryan Phelps's Talk at ReasonConf

Overview

The topic of "designing" user interfaces can be a bit subjective, we won't be going into design trends like say, how you can achieve Neumorphic Design or Material Design in your applications, but the underlying graphics API architecture that drives user interfaces, the sequence of events that takes a Web Application from DOM to graphics API calls, or how React Native goes to OS User interfaces which then executes raster calls to the GPU, or game engine user interfaces that are custom built and immersive [Victor 2012] [Sakamoto 2015]. We'll be discussing:

1. Current Solutions - A very brief review of currently available solutions to solving the creation of user interfaces for native applications, web applications, and even projects that attempt to bridge the gap between the two paradigms.

   • Qt
   • React Native
   • ImGUI
   • Revery

2. UI Abstractions - What a declarative API is, what an immediate mode API is, and how these could be written out in your own applications. We'll be using C++ as our primary programming language here.

3. Implementation Details - All UI frameworks build on top of the same fundamental draw calls and architecture, so we'll be discussing the implementation details of these abstractions using graphics programming terminology. This somewhat touches on game engine architecture.

4. Case Studies - We'll be reviewing the underlying graphics calls that drive user interfaces in commercial applications such as:

   • Unreal Engine 4
   • Unity 3D

Current Solutions

Qt

Qt is the most popular cross platform UI framework in the world, used by developers to design user interfaces with native abstractions or to even interface with graphics APIs to design custom interfaces.

Users include us at Marmoset with Marmoset Hexels, BMW with certain iterations of their vehicles, medical user interfaces. It is licensed LGPL-3 which can be a bit overbearing of a license since it stipulates that any changes to Qt that you do must be provided to them, so bear that in mind if you decide to start a project with Qt. that being said, Qt can even be used along side some of these other UI solutions.

React Native

React Native is an open source graphical user interface library designed to make it easy to convert web UIs written in the DOM abstraction library React into native applications and UIs. (so essentially reusing the same code across codebases).

Users include facebook apps like Facebook, Messenger, Instagram, and apps like Uber, Tesla, and this can even be found in automotive user interfaces as well. User interfaces built using web abstractions can be easy to update and maintain, so it's perfectly understandable to use this project in greenfield applications.

Revery

Revery is a cross platform GUI abstraction written in a Reason, a dialect of Ocaml that looks very similar to TypeScript/JavaScript. I put this project on this list because of just how similar its goals are to the goals of game engine UI renderers.

It's currently a research project by Facebook and uses GPU draw calls to render UIs on both native platforms and the Web.

ImGui

ImGui is an open source immediate mode graphical user interface API written in C++. It's currently the most common graphical user interface library for greenfield projects due to its extreme ease of use and performance. For instance, EA's research uses ImGui for their real time ray tracing seed demo, NVIDIA uses ImGui for their Falcor research renderer library, it's used by countless indie projects such as LightTracer and it's even found in commercial applications such as Fallout 76, Tom Clancy's Ghost Recon, and much more.

And there's commercial game engines such as:

• Unreal Engine - Unreal Engine's open source architecture makes it easily extendable, and its active community can help solve any problems you might run into.

• Unity - Unity's critical mass adoption makes it easy to solve similar problems, though it is not open source.

There's no right answer here, but there's things you as a developer should be concerned about. We're building software user interfaces for people, and so using custom interfaces introduces the challenge of accessability support (which native user interfaces and web DOM elements can handle quite well).

That being said, custom user interfaces can enable more ease of use and programmatic control by means of real time rendering techniques such as shader execution to generate custom effects. (The grid for instance is a custom shader).

UI Abstractions

Solving the problem of rendering user interfaces for interactive applications can involve a variety of different domains:

• Graphics API - DirectX 12, Vulkan, Metal, WebGPU, WebGL, OpenGL

• System UIs - Apple SwiftUI, NSObjects, Android UIs, Qt XML, HTML5, etc.

But no matter the underlying method for drawing your UIs, there's design patterns that persist everywhere:

Declarative APIs

With declarative APIs, you write out the data that describes what you want to have executed. This is in contrast to imperative programming, where you ask the application what to execute:

<!--💬 Declarative-->
<path d="M938.5,432c-5.5,9.6-15.9,16-27.7,16c-17.7,0-32-14.3-32-32"/>

// 💪 Imperative
var canvas = document.getElementById("mycanvas");
var ctx = canvas.getContext('2d');
ctx.moveTo(938.5, 432);
ctx.bezierCurveTo(-5.5, 9.6, -15.9, 16, -27.7, 16);
ctx.bezierCurveTo(-17.7, 0, -32, -14.3, -32, -32);

Immediate Mode

Immediate Mode programming is a polling based paradigm where UIs are designed by means of stack based calls every frame.

// 🏃‍♀️ Immediate Mode (Polling Based)
Begin();
Text("Hello World %d", 123);
if (Button("Save"))
{
    // Do Something...
}
End();

Retained Mode

Compare this with Retained Mode APIs such as those found in common user interface application and you'll find that immediate mode programming allows for more dynamically changing UIs at the cost of performance due to constantly recreating the UI each frame. [Guillemot 2016]

// 👨‍👩‍👧‍👦  Retained Mode (Interrupt Based)
Window* window = new Window();
window->add(new Text("Hello, world %d", 123));
Button* button = new Button("Save");
button->onClick = []()
{
    // Do something lambda...
};
window->add(button);

Either way your API calls would eventually become draw lists that your application requests to be executed by your target graphics API (be it OpenGL, WebGL, DirectX 11, 12, Vulkan, WebGPU, Metal, etc.)

Conclusion

GUI architecture tends to follow fairly clear design patterns, at a high level it's possible to architect your GUI either with a Retained Mode architecture, an Immediate Mode architecture, or naively with imperative programming. Either way, you would build a description of your GUI, which would then be converted into a list of draw commands, which is then converted to the draw calls of your graphics API of choice (OpenGL, Vulkan, WebGL, DirectX, etc.)

Resources

• Falcor - GitHub.com/NVIDIAGameWorks/Falcor

• ImGui - Github.com/ocornut/imgui

• Qt - Qt.io

• React Native - ReactNative.dev/

• Revery - Github.com/revery-ui/revery

• Skia - Skia.org

• wxWidgets - wxWidgets.org