OpenBoard on Wayland - letsfindaway/OpenBoard GitHub Wiki

At some point in time, X11 will no longer be maintained and will vanish from the Linux distributions. OpenBoard must be prepared for this.

There are however several issues people have created which use OpenBoard on Wayland. Problems range from crashes to the inability to position windows or to create snapshots. Here I want to collect some problematic topics related to the way how Wayland operates, mainly for screenshots, screen mirroring and screen recording.

Screenshots

To put the content of a screen area in a QPixmap Qt provides two completely different ways:

  • If the screen area is part of a widget of the application, then this widget can be rendered in a QPixmap by using a QPainter.
  • For other cases, it is however also possible to retrieve the QScreen object representing a screen and call the grab() function on the screen.

The first method is independent of the windowing system and works in all cases. The second method however does not work with Wayland, because it disallows grabbing arbitrary screen areas.

The Wayland alternative is using the XDG Desktop Portal. Here a screenshot can be requested via DBus. The system implementation of the portal may notify the user and ask for permission before taking the snapshot. The result is then delivered as a file and the application is notified. This sounds practicable, but has several problems in practice:

  • The implementation of the screenshot portal function is different for each desktop environment. So Gnome, KDE and all others have their own implementation, which might differ in user behavior. Whether and how the user is notified is the choice of the DE.
  • The screenshot takes much longer than screen grabbing. On my system with KDE a single screenshot takes about 350ms.
  • Taking a screenshot always involves recording the content of all connected monitors. The relevant part must then be cut out by the application. This makes taking screenshots not faster.
  • At least on KDE the screenshot file is created in the Images folder and must be deleted afterwards. So each screenshot involves a disk operation.

So in the end this function might be useful for screenshots in Web and Desktop modes. But it is not applicable for screen mirroring in Desktop mode, as it is way too slow. For the same reason it is also impossible to use it for Podcast recording.

ScreenCast

But there is an alternative function in the XDG Desktop Portal: ScreenCast. This function is intended for screen sharing applications. It grabs a screen content as a video stream. Such a stream could be recorded in a file or replayed in a widget on another screen.

But my first tests revealed other problems:

  • The application cannot tell the portal which screen to record. Instead a dialog is presented to the user, where they have to select the proper monitor or application for sharing. This dialog is not intuitive for an OpenBoard user and it is likely that the user selects the wrong option.
  • At least in my first tests this dialog was somehow hidden behind the mask when I was using it in Desktop mode, so there was no way to interact with the dialog. Further tests revealed that we have to hide the desktop annotation widget before calling Start on the ScreenCast portal and show it again afterwards. Edit Problem solved as described.
  • Sometimes we get a response from the SelectSources call, sometimes we don't. FlameShot already has a comment on that (Hahaha, ist true und geht nicht). Have to work on that - or try some workarounds. Edit Problem was that the responses arrived faster than we could connect to the DBus to listen for them. Solution is to connect to the DBus and listen for the response before we make the call. This is possible as we can anticipate the path of the response.

We can then use the pipewire API to get the data from that stream and to create images from the frames. These images can then be displayed on the Display screen just as images taken by the grab() function. The same mechanism can be used for podcast recording.

Window positioning

It is essential for OpenBoard to have control over window positions. However with Wayland, the compositor has the ultimate control over window positions. This means that you cannot move a top-level window to an arbitrary screen or position.

The only workaround I found so far is as follows:

  • It seems that Wayland at least takes the choice of the screen for a full-screen window into account. This helps for most of OpenBoard's use cases.
  • When we later convert a full-screen window to a normal window, then this stays on the same screen. We can use this to position the screen labels for screen configuration in the preferences dialog.

These findings are for KDE KWin-wayland and have to be checked for any other compositor.

Stay on top

With Wayland it is not possible to create a window which always stays on top of all other windows. This is however important for Desktop mode, where the desktop palette should always stay visible.

For KDE there is a workaround: even if the program itself cannot create such a window it is possible to instruct kwin to keep a window always above all others using a KWin script. The solution for KDE is therefore to provide such a script and to execute it after switching to Desktop mode.

Portal implementations

As already stated above, the XDG Desktop Portal is just a thin layer which relies on implementations provided by the individual desktop environments. When looking at the bug trackers of both Gnome and KDE it seems that there are sill many unresolved problems.

The different implementations also increase the testing effort necessary for OpenBoard. We have to test at least with Gnome and KDE, but probably also with others. I can also imagine that we get bug reports from users using another desktop environment and experience different behavior. This makes life for an application developer not easier.

Consequences

In general we have to state that using the XDG Desktop Portal may involve user interactions which cannot be influenced or inhibited by the application. These interactions even depend on the desktop environment. So what can we do?

  • We can try to explain to the user what they have to do when such a dialog pops up. Still, these dialogs are problematic for the user experience.
  • We can remove functions from OpenBoard which require these things. This would especially be screen mirroring in Desktop mode and the Podcast function.
  • We should further investigate what "persistence" means for such a selection. The documentation is not clear about that. At least it can be for one session, so the user only has to do that once after OpenBoard was started. Edit In a screencast session we get a token which can be used in the next session to skip the dialog providing the screen configuration has not changed. I think this provides a good enough user experience.