ChanServ changed the topic of #wayland to: https://wayland.freedesktop.org | Discussion about the Wayland protocol and its implementations, plus libinput
pyromancy has left #wayland [#wayland]
co1umbarius has joined #wayland
columbarius has quit [Ping timeout: 480 seconds]
Dami-star has quit [Ping timeout: 480 seconds]
Dami-star has joined #wayland
kayve has joined #wayland
<kayve>
hello I am having an authentication problem I'm hoping someone can help me with
<kayve>
is anyone here?
kayve has quit [Quit: Page closed]
kayve has joined #wayland
<kayve>
hello I'm having an authentication problem I'm hoping to get help with. Is anyone here?
<soreau>
kayve: usually on irc, you just ask your question and then wait to see if anyone responds with an answer
<kayve>
I created a tarball from a HostGator cPanel of a website called regalchess.com. I unzipped the tarball on my Ubuntu system that apparently runs Wayland. I un-tarred the files and re-tarred the files and my issue is seen in the videos on imgur
<RAOF>
That does not appear to be a Wayland problem; that's probably a file-permissions problem.
<kayve>
I am able to use command lines to work with the files but for looking at things like images this is less efficient.
<Oro>
Didn't want to deal with IRC clients and allat
<Oro>
anyways
<Oro>
Wayland stuff
<Oro>
I had an idea for a new protocol, and I was wondering if anybody wanted to take a look at the idea before writing a protocol for it
<Oro>
I want to be able to easily overlay stuff on other Wayland windows (that aren't mine)
<Oro>
The best way I've thought of that so far is a protocol that takes the normal Wayland stuff and just overlays that onto another window, identified with a Wayland handle
<vyivel>
hm what's the use case?
<Oro>
How the clients communicate which handle to use is dealt with by them, for example with Steam it would probably force apps to do so via LD_PRELOAD
<Oro>
vyivel: The Steam Overlay, mainly
<Oro>
Currently it's very hacky and relies on LD_PRELOAD to modify EGL and related stuff iirc
<Oro>
Doesn't work on native Wayland clients
<Oro>
Some slight modifications might need to be made for Xwayland, or let Steam handle that otherwise, like most apps do for X
<Oro>
So native Wayland clients communicate to Steam through their preferred method, I'd imagine forced to do so via dbus and ld_preload, and Xwayland clients do so via the same methods but do X stuff instead to get the overlay going
<Oro>
Unless a Wayland handle can be returned for Xwayland too?
<Oro>
Might need some translation
<Oro>
X11:XID -> wayland:handle
<drakulix[m]>
I had proposed such a protocol in the past, I believe. Essentially what you are asking for is XEmbed like functionality (no matter if initiated by the client or the overlay) and the response was, that synchronization across clients is a very large can of worms nobody wants to open and to use an embedded compositor instead.
<emersion>
are there other use-cases apart from steam?
<emersion>
this has security implications btw, we can't let any client create overlays
<Oro>
drakulix[m]: Why would synchronization matter here? It would literally be a Wayland window on top of another Wayland window
<drakulix[m]>
Panels were my use-case. Our panel is now just a compositor itself.
<emersion>
Oro: then the user resizes the window...
<Oro>
emersion: Any client can create an overlay, but it can't just toss it on top of any client it likes
<Oro>
The information to do that would be communicated between those two processes via something like dbus
kts has quit [Quit: Konversation terminated!]
<emersion>
tbh, if you need LD_PRELOAD anyways, you can insert a sub-surface
<Oro>
emersion: Yeah, that was an issue I thought of too. Best I can think of is the overlay gets the resize info and resizes with it, with a slight delay. The overlay could tell the compositor some preference for where to be, maybe?
Dami-star has quit [Ping timeout: 480 seconds]
<emersion>
wayland aims to be frame-perfect
<emersion>
ie, no delays, no races
<drakulix[m]>
Oro: Right, my proposal was talking specifically about subsurfaces, which obviously could be synchronized to parent surfaces.
<drakulix[m]>
But even without that you get issues. Like when the overlayed window is resized, you certainly want a notification for that size change.
<drakulix[m]>
What if the new size is smaller than the overlay? Does the compositor wait for a new overlay commit?
<Oro>
emersion: LD_PRELOAD might not always work though, like in a sandbox
<Oro>
This would be more sandbox friendly
<Oro>
Because the processes can communicate through dbus too
<Oro>
Or their preferred IPC mechanism
<emersion>
this sounds like a pretty weak use-case to me fwiw
<drakulix[m]>
Then just build an embedded compositor, I guess? E.g. gamescope could just render the steam overlay itself conceptually. That is sandbox friendly.
<Oro>
What if you want to run a game in its own sandbox though?
<Oro>
Then there's no way to get LD_PRELOAD to work
<drakulix[m]>
Oro: you have to pass a wayland-socket anyway? I don’t think nested sandboxes are an issue here.
<Oro>
Or to launch the desired game in gamescope
<Oro>
drakulix[m]: I mean completely different sandboxes
<drakulix[m]>
you can also expose the wayland-socket from the embedded compositor to the host and pass it down the game sandbox.
<Oro>
A game doesn't have to be in the launcher's sandbox to have integration with the overlay
<Oro>
Is the idea here
<Oro>
And won't have to deal with hacks like LD_PRELOAD which are very inconsistent
<Oro>
I actually do have a game that runs separately from Steam: FFXIV. It has a third party launcher that runs in its own sandbox, and LD_PRELOAD hacks applied by Steam won't work at all here, as Steam could be in a sandbox too.
<drakulix[m]>
If you launcher (supposedly running in something like flatpak) wants to overlay with something launched outside its sandbox, it can spawn processes outside the sandbox anyway.
<drakulix[m]>
Sounds more like a portal problem.
<drakulix[m]>
So it has some form of communication to outside the sandbox. That part could help to expose the wayland-socket as well.
<Oro>
The communication outside the sandbox here would be dbus
<Oro>
Ideally
<Oro>
Since that's the most sandbox friendly solution
<drakulix[m]>
right, it needs cooperation with some process running without a sandbox, but it is certainly possible.
<Oro>
--talk-name=com.valvesoftware.Steam, games pass through their Wayland handles and Steam IDs, and Steam tosses an overlay into them using the protocol.
<Oro>
if a game needs to be forced to pass through the needed information, a simple LD_PRELOAD workaround could be build and applied in the appropriate sandbox
rasterman has joined #wayland
<Oro>
Thus, no missing file hell like before
<drakulix[m]>
You could pass a wayland-socket the same way.
<Oro>
Why start up another compositor just for one game though? Not everything would need or benefit from that
Dami-star has joined #wayland
<Oro>
ideally it blends in more seamlessly than a nested compositor
<drakulix[m]>
But its not a huge overhead either. Its mostly passing through dmabufs anyway and provides a clear way to modify the rendered image.
<Oro>
Not fully seamless, like resizing delays, but better than nothing
<Oro>
drakulix[m]: What about desktop integration?
<Oro>
The nested compositor might not provide the features of the host compositor
<drakulix[m]>
A nested compositor could blend very well. It could even just pass through everything just inject another subsurface.
<Oro>
That still feels like a lot more work than just a simple protocol for tossing a wayland window on top of another
<emersion>
i really don't think such a protocol would be "simple"
<drakulix[m]>
Also you are talking about sandboxed clients anyway, they don’t need access to every protocol under the sun. In fact once something like the security-context protocol is happening, the protocol surface is expected to be restricted anyway.
<drakulix[m]>
So that is kinda non-issue.
<emersion>
the synchronization problem can't just be shrugged off
<drakulix[m]>
Oro: That was exactly my argument back then, I have been convinced otherwise since.
<drakulix[m]>
Maybe I should dig up that issue, maybe you want to give it a read?
<Oro>
There wouldn't be much to synchronize
<Oro>
Yes please
<emersion>
not much, ie. the whole overlay state really
<Oro>
Continuing on sync: the overlay could add some preference for where it should be placed on the window. In the unlikely event the window is moved while the overlay is open, it stays centered at the preferred area until it receives the hint to resize too
<emersion>
no, there must be a way to implement a frame-perfect overlay
<emersion>
with the compositor waiting for both surfaces to resize before repainting
<emersion>
(both = game and overlay)
<Oro>
So, for example, an overlay showing the currently speaking users in a voice chat would stay at the preferred region of a window with little issue. Moving around is fine, resizing is mostly fine
<Oro>
Mmmm
<emersion>
the synchronization issues happen more often than one would think
<Oro>
emersion: Without the compositor syncing both clients automatically, there's not much that can be done. But imo this is still way better than anything a nested compositor can do.
<drakulix[m]>
Also overlays like voice chat could just be layer-shell overlay surfaces?
<drakulix[m]>
If you want to hide those, if no game is visible, ext-foreign-toplevel could expose enough information.
<emersion>
also, keep in mind your protocol would require buy-in from steam and all compositors
<Oro>
I have no doubt Steam would like this very much, especially because they're suffering from lack of Wayland support. This would make it easier to do.
<Oro>
Compositors, that's another problem. Would help to get a dev from each to toss their thoughts into the ring on how it could be done.
iomari891 has joined #wayland
<Oro>
Actually, going back to resizing, is there too much delay between sending two resize events to two clients and waiting for them both to resize before putting it onscreen?
<Oro>
If the overlay dies no worries, and if it's not dead then it shouldn't be too hard to resize both within a reasonable timeframe
<davidre>
OOI are you working on Steam or is it an example to illustrate things? If the latter what's the use-case/problem you are interested in?
<Oro>
I want to have an overlay protocol for any potential use cases and maybe some I have in the future too. I'd like to rope Steam in when a protocol is actually submitted
<d_ed[m]>
We've definitely seen issues with Orca, a screen reader which on X11 shows highlight boxes of entries as a floating window above another window.
<d_ed[m]>
(or on wayland, randomly on the screen)
<d_ed[m]>
Dallas Strouse (Oro, any/all): I would strongly suggest collecting the use cases first, rather than starting with a proposal for potential users
<davidre>
I fear that's the wrong way around to go about this, creating a new protocol in case somebody might have a use-case in the future
<psykose>
the use case is already presented though
kts has joined #wayland
<pq>
Concrete use cases help move discussions forward, hand-waving just gets stuck in circles forever. To get attention, I guess a new issue in wayland-protocols would be fine to collect the discussion.
<Oro>
davidre: I know someone has a use case already, that would be Steam, but there are many potential users too, all of which are more or less the same or simple to implement
<zamundaaa[m]>
I can think of at least four other use cases
<d_ed[m]>
yeah, I'm not opposed to such a protocol
<emersion>
what are these?
<Oro>
Want me to pull up a document to keep track of everything?
<davidre>
Oro: You dont know if they would use it
<davidre>
maybe they are happy with their current approach?
<MrCooper>
for the Steam overlay, seems like a sub-surface should do, either directly or via a nested compositor
<Oro>
The steam overlay should be doable via an overlay protocol too though
<zamundaaa[m]>
emersion: Game performance overlays like Mangohud, Streaming overlays like in Zoom (for painting on top of the window), recording overlays like in OBS (that shows keypresses and stuff), community tools for games (don't remember which game had that exactly, can look it up later)
<Oro>
Then it's just figuring out how to sync the two windows
<MrCooper>
any use case could be covered by a bespoke protocol for it, doesn't mean it's a good idea
<Oro>
d_ed[m]: On Windows, some games have Discord integration that will allow Discord to put a transparent box at the corner of the game window with some user profiles and names. If the user is talking, their profile picture lights up green.
<davidre>
If we want to support Steam with such an procotol we should know their actual requirements, instead of designing an overlay protocol that could or could not work for them
<emersion>
that discord thing is completely inside the client iirc
<emersion>
no need for an external overlay
<Oro>
Discord getting their user overlay info onto a game window?
<Oro>
That definitely sounds like it needs an external overlay
<emersion>
the game can just use libdiscord and let that paint a sub-surface
<Oro>
Wouldn't that need the game to know how to handle that though?
<emersion>
no
<emersion>
the OBS keypress overlay, isn't that on the whole screen, not tied to a specific window?
<zamundaaa[m]>
emersion: not if you're recording a window
<Oro>
That's something slightly different. This is focused on overlaying on top of individual windows, not everything
<Oro>
Doing everything would probably need a protocol like wlroots has right now
<d_ed[m]>
Oro: so is that more like a picture-in-picture type floating overlay generically on the screen rather than related to another window?
<Oro>
d_ed[m]: More or less. You do want the floating overlay to move with and be on top of a specific window though
<Oro>
It's not too bad to have it lag behind when resizing and moving around
<Oro>
Not ideal, but not bad
pochu has joined #wayland
<Oro>
I don't believe my question earlier was answered, how easy would synchronization be?
<Oro>
Can't you wait for both windows to say they're ready to be displayed during a resize?
<d_ed[m]>
not trivially
<Oro>
Hmmm, what about if you downscaled the overlay?
cmichael has joined #wayland
<Oro>
Render at a slightly bigger size than needed, at this size it's not an issue, and just downscale that
<Oro>
I don't believe many games allow arbitrary sizes, so this could work
<Oro>
You'd just need a way for the compositor to know that information
<Oro>
For arbitrary sizes, wait for the overlay to give the okay
<Oro>
Then it's just a matter of does the overlay respond fast enough?
<Oro>
I believe that's similar to how a nested compositor would work
<d_ed[m]>
yeah, it's not trivial in a nested compositor either
<Oro>
If the compositor can't resize, the application won't either
<Oro>
Except in this case, if the overlay dies, we don't have to wait on it
<Oro>
And nothing completely breaks
<Oro>
If the process that handles the overlay gets killed, we just.... Stop overlaying it
<Oro>
Or if it hangs and doesn't respond
<Oro>
So, for resizing: why not try and apply the nested compositor logic? It's similar enough here, and it's an alternative solution (albeit maybe a worse one)
<Oro>
Wait for the overlay to resize if it's up, and if not, don't block on it
kts has quit [Quit: Konversation terminated!]
<Oro>
This might actually be slightly better than a nested compositor because then both windows can resize individually, with a nested compositor you wait for one to send the signal to the other
<Oro>
s/other/application /
<Oro>
Then you wait for the application to resize, the compositor, and the host gets the signal to resize and recomposite
<Oro>
With an overlay, both can resize at the same time and say they're ready in less time
<Oro>
If the overlay is up, the compositor would block on the overlay
<drakulix[m]>
Just forcing the latest overlay's buffer into a the same relative space of the overlays window by applying scaling is still against wayland's "every frame is perfect" philosophy. Essentially you would need to document inside the protocol, that your surface might be rescaled in completely unpredictable ways.
<kennylevinsen>
Oro: have you looked at xdg-foreign? That allows you to make your own window a child of another, and if you give it a null input region it'll be input transparent.
<kennylevinsen>
but it requires some initial cooperation, and of course means that composition will might be required as multiple surfaces are present
<Oro>
The hope is to require very little implementation for the overlayed game/app
<Oro>
If it absolutely refuses to implement it, an LD_PRELOAD hack can be done. If it cooperates, it'll communicate the handle for the overlay in the overlay's preferred fashion
<kennylevinsen>
Some compositors also have protocols for "desktop environment" level surfaces like bars, notification panels and launchers which can float over other things (wlr-layer-shell). That can give you a desktop-wide overlay, which might be useful enoufh
<emersion>
i don't think that's a good goal. any impl can be put into a library
<kennylevinsen>
A protocol to force an overlay on something else is unlikely to be accepted as it would be an obvious protocol for clickjacking/phishing/others
<drakulix[m]>
xdg-foreign also doesn't allow you to pick, where you window is positioned relative to the parent window. So it is not really a solution here.
<kennylevinsen>
that isn't necessarily a requirement
<drakulix[m]>
Though I personally think the discord or zoom use-case is covered by layer-shell pretty well.
<kennylevinsen>
Can just have the user put it where they want - might be better than a rigidly positioned overlay tbh
<Oro>
Desktop-wide overlays might want a protocol like wlroots has, not this one.
<Oro>
Also, this is for application-specific overlays like Steam, the user shouldn't put it where they want
<Oro>
The overlay should be over the game or app window and only that
<kennylevinsen>
that's sort of statement always needs to be challenged
<Oro>
Trying to move it, like a screen reader overlay, will not be fun
<kennylevinsen>
I.e. I might disagree with where the overlay went as a user. Top right? Maybe it blocks something and I want it bottom left.
<Oro>
The overlay should have some preferences for that then
<kennylevinsen>
Maybe all corners have important stuff and I want it a bit in
<kennylevinsen>
Forcing positioning should always be last resort where there is no alternative, and "we did it like this before" is not a reason :)
<drakulix[m]>
Also I don't think games are a good use-case either. Given these overlays are likely supposed to work over a range of different operating systems anyway, something that works on the GL/Vulkan API layer (like vulkan layers, which steam is already using anyway) is a much easier solution.
<drakulix[m]>
Which leaves the screen-reader use-case, which is a very legitimate problem, but also a much more isolated one, which might not need such a very broadly applicable protocol.
<Oro>
Maybe the protocol should have a way to say "hey, I'm fine with being in a different position, feel free to move me" so it doesn't break things that shouldn't be moved
<kennylevinsen>
I don't see an argument for not always allowing movement yet
<Oro>
Screen reader highlights
<Oro>
Mainly things that are closely tied to application content placement
<kennylevinsen>
screen readers cannot see content of a window without integration
<kennylevinsen>
So that might take something else anyway
<kennylevinsen>
Unless you screen record and OCR everything, in which case please don't :/
<Oro>
Yes, they'd provide that information through the a11y interfaces; but to get the screen reader overlay on the app, it could use this protocol
<drakulix[m]>
Screen-readers are a specific use-case. That is just like the same discussion about "wayland doesn't let me arbitrarily position my window". It doesn't do that for good reason and applications still manage to work in those constrains. If a specific use-case comes up a protocol is designed for that use-case alone, not for a dozen hypothetical ones.
<kennylevinsen>
The a11y use case could be covered through something similar to the IME protocols
<pq>
I get the feeling there are too many somewhat different use cases being shoved into a single extension that it falls apart in both design and discussion. :-)
<kennylevinsen>
Indeed :)
<pq>
like having decided on a solution and then looking for problems to fit
<kennylevinsen>
start with a specific problem to solve and let us find the best solution for that, and then one can consider making it generic enough to cover other issues as well
<kennylevinsen>
don't start from an overly generic solution - especially something as powerful as arbitrary window hijacking
<Oro>
Going with screen readers first: it should probably allow it to be precisely on the window, so any highlights don't break
<Oro>
For it to obtain the information to draw that overlay to the app, it provides the Wayland handle via a11y
<kennylevinsen>
Look at the IME protocols, they provide very specific positioning (i.e. adjacent to text input fields) without giving away unnecessary control
<Oro>
How would the screen reader put the highlights over the app?
<pq>
Why would the application not highlight itself the parts it gives to screenreader to read out loud? Does it not work like that?
<kennylevinsen>
Oro: as I said, look at the IME protocols, they solve "how does an IME put word suggestions and completions over the app"
<Oro>
You get into some very annoying inconsistencies there
<kennylevinsen>
similar problem, so we might be looking at a similar solution
<Oro>
IME seems to only be for text-based input
<Oro>
If I understand it correctly, it doesn't do things like app widget positioning
<pq>
"application" also includes any (standard?) a11y libraries it might be using in order to connect to a11y services, the highlight does not need to be application-invented. Or is that infrastructure just not there today, and the highlight must be done in a different process for infrastructure reasons?
<kennylevinsen>
it accurately places the IME surface in the correct position for the text input
<Oro>
Do you have anything I can look at for how it handles that?
<Oro>
<pq> ""application" also includes any..." <- Honestly, I'm not all that sure on the current state of a11y infra right now. It seems to be in a very sorry state. I do know TalkBack on Android has them though, I just need to find the technical details.
<pq>
Would be good to know that before suggesting they should use a specific protocol. :-)
<Oro>
* On the Matrix side here's a reply, @d_ed mentioned highlights so maybe the APIs are there.
<Oro>
You'll probably know more on this
<kennylevinsen>
Yeah it's plausible that there is a need for improvements in a11y, but we should take a step back and understand a11y first
<kennylevinsen>
you started with a steam overlay issue, which is something quite orthogonal to screen readers
<Oro>
Yes, then screen readers were mentioned, and some other small use cases
<Oro>
And then it all got to be a mess
<Oro>
For a11y, it sounds like it provides the needed info already, just no way to properly utilize it on Wayland
Berty has joined #wayland
<Oro>
@d_ed can you confirm?
<d_ed[m]>
I meant accerciser not orca; we get relative positions of textboxes and such sent over DBus
<d_ed[m]>
and we get sent what the client thinks is the window position
<Oro>
d_ed[m]: And you can apply highlights over the positions you're given?
<Oro>
At least on X
<Oro>
Bleh, continuing on
<Oro>
The information is there
<Oro>
But not all screen readers or environments will utilize it in the same way
<Oro>
A simple box drawn by an application might not be desired unless it's heavily customizable, such as for contrast. It's probably easier to let a screen reader to overlay its own box on top
<Oro>
It could do that by tightly integrating with the DE, but a simple Wayland protocol that will work on all environments will be even better as it's way less work done
<kennylevinsen>
any protocol that gives you a surface will allow the client to show whatever content with whatever contrast
<kennylevinsen>
say, a protocol where a surface could be placed on a "token", could do that
<kennylevinsen>
a generic window hijack protocol would be simpler to define, and cover many things, but would be a much bigger discussion due to the high risk
MrCooper has quit [Remote host closed the connection]
<kennylevinsen>
definitely needs to be a highly privileged protocol
<Berty>
Guys, what's the deal with Nvidia? I have been using Qtile for almost an year now, but as soon as I switch to Wayland, it starts to misbehave. Last week only, I tried building Hyprland, but wasn't successful.
<Oro>
That only matters if the protocol can scan for arbitrary windows though, right?
MrCooper has joined #wayland
<kennylevinsen>
(but a generic protocol would have significant synchronization issues, so it would likely not be glitch free)
<Oro>
If the protocol just accepts something like a Wayland handle, without specifying how the client obtains it, the client can get it though more secure methods
<Oro>
Like directly from the app in a11y
<kennylevinsen>
yeah if the original client "hands off" permission, taht would be better
<kennylevinsen>
but if content moves (reflow, resize), it would be hard to synchronize
<Oro>
For this protocol, we get the handle through a11y and the screen reader draws the desired contents on top of the app via the overlay
<kennylevinsen>
if the protocol instead specified anchors that the original client moved with content, that issue would go away
<kennylevinsen>
e.g., each word being a token/anchor
<Oro>
That would probably increase complexity vs just blocking on the overlay size though, right?
<Oro>
s/size/resize/
<kennylevinsen>
I imagine it would be less complexity than synchronizing surfaces from two clients in a non-glitchy manner
<Oro>
If the main window gets told to resize when the overlay does though, and it's only done when both say they're ready, wouldn't that work?
<Oro>
So the app says it's done resizing, blocked on the overlay. Overlay relies on the app to resize to a position before resizing
<kennylevinsen>
any buffer commit can be a resize, and words can reflow without resize (e.g., dynamic content)
<Oro>
Or both resize at the same time and the result is displayed when both say they're done
<Oro>
Then it's expected that the information would be sent through a11y and the overlay would update on that
<kennylevinsen>
that won't work, the buffer is already committed
<Oro>
At least on Android, it seems TalkBack will lag just slightly behind the app
<Oro>
Assuming you're doing something like scrolling
<Oro>
(just tested)
<Oro>
So the overlay would always be just slightly older than the apl
<kennylevinsen>
if we're going out of our way to make a protocol, wouldn't it be best to solve that? :)
<Oro>
s/apl/aplapp/
<Oro>
s/apl/app/
<kennylevinsen>
with an anchor approach (similar in nature to the input-method protocol), the compositor can keep the content aligned
<kennylevinsen>
every frame perfect
<Oro>
kennylevinsen: Well, we could block the result being displayed on the overlay being done, but it's not such a large issue that it should be considered imo
<pq>
The overlay can commit on its own volition, so the compositor could mistake an overlay commit to match a resize of the window.
<Oro>
And what if the a11y content changes, like box sizes? An anchor wouldn't be frame perfect either
<Oro>
Oro: (or afaiu)
<Oro>
pq: That shouldn't happen in the first place though
<pq>
why not?
<Oro>
They should only resize both
<pq>
cannot the overlay have animations?
<pq>
who's to say the overlay must change size to match a window resize?
<Oro>
The overlay would be tied to the app window in window size and everything. So a resize of the overlay should make the window resize too.
<kennylevinsen>
having every frame be perfect (by policy, not happenstance) is sort of the whole idea with wayland - so "content is not synchronized at all" seems like a pretty big issue to me
<Oro>
There are two solutions here: you're either just slightly behind, which is quite frankly not a large issue, or you need both overlay and app to update before pushing a new frame
<kennylevinsen>
and a mechanism to tie the two updates together
jmdaemon has quit [Ping timeout: 480 seconds]
<Oro>
You'd send the same resize events to the overlay as you would the window, and when both are done, you push the new frame
<Oro>
Sounds simple enough
<kennylevinsen>
resize isn't an event
<Oro>
If ones not done, don't push a new frame
<kennylevinsen>
resize is what happens when a client commits a buffer of a different size than the previous one
<kennylevinsen>
can happen on any frame
<kennylevinsen>
and content can move without the window
<kennylevinsen>
scroll being your own example
<Oro>
In that case, you compare the size of one to the other, or add some way for a normal client in Wayland (not an overlay) to always specify its size
<Oro>
Comparing the two sizes should work here without being too bad
<drakulix[m]>
It does sound simple enough, in practice you make synchronization the issue of the compositor. and you simplify because you are only looking at resizing, while I bet there are far more nuanced interactions here, that are problematic.
<kennylevinsen>
size isn't an indicator for content
<Oro>
The content is not something we can or probably should be handling
<drakulix[m]>
Like this isn't an easy problem, just because the solution "sounds simple enough".
<kennylevinsen>
speaking from experience in sway's window transaction system, which uses size as a fallback for X11 windows when updating tiling layout
<kennylevinsen>
it's bad
<kennylevinsen>
any synchronization should be on a serial, token, whatever - an explicit thing where you can say "the overlay has ack'd content Y, so all is good"
<drakulix[m]>
Speaking from the transaction logic in smithay: yeah, it really is
<pq>
Content and size are inseparable, like is other window state. They all form a single atomic unit that must not be broken.
<drakulix[m]>
kennylevinsen: At that point you could just send every new frame to the overlay client and let it commit in place of the original surface. If the overlay has to ack all content anyway, we could basically treat the overlays commits as commits for the original window.
<Oro>
kennylevinsen: That sounds better, rather than letting the compositor handle it. When the app sends some new stuff over, the overlay has to put it out (where it makes sure it's up to date) and acks it
<Oro>
Or that?
<Oro>
Honestly I don't like the idea of the two being tied together like that
<drakulix[m]>
That is not a good solution either, because you can't serialize "all information relevant for a commit", because every new protocol can extend that state.
<pq>
The window and overlay contents are inherently connected, because the whole premise is that you want them to be coherent with each other.
<drakulix[m]>
E.g. your resizing logic gets much more complex as soon as you take wp_viewporter into account for example. And that is not just one new challenge to overcome. Every protocol adds state, that might be potentially relevant here.
<kennylevinsen>
and you wouldn't want to do this for every commit, only every commit with relevant changes...
<kennylevinsen>
and then we get back to having something like explicit tokens/anchors/boxes to update
<Oro>
drakulix[m]: So it's just simpler to do as you suggested and have the overlay handle committing it?
<Oro>
Better than the idea I was stuck on of having the two separate
<kennylevinsen>
the overlay cannot commit the original window state
<kennylevinsen>
I mean, the overlay "client" could be a full compositor proxy thingie wrapping stuff, but... That seems like an "interesting" solution :P
<drakulix[m]>
Oro: No, because the original commit would still be received by the compositor and needs to be send to the overlay. The overlay needs to know about viewporter (and every other protocol involved here).
<drakulix[m]>
Acking commits of the parent surface in the overlay is a more reasonable solution. But that is also just "cross-client synchronization".
<drakulix[m]>
Let me rephrase Dallas Strouse (Oro, any/all). I see no solution to this problem, that isn't either:
<drakulix[m]>
- cross-client synchronization, which so far has been a huge NACK from compositor developers.
<drakulix[m]>
- or solving the problem completely on the client-side, e.g. with a nested compositor, some wrapper, vulkan-layer, LD_PRELOAD-hack, you name it.
<drakulix[m]>
- (Or ignoring synchronization, which isn't flying with wayland's principles.)
<Oro>
And I'm strongly against handling it client-side with things like a nested compositor
<Oro>
For reasons stated before, they feel too icky
<Oro>
If I understand how subsurfaces work, they wouldn't be useful here, since an overlay needs to be applied at any time
<Oro>
Not when the app starts up
<kennylevinsen>
(note that a nested compositor moves it outside the target client - the nested compositor is the screen reader - but ickiness factor is still there)
<drakulix[m]>
No matter the type of surface, what you want to do then, is convincing compositor devs, that synchronization across clients, is the better solution. And "being" icky for a solution that is deployed and working for most users, is a weak use-case.
<kennylevinsen>
it would definitely be a good idea to have a good understanding of the various surface roles/shells, and the various synchronization issues we have already dealt with
<kennylevinsen>
can we not talk in matrix.org pastebin links please
<Oro>
(trying)
<drakulix[m]>
(sorry, to used to matrix reply feature, will try my best)
<Oro>
I would love to implemented it, but I probably can't do much outside of the sync part itself
<kennylevinsen>
one line messages :)
<Oro>
s/implemented/implement/
<Oro>
So give me an implementation that handles the overlay like I want, a handle that the overlay can use to draw on top of with another surface, and I can sync em together
Company has joined #wayland
<Oro>
And that would be the screen reader case done: just arbitrary content that the overlay has to ack before it's ever put on screen. Unless the app lies about where the content is, should be no issues with frame perfectness
<drakulix[m]>
Honestly, I envy your drive to just tackle this, but multiple people with good understanding of different compositor code bases, are telling you, that it is not that easy.
<Oro>
And apps lying is not a good thing for a11y anyways
<pq>
Pass libwayland objects and a11y metadata to a library, let it create any sub-surfaces it wants, and coordinate with surface commits to achieve atomic presentation. That's not different from any other library that wants to employ a sub-surface, say, a video player library. Keep everything under the same Wayland connection, so ordering can be guaranteed without compositor help. Might be totally not what
<pq>
existing a11y/whatever infra expects.
<drakulix[m]>
And they need to review, maintain and work with that. Even if I could build a prototype of that in day, I wouldn't want to maintain it, given all the pitfalls.
<drakulix[m]>
Right for the a11y use-case, you require client cooperation anyway in the first place. There is no reason that same client couldn't commit the screen reader overlays.
<Oro>
Wouldn't that client need to have the overlays in the surface as well, though?
<pq>
"overlays in the surface"?
<Oro>
Highlighting drawn inside of the app window
<Oro>
If those terms are correct
<Oro>
And that's an issue because then youd have dozens of different highlights drawn by different toolkits
<pq>
overlay is an overlay, a sub-surface, not painted into the original wl_surface
<Oro>
Hmmm, could you toss a subsurface into a client at runtime, after the original surface is done and made? You'd also need it able to handle applying that subsurface
<Oro>
(done and made being the app is already in use)
<pq>
what do you mean "toss"?
<Oro>
Apply any highlighting and everything from the subsurface into the main surface
<pq>
By sub-surface we mean wl_subsurface protocol object, in case that wasn't clear.
<Company>
question: does Wayland have an env var or something to downgrade versions of globals for testing?
<emersion>
Company: nope
<kennylevinsen>
Oro: subsurface is a specific surface role, any new protocol would introduce a new role with the relevant functionality
<pq>
Company, no. But middle-man debuggers might?
<Company>
I might just add one to GTK
<kennylevinsen>
but as mentioned, the input-method protocol allows a separate IME client to attach a surface windows with text fields
<Oro>
Just text fields aren't suitable here
<kennylevinsen>
so you can think of it like a "sub-surface" just not wl_subsurface
<Oro>
So this would be more expanding on that
<kennylevinsen>
the point of reference to that protocol is not to imply that it you're dealing with text fields, just to note how protocols allow foreign surfaces
<Oro>
Reading some stuff on wl_subsurface again, definitely sounds like what's desired here
<kennylevinsen>
subsurface would not apply, but like with input-method you can make a role that has the behavior you need
<Oro>
Going back to wl_subsurface though, apparently it will work, clients just need to use them
Fxzxmic has joined #wayland
<Oro>
And that's where it gets nasty on that end, I believe
<Oro>
Unless the compositor can handle that?
<kennylevinsen>
subsurface will only ever allow a client to put content within their own surfaces - you need a new role for clients to place surfaces on a foreign surface
<pq>
My main point though is to completely give up the notion of multiple Wayland clients, and do everything though the original Wayland connection opened by the application. Then the compositor does not need to handle anything new.
<kennylevinsen>
pq: yeah an a11y library would be easier
<pq>
*through
<kennylevinsen>
but then it wouldn't even strictly speaking need to know about wayland, the application could render the returned content itself
<Oro>
Mmmm, alright, I'll mark it off
<Oro>
Should we get into things like game overlays?
<pq>
kennylevinsen, indeed, you could trade off Wayland to some buffer passing API or drawing API.
<Oro>
The steam overlay works by injecting itself into arbitrary windows (that had LD_PRELOAD hacks) and managing that
<Oro>
The point here is that games don't need to opt in to the overlay
<drakulix[m]>
I don't think it does? It just uses a vulkan-layer, no?
Berty has quit [Quit: Leaving.]
<Oro>
Or EGL, yes
<pq>
Given the premise of the Steam overlay is to work on non-cooperating applications that cannot be modified, the use case is fundamentally different.
<Oro>
But not all apps will work with this
<Oro>
The steam overlay doesn't work on native Wayland games, for some reason or another
<Oro>
That appears to need a fix on their end
<kennylevinsen>
"mess with content of an arbitrary window without its involvement" is a bit of a red flag for a protocol though. If implemented, it would have to be considered a privileged protocol that clients generally are not allowed to use - like wlr-layer-shell is meant for desktop environment widgets, not apps.
<Oro>
It's not exactly without involvement
<Oro>
The app being modified still has to provide its handle somehow
<drakulix[m]>
But that is an issue with their vulkan layer. In theory every vulkan-layer can work with every windowing system. It wouldn't surprise me, if they use the same layer for windows. The issue is more likely that almost nobody is shipping wayland-only games.
Fxzx_mic has joined #wayland
<Oro>
(except SDL games)
<kennylevinsen>
right but you said "don't need to opt in" though
<Oro>
Assuming they're using the same LD_PRELOAD hacks mentioned before
<drakulix[m]>
Steam can't and won't modify decade old games.
<Oro>
The difference is the overlay is applied through the Wayland protocol
<Oro>
And they usually don't need to
<Oro>
Eventually, one way or another, they'll run on Wayland
<pq>
I see no reason why one could not use the same hook-up points and just create wl_subsurfaces, like one hooks up on X11. Maybe there is something about event delivery having surprising wl_surface arguments the game does not expect?
<Oro>
Be it SDL or a nested compositor
<drakulix[m]>
If the client has to opt-in, you have to make the game send that opt-in command, no?
<Oro>
That's the point of LD_PRELOAD
<Oro>
Steam gets sent the Wayland handles of the game
<drakulix[m]>
If you don't get rid of LD_PRELOAD, how is the solution better using the wayland-protocol than the status quo.
<Oro>
This could be applied to any other protocol or application type, but the main use case is games
<Oro>
Because you dont have to modify EGL and use Vulkan layers and other potentially nasty hacks
<Oro>
It doesn't seem to be very well designed to get it onto the window right now
<drakulix[m]>
vulkan layers are no nasty hacks.
<drakulix[m]>
They are part of the vulkan-spec.
<kennylevinsen>
to clarify, you don't need to just read a handle, the LD_PRELOAD will need to hijack the wayland display to chat with the compositor and create said handles
<Oro>
Referring to the layer itself
<kennylevinsen>
so that would be a hack iself
<drakulix[m]>
If their layer is bad code, you don't work around that by inventing a new convoluted protocol. You tell them to fix their vulkan-layer.
<kennylevinsen>
at least while using LD_PRELOAD - more so than a vulkan layer - although it could be clean without
Fxzxmic has quit [Ping timeout: 480 seconds]
<Oro>
Fair enough
<Oro>
But what if you want an overlay outside of a game?
<drakulix[m]>
(Big if I don't know their overlay code. I would assume it doesn't work on wayland, because its not a high priority.)
<drakulix[m]>
Like for example?
<Oro>
Think system monitors
<Oro>
I could bind an overlay to a keyboard shortcut to pull up my system information and current info
<kennylevinsen>
don't search for problems fitting the solution
<drakulix[m]>
layer-shell.
<Oro>
s/information and current info/infoinformation/
agd5f has quit [Ping timeout: 480 seconds]
<Oro>
drakulix[m]: And to include application-specific content there too?
<Oro>
Like an fps counter
<pq>
Compositors can have an UI to designate arbitrary windows on a "overlay window stacking plane". There is no need for protocol to do that.
<drakulix[m]>
FPS counters can be overlayed via layer-shell just the same. Where you get the information from is another question.
<Oro>
Layer shell can limit an overlay to a specific window?
<drakulix[m]>
no. if you are talking windowed games, I would say vulkan-layers like MangoHUD are a good solution.
<kennylevinsen>
attaching to a specific window is also only visually different if the window is occluded
<Oro>
Wayland feels like a much better place to handle overlays, with the information being handled outside of it
<pq>
The cleanest generic design to handle the Steam overlay case really seems to boil down to an intermediate compositor between the app and the host compositor. It has a bit of overhead, but it's a very simple design with no new protocol needed.
<kennylevinsen>
*cough* gamescope *cough*
<pq>
mmhm
<Oro>
Ideally, an intermediate compositor would perfectly blend in with the host save for the addons
<Oro>
Is that possible at all?
<kennylevinsen>
the imtermediate compositor will be a window just like the original, and can do whatever it would like to the game - including direct pass-through
<kennylevinsen>
it's what gamescope does
Fxzxmic has joined #wayland
<pq>
what does "blend in" mean?
<Oro>
Make it look like the app is running as it would normally
<Oro>
No windoe decorations like gamescope hsd
<pq>
that's easy like kennylevinsen said: just pass through
<Oro>
Wouldn't you need to implement all of the protocols there too though? Or could you put a shim between the one you want and pass everything else though?
<drakulix[m]>
gamescope has decoration, because it just creates an SDLWindow and leaves it at that. There is nothing passed through like decoration-state of the internal window, etc. You can achieve all of that.
<pq>
one may need pass-through code for all protocol interface, but that could probably be automated - we have XML to describe all messages
<drakulix[m]>
You could probably also build a compositor, which just proxies through messages and only intercepts e.g. client buffers to draw an overlay on top.
<drakulix[m]>
Then you don't need to understand every protocol under the sun.
<kennylevinsen>
you probably wouldn't draw on top of them, just attach a subsurface on the way out
<drakulix[m]>
That said I doubt games will use more than a couple of well-known protocols.
<Oro>
So I guess the goal from here would be making a really small and really efficient Wayland compositor that applies the overlay
Fxzx_mic has quit [Ping timeout: 480 seconds]
<pq>
It does not even need to apply an overlay, just make another wl_surface sub-surface for it, while passing all other messages through.
<Oro>
kennylevinsen I thought Wayland clients had to know how to handle subsurfaces in order to use them at all?
<pq>
the game would never know a sub-surface even exists
<davidre>
luckily valve has their own compositor already
<kennylevinsen>
Oro: From the parent compositor perspective, the nested compositor is the client with a surface containining the game. It can use subsurfaces there.
<kennylevinsen>
the game will only have its own main surface and never know about the subsurface
<Oro>
oh
<kennylevinsen>
(because what happens between host compositor and nested compositor, and what happens between nested compositor and game is completely distinct - the nested compositor would just use the game buffers)
<kennylevinsen>
and indeed - that's gamescope
<drakulix[m]>
If you have problems with gamescope - like decorations - I would rather suggest to just open issues and contribute to gamescope to fix these.
<Oro>
Will do
<pq>
The middle-man compositor cannot blindly pass messages through, it needs to translate object ids and filter out the messages not meant for the game, so that kind of overhead exists.
Fxzx_mic has joined #wayland
<Oro>
Oh, I guess a nested compositor like this could work for several clients too, right?
<Oro>
So, I could run a steam game and a non steam game in the same nested compositor and have the overlay on both
<Oro>
The compositor just needs to know how to make those two games separate windows
<Oro>
(gamescope right now only does one)
<kennylevinsen>
if you wanted to, but it doesn't need to
<pq>
yeah, it could, but if you want them separate, then why not run separate nested compositor instances?
<drakulix[m]>
If you wanted to avoid spawning multiple compositors (though there is no real harm in doing that), you could.
<Oro>
Alright, I just didn't want there to have to be one different socket for each game
Fxzxmic has quit [Ping timeout: 480 seconds]
<pq>
I'm not actually sure what would be simpler as the nested compositor implemententation in the end, if one would not want to assume a game has only one window at a time.
<pq>
Oro, why would a different socket be bad?
<Oro>
A different Wayland socket for each game feels unnecessary, since you could probably just reuse what already exists for the first one
<drakulix[m]>
If all you want to avoid is sockets cluttering up XDG_RUNTIME_DIR, you can use WAYLAND_SOCKET and just pass the file descriptor directly to the spawned client.
<pq>
you'd never configure a socket manually anyway
<pq>
a socket per game might even be useful, in case the game has multiple programs each connecting to Wayland either in turn or simultaneously