This seems to be an issue with CEF and applications running inside an iFrame. getDisplayMedia seems to be working as expected and show a popup to select either a window or the entire desktop to share when it is invoked in the developer console of the Genesys application in the root scope. If it is called from the scope of an application running in an iframe, the same popup is not shown.
After going through the issues of the CEF project, there is a workaround. You can add this flag to the Genesys application executable --use-fake-ui-for-media-stream but it is not ideal, from an administrative point of view. All the agents would have to somehow add this flag...
Please let us know if there is a fix on the way.