Not yet quite sure what exactly is the culprit, but this seems to help for all
tests (!) to pass with Xvfb locally.
For now only scoped to Qt 6.9.0. Will probably already need to reevaluate with
the RC, but definitely with the final release.
See #8444
For tests that do the same prompt in the same session, they are failing
on Qt6.8 and PyQWebEnginet6.7 because we can't disable the new WebEngine
behaviour.
We can work around this by getting a fresh instance for each test, but I
don't want to make more tests slower if I don't have to. So move that
logic into a custom "fresh instance" prompt that will only create one
when needed.
QtWebEngine has a new feature where it will remember what permissions
you have granted or denied. It has three options regarding permissions
storage:
AskEveryTime -- don't store
StoreInMemory -- store in memory only
StoreOnDisk -- store in memory and on disk
By default it does the StoreOnDisk behavior. Having webengine remember
whether you granted or denied a permission would make the qutebrowser UX
around that area inconsistent. For example the default y/n actions for a
permission prompt in qutebrowser will only accept that single
permission request, and you'll be re-prompted if you reload the page.
Users may be used to this and if webengine started remembering
permission grants the users may be surprised to find that the page was
accessing features without prompting them for permission anymore.
Additionally we already have our own permission storage machinery in
autoconfig.yml.
This commit will set the webengine feature to AskEveryTime, which disables
any storing of permission grants.
Also adjusts the skip marker of the affected tests so they'll be enabled
again on Qt versions with the appropriate permissions API.
Qt have updated their permission storage feature so it respects our
the setting our basedir feature uses, so now all the tests that use
"Given I have a fresh instance" are passing.
The remaining failing ones do pass if I make them run in a fresh
instance, but I am leaving them as xfail because a) opening a new
instance is slow b) the new upstream behaviour causes a regression in
the qutebrowser behavior (you don't get re-prompted where you would have
been previously) so I feel like it is correct for some tests to be
failing! We have to set AskEveryTime at some point and we can address
them then.
The message is:
The following paths were searched for Qt WebEngine dictionaries:
/tmp/qutebrowser-basedir-qrhbqblr/data/qtwebengine_dictionaries
but could not find it.
Spellchecking can not be enabled.
Tests are failing on "Logged unexpected errors".
This commit takes a screenshot of the active browser window when an
end2end test fails. When running on CI a zip file of screenshots will be
attached to the run summary as an artifact. When run locally screenshots
will be left in /$TMPDIR/pytest-screenshots/.
The screenshot is of the Xvfb screen that the tests are running under.
If there are multiple windows open it will likely only show the active
window because a) we aren't running with a window manager b) the Xvfb
display is, by default, the same size as the browser window.
I'm not sure if xvfb is used on the Window runs in CI. We could fall
back to trying to take screenshots if not running under xvfb but I'm a
bit wary of an automatic feature that takes screenshots of people's
desktops when running locally. Even if they just to to /tmp/ it might be
surprising. We can change it later if it turns out we need to try to
take screenshots in more cases.
I'm using pillow ImageGrab, the same as pyvirtualdisplay.smartdisplay. I'm
getting the display number from the pytest-xvfb plugin and formatting it
appropriately (pyvirtualdisplay has an already formatted one which is used by
the smartdisplay, but that's not accessible).
Pillow is now a requirement for running the tests. I thought about making
it gracefully not required but I'm not sure how to inform the user with
a warning from pytest, or if they would even want one. Maybe we could
add a config thing to allow not taking screenshots?
I had to bump the colordepth config for pytest-xvfb otherwise pillow
complained that the default 16bit color depth wasn't supported as it
only supports 24bit, see https://github.com/python-pillow/Pillow/blob/1138ea5370cbda5eb328ec949
8c314d376c81265/src/display.c#L898
I'm saving screenshots to a temp dir because I don't want to put them in
my workdir when running locally. I want to clear the directory for each
run so that you don't get confused by looking at old images. I'm not
100% sure about the lifecycle of the process classes though. Eg if we
have two processes they might both try to create the output directory.
I'm using pytest.session.stash to save the directory so perhaps the
lifecycle of the stash will handle that? Not sure.
Ideally the images would be uploaded somewhere where we could click
through and open them in the browser without having to download a zip
file, but I'm not sure how to achieve that.
It would be nice to print in the test logs that a screenshot was saved
and where to. Just so you could copy paste the filename instead of
having to match the sanitized filename against failing test names. But I
don't know how to log stuff from this stage in the pytest lifecycle.
TODO:
* I'm not sure that the screenshot captures the whole browser window?
Maybe the browser windows is bigger than the X11 display?
Closes: #7625
The "Async question interrupted by async one" test was failing on CI but
not locally. Not sure why, changing the tests to skip instead of xfail.
The downside is that we won't bet a notification when upstream fixes the
issues, hopefully they mark the QTBUG as closed and we see it that way.
I think we do have to do something else to deal with this persistent
permission thing anyway, assuming they don't change it to be
off-by-default, so I'm sure we'll be looking in this area again!
They'll at the very least be re-enabled when we get a PyQt 6.8.
The test (`test_failing_flush()`) is deliberately trying to write to a
close file. This is a new error in Qt 6.8 it seems.
Ignore just the specific file for this test in case it pops up somewhere
else later.
WebEngine now has it's own mechanism to remember permission, and it's
turned on by default. We can't disable it until PyQt picks up the new
`QWebEngineProfile::setPersistentPermissionsPolicy()`. So the first test
that prompts for a permission will persist that setting and later ones
will fail because they don't get the prompt they expect.
For now lets set them to xfail while we figure out what to do with
permission persisting for actual users. That we we can reduce the noise
in the test results!
The WebEngine permissions persistence mechanism doesn't seem to
respect whatever we are doing to separate storage per-basedir. Testing
with a temp basedir like so:
python3 -m qutebrowser -T https://web-push-book.gauntface.com/demos/notification-examples/
I see this file has been created under my home directory:
$ cat ~/.local/share/qutebrowser/qutebrowser/QtWebEngine/Default/permissions.json
{"Notifications":{"https://web-push-book.gauntface.com/":true}}
I've raised an issue upstream about that here: https://bugreports.qt.io/browse/QTBUG-126595
We've got two things to think about regarding how to deal with this new
on-by-default feature:
1. what do we do for the tests? We can Disable the feature (if on new
enough PyQt) or add a test setup step that ... restarts the browser
and deletes the permissions.json. That's not great
2. what do we do for real users? See below
By default I would recommend disabling the webengine one since we
already have our own. BUT we can't actually disable it until PyQt
updates with the new APIs, which can take a while, and it's pretty
likely people will be using the new Qt version before PyQt updates. So
it would be best to figure out what we can do before that! Can we make
it respect the basedir data path? Can we make it write to some fake file
we throwaway? chmod +i?
Hopefully Qt makes the permission JSON respect the data path we set and
then at the very least we can remove the JSON file after a permission is
set. It'll still be a change in behavior for users on Qt 6.8 and PyQt
6.7 though as it'll likely remember permissions within a browser
instance by default, which isn't the case for our implementation
currently.
Related to: https://github.com/qutebrowser/qutebrowser/issues/8242#issuecomment-2175949686
This starts happening reliably with the "Using session completion" end
to end test after the previous change where I made it so we don't set
the completion widget model to none, just call `deleteLater()` on the old
one. The relevant part of the test is:
And I run :session-save hello
# This loads a completion model with one entry
When I run :cmd-set-text -s :session-load
# This one selects the entry, which clears the model because the
# session completion only applies to the first word after the
# command, which we've just selected.
And I run :completion-item-focus next
# This does nothing, since we have no completion model loaded,
# not sure why it is in the test, presumably making sure no
# exception is thrown.
And I run :completion-item-focus next
The log message is reliable in the test but it's a bit flaky to
reproduce manually. I've reproduced it via `for f in 1 2;do qute-send -i
/tmp/qutebrowser-basedir-npzdpupy/runtime/ipc-8075cf54f63b8e1bc05db34f41292c38
":completion-item-focus next" ;done` if I manually go through the
scenario a few times. Possibly it would reproduce easier if I pin it to
one process using `taskset`.
I think the reason for this is that the model is marked for deletion in
the next event loop, then a signal goes out, then the selection model is
marked for deletion on the next event loop. And possibly this only
happens between the model and the selection model being deleted?
In the bleeding edge CI tests I'm seeing:
IGNORED: Path override failed for key base::DIR_APP_DICTIONARIES and path '/__w/qutebrowser/qutebrowser/.tox/bleeding/lib/python3.11/sitePath override failed for key base::DIR_APP_DICTIONARIES and path '/__w/qutebrowser/qutebrowser/.tox/bleeding/lib/python3.11/site-packages/PyQt6/Qt6/libexec/qtwebengine_dictionaries'
INVALID: -packages/PyQt6/Qt6/libexec/qtwebengine_dictionaries'
Where the second line looks like maybe spill over from the first one
that is for some reason being printed twice and interleaved! Maybe
different processes or threads?
I did a regex for "line ending with this and with no spaces in it" which
is hopefully safe enough to not pick up a wrong line accidentally.
Florian Bruhin
toofar