It's broken in weird ways since recently (`:version` not loading,
segfault in test_version.py). Since nobody should be using it anyways,
there is no point in spending time on debugging a tricky issue.
Next step is probably ripping it out completely, but that's a separate
can of worms.
See #4039
Ubuntu 20.04 will be EOL in April 2025, and PyQt 6.8 does not support being
installed on it anymore:
https://pyqt-builder.readthedocs.io/en/stable/releases.html
Other than for the oldest Qt 5 / Qt 6 envs, and for utility envs, let's use
Ubuntu 22.04 or 24.04.
- Newest Linux/macOS/Windows environments (should be roughly same as release,
especially for Windows/macOS)
- Nightly binary builds
- Release automation
Closes#8205
The upload artifact action can't collect artifacts from /tmp/ in the test
runners. So now that we are writing the screenshots that we want to collect
later to the pytest `tmp_path` location we need to make sure that lives
somewhere the later actions can find it.
Pytest uses `tempfile.gettempdir()` to find the temp dir, and that respects a
number of environment variables including `TMPDIR`. This commits sets TMPDIR
to RUNNER_TEMP in in our test runners to make pytest uses the temp dir that's
mounted into the action containers.
For the docker based runners I can use the `env` map, but for the ubuntu ones
it didn't let me expand `${{ runner.temp }}` in the end map under `step`, so
I'm writing it to the env file for the runner instead. It failed to parse the
action yaml and said:
> Unrecognized named-value: 'runner'. Located at position 1 within expression: runner.temp
For the user part of the `pytest-of-$user` directory, I looked at the new
screenshot related test summary lines to see what the user was called.
`runner` on the ubuntu containers and `user` in our docker containers. Pytest
maintains the "pytest-current" symlink to the latest temp folder.
This commit takes a screenshot of the active browser window when an
end2end test fails. When running on CI a zip file of screenshots will be
attached to the run summary as an artifact. When run locally screenshots
will be left in /$TMPDIR/pytest-screenshots/.
The screenshot is of the Xvfb screen that the tests are running under.
If there are multiple windows open it will likely only show the active
window because a) we aren't running with a window manager b) the Xvfb
display is, by default, the same size as the browser window.
I'm not sure if xvfb is used on the Window runs in CI. We could fall
back to trying to take screenshots if not running under xvfb but I'm a
bit wary of an automatic feature that takes screenshots of people's
desktops when running locally. Even if they just to to /tmp/ it might be
surprising. We can change it later if it turns out we need to try to
take screenshots in more cases.
I'm using pillow ImageGrab, the same as pyvirtualdisplay.smartdisplay. I'm
getting the display number from the pytest-xvfb plugin and formatting it
appropriately (pyvirtualdisplay has an already formatted one which is used by
the smartdisplay, but that's not accessible).
Pillow is now a requirement for running the tests. I thought about making
it gracefully not required but I'm not sure how to inform the user with
a warning from pytest, or if they would even want one. Maybe we could
add a config thing to allow not taking screenshots?
I had to bump the colordepth config for pytest-xvfb otherwise pillow
complained that the default 16bit color depth wasn't supported as it
only supports 24bit, see https://github.com/python-pillow/Pillow/blob/1138ea5370cbda5eb328ec949
8c314d376c81265/src/display.c#L898
I'm saving screenshots to a temp dir because I don't want to put them in
my workdir when running locally. I want to clear the directory for each
run so that you don't get confused by looking at old images. I'm not
100% sure about the lifecycle of the process classes though. Eg if we
have two processes they might both try to create the output directory.
I'm using pytest.session.stash to save the directory so perhaps the
lifecycle of the stash will handle that? Not sure.
Ideally the images would be uploaded somewhere where we could click
through and open them in the browser without having to download a zip
file, but I'm not sure how to achieve that.
It would be nice to print in the test logs that a screenshot was saved
and where to. Just so you could copy paste the filename instead of
having to match the sanitized filename against failing test names. But I
don't know how to log stuff from this stage in the pytest lifecycle.
TODO:
* I'm not sure that the screenshot captures the whole browser window?
Maybe the browser windows is bigger than the X11 display?
Closes: #7625
Will be dropped on GitHub Actions tomorrow:
https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/
For unit tests, we now run them on macOS 13 instead, thus testing on all three
macOS versions we currently support.
For releases, this forces us to now support macOS 12 as the oldest supported
version and drop macOS 11 support. Thus, we should not have a v3.2.2 release.
Not backporting this commit so CI fails there rather than silently bumping up
requirements.
With GitHub Actions now providing macOS 14 runners with M1 chips, we can
build a separate Apple Silicon release there and upload it.
Universal wheels are currently not possible, see #8229 for details.
Closes#6478
This installs pdf.js in a selection of CI jobs. Previously the PDF.js
tests (in qutescheme.feature) were skipped in CI because it wasn't
installed anywhere. There has been a couple of recent cases where pdf.js
started depending on javascript features that are too new for even the
most recent QtWebEngine to support. The aim of this commit is to catch
that case. This doesn't add coverage for older webengine releases.
This also incidentally updates the ace editor in these test jobs, since that is
also updated by default by the update_3rdparty script. Hopefully that
doesn't cause issues.
The reasoning for installing on each type of job:
*ubuntu jobs*: not installed - while our main test runs are on ubuntu
there is an upstream issue where many assets used by pdf.js (like icons
used in the toolbar) aren't packaged, see #7904. This causes warning
messages because assets requested via qutescheme can't be found, which
causes the tests to fail. We could a) install pdf.js from source instead
of using the ubuntu one b) ignore the warning logs c) skip this
environment and rely on tests elsewhere. I've chosen to do (c). I don't
see a huge benefit in testing pdf.js across multiple environments if we
aren't using it installed from the OS anyway. We could install from
source but currently all the Qt < 6.5 tests are failing from some other
JS error, and I think fixing that is out of scope of this issue.
*docker Qt6*: installed - the archlinux pdfjs package works fine and we are
only testing the most recent Qt versions because arch users are expected
to stay up to date.
*docker Qt5*: not installed - doesn't support JS features required by
PDF.js. I guess we could install the legacy build from source here. I'm
mostly worried about catching new breakages for this commit though
*windows*: installed - we install pdf.js from source when making a
release so it would be nice to do that in tests too.
*macos*: not installed - the tests that were catching the breakages are
end2end tests which we don't run on mac. And I think there was an
error from the :versions tests here, don't remember.
*bleeding edge*: installed - from source
pdf.js tests fail on Qt < 6.5 with `Uncaught TypeError: Cannot read
properties of null (reading 'mainContainer')`
The `TestPDFJSVersion.test_real_file` unit tests currently fails because
`version._pdfjs_version()` returns `unknown (bundled)`, not sure why. I
think this is pre-existing and it also wasn't being run on CI.
6.7 is released now, some distros are already shipping it!
This commit:
1. adds a new 6.7 requirements file (the plain 6 one has already been
updated by the bot)
2. adds a new tox env referring to the new requirements file
3. updates the mac and windows installer jobs to run with pyqt67 with the
assumption we'll be including that in our next release
4. adds two new CI environments for 6.7, one each for python 3.11 and 3.12
(3.12 only came out like 6 months ago)
5. updates a couple of references to the py37 tox env that looked like they
were missed, 3.7 support was dropped in 93c7fdd
6. updates various ubuntu containers to the latest LTS instead of the previous
related one - this is quite unrelated to this change but I thought I would
give it a go, no need to use the old one unless we are specifically testing
on it?
- linters - they use tox but probably use system libraries
- these all run in nested containers anyway, should be fully isolated
- codeql - eh, uses a third party action, check the docs if it fails
- irc - as above
It looks like our last release builds were done with python 3.11 and
PyQt 6.5.3. I'm expecting that since PyQt6.6 is out now our next release
will be on 6.6. So lets update the CI to match.
Questions:
* what about python12? I don't think there is a benefit to updating to
that, so lets leave it.
* what about pyqt6.5? Do we care about testing that? Maybe for homebrew
users? We aren't providing new builds with an old Qt right?
last release builds: https://github.com/qutebrowser/qutebrowser/actions/runs/6578864884
ref: https://github.com/qutebrowser/qutebrowser/issues/7989
I'm not sure if we need a py3.11 pyqt6.5 variant or a py3.10 pyqt6.6
one? Those might well be combinations that people have (debian has 3.11
and 6.5 at the moment) but how much coverage do we need?
ref: https://github.com/qutebrowser/qutebrowser/issues/7989
Florian Bruhin
dependabot[bot]
toofar
Philipp Albrecht