- Newest Linux/macOS/Windows environments (should be roughly same as release,
especially for Windows/macOS)
- Nightly binary builds
- Release automation
Closes#8205
They require maintenance, but we don't have a great need for Qt5 builds.
See https://github.com/qutebrowser/qutebrowser/issues/8260
All the Qt5 switches are still in tox, the build release script and the
nsis installer.
The upload artifact action can't collect artifacts from /tmp/ in the test
runners. So now that we are writing the screenshots that we want to collect
later to the pytest `tmp_path` location we need to make sure that lives
somewhere the later actions can find it.
Pytest uses `tempfile.gettempdir()` to find the temp dir, and that respects a
number of environment variables including `TMPDIR`. This commits sets TMPDIR
to RUNNER_TEMP in in our test runners to make pytest uses the temp dir that's
mounted into the action containers.
For the docker based runners I can use the `env` map, but for the ubuntu ones
it didn't let me expand `${{ runner.temp }}` in the end map under `step`, so
I'm writing it to the env file for the runner instead. It failed to parse the
action yaml and said:
> Unrecognized named-value: 'runner'. Located at position 1 within expression: runner.temp
For the user part of the `pytest-of-$user` directory, I looked at the new
screenshot related test summary lines to see what the user was called.
`runner` on the ubuntu containers and `user` in our docker containers. Pytest
maintains the "pytest-current" symlink to the latest temp folder.
This commit takes a screenshot of the active browser window when an
end2end test fails. When running on CI a zip file of screenshots will be
attached to the run summary as an artifact. When run locally screenshots
will be left in /$TMPDIR/pytest-screenshots/.
The screenshot is of the Xvfb screen that the tests are running under.
If there are multiple windows open it will likely only show the active
window because a) we aren't running with a window manager b) the Xvfb
display is, by default, the same size as the browser window.
I'm not sure if xvfb is used on the Window runs in CI. We could fall
back to trying to take screenshots if not running under xvfb but I'm a
bit wary of an automatic feature that takes screenshots of people's
desktops when running locally. Even if they just to to /tmp/ it might be
surprising. We can change it later if it turns out we need to try to
take screenshots in more cases.
I'm using pillow ImageGrab, the same as pyvirtualdisplay.smartdisplay. I'm
getting the display number from the pytest-xvfb plugin and formatting it
appropriately (pyvirtualdisplay has an already formatted one which is used by
the smartdisplay, but that's not accessible).
Pillow is now a requirement for running the tests. I thought about making
it gracefully not required but I'm not sure how to inform the user with
a warning from pytest, or if they would even want one. Maybe we could
add a config thing to allow not taking screenshots?
I had to bump the colordepth config for pytest-xvfb otherwise pillow
complained that the default 16bit color depth wasn't supported as it
only supports 24bit, see https://github.com/python-pillow/Pillow/blob/1138ea5370cbda5eb328ec949
8c314d376c81265/src/display.c#L898
I'm saving screenshots to a temp dir because I don't want to put them in
my workdir when running locally. I want to clear the directory for each
run so that you don't get confused by looking at old images. I'm not
100% sure about the lifecycle of the process classes though. Eg if we
have two processes they might both try to create the output directory.
I'm using pytest.session.stash to save the directory so perhaps the
lifecycle of the stash will handle that? Not sure.
Ideally the images would be uploaded somewhere where we could click
through and open them in the browser without having to download a zip
file, but I'm not sure how to achieve that.
It would be nice to print in the test logs that a screenshot was saved
and where to. Just so you could copy paste the filename instead of
having to match the sanitized filename against failing test names. But I
don't know how to log stuff from this stage in the pytest lifecycle.
TODO:
* I'm not sure that the screenshot captures the whole browser window?
Maybe the browser windows is bigger than the X11 display?
Closes: #7625
Will be dropped on GitHub Actions tomorrow:
https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/
For unit tests, we now run them on macOS 13 instead, thus testing on all three
macOS versions we currently support.
For releases, this forces us to now support macOS 12 as the oldest supported
version and drop macOS 11 support. Thus, we should not have a v3.2.2 release.
Not backporting this commit so CI fails there rather than silently bumping up
requirements.
With GitHub Actions now providing macOS 14 runners with M1 chips, we can
build a separate Apple Silicon release there and upload it.
Universal wheels are currently not possible, see #8229 for details.
Closes#6478
This installs pdf.js in a selection of CI jobs. Previously the PDF.js
tests (in qutescheme.feature) were skipped in CI because it wasn't
installed anywhere. There has been a couple of recent cases where pdf.js
started depending on javascript features that are too new for even the
most recent QtWebEngine to support. The aim of this commit is to catch
that case. This doesn't add coverage for older webengine releases.
This also incidentally updates the ace editor in these test jobs, since that is
also updated by default by the update_3rdparty script. Hopefully that
doesn't cause issues.
The reasoning for installing on each type of job:
*ubuntu jobs*: not installed - while our main test runs are on ubuntu
there is an upstream issue where many assets used by pdf.js (like icons
used in the toolbar) aren't packaged, see #7904. This causes warning
messages because assets requested via qutescheme can't be found, which
causes the tests to fail. We could a) install pdf.js from source instead
of using the ubuntu one b) ignore the warning logs c) skip this
environment and rely on tests elsewhere. I've chosen to do (c). I don't
see a huge benefit in testing pdf.js across multiple environments if we
aren't using it installed from the OS anyway. We could install from
source but currently all the Qt < 6.5 tests are failing from some other
JS error, and I think fixing that is out of scope of this issue.
*docker Qt6*: installed - the archlinux pdfjs package works fine and we are
only testing the most recent Qt versions because arch users are expected
to stay up to date.
*docker Qt5*: not installed - doesn't support JS features required by
PDF.js. I guess we could install the legacy build from source here. I'm
mostly worried about catching new breakages for this commit though
*windows*: installed - we install pdf.js from source when making a
release so it would be nice to do that in tests too.
*macos*: not installed - the tests that were catching the breakages are
end2end tests which we don't run on mac. And I think there was an
error from the :versions tests here, don't remember.
*bleeding edge*: installed - from source
pdf.js tests fail on Qt < 6.5 with `Uncaught TypeError: Cannot read
properties of null (reading 'mainContainer')`
The `TestPDFJSVersion.test_real_file` unit tests currently fails because
`version._pdfjs_version()` returns `unknown (bundled)`, not sure why. I
think this is pre-existing and it also wasn't being run on CI.
6.7 is released now, some distros are already shipping it!
This commit:
1. adds a new 6.7 requirements file (the plain 6 one has already been
updated by the bot)
2. adds a new tox env referring to the new requirements file
3. updates the mac and windows installer jobs to run with pyqt67 with the
assumption we'll be including that in our next release
4. adds two new CI environments for 6.7, one each for python 3.11 and 3.12
(3.12 only came out like 6 months ago)
5. updates a couple of references to the py37 tox env that looked like they
were missed, 3.7 support was dropped in 93c7fdd
6. updates various ubuntu containers to the latest LTS instead of the previous
related one - this is quite unrelated to this change but I thought I would
give it a go, no need to use the old one unless we are specifically testing
on it?
- linters - they use tox but probably use system libraries
- these all run in nested containers anyway, should be fully isolated
- codeql - eh, uses a third party action, check the docs if it fails
- irc - as above
Currently the unstable docker images are failing to build
(undefined symbol: _ZN5QFont11tagToStringEj, version
Qt_6. Looks like Qt has upgraded to 6.7 but pyqt6 hasn't been patched to
remove some symbols that are gone now).
But we might as well let the stable ones rebuild right?
It seems `sed -i` is not very portable. Initially we were using this
command:
sed -i '' '/.-d., .--debug.,/s/$/ default=True,/' qutebrowser/qutebrowser.py
and then that started breaking on windows, I'm not sure why, with "can't
read /.-d., .--debug.,/s/$/ default=True,/: No such file or directory".
Then we changed to:
sed -i '/.-d., .--debug.,/s/$/ default=True,/' qutebrowser/qutebrowser.py
so without the extension argument, but that broke on mac with "1:
"qutebrowser/qutebrowser.py": extra characters at the end of q command"
then we tried:
sed -i'' '/.-d., .--debug.,/s/$/ default=True,/' qutebrowser/qutebrowser.py
and that also broke on mac with the same error. On the recommendation of
stackoverflow I just changed it no not use in-place editing and do a
good old fashioned move afterwards. https://unix.stackexchange.com/questions/92895/how-can-i-achieve-portability-with-sed-i-in-place-editing
... record scratch ...
Apparently these GHA steps are being run in powershell in windows where
`mv` is implemented by `Move-Item` where you have to use -Force to
overwrite destination files.
But that's not portable. cp does happily overwrite without any
additional instruction though. So I'm doing cp instead of mv and then
removing the temp file.
Probably if this drags out anymore we should download something off of
pypi which is platform independent to handle it.
The nightly jobs have a `workflow_dispatch` action, which means you can
kick the job off on any branch. But the build steps has the branch to
build on hardcoded. I would like to be able to build windows and mac
builds without having a local build environment setup.
The docs for the checkout action says it default to the main branch,
so the scheduled actions should keep working fine. But now we'll be able
to create builds off of other branches too.
docs: https://github.com/actions/checkout#usage
ref: https://github.com/qutebrowser/qutebrowser/issues/7989
It looks like our last release builds were done with python 3.11 and
PyQt 6.5.3. I'm expecting that since PyQt6.6 is out now our next release
will be on 6.6. So lets update the CI to match.
Questions:
* what about python12? I don't think there is a benefit to updating to
that, so lets leave it.
* what about pyqt6.5? Do we care about testing that? Maybe for homebrew
users? We aren't providing new builds with an old Qt right?
last release builds: https://github.com/qutebrowser/qutebrowser/actions/runs/6578864884
ref: https://github.com/qutebrowser/qutebrowser/issues/7989
I'm not sure if we need a py3.11 pyqt6.5 variant or a py3.10 pyqt6.6
one? Those might well be combinations that people have (debian has 3.11
and 6.5 at the moment) but how much coverage do we need?
ref: https://github.com/qutebrowser/qutebrowser/issues/7989
By default, we only get a narrow checkout, so we don't know about any
other branches. Use the GitHub API and some JS to get the release branch
instead.
I don't think the dependencies jbos needs to be installing asciidoc anymore either but oh
well.
I just grepped for libxcb-shape0 and constituently added libxcb-cursor0
after it. I haven't checked it's in stretch but it should be.
ref: 6f5de192e0
- Make sure we still test Python 3.7 and 3.8 after dropping old PyQt
versions in c5a51eb0bc
- Keep a modern Python version (3.11) with Qt 5 around, however
- Make sure we still test Python 3.10 too
- Also start testing the Python 3.12 alpha
Error: qutebrowser/browser/webengine/certificateerror.py has 90.91% line and 50.00% branch coverage!
Error: qutebrowser/browser/webengine/darkmode.py has 96.88% line and 100.00% branch coverage!
Error: qutebrowser/keyinput/basekeyparser.py has 97.59% line and 100.00% branch coverage!
Error: qutebrowser/keyinput/keyutils.py has 92.64% line and 96.30% branch coverage!
Error: qutebrowser/utils/debug.py has 94.51% line and 92.19% branch coverage!
Error: qutebrowser/utils/qtutils.py has 98.67% line and 96.74% branch coverage!
Error: qutebrowser/utils/usertypes.py has 99.20% line and 96.30% branch coverage!
Error: qutebrowser/utils/version.py has 97.66% line and 97.83% branch coverage!
Maybe we should open an issue to get them back up.
On CI were were getting "Could not import sip" because link_pyqt was
looking for PyQt5.sip.
I made that look at QUTE_QT_WRAPPER since that's being set already on
tox.ini
There are probably a few other changes around link_pyqt and the makefile
etc we need to change when we switch the default wrapper.
I overrode the default `py` tox environment with py-qt6 to override
those wrapper related variables. I probably could have done something
sneaky with curly braces to make it so we don't have to add a few more
lines to the file. But in my opinion in config file is far to obfuscated
and hard to maintain already.
I changed the docker file to call the new py-qt6 env if it's a qt6
container. I'm not 100% sure that is required though since there is also
a tox invocation in the GH action definition, maybe that overrides the
container entrypoint? Also changed the indentation in the dockerfile
template a bit to make it easier to see where the conditionals start and
end.
Speaking of which I changed the matrix definition and tox invocation to
match a later one to hopefully make it so we can invoke different tox
environments in the containers without having to rebuild the containers.
Not sure I did that right, I'll see soon.
I added the unstable-qt6 container generation line so we can use it in
the future, and to match the not-qt6 one. I'm not switching to that in
CI though because the pyqt used by that is broken at the moment
(ref https://www.riverbankcomputing.com/pipermail/pyqt/2023-March/045214.html)
Also fixed the vim modeline in generate.py so my syntax highlighting
works.
6.4 tests should be passing now. 6.5 is still waiting on PyQt on PyPI
getting fixed. That goes for kde-unstable-qt6 too as that's pulling down
6.5.
We should take the opportunity to revisit the test matrix configuration
again when we add 6.5 in there. For example py3.11 on ubuntu 20.04 is a
bit odd as it only got released last year.
Florian Bruhin
dependabot[bot]
toofar
Philipp Albrecht