The normal PDF.js build only officially supports the latest Chromium, so things
might break every once in a while with QtWebEngine (e.g. #8199, #7335).
Let's instead bundle and recommend the legacy build.
Closes#8332Closes#7721 (reworded)
Also see #7135
Similarly to #5998, XHR requests should be able to set their custom
Accept-Language values - and for some odd reason, stuff breaks on websites
sometimes when that's not respected.
There's no way for qutebrowser to know if a given header value is already set in
a request (i.e. whether we're adding or overriding). Thus we only really have
two options here:
1) Don't set any shared.custom_headers() for XHR requests at all.
2) Special-case Accept-Language here, because the issue usually is triggered by
the global override, but that already gets set just fine via QWebEngineProfile
anyways.
Given that 2) is the thing causing trouble in the wild and it's unclear what the
desired behavior for 1) is (e.g. for the DNT header), let's go for 2) here.
With the upgrade to MarkupSafe 3.0, something funny happened when trying to pass
the GUIProcess object to jinja after launching a userscript:
[...]
File "[...]/qutebrowser/browser/qutescheme.py", line 291, in qute_process
src = jinja.render('process.html', title=f'Process {pid}', proc=proc)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]/qutebrowser/utils/jinja.py", line 123, in render
return environment.get_template(template).render(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[...]
File "html/process.html", line 11, in block 'content'
File "[...]/lib/python3.11/site-packages/markupsafe/__init__.py", line 42, in escape
if hasattr(s, "__html__"):
^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: wrapped C/C++ object of type GUIProcess has been deleted
This can be reproduced with:
qutebrowser --temp-basedir ':cmd-later 0 spawn -u -o /bin/echo test'
We pass the `GUIProcess` to the Jinja template as `proc`, which then formats it as
`{{ proc }}`` (to stringify it). For some reason, with the newest MarkupSafe/Jinja
versions, this now triggers the `if hasattr(s, "__html__")` check in MarkupSafe
(which has been around for a while). That then presumably causes PyQt to try and
access the underlying C++ object for `GUIProcess``, but that has already been
deleted.
But why is it deleted in the first place, if we keep track of even completed
processes data ever since we added `:process` in a3adba81c? It looks like the Qt
parent-child relationship is the culprit here: When we pass a parent to the
`GUIProcess`` from the userscript runner, it will get deleted as soon as said
runner is cleaned up (which happens after the userscript has finished).
We probably never noticed this before because we only accessed data from the
Python wrapper and not from the C++ side, but it still seems like a good idea
to avoid passing a parent for a long-lived object (with time-based cleanup) in
the first place.
This flag is vital for the allow-list configuration to be picked up.
It should be set globally and `dev-qt/qtwebengine` should be
recompiled after it's enabled.
Ref #8313.
With PyQt 6, this gets represented as
QWebEnginePage.RenderProcessTerminationStatus(-1)
which is != -1, thus leading to a KeyError.
Updating to a RendererProcessTerminationStatus
enum value works fine on both PyQt5 and PyQt6.
Will be dropped on GitHub Actions tomorrow:
https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/
For unit tests, we now run them on macOS 13 instead, thus testing on all three
macOS versions we currently support.
For releases, this forces us to now support macOS 12 as the oldest supported
version and drop macOS 11 support. Thus, we should not have a v3.2.2 release.
Not backporting this commit so CI fails there rather than silently bumping up
requirements.
My virtualenv I used to run webkit has rotted long ago and I don't remember
how I set it up. There is a PyQtWebKit project on PyPI but I don't know
who that's published by.
So I figured I would write some notes for myself on using the docker container
used for CI instead. I chose to mount the current directory (which is
presumably a qutebrowser checkout!) directly into the container instead of
cloning it so I could have quicker feedback between making code changes and
running tests.
Then there's a couple of things that stem from that. Since the user in the
container is different from the one in the host we have to move some things
that are normally written to the current directory to be written elsewhere.
There are other ways to approach this (eg you can add `-u $(id -u)` to the
docker command line, although that makes things a bit confusing in the
container) but arguably it's good for the container not to be able to write to
the host, hence making that volume read only.
The TOX_WORK_DIR trick is from
[here](https://github.com/tox-dev/tox/issues/20), apart from with
`{toxinidir}` in it too because the pyroma env was failing with just
`.tox`, saying the pyroma binary needed to be in the allowlist, possibly
it was doing full path matching without normalizing.
The hypothesis folks
[here](https://github.com/HypothesisWorks/hypothesis/issues/2367#issuecomment-595524571)
say if you want to override the examples DB location with an env var to
do it yourself. It's actually only a warning from hypothesis, it says it
falls back to an in-memory DB, but I guess the tests run with
warnings-are-errors. You can also pass `database=None` to make
hypothesis skip example storage altogether.
I'm using tox to run commands in a virtualenv with the right stuff in it
because, uh, because I was copying the CI workflow actually. I just found out
about the `exec` subcommand to override the `commands` defined for the env,
neat! One point of awkwardness about that is that since we are using the
PyQt from the OS we need any virtualenv we use to have access to the OS
packages, which isn't the default for virtualenvs created by tox. The
text envs use the link_pyqt script for that but if you are using this
container and the first thing you do is run `tox exec` then that
wouldn't have been run. So I'm setting `VIRTUALENV_SYSTEM_SITE_PACKAGES`
to tell tox to always make the system packages available in the
virtualenvs it manages.
I did try using the mkvenv script instead of tox but it complained when
trying to install the current directory in editable mode because
setup.py tries to write to a git-commit-id file.
With GitHub Actions now providing macOS 14 runners with M1 chips, we can
build a separate Apple Silicon release there and upload it.
Universal wheels are currently not possible, see #8229 for details.
Closes#6478
I was getting crash reports from someone about this. Not sure what's going wrong
there (hence the additional information in the exception).
What's clear however is that we're raising ParseError, but only handling that
when actually parsing. The code calling copy_/_find_webengine_resources only
handles OSError. So let's raise a FileNotFoundError instead.
When qutebrowser is running but its installation has been deleted/moved, it
fails in somewhat mysterious but predictable ways. This is e.g. the case
currently, when people upgrade their Archlinux packages and upgrade from Python
3.11 to 3.12. When doing that with qutebrowser open, on the next page load, it
will:
- Have a crashed renderer process, because (assumingly) the process executable
is gone on disk.
- Which then causes us trying to render an error page, but that fails due to
broken_qutebrowser_logo.png being gone from disk.
- The FileNotFoundError then causes jinja2 to import jinja2.debug at runtime,
but that *also* fails because the jinja2 package is gone.
We work around this by loading the PNG into RAM early, and then using the cached
version instead. This amends b4a2352833 which did
the same with HTML/JS resources, but never for this PNG, which (looking at crash
logs) seems to be a somewhat common breakage.
Alternatives I've considered:
- Catching the FileNotFoundError and not showing an error page at all.
- Generating a PNG with an explanatory text via QPainter and returning that.
However, with the renderer process crash happening in the first place for
unknown reasons, it's unclear if the error page ever gets actually displayed...
Let's roll with this for now, and if this causes a repeating renderer process
crash, fix that separately (also see #5108).
Fixes#8145, see #5390.
As long as we don't have a solution to get notified about focus happening
(#2471 possibly?), it looks like there is no better way to get notified
about this, so a delay will need to do for now.
Webengine added a getter for their chromium patch level back in Qt 6.3,
since they backport security fixes from chromium in the periods between
doing major chromium feature upgrades.
It's pulled from a hardcoded string in the webengine source
`src/core/web_engine_context.cpp` that's manually updated when they
backport something.
The "(plus any distribution patches)" bit in there is because it was
pointed out that some distributions backport their own security patches
or even use webengine from a branch when the hardcoded string only gets
updated at release time, despite patches being backported in the
meantime.
Closes: https://github.com/qutebrowser/qutebrowser/issues/7187
I would like to make merging PRs lower friction. One aspect of that for
me is having to think about where to add the changelog info, whether it
should go in an existing section, whether I should create a new section,
what the format changelog is supposed to be in. These questions are a
bit coupled with the decision of whether to backport a change or not.
Those aren't hard questions but I don't usually have a long stretch of
time for open source work. So making it so I don't have to make those
decisions at merge time makes it easier for me to fit that work into my
day. Previously it seemed to me that the norm was to only have a future
changelog entry for the next patch release. Occasionally I would merge
stuff and add it to the patch release changelog entry and then think
about how I would have made getting any security fixes out harder or how
it would have to be corrected at backport time. So this is kind of a
pre-commitment that yes, we are going to be merging stuff to main that
won't make it to the next release.
A lot, but not all, of the above rambling will also be mitigated by
adopting a fragment based changelog management system (#7101), because
that means that more of the stuff we have to worry about when merging is
only in the context of the PR. Eg just describe that the change does,
don't worry too much about where that description is going to end up.
Other follow up stuff we could do if norms are established or need
re-enforcing:
* update contributor docs to describe more of the branching strategy as
it applies to merging
* update contributor docs to describe backporting steps and philosophy
* link changelog entries to milestones?
Almost 7 years ago, it was observed that hiding the status bar causes some
websites being scrolled to the top: #2236.
Back then, it never really was clear why this happens. However, with the v3.0.0
release, we had a regression causing the same thing to happen when leaving
prompt mode: #7885.
Thanks to "git bisect", the culprit was found to be 8e152aa, "Don't give
keyboard focus to tab bar", which was a fix for #7820. However, it still wasn't
clear why this phenomenon happens.
What made things clearer to me was a combination of debugging and an old comment
by pevu: https://github.com/qutebrowser/qutebrowser/issues/2236#issuecomment-337882102
> Chromium-browser has the same issue. When you open lipsum.com, scroll down,
> then focus the location bar (url box), then press Tab, it will jump to the
> top of the page and focus the first link. This doesn't happen when you
> switch focus using the mouse.
>
> It seems to be an issue of how the view containing the website is focused
> when all qutebrowser ui elements disappear.
And indeed, tabbing into the web contents from the UI elements via the tab key
in Chromium causes the website to start at the top, presumably as an
accessibility feature?
Essentially, this is also what happens in qutebrowser when an UI element is
hidden while it still has focus: In QWidget::hide() (or, rather,
QWidgetPrivate::hide_helper()), Qt moves the focus to the next widget by
calling focusPrevNextChild(true):
https://github.com/qt/qtbase/blob/v6.6.1/src/widgets/kernel/qwidget.cpp#L8259-L8271
And apparently, focusPrevNextChild() basically does the same thing as pressing
the tab key, to the point that there is some code in Qt Declarative actually
making tab keypresses out of it (which I'm still not sure is related, or maybe
just the cause of #4579):
https://github.com/qt/qtdeclarative/blob/v6.6.1/src/quickwidgets/qquickwidget.cpp#L1415-L1429
Some debugging confirms that this is exactly what happening:
1) We hide the status bar (or prompt) which has keyboard focus
2) Qt focuses the web view, which triggers the Chromium feature (?) scrolling it
to the very top.
3) Only then, in TabbedBrowser.on_mod_left(), we noticed that the command or
prompt mode was left, and reassign focus to the web view properly.
In step 2), before this change, Qt happened to focus the tab bar (before we set
the focus manually to the web contents), and thus this didn't happen.
Not sure why it didn't focus the tab bar when we hid the status bar (maybe
because how our widget hierarchy works with TabbedBrowser?).
Python stacktrace of hiding prompt:
Traceback (most recent call first):
<built-in method hide of DownloadFilenamePrompt object at remote 0x7fffb8bc65f0>
File ".../qutebrowser/mainwindow/prompt.py", line 204, in _on_mode_left
self.show_prompts.emit(None)
File ".../qutebrowser/keyinput/modeman.py", line 434, in leave
self.left.emit(mode, self.mode, self._win_id)
File ".../qutebrowser/keyinput/modeman.py", line 445, in mode_leave
self.leave(self.mode, 'leave current')
C++ stacktrace, with the focus change presumably being passed of to Chromium
here: https://github.com/qt/qtwebengine/blob/dev/src/core/render_widget_host_view_qt_delegate_client.cpp#L714#0 QtWebEngineCore::RenderWidgetHostViewQtDelegateClient::handleFocusEvent(QFocusEvent*) () at /usr/src/debug/qt6-webengine/qtwebengine-everywhere-src-6.6.0/src/core/render_widget_host_view_qt_delegate_client.cpp:708
#1 QtWebEngineCore::RenderWidgetHostViewQtDelegateClient::handleFocusEvent(QFocusEvent*) () at /usr/src/debug/qt6-webengine/qtwebengine-everywhere-src-6.6.0/src/core/render_widget_host_view_qt_delegate_client.cpp:705
#2 0x00007fffe5fea70c in QtWebEngineCore::RenderWidgetHostViewQtDelegateClient::forwardEvent(QEvent*) () at /usr/src/debug/qt6-webengine/qtwebengine-everywhere-src-6.6.0/src/core/render_widget_host_view_qt_delegate_client.cpp:300
#3 0x00007fffe4dd5c79 in QQuickItem::event(QEvent*) (this=0x555556b6cd20, ev=0x7fffffffa320) at /usr/src/debug/qt6-declarative/qtdeclarative-everywhere-src-6.6.0/src/quick/items/qquickitem.cpp:8871
#4 0x00007ffff1f7318b in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x555556b6cd20, e=0x7fffffffa320)
at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qapplication.cpp:3290
#5 0x00007ffff295e4a7 in () at /usr/lib/python3.11/site-packages/PyQt6/QtWidgets.abi3.so
#6 0x00007ffff59626d8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x555556b6cd20, event=0x7fffffffa320) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/corelib/kernel/qcoreapplication.cpp:1118
#7 0x00007ffff596271d in QCoreApplication::sendEvent(QObject*, QEvent*) (receiver=<optimized out>, event=<optimized out>) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/corelib/kernel/qcoreapplication.cpp:1536
#8 0x00007fffe4f33f15 in QQuickDeliveryAgentPrivate::setFocusInScope(QQuickItem*, QQuickItem*, Qt::FocusReason, QFlags<QQuickDeliveryAgentPrivate::FocusOption>)
(this=<optimized out>, scope=<optimized out>, item=<optimized out>, reason=<optimized out>, options=...) at /usr/src/debug/qt6-declarative/qtdeclarative-everywhere-src-6.6.0/src/quick/util/qquickdeliveryagent.cpp:439
#9 0x00007fffe4dd348a in QQuickItem::setFocus(bool, Qt::FocusReason) (this=0x555556b724d0, focus=<optimized out>, reason=Qt::TabFocusReason) at /usr/include/qt6/QtCore/qflags.h:73
#10 0x00007fffe4e7239b in QQuickWindow::focusInEvent(QFocusEvent*) (this=<optimized out>, ev=<optimized out>) at /usr/src/debug/qt6-declarative/qtdeclarative-everywhere-src-6.6.0/src/quick/items/qquickwindow.cpp:231
#11 0x00007ffff1fc3a05 in QWidget::event(QEvent*) (this=0x555556457b50, event=0x7fffffffa770) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qwidget.cpp:9111
#12 0x00007ffff1f7318b in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x555556457b50, e=0x7fffffffa770)
at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qapplication.cpp:3290
#13 0x00007ffff295e4a7 in () at /usr/lib/python3.11/site-packages/PyQt6/QtWidgets.abi3.so
#14 0x00007ffff59626d8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x555556457b50, event=0x7fffffffa770) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/corelib/kernel/qcoreapplication.cpp:1118
#15 0x00007ffff596271d in QCoreApplication::sendEvent(QObject*, QEvent*) (receiver=<optimized out>, event=<optimized out>) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/corelib/kernel/qcoreapplication.cpp:1536
#16 0x00007ffff1f7f1b2 in QApplicationPrivate::setFocusWidget(QWidget*, Qt::FocusReason) (focus=0x555556457b50, reason=<optimized out>) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qapplication.cpp:1538
#17 0x00007ffff1fca29d in QWidget::setFocus(Qt::FocusReason) (this=0x555556b1ceb0, reason=<optimized out>) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qwidget.cpp:6580
#18 0x00007ffff1fb4f1b in QWidget::focusNextPrevChild(bool) (this=<optimized out>, next=<optimized out>) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qwidget.cpp:6844
#19 0x00007ffff298d0ac in () at /usr/lib/python3.11/site-packages/PyQt6/QtWidgets.abi3.so
#20 0x00007ffff298d0ac in () at /usr/lib/python3.11/site-packages/PyQt6/QtWidgets.abi3.so
#21 0x00007ffff298d0ac in () at /usr/lib/python3.11/site-packages/PyQt6/QtWidgets.abi3.so
#22 0x00007ffff1fbdb76 in QWidgetPrivate::hide_helper() (this=this@entry=0x55555646a360) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qwidget.cpp:8271
#23 0x00007ffff1fbf158 in QWidgetPrivate::setVisible(bool) (this=0x55555646a360, visible=<optimized out>) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.0/src/widgets/kernel/qwidget.cpp:8447
[...]
We fix this problem by explicitly handling focus before hiding the UI elements.
This is done with a new TabbedBrowser.on_release_focus() slot, which is bound to
signals emitted just before things are hidden: The existing Command.hide_cmd()
for the status bar, and a new release_focus() signal for prompts.
Additionally, we make sure to not double-handle hiding in the statusbar
code when it's already handled separately for comamnd mode.
Unfortunately, no tests for this, as application window focus is required to
reproduce the issue. In theory, a test in scroll.feature could be added though,
which loads simple.html, scrolls down, shows/hides a prompt or the status bar,
and then checks the vertical scroll position is != 0.
Fixes#2236Fixes#7885
Florian Bruhin
toofar
Harm te Molder
Jun Chen
qutebrowser bot
Sviatoslav Sydorenko
Fernando Ramos
owl
Evan Chen