The previous way of obtaining the Bang's URL was using the
response's body, using the url atrribute of a meta tag. The way
this was handled was a bit problematic: some URLs were of the format
/l/?uddg=<URL>&<some-other-query-arguments>, while others (such as google)
is simply <URL>. This could have been fixed by using an if-else, but I
preferred another approach. Lite DuckDuckGo sends a 303 response when
using Bangs, instead of a 200 response which makes the redirect. The
location header of the 303 response is the search engine's URL, so it's
easy to implement this function, working for both afformentioned URL
templates.
Looks like the upstream bug is fixed now.
I didn't run recompile-requirements because it requires py3.7 for the
pylint file and I don't have that. The robot will run overnight anyway
(and pick up a new pylint release)
ref: 085a7956eb
The changes in requirements-mypy.txt would get overwritten on the next
dependency update. Also, it looks like we don't actually need PyQt6 (or
the PyQt6 stubs) available for checking PyQt 5 code if all Qt 6 imports
are appropriately gated by conditionals mypy knows about.
Follow-up to c1738ca550.
Would be nice to have a bare `mypy` env which ran both the more specific ones
in sequence but I don't know how to do that.
Not sure if there is a way to pull the CONSTANTS_ARGS stuff out to a non-env
parameter and pass it into commands but I couldn't figure out a way. So via
the environment it is.
TODO: compare PyQt6 as-is with the WIP PyQt6-Stub
For some unknown reason, those new stubs cause a *lot* of things now to be
checked by mypy which formerly probably got skipped due to Any being implied
somewhere.
The stubs themselves mainly improved, with a couple of regressions too.
In total, there were some 337 (!) new mypy errors. This commit fixes almost all
of them, and the next commit improves a fix to get things down to 0 errors
again.
Overview of the changes:
==== qutebrowser/app.py
- Drop type ignore due to improved stubs.
==== qutebrowser/browser/browsertab.py
- Specify the type of _widget members more closely than just QWidget.
This is debatable: I suppose the abstract stuff shouldn't need to know
anything about the concrete backends at all. But it seems like we cut some
corners when initially implementing things, and put some code in browsertab.py
just because the APIs of both backends happened to be compatible. Perhaps
something to reconsider once we drop QtWebKit and hopefully implement a dummy
backend.
- Add an additional assertion in AbstractAction.run_string. This is already
covered by the isinstance(member, self.action_base) above it, but that's too
dynamic for mypy to understand.
- Fix the return type of AbstractScroller.pos_px, which is a QPoint (with x
and y components), not a single int.
- Fix the return type of AbstractScroller.pos_perc, which is a Tuple (with x
and y components), not a single int.
- Fix the argument types of AbstractScroller.to_perc, as it's possible to pass
fractional percentages too.
- Specify the type for AbstractHistoryPrivate._history. See above (_widget) re
this being debatable.
- Fix the return type of AbstractTabPrivate.event_target(), which can be None
(see #3888).
- Fix the return type of AbstractTabPrivate.run_js_sync, which is Any (the JS
return value), not None.
- Fix the argument type for AbstractTabPrivate.toggle_inspector: position can
be None to use the last used position.
- Declare the type of sub-objects of AbstractTab.
- Fix the return value of AbstractTab.icon(), which is the QIcon, not None.
==== qutebrowser/browser/commands.py
- Make sure the active window is a MainWindow (with a .win_id attribute).
==== qutebrowser/browser/downloadview.py
- Add _model() which makes sure that self.model() is a DownloadModel, not None
or any other model. This is needed because other methods access a variety of
custom attributes on it, e.g. last_index().
==== qutebrowser/browser/greasemonkey.py
- Add an ignore for AbstractDownload.requested_url which we patch onto the
downloads. Probably would be nicer to add it as a proper attribute which always
gets set by the DownloadManager.
==== qutebrowser/browser/hints.py
- Remove type ignores for QUrl.toString().
- Add a new type ignore for combining different URL flags (which works, but is
not exactly type safe... still probably a regression in the stubs).
- Make sure the things we get back from self._get_keyparser are what we actually
expect. Probably should introduce a TypedDict (and/or overloads for
_get_keyparser with typing.Literal) to teach mypy about the exact return value.
See #7098.
This is needed because we access Hint/NormalKeyParser-specific attributes such
as .set_inhibited_timout() or .update_bindings().
==== qutebrowser/browser/inspector.py
- Similar changes than in browsertab.py to make some types where we share API
(e.g. .setPage()) more concrete. Didn't work out unfortunately, see next
commit.
==== qutebrowser/browser/network/pac.py
- Remove now unneeded type ignore for signal.
==== qutebrowser/browser/qtnetworkdownloads.py
- Make sure that downloads is a qtnetworkdownloads.DownloadItem (rather than an
AbstractDownload), so that we can call ._uses_nam() on it.
==== qutebrowser/browser/qutescheme.py
- Remove now unneeded type ignore for QUrl flags.
==== qutebrowser/browser/urlmarks.py
- Specify the type of UrlMarkManager._lineparser, as those only get initialized
in _init_lineparser of subclasses, so mypy doesn't know it's supposed to exist.
==== qutebrowser/browser/webelem.py
- New casts to turn single KeyboardModifier (enum) entries into
KeyboardModifiers (flags). Might not be needed anymore with Qt 6.
- With that, casting the final value is now unneeded.
==== qutebrowser/browser/webengine/notification.py
- Remove now unneeded type ignore for signal.
- Make sure the self.sender() we get in HerbeNotificationAdapter._on_finished()
is a QProcess, not just any QObject.
==== qutebrowser/browser/webengine/webenginedownloads.py
- Remove now unneeded type ignores for signals.
==== qutebrowser/browser/webengine/webengineelem.py
- Specify the type of WebEngineElement._tab.
- Remove now unneeded type ignore for mixed flags.
==== qutebrowser/browser/webengine/webengineinspector.py
- See changes to inspector.py and next commit.
- Remove now unneeded type ignore for signal.
==== qutebrowser/browser/webengine/webenginequtescheme.py
- Remove now unneeded type ignore for mixed flags.
==== qutebrowser/browser/webengine/webenginesettings.py
- Ignore access of .setter attribute which we patch onto QWebEngineProfile.
Would be nice to have a subclass or wrapper-class instead.
==== qutebrowser/browser/webengine/webenginetab.py
- Specified the type of _widget members more closely than just QWidget.
See browsertab.py changes for details.
- Remove some now-unneeded type ignores for creating FindFlags.
- Specify more concrete types for WebEngineTab members where we actually need to
access WebEngine-specific attributes.
- Make sure the page we get is our custom WebEnginePage subclass, not just any
QWebEnginePage. This is needed because we access custom attributes on it.
==== qutebrowser/browser/webengine/webview.py
- Make sure the page we get is our custom WebEnginePage subclass, not just any
QWebEnginePage. This is needed because we access custom attributes on it.
==== qutebrowser/browser/webkit/network/networkreply.py
- Remove now unneeded type ignores for signals.
==== qutebrowser/browser/webkit/webkitinspector.py
- See changes to inspector.py and next commit.
==== qutebrowser/browser/webkit/webkittab.py
- Specify the type of _widget members more closely than just QWidget.
See browsertab.py changes for details.
- Add a type ignore for WebKitAction because our workaround needs to
treat them as ints (which is allowed by PyQt, even if not type-safe).
- Add new ignores for findText calls: The text is a QString and can be None; the
flags are valid despite mypy thinking they aren't (stubs regression?).
- Specify the type for WebKitHistoryPrivate._history, because we access
WebKit-specific attributes. See above (_widget) re this being debatable.
- Make mypy aware that .currentFrame() and .frameAt() can return None (stubs
regression?).
- Make sure the .page() and .page().networkAccessManager() are our subclasses
rather than the more generic QtWebKit objects, as we use custom attributes.
- Add new type ignores for signals (stubs regression!)
==== qutebrowser/browser/webkit/webpage.py
- Make sure the .networkAccessManager() is our subclass rather than the more
generic QtWebKit object, as we use custom attributes.
- Replace a cast by a type ignore. The cast didn't work anymore.
==== qutebrowser/browser/webkit/webview.py
- Make sure the .page() is our subclass rather than the more generic QtWebKit
object, as we use custom attributes.
==== qutebrowser/commands/userscripts.py
- Remove now unneeded type ignore for signal.
==== qutebrowser/completion/completer.py
- Add a new _completion() getter (which ensures it actually gets the completion
view) rather than accessing the .parent() directly (which could be any QObject).
==== qutebrowser/completion/completiondelegate.py
- Make sure self.parent() is a CompletionView (no helper method as there is only
one instance).
- Remove a now-unneeded type ignore for adding QSizes.
==== qutebrowser/completion/completionwidget.py
- Add a ._model() getter which ensures that we get a CompletionModel (with
custom attributes) rather than Qt's .model() which can be any QAbstractItemModel
(or None).
- Removed a now-unneeded type ignore for OR-ing flags.
==== qutebrowser/completion/models/completionmodel.py
- Remove now unneeded type ignores for signals.
- Ignore a complaint about .set_pattern() not being defined. Completion
categories don't share any common parent class, so it would be good to introduce
a typing.Protocol for this. See #7098.
==== qutebrowser/components/misccommands.py
- Removed a now-unneeded type ignore for OR-ing flags.
==== qutebrowser/components/readlinecommands.py
- Make sure QApplication.instance() is a QApplication (and not just a
QCoreApplication). This includes the former "not None" check.
==== qutebrowser/components/scrollcommands.py
- Add basic annotation for "funcs" dict. Could have a callable protocol to
specify it needs a count kwarg, see #7098.
==== qutebrowser/config/stylesheet.py
- Correctly specify that stylesheet apply to QWidgets, not any QObject.
- Ignore an attr-defined for obj.STYLESHEET. Perhaps could somehow teach mypy
about this with overloads and protocols (stylesheet for set_register being None
=> STYLESHEET needs to be defined, otherwise anything goes), but perhaps not
worth the troble. See #7098.
==== qutebrowser/keyinput/keyutils.py
- Remove some now-unneeded type ignores and add a cast for using a single enum
value as flags. Might need to look at this again with Qt 6 support.
==== qutebrowser/keyinput/modeman.py
- Add a FIXME for using a TypedDict, see comments for hints.py above.
==== qutebrowser/mainwindow/mainwindow.py
- Remove now-unneeded type ignores for calling with OR-ed flags.
- Improve where we cast from WindowType to WindowFlags, no int needed
- Use new .tab_bar() getter, see below.
==== qutebrowser/mainwindow/prompt.py
- Remove now-unneeded type ignores for calling with OR-ed flags.
==== qutebrowser/mainwindow/statusbar/bar.py
- Adjust type ignores around @pyqtProperty. The fact one is still needed seems
like a stub regression.
==== qutebrowser/mainwindow/statusbar/command.py
- Fix type for setText() override (from QLineEdit): text can be None
(QString in C++).
==== qutebrowser/mainwindow/statusbar/url.py
- Adjust type ignores around @pyqtProperty. The fact one is still needed seems
like a stub regression.
==== qutebrowser/mainwindow/tabbedbrowser.py
- Specify that TabDeque manages browser tabs, not any QWidgets. It accesses
AbstractTab-specific attributes.
- Make sure that the .tabBar() we get is a tabwidget.TabBar, as we access
.maybe_hide.
- Fix the annotations for stored marks: Scroll positions are a QPoint, not int.
- Add _current_tab() and _tab_by_idx() wrappers for .currentWidget() and
.widget(), which ensures that the return values are valid AbstractTabs (or None
for _tab_by_idx). This is needed because we access AbstractTab-specific
attributes.
- For some places, where the tab can be None, continue using .currentTab() but
add asserts.
- Remove some now-unneeded [unreachable] ignores, as mypy knows about the None
possibility now.
==== qutebrowser/mainwindow/tabwidget.py
- Add new tab_bar() and _tab_by_idx() helpers which check that the .tabBar() and
.widget() are of type TabBar and AbstractTab, respectively.
- Add additional assertions where we expect ._tab_by_idx() to never be None.
- Remove dead code in get_tab_fields for handling a None y scroll position. I
was unable to find any place in the code where this could be set to None.
- Remove some now-unneeded type ignores and casts, as mypy now knows that
_type_by_idx() could be None.
- Work around a strange instance where mypy complains about not being able to
find the type of TabBar.drag_in_progress from TabWidget._toggle_visibility,
despite it clearly being shown as a bool *inside* that class without any
annotation.
- Add a ._tab_widget() getter in TabBar which ensures that the .parent() is in
fact a TabWidget.
==== qutebrowser/misc/crashsignal.py
- Remove now unneeded type ignores for signals.
==== qutebrowser/misc/editor.py
- Remove now unneeded type ignores for signals.
==== qutebrowser/misc/ipc.py
- Remove now unneeded type ignores for signals.
- Add new type ignores for .error() which is both a signal and a getter
(stub regression?). Won't be relevant for Qt 6 anymore, as the signal was
renamed to errorOccurred in 5.15.
==== qutebrowser/misc/objects.py
- Make sure mypy knows that objects.app is our custom Application (with custom
attributes) rather than any QApplication.
==== qutebrowser/utils/objreg.py
- Ignore attr-defined for .win_id attributes. Maybe could add a typing.Protocol,
but ideally, the whole objreg stuff should die one day anyways.
==== tests/unit/completion/test_completer.py
- Make CompletionWidgetStub inherit from CompletionView so that it passes the
new isinstance() asserts in completer.py (see above).
Fix qute-bitwarden userscript URL matching for multi level subdomains
The qute-bitwarden userscript is not able to autofill for a multi-level subdomain URLs (e.g. `http://my.sub.domain.local/`).
qute-bitwarden exits with status code 2 and returns the following on stderr:
```
No pass candidates for URL 'http://my.sub.domain.local/' found!
```
Joining the extracted subdomain and domain with a `.` solves the issue.
The PyQt resources system is gone in 6.2 and deprecated before that. This
should be the last usage of it.
Switches icons to be read with `utils.resources.read_file_binary()` in
`notification.py` (fallback desktop notification icon) and `app.py` (icon for
the desktop window).
importlib only loads resources under a package, so the icons are moved under
the `qutebrowser/` directory.
Closes: #6062
Then regenerate the relevant files.
Also drop dataclasses from requirements files.
TODO: should we drop the dataclasses-types requirement for mypy too?
I am making the yt-dl program used in the `cast` userscript
configurable, via a pattern borrowed from some of the other
userscripts here (e.g. `password_fill` and `kodi`): we look for an
optional "cast_rc" file in the qutebrowser config directory and source
it if it exists.
(Technically this allows for overriding any variables used in `cast`,
but this is in line with how the pattern works in the other scripts
already.)
If the config file is not found, we default to yt-dlp, and if that
doesn't exist then youtube-dl. If after all this no program is
available, we emit an error message
(note, the error messaging function as currently written in the cast script
seems broken and doesn't display the full error message, but fixing this
existing bug is outside of the scope of this change. May be good for a
followup).
I recognize there's some danger of breakage for some users by switching
the default to yt-dlp, but I think it's reasonable to assume that almost
everybody who has yt-dlp installed would prefer it to be used anyway.
Those who don't will experience no difference.
The `cast` userscript hasn't worked for me in a while, because it attempts to launch youtube-dl which has been replaced by yt-dlp for some time. Switching this one command gets `cast` working for me.
I wonder how many else will fail.
Also it would be nice if the compiled requirements files said where the
requirements come from, as in what they are dependencies of, if
anything. Like pip-compile does.
Hopefully that works. This was confusing. Grepping for 3.6 to see how
other places pin it there were a bunch of normal pins in .txt files and
then some comment-but-actually-instruction comments in the -raw files.
There is a readme in the requirements folder but it just says what
instructions are allowed. Not why you would want to add them, seems to
very high cognitive load setup.
Trying to run the recompile_requirements.py script it did some stuff
then quit because it couldn't find python3.7. Yeah I don't have that
installed. It doesn't try to use pyenv or anything, just runs venv
expecting you have everything installed?
Turns out you take the name of the requirements file you added the
instructions to (eq requirements-test.txt-raw -> test) and run it with
that as argument. The requirements file you modify should be the highest
level one that mentions the requirement you want to pin.
Anyway I edited the qutebrowser.text-raw one because that seems like the
main one and it mentions MarkupSafe (because of jinja2?) and ran
`scripts/dev/recompile_requirements qutebrowser` in my existing venv.
Lets see what the CI thinks.
Also got lots of `pkg_resources==0.0.0` in a bunch of requirements files
which seems problematic.
This allows it to work on Wayland (my use case), but also opens up the
possibility of using the system keychain to manage the password, or any
other method the user might want.
This was originally intended as a fix for CVE-2021-41146, but it turned out the
same exploit works via e.g. https:// just as well. Still, it makes sense to
remove it.
On Windows, if an application is registered as an URL handler like this:
HKEY_CLASSES_ROOT
https
URL Protocol = ""
[...]
shell
open
command
(Default) = ".../qutebrowser.exe" "%1"
one would think that Windows takes care of making sure URLs can't inject
arguments by containing a quote. However, this is not the case, as
stated by the Microsoft docs:
https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/aa767914(v=vs.85)
Security Warning: Applications that handle URI schemes must consider how to
respond to malicious data. Because handler applications can receive data
from untrusted sources, the URI and other parameter values passed to the
application may contain malicious data that attempts to exploit the handling
application.
and
As noted above, the string that is passed to a pluggable protocol handler
might be broken across multiple parameters. Malicious parties could use
additional quote or backslash characters to pass additional command line
parameters. For this reason, pluggable protocol handlers should assume that
any parameters on the command line could come from malicious parties, and
carefully validate them. Applications that could initiate dangerous actions
based on external data must first confirm those actions with the user. In
addition, handling applications should be tested with URIs that are overly
long or contain unexpected (or undesirable) character sequences.
Indeed it's trivial to pass a command to qutebrowser this way - given how
trivial the exploit is to recreate given the information above, here's a PoC:
https:x" ":spawn calc
(or qutebrowserurl: instead of https: if qutebrowser isn't registered as a
default browser)
Some applications do escape the quote characters before calling
qutebrowser - but others, like Outlook Desktop or .url files, do not.
As a fix, we add an --untrusted-args flag and some early validation of the raw
sys.argv, before parsing any arguments or e.g. creating a QApplication (which
might already allow injecting Qt flags there).
We assume that there's no way for an attacker to inject flags *before* the %1
placeholder in the registry, and add --untrusted-args as the last argument of
the registry entry. This way, it'd still be possible for users to customize
their invocation flags without having to remove --untrusted-args.
After --untrusted-args, however, we have some rather strict checks:
- There should be zero or one arguments, but not two (or more)
- Any argument may not start with - (flag) or : (qutebrowser command)
We also add the --untrusted-args flag to the Linux .desktop file, though it
should not be needed there, as the specification there is sane:
https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html#exec-variables
Implementations must take care not to expand field codes into multiple
arguments unless explicitly instructed by this specification. This means
that name fields, filenames and other replacements that can contain spaces
must be passed as a single argument to the executable program after
expansion.
There is no comparable mechanism on macOS, which opens the application without
arguments and then sends an "open" event to it:
https://doc.qt.io/qt-5/qfileopenevent.html
This issue was introduced in qutebrowser v1.7.0 which started registering it as
URL handler: baee288890 / #4086
This is by no means an issue isolated to qutebrowser. Many other projects have
had similar trouble with Windows' rather unexpected behavior:
Electron / Exodus Bitcoin wallet:
- http://web.archive.org/web/20190702112128/https://medium.com/0xcc/electrons-bug-shellexecute-to-blame-cacb433d0d62
- https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1000006
- https://medium.com/hackernoon/exploiting-electron-rce-in-exodus-wallet-d9e6db13c374
IE/Firefox:
- https://bugzilla.mozilla.org/show_bug.cgi?id=384384
- https://bugzilla.mozilla.org/show_bug.cgi?id=1572838
Others:
- http://web.archive.org/web/20210930203632/https://www.vdoo.com/blog/exploiting-custom-protocol-handlers-in-windows
- https://parsiya.net/blog/2021-03-17-attack-surface-analysis-part-2-custom-protocol-handlers/
- etc. etc.
See CVE-2021-41146 / GHSA-vw27-fwjf-5qxm:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41146https://github.com/qutebrowser/qutebrowser/security/advisories/GHSA-vw27-fwjf-5qxm
Thanks to Ping Fan (Zetta) Ke of Valkyrie-X Security Research Group
(VXRL/@vxresearch) for finding and responsibly disclosing this issue.
The default behaviour of qute-pass userscript will insert a password
automatically if there is only a single candidate found. This option will
force the selection prompt (i.e. dmenu) to show regardless of how many
password candidates are found.
I just added the readability package to the [AUR](https://aur.archlinux.org/packages/nodejs-readability-git/).
Not much of a difference from using npm directly, but, personally, (i) I prefer to manage all my (important) packages using yay, and (ii) like to perform all system upgrades from one place (i.e. yay --devel).
qutebrowser bot
Florian Bruhin
Eolien55
toofar
Takahashi, Naoki
Jon Higgs
Alessio Bogon
kt programs
mkonig
Rijnder Wever
Markus Blöchl
toofar
Lars Rustand
Benjamin Schnitzler
bitraid
Philipp Albrecht
David Vaughan
DavidRV00
Joakim Hansen
Daniel Dimijian
Marco A Parillo
Pierre-Albéric TROUPLIN
Jonas Belouadi
tinywrkb
Eric Engestrom
Mirko König
Adrián Lattes
mhmdanas
André Keller
Jason Rosenzweig
zocker-160
Sidharth Arya