Respect Accept-Language set via XHR

Similarly to #5998, XHR requests should be able to set their custom
Accept-Language values - and for some odd reason, stuff breaks on websites
sometimes when that's not respected.

There's no way for qutebrowser to know if a given header value is already set in
a request (i.e. whether we're adding or overriding). Thus we only really have
two options here:

1) Don't set any shared.custom_headers() for XHR requests at all.
2) Special-case Accept-Language here, because the issue usually is triggered by
the global override, but that already gets set just fine via QWebEngineProfile
anyways.

Given that 2) is the thing causing trouble in the wild and it's unclear what the
desired behavior for 1) is (e.g. for the DNT header), let's go for 2) here.

doc/changelog.asciidoc

@@ -48,6 +48,11 @@ Changed
   `image/webp`.
 - Updated mimetype information for getting a suitable extension when downloading
   a `data:` URL.
+- If a XHR request made via JS sets a custom `Accept-Language` header, it now
+  correctly has precedence over the global `content.headers.accept_language`
+  setting (but not per-domain overrides). This fixes subtle JS issues on
+  websites that rely on the custom header being sent for those requests, and
+  e.g. block the requests server-side otherwise. (#8370)
 
 Fixed
 ~~~~~

qutebrowser/browser/shared.py

@@ -42,8 +42,8 @@ def custom_headers(url):
         headers[encoded_header] = encoded_value
 
     accept_language = config.instance.get('content.headers.accept_language',
-                                          url=url)
-    if accept_language is not None:
+                                          url=url, fallback=False)
+    if accept_language is not None and not isinstance(accept_language, usertypes.Unset):
         headers[b'Accept-Language'] = accept_language.encode('ascii')
 
     return sorted(headers.items())

tests/end2end/data/misc/xhr_headers.html

@@ -8,6 +8,7 @@
                 const xhr = new XMLHttpRequest();
                 xhr.open("GET", "/headers");
                 xhr.setRequestHeader("X-Qute-Test", "from XHR");
+                xhr.setRequestHeader("Accept-Language", "from XHR");
 
                 const elem = document.getElementById("output");
                 xhr.addEventListener("load", function(event) {

tests/end2end/features/misc.feature

@@ -387,9 +387,11 @@ Feature: Various utility commands.
     @qtwebkit_skip
     Scenario: Custom headers via XHR
         When I set content.headers.custom to {"Accept": "config-value", "X-Qute-Test": "config-value"}
+        When I set content.headers.accept_language to "config-value"
         And I open data/misc/xhr_headers.html
         And I wait for the javascript message "Got headers via XHR"
         Then the header Accept should be set to '*/*'
+        And the header Accept-Language should be set to 'from XHR'
         And the header X-Qute-Test should be set to config-value
 
     ## https://github.com/qutebrowser/qutebrowser/issues/1523

tests/unit/browser/test_shared.py

@@ -6,6 +6,7 @@ import logging
 
 import pytest
 
+from qutebrowser.qt.core import QUrl
 from qutebrowser.browser import shared
 from qutebrowser.utils import usertypes
 
@@ -35,6 +36,16 @@ def test_custom_headers(config_stub, dnt, accept_language, custom_headers,
     assert shared.custom_headers(url=None) == expected_items
 
 
+@pytest.mark.parametrize("url, expected", [
+    (None, True),  # url is never None in the wild, mostly sanity check
+    (QUrl("http://example.org"), False),
+])
+def test_accept_language_no_fallback(config_stub, url ,expected):
+    config_stub.val.content.headers.accept_language = "de, en"
+    has_header = b"Accept-Language" in dict(shared.custom_headers(url=url))
+    assert has_header == expected
+
+
 @pytest.mark.parametrize(
     (
         "levels_setting, excludes_setting, level, source, msg, expected_ret, "