Skip to content

Skyline, nginx, and Keystone: Fixing "426 Upgrade Required"

When I turned on OIDC login for Skyline, the OpenStack dashboard, every call it proxied to Keystone came back with 426 Upgrade Required. Nothing in those requests asked to upgrade anything. The real problem was that Skyline's nginx was talking HTTP/1.0 to Keystone, and the Keystone side wanted HTTP/1.1. One line of nginx configuration fixes it.

The setup

Everything is deployed with openstack-helm, split across two Kubernetes clusters:

  • Keystone runs in one cluster. Like most Keystone deployments, it has Apache httpd in front of the Python WSGI application.
  • Skyline runs in a different cluster. The skyline-apiserver image includes an nginx that serves the console and acts as a reverse proxy for the OpenStack APIs: when the browser calls Keystone, Nova, Neutron and so on, the request goes to Skyline's nginx, which forwards it to the service endpoint.
  • Both clusters have Envoy in front, so traffic from Skyline to Keystone leaves one cluster through Envoy and enters the other through Envoy.

A request from the browser to Keystone, by way of Skyline, looks like this:

browser -> Envoy -> nginx (Skyline) -> Envoy -> Apache -> Keystone (WSGI)

That is a lot of proxies, and each hop is its own HTTP connection with its own protocol version.

The symptom

With OIDC authentication configured, logging in to Skyline failed. The requests Skyline proxied to Keystone, including the role and project lookups behind the project switcher, all returned:

426 Upgrade Required

This had me confused at first. A 426 usually shows up when a server wants the client to switch protocols, such as moving to TLS or to a WebSocket. Nothing in Skyline was trying to do either.

What the 426 actually meant

The clue is the HTTP version. Browsers and curl speak HTTP/1.1 or newer. nginx's proxy_pass does not: unless you set proxy_http_version, nginx forwards proxied requests to the upstream as HTTP/1.0.

The Keystone side refused HTTP/1.0 and answered with 426. The "upgrade" it was asking for was simply "use HTTP/1.1, please." Envoy behaves this way out of the box: it rejects HTTP/1.0 requests with 426 Upgrade Required unless HTTP/1.0 support is explicitly turned on. So if you get a 426 anywhere in a chain of proxies, check which HTTP version each hop is using before anything else.

You can reproduce it by forcing curl down to HTTP/1.0:

# Fails with 426 Upgrade Required
curl -si --http1.0 https://keystone.example.com/v3 | head -n 1

# Works
curl -si --http1.1 https://keystone.example.com/v3 | head -n 1

The fix

Tell nginx to use HTTP/1.1 when it talks to the upstream OpenStack services. Skyline generates its nginx configuration from skyline_apiserver/templates/nginx.conf.j2, with one location block per service endpoint. Adding proxy_http_version 1.1; to those blocks is all it takes:

location {{ endpoint["location"] }} {
    proxy_pass {{ endpoint["url"] }};
    proxy_redirect {{ endpoint["url"] }} {{ endpoint["location"] }};
    proxy_buffering off;
    # Talk HTTP/1.1 to upstream OpenStack services. nginx defaults to
    # HTTP/1.0 for proxied requests, and some service front-ends (e.g.
    # an Apache fronting Keystone) reject HTTP/1.0 with "426 Upgrade
    # Required", which breaks every proxied API call.
    proxy_http_version 1.1;
    proxy_ssl_server_name on;
}

With that in place, nginx makes HTTP/1.1 requests, the Keystone side accepts them, and OIDC login and project switching in Skyline work.

I've submitted this upstream as Fix 426 errors on proxied APIs by forcing HTTP/1.1 upstream. Until it merges, you can carry the same one-line change in your own Skyline image, or add it to the generated nginx.conf.

Takeaways

  • nginx proxies with HTTP/1.0 by default. If an upstream needs HTTP/1.1, set proxy_http_version 1.1; in the location block.
  • A 426 from a proxy chain usually means a protocol version problem, not a request to switch to TLS or WebSockets. Envoy in particular returns 426 for HTTP/1.0 requests by default.
  • Testing with curl can hide the problem, because curl already uses HTTP/1.1. Use --http1.0 to see what the proxy sees.