We’re in the process deploying a major upgrade to the GrapheneOS server infrastructure to provide faster connections to our services and higher robustness.

If you notice any new issues connecting to our services, please report them in our Infrastructure chat room (https://grapheneos.org/contact#community-chat).

Since August 2021, we’ve been hosting authoritative DNS for GrapheneOS domains with our own servers. Authoritative DNS is where the mapping from names like releases.grapheneos.org to the IP addresses and other information is provided. DNS resolver servers query authoritative DNS and cache results.

We moved to self-hosting authoritative DNS to deploy GeoDNS for steering traffic to nearby servers combined with health checks to direct traffic to servers which are still up when there’s downtime. We investigated various providers but decided to self-host for independence and minor privacy reasons.

In June 2023, we moved ns2.grapheneos.org to 3 servers on BuyVM (New York, Las Vegas and Luxembourg). We use BuyVM’s anycast IPv4 support to send IPv4 DNS traffic to the nearest server, which improves latency. It’s a very basic anycast feature without IPv6, failover or many locations but works well.

DNS has automatic fallback if an authoritative nameserver is down so our DNS continues working if either OVH or BuyVM is completely down. Our main 3 services (website, network and update servers) are multi-provider with 5 minute DNS Time-To-Live so those keep working if 1 provider goes down.

Our initial setup had ns1.grapheneos.org on an OVH VPS in Beauharnois. BuyVM was used to replace the initial ns2.grapheneos.org on an OVH VPS in Gravelines.

The major upgrade we’ve started deploying today is replacing ns1.grapheneos.org with a dynamic anycast network for IPv4 and IPv6.

We’re using the Rage4 AnycastIP service for the ns1.grapheneos.org IPv4 and IPv6 addresses. This is a more advanced anycast service with 21 locations and dynamic routing to our instances based on us announcing IPs to their nodes with BGP. We’ve added a server in Frankfurt and will be adding more.

This service allows us to continue hosting authoritative DNS ourselves with multiple providers for redundancy while using our own GeoDNS and failover. We now have anycast for both ns1.grapheneos.org and ns2.grapheneos.org, but it’s more advanced for ns1 with IPv6, failover and more location options.

We plan to replace the Beauharnois VPS with New York to have it near a Rage4 New York node. Our new Frankfurt VPS is already right near the Rage4 Frankfurt node. We also plan to add Seattle and Singapore soon. Adding more can be done alongside new website and network server instances in the future.

We also plan to make a small upgrade for ns2.grapheneos.org by adding the optional Miami location via BGP once BuyVM has stock available. BuyVM will hopefully add a Singapore location and IPv6 support in the future, which are the main weaknesses other than lack of failover beyond Miami going to NY.

One of the side benefits of hosting our own authoritative DNS is providing opportunistic DNS-over-TLS (DoT) between resolver servers and our authoritative servers (https://datatracker.ietf.org/doc/rfc9539/). DoT isn’t needed for authentication since reasonable servers enforce DNSSEC but it improves privacy.

Several large public resolvers including Google Public DNS and several European ISPs have adopted opportunistic ADoT (Authoritative DNS-over-TLS) and use it with our servers. There’s no way to ENFORCE using ADoT yet but we’re keeping an eye on it. We have valid WebPKI certs + DANE TLSA for it.

If you’re curious about the configurations we use on our servers, most of that is available in these 2 repositories:

https://github.com/GrapheneOS/ns1.grapheneos.orghttps://github.com/GrapheneOS/infrastructure

Our bird.conf for announcing our anycast IPs via BGP to Rage4 and the routing setup script aren’t published yet.

We’ve added OVH instances for ns1.grapheneos.org in Singapore and Hillsboro, Oregon for now. We can consider using Vultr in the future for more locations and closer proximity but metered bandwidth isn’t appealing. We could also eventually handle managing anycast ourselves.

If we owned an IPv4 /24 and IPv6 /48, we could announce anycast IPs from Vultr via BGP ourselves. That would mean we’d need to manage optimizing anycast routing which we aren’t interested in doing right now, so we’ll leave that up to Rage4 for now. We also don’t own IP space yet.

You can see the quite decent results we’re getting with only 4 VPS instances with 2 in imperfect locations:

https://ping.pe/ns1.grapheneos.org

Here’s how that compares to the single server:

https://ping.pe/0.ns1.grapheneos.org

OVH does have many locations via Local Zones but not for what we use.