Rover: DNS name causing problems

I’ve ran into a problem I can’t reproduce anywhere but inside a container on a server.

When execSync("rover supergraph compose --config ... --output ... --elv2-license=accept --log=debug") is run I get this error:

Make sure the endpoint is accepting connections and is spelled correctly
    E028: error sending request for url (http://plugin_webbuilder_api/graphql): error trying to connect: dns error: proto error: Label contains invalid characters: Err(Errors { invalid_mapping, disallowed_by_std3_ascii_rules }) while resolving the schema for the 'webbuilder' subgraph

My supergraph.yaml config looks like this:

federation_version: 2
subgraphs:
  # other similar subgraphs
  plugin_webbuilder_api:
    routing_url: http://plugin_webbuilder_api/graphql
    schema:
      subgraph_url: http://plugin_webbuilder_api/graphql
  # other similar subgraphs

Result of running it with --log=debug

   INFO rover_std::fs: creating /home/node/.config/rover directory
    at crates/rover-std/src/fs.rs:77

   INFO rover_std::fs: writing /home/node/.config/rover/machine.txt to disk
    at crates/rover-std/src/fs.rs:66

  DEBUG rover::utils::telemetry: serialized_command: Command { name: "supergraph compose", arguments: {"opts": Object {"elv2_license_accepter": Object {"elv2_license_accepted": Bool(true)}, "profile": Object {}, "skip_update": Bool(false)}} }
    at src/utils/telemetry.rs:79

   INFO rover::utils::telemetry: Telemetry is enabled. To disable, set $APOLLO_TELEMETRY_DISABLED=1
    at src/utils/telemetry.rs:89

  DEBUG rover::utils::version: Failed to read version file containing last update check time: No such file or directory (os error 2)
    at src/utils/version.rs:92

  DEBUG sputnik::session: POSTing to https://rover.apollo.dev/telemetry
    at crates/sputnik/src/session.rs:140

  DEBUG sputnik::session: {"command":{"name":"supergraph compose","arguments":{"opts":{"elv2_license_accepter":{"elv2_license_accepted":true},"profile":{},"skip_update":false}}},"machine_id":"d73b980f-6121-4580-93cf-e8ae31a9afd5","session_id":"629bf165-b474-42b6-ab29-ea980b265133","cwd_hash":"c06ac3bca5226a64c6608698b04cf53c583b6eacc13c5334b27e576e5f69d544","remote_url_hash":null,"platform":{"os":"linux","continuous_integration":null},"cli_version":"0.13.0"}
    at crates/sputnik/src/session.rs:141

  DEBUG reqwest::connect: starting new connection: https://github.com/
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.14/src/connect.rs:429

  DEBUG trust_dns_proto::xfer::dns_handle: querying: github.com A
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/xfer/dns_handle.rs:67

  DEBUG trust_dns_resolver::name_server::name_server_pool: sending request: [Query { name: Name("github.com"), query_type: A, query_class: IN }]
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/name_server/name_server_pool.rs:257

  DEBUG trust_dns_resolver::name_server::name_server: reconnecting: NameServerConfig { socket_addr: 127.0.0.11:53, protocol: Udp, tls_dns_name: None, trust_nx_responses: false, bind_addr: None }
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/name_server/name_server.rs:115

  DEBUG trust_dns_proto::xfer: enqueueing message:QUERY:[Query { name: Name("github.com"), query_type: A, query_class: IN }]
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/xfer/mod.rs:171

  DEBUG reqwest::connect: starting new connection: https://rover.apollo.dev/
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.14/src/connect.rs:429

  DEBUG trust_dns_proto::xfer::dns_handle: querying: rover.apollo.dev A
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/xfer/dns_handle.rs:67

  DEBUG trust_dns_resolver::name_server::name_server_pool: sending request: [Query { name: Name("rover.apollo.dev"), query_type: A, query_class: IN }]
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/name_server/name_server_pool.rs:257

  DEBUG trust_dns_resolver::name_server::name_server: existing connection: NameServerConfig { socket_addr: 127.0.0.11:53, protocol: Udp, tls_dns_name: None, trust_nx_responses: false, bind_addr: None }
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/name_server/name_server.rs:128

  DEBUG trust_dns_proto::xfer: enqueueing message:QUERY:[Query { name: Name("rover.apollo.dev"), query_type: A, query_class: IN }]
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/xfer/mod.rs:171

  DEBUG trust_dns_proto::udp::udp_client_stream: final message: ; header 29724:QUERY:RD:NoError:QUERY:0/0/0
; query
;; github.com. IN A

    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/udp/udp_client_stream.rs:190

  DEBUG trust_dns_proto::udp::udp_client_stream: final message: ; header 45795:QUERY:RD:NoError:QUERY:0/0/0
; query
;; rover.apollo.dev. IN A

    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/udp/udp_client_stream.rs:190

  DEBUG trust_dns_proto::udp::udp_stream: created socket successfully
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/udp/udp_stream.rs:254

  DEBUG trust_dns_proto::udp::udp_stream: created socket successfully
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/udp/udp_stream.rs:254

  DEBUG trust_dns_proto::udp::udp_client_stream: received message id: 29724
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/udp/udp_client_stream.rs:305

  DEBUG trust_dns_resolver::error: Response:; header 29724:RESPONSE:RD,RA:NoError:QUERY:1/0/0
; query
;; github.com. IN A
; answers 1
github.com. 14 IN A 20.27.177.113
; nameservers 0
; additionals 0

    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/error.rs:148

  DEBUG trust_dns_resolver::error: Response:; header 29724:RESPONSE:RD,RA:NoError:QUERY:1/0/0
; query
;; github.com. IN A
; answers 1
github.com. 14 IN A 20.27.177.113
; nameservers 0
; additionals 0

    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/error.rs:148

  DEBUG hyper::client::connect::http: connecting to 20.27.177.113:443
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/connect/http.rs:537

  DEBUG trust_dns_proto::udp::udp_client_stream: received message id: 45795
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-proto-0.22.0/src/udp/udp_client_stream.rs:305

  DEBUG trust_dns_resolver::error: Response:; header 45795:RESPONSE:RD,RA:NoError:QUERY:3/0/0
; query
;; rover.apollo.dev. IN A
; answers 3
rover.apollo.dev. 269 IN CNAME apollo-orbiter.netlify.app.
apollo-orbiter.netlify.app. 20 IN A 18.139.194.139
apollo-orbiter.netlify.app. 20 IN A 34.142.199.10
; nameservers 0
; additionals 0

    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/error.rs:148

  DEBUG trust_dns_resolver::error: Response:; header 45795:RESPONSE:RD,RA:NoError:QUERY:3/0/0
; query
;; rover.apollo.dev. IN A
; answers 3
rover.apollo.dev. 269 IN CNAME apollo-orbiter.netlify.app.
apollo-orbiter.netlify.app. 20 IN A 18.139.194.139
apollo-orbiter.netlify.app. 20 IN A 34.142.199.10
; nameservers 0
; additionals 0

    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/trust-dns-resolver-0.22.0/src/error.rs:148

  DEBUG hyper::client::connect::http: connecting to 18.139.194.139:443
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/connect/http.rs:537

  DEBUG hyper::client::connect::http: connected to 20.27.177.113:443
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/connect/http.rs:540

  DEBUG hyper::proto::h1::io: flushed 138 bytes
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/io.rs:342

  DEBUG hyper::proto::h1::io: parsed 17 headers
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/io.rs:207

  DEBUG hyper::proto::h1::conn: incoming body is empty
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/conn.rs:222

  DEBUG hyper::client::pool: pooling idle connection for ("https", github.com)
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/pool.rs:376

  DEBUG reqwest::async_impl::client: redirecting 'https://github.com/apollographql/rover/releases/latest' to 'https://github.com/apollographql/rover/releases/tag/v0.13.0'
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.14/src/async_impl/client.rs:2032

  DEBUG hyper::client::pool: reuse idle connection for ("https", github.com)
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/pool.rs:250

  DEBUG hyper::proto::h1::io: flushed 208 bytes
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/io.rs:342

  DEBUG hyper::client::connect::http: connected to 18.139.194.139:443
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/connect/http.rs:540

  DEBUG hyper::proto::h1::io: flushed 607 bytes
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/io.rs:342

  DEBUG hyper::proto::h1::io: parsed 18 headers
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/io.rs:207

  DEBUG hyper::proto::h1::conn: incoming body is empty
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/conn.rs:222

  DEBUG hyper::client::pool: pooling idle connection for ("https", github.com)
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/pool.rs:376

   INFO rover_std::fs: writing /home/node/.config/rover/version.toml to disk
    at crates/rover-std/src/fs.rs:66

resolving SDL for subgraphs defined in /erxes-gateway/dist/gateway/src/apollo-router/temp/supergraph.yaml
   INFO rover_std::fs: reading /erxes-gateway/dist/gateway/src/apollo-router/temp/supergraph.yaml from disk
    at crates/rover-std/src/fs.rs:29

  DEBUG apollo_federation_types::config::supergraph: SupergraphConfig { subgraphs: {"contacts": SubgraphConfig { routing_url: Some("http://plugin_contacts_api/graphql"), schema: SubgraphIntrospection { subgraph_url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("plugin_contacts_api")), port: None, path: "/graphql", query: None, fragment: None } } }, "core": SubgraphConfig { routing_url: Some("http://plugin_core_api/graphql"), schema: SubgraphIntrospection { subgraph_url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("plugin_core_api")), port: None, path: "/graphql", query: None, fragment: None } } }, "tags": SubgraphConfig { routing_url: Some("http://plugin_tags_api/graphql"), schema: SubgraphIntrospection { subgraph_url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("plugin_tags_api")), port: None, path: "/graphql", query: None, fragment: None } } }, "workers": SubgraphConfig { routing_url: Some("http://plugin_workers_api/graphql"), schema: SubgraphIntrospection { subgraph_url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("plugin_workers_api")), port: None, path: "/graphql", query: None, fragment: None } } }}, federation_version: Some(ExactFedTwo(Version { major: 2, minor: 3, patch: 1 })) }
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/apollo-federation-types-0.7.1/src/config/supergraph.rs:30

  DEBUG introspector_gadget::blocking::client: Request Body: {"variables":null,"query":"query SubgraphIntrospectQuery {\n    # eslint-disable-next-line\n    _service {\n        sdl\n    }\n}","operationName":"SubgraphIntrospectQuery"}
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/introspector-gadget-0.2.0/src/blocking/client.rs:89

  DEBUG reqwest::connect: starting new connection: http://plugin_workers_api/
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.14/src/connect.rs:429

  DEBUG introspector_gadget::blocking::client: Request Body: {"variables":null,"query":"query SubgraphIntrospectQuery {\n    # eslint-disable-next-line\n    _service {\n        sdl\n    }\n}","operationName":"SubgraphIntrospectQuery"}
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/introspector-gadget-0.2.0/src/blocking/client.rs:89

  DEBUG reqwest::connect: starting new connection: http://plugin_core_api/
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.14/src/connect.rs:429

  DEBUG introspector_gadget::blocking::client: Request Body: {"variables":null,"query":"query SubgraphIntrospectQuery {\n    # eslint-disable-next-line\n    _service {\n        sdl\n    }\n}","operationName":"SubgraphIntrospectQuery"}
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/introspector-gadget-0.2.0/src/blocking/client.rs:89

  DEBUG reqwest::connect: starting new connection: http://plugin_contacts_api/
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.14/src/connect.rs:429

  DEBUG introspector_gadget::blocking::client: Request Body: {"variables":null,"query":"query SubgraphIntrospectQuery {\n    # eslint-disable-next-line\n    _service {\n        sdl\n    }\n}","operationName":"SubgraphIntrospectQuery"}
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/introspector-gadget-0.2.0/src/blocking/client.rs:89

  DEBUG reqwest::connect: starting new connection: http://plugin_tags_api/
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.14/src/connect.rs:429

  DEBUG hyper::proto::h1::io: parsed 8 headers
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/io.rs:207

  DEBUG hyper::proto::h1::conn: incoming body is content-length (15 bytes)
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/conn.rs:222

  DEBUG hyper::proto::h1::conn: incoming body completed
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/proto/h1/conn.rs:298

  DEBUG hyper::client::pool: pooling idle connection for ("https", rover.apollo.dev)
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/pool.rs:376

Turns out it was because the service names contained underscore _.