DNS problem with Algorand Server on Ubuntu 20.04

Vytek · April 5, 2021, 5:10pm

Hi,
I have this problem:

{“file”:“bootstrap.go”,“function”:“github.com/algorand/go-algorand/tools/network.ReadFromSRV",“level”:“info”,“line”:43,“msg”:"ReadFromBootstrap: DNS LookupSRV failed when using system resolver: no answer for (_algobootstrap._tcp.mainnet.algorand.network., 33) from DNS servers [8.8.8.8:53 8.8.4.4:53]”,“time”:“2021-04-05T17:06:22.870321Z”}
{“file”:“wsNetwork.go”,“function”:“github.com/algorand/go-algorand/network.(*WebsocketNetwork).meshThread”,“level”:“info”,“line”:1341,“msg”:“got no DNS addrs for network mainnet”,“name”:"",“time”:“2021-04-05T17:06:22.898683Z”}

The server is Linux:

5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Dig works fine. What is the problem? How can I fix the issue?

Thank you very much for you help!

fabrice · April 5, 2021, 6:14pm

Is it possible that the Google DNS is blocked on your network?

Check by running in a console:

dig @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv

Algorand uses 8.8.8.8 DNS server by default because of DNSSec.

trekianov · April 6, 2021, 5:05pm

I probably have the same issue with Ubuntu 20.04, my node.log is slightly different:
{“file”:“bootstrap.go”,“function”:“github.com/algorand/go-algorand/tools/network.ReadFromSRV",“level”:“info”,“line”:43,“msg”:"ReadFromBootstrap: DNS LookupSRV failed when using system resolver: DNS error: SERVFAIL”,“time”:“2021-04-06T18:56:26.164530+02:00”}
{“file”:“wsNetwork.go”,“function”:“github.com/algorand/go-algorand/network.(*WebsocketNetwork).meshThread”,“level”:“info”,“line”:1341,“msg”:“got no DNS addrs for network mainnet”,“name”:"",“time”:“2021-04-06T18:56:26.189179+02:00”}

The result of the above dig gives the following:
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.16.1-Ubuntu <<>> @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22272
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: Message has 36 extra bytes at end

;; QUESTION SECTION:
;_algobootstrap._tcp.testnet.algorand.network. IN SRV

;; ANSWER SECTION:
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-dublin-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-london-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-singaporeg1-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 r19.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-singaporeg2-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 r20.algorand-testnet.network.

;; Query time: 11 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: mar apr 06 19:04:27 CEST 2021
;; MSG SIZE rcvd: 512

fabrice · April 6, 2021, 5:56pm

is not normal. Are you on your personal network or a company network with some firewall/proxy?

Can you try to disable DNSSec? Option DNSSecurityFlags in config.json (Node configuration settings - Algorand Developer Portal)
Concretely, update the file $ALGORAND_DATA/config.json (assuming $ALGORAND_DATA is your data directory with the file genesis.json) so that it looks like:

{
    "DNSSecurityFlags": 0,
    ...
}

trekianov · April 6, 2021, 6:33pm

I set DNSSecurityFlags to 0 and now the message is sligthly different (errorID and extra bytes), but still the node is at 0 block:
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.16.1-Ubuntu <<>> @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24564
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: Message has 5 extra bytes at end

;; QUESTION SECTION:
;_algobootstrap._tcp.testnet.algorand.network. IN SRV

;; ANSWER SECTION:
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-dublin-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-washington-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 80 relay-switzerland1-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 r20.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-singaporeg2-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 300 IN SRV 1 1 4161 relay-london-testnet-algorand.algorand-testnet.network.

;; Query time: 11 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: mar apr 06 20:30:10 CEST 2021
;; MSG SIZE rcvd: 512

I am at home with an usual router: do I have to forward specific ports?

fabrice · April 6, 2021, 6:51pm

Can you copy the error message you see in node.log?
Can you also show the output of:

goal version -v
goal node status

(you can copy text between tripe backquotes ``` to improve formatting)

trekianov · April 6, 2021, 7:38pm

{"file":"bootstrap.go","function":"github.com/algorand/go-algorand/tools/network.ReadFromSRV","level":"info","line":43,"msg":"ReadFromBootstrap: DNS LookupSRV failed when using system resolver: lookup _algobootstrap._tcp.mainnet.algorand.network on 127.0.0.53:53: read udp 127.0.0.1:54597-\u003e127.0.0.53:53: i/o timeout","time":"2021-04-06T21:35:02.726053+02:00"}

Version: [v1 v2] 
GenesisID: mainnet-v1.0 
Build: 2.4.1.stable [rel/stable] (commit #573a34c4)

Last committed block: 0
Time since last block: 0.0s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0
Round for next consensus protocol: 1
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: mainnet-v1.0
Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=

fabrice · April 6, 2021, 10:11pm

That is very strange.
Are you running goal / algod directly on the same server you are running dig, without docker or anything like that?
Are you using any security software such as AppArmor with a profile blocking algod?
How did you install algod, using the package or directly?
Can you try to kill all algod processes, create a fresh new ~/new-algorand-data folder, copy just genesis.json inside, and start again goal node start -d ~/new-algorand-data?

fabrice · April 8, 2021, 3:13pm

I’ve written a very simple software to help debug: GitHub - fabrice102/go-algorand-test-dns-bootstrap: Small software to help debug/test DNS bootstrap of Algorand
Can you run it and post the results?

trekianov · April 9, 2021, 9:43am

sorry for late reply.
I am currently running on the same server , no docker, my own personal pc at home, behind a normal router. No Apparmor o similar, I installed following the instructions here:
https://developer.algorand.org/docs/run-a-node/setup/install/
After having copied to /new-algorand I cannot make it work:

(base) msi:~$ systemctl stop algorand.service
(base) msi:~$ goal node start -d /var/lib/new-algorand/
error creating file for capturing stderr: open /var/lib/new-algorand/algod-err.log: permission denied
error creating file for capturing stdout: open /var/lib/new-algorand/algod-out.log: permission denied
Algorand node failed to start: node exited with an error code, check node.log for more details : exit status 1
(base) msi:~$ goal node status -d /var/lib/new-algorand/
Cannot contact Algorand node: Get "http://127.0.0.1:8080/v2/status": dial tcp 127.0.0.1:8080: connect: connection refused

trekianov · April 9, 2021, 9:49am

here is the result:

go: finding golang.org/x/net v0.0.0-20200904194848-62affa334b73
INFO[0000] Trying: fallback="", secure=true              file=main.go function=main.testBootstrap line=16
INFO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver: DNS error: SERVFAIL  file=bootstrap.go function=github.com/algorand/go-algorand/tools/network.ReadFromSRV line=43
ERRO[0000] [Stack] goroutine 1 [running]:
runtime/debug.Stack(0xc0002228c0, 0xc0001da058, 0xc000223420)
	/usr/lib/go-1.13/src/runtime/debug/stack.go:24 +0x9d
github.com/algorand/go-algorand/logging.logger.Error(0xc0002228c0, 0xc0001da058, 0xc0001dea70, 0x1, 0x1)
	/home/treccani/go/pkg/mod/github.com/algorand/go-algorand@v0.0.0-20210122175925-573a34c457e9/logging/log.go:219 +0x48
main.testBootstrap(0x0, 0x0, 0xc000026101)
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:26 +0x201
main.main()
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:34 +0x55  file=main.go function=main.testBootstrap line=26
ERRO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver(DNS error: SERVFAIL), fallback resolver(<nil>), as well as using default resolver due to no answer for (_algobootstrap._tcp.mainnet.algorand.network., 33) from DNS servers [1.1.1.1:53 208.67.222.222:53 8.8.8.8:53 77.88.8.8:53 8.26.56.26:53 180.76.76.76:53]  file=main.go function=main.testBootstrap line=26
INFO[0000] Found 0 SRV records:                          file=main.go function=main.printAddress line=12
INFO[0000] Trying: fallback="8.8.8.8", secure=true       file=main.go function=main.testBootstrap line=16
INFO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver: DNS error: SERVFAIL  file=bootstrap.go function=github.com/algorand/go-algorand/tools/network.ReadFromSRV line=43
ERRO[0000] [Stack] goroutine 1 [running]:
runtime/debug.Stack(0xc0002228c0, 0xc0001da058, 0xc00025e3f0)
	/usr/lib/go-1.13/src/runtime/debug/stack.go:24 +0x9d
github.com/algorand/go-algorand/logging.logger.Error(0xc0002228c0, 0xc0001da058, 0xc0001df1f0, 0x1, 0x1)
	/home/treccani/go/pkg/mod/github.com/algorand/go-algorand@v0.0.0-20210122175925-573a34c457e9/logging/log.go:219 +0x48
main.testBootstrap(0xd80782, 0x7, 0xc000026101)
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:26 +0x201
main.main()
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:35 +0x73  file=main.go function=main.testBootstrap line=26
ERRO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver(DNS error: SERVFAIL), fallback resolver(<nil>), as well as using default resolver due to no answer for (_algobootstrap._tcp.mainnet.algorand.network., 33) from DNS servers [1.1.1.1:53 208.67.222.222:53 8.8.8.8:53 77.88.8.8:53 8.26.56.26:53 180.76.76.76:53]  file=main.go function=main.testBootstrap line=26
INFO[0000] Found 0 SRV records:                          file=main.go function=main.printAddress line=12
INFO[0000] Trying: fallback="", secure=false             file=main.go function=main.testBootstrap line=16
INFO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver: DNS error: SERVFAIL  file=bootstrap.go function=github.com/algorand/go-algorand/tools/network.ReadFromSRV line=43
ERRO[0000] [Stack] goroutine 1 [running]:
runtime/debug.Stack(0xc0002228c0, 0xc0001da058, 0xc00025f3b0)
	/usr/lib/go-1.13/src/runtime/debug/stack.go:24 +0x9d
github.com/algorand/go-algorand/logging.logger.Error(0xc0002228c0, 0xc0001da058, 0xc0001df960, 0x1, 0x1)
	/home/treccani/go/pkg/mod/github.com/algorand/go-algorand@v0.0.0-20210122175925-573a34c457e9/logging/log.go:219 +0x48
main.testBootstrap(0x0, 0x0, 0xc000026100)
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:26 +0x201
main.main()
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:36 +0x84  file=main.go function=main.testBootstrap line=26
ERRO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver(DNS error: SERVFAIL), fallback resolver(<nil>), as well as using default resolver due to no answer for (_algobootstrap._tcp.mainnet.algorand.network., 33) from DNS servers [1.1.1.1:53 208.67.222.222:53 8.8.8.8:53 77.88.8.8:53 8.26.56.26:53 180.76.76.76:53]  file=main.go function=main.testBootstrap line=26
INFO[0000] Found 0 SRV records:                          file=main.go function=main.printAddress line=12
INFO[0000] Trying: fallback="8.8.8.8", secure=false      file=main.go function=main.testBootstrap line=16
INFO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver: DNS error: SERVFAIL  file=bootstrap.go function=github.com/algorand/go-algorand/tools/network.ReadFromSRV line=43
ERRO[0000] [Stack] goroutine 1 [running]:
runtime/debug.Stack(0xc0002228c0, 0xc0001da058, 0xc000284380)
	/usr/lib/go-1.13/src/runtime/debug/stack.go:24 +0x9d
github.com/algorand/go-algorand/logging.logger.Error(0xc0002228c0, 0xc0001da058, 0xc00028a0e0, 0x1, 0x1)
	/home/treccani/go/pkg/mod/github.com/algorand/go-algorand@v0.0.0-20210122175925-573a34c457e9/logging/log.go:219 +0x48
main.testBootstrap(0xd80782, 0x7, 0xc000026100)
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:26 +0x201
main.main()
	/home/treccani/algo-test/go-algorand-test-dns-bootstrap/main.go:37 +0xa2  file=main.go function=main.testBootstrap line=26
ERRO[0000] ReadFromBootstrap: DNS LookupSRV failed when using system resolver(DNS error: SERVFAIL), fallback resolver(<nil>), as well as using default resolver due to no answer for (_algobootstrap._tcp.mainnet.algorand.network., 33) from DNS servers [1.1.1.1:53 208.67.222.222:53 8.8.8.8:53 77.88.8.8:53 8.26.56.26:53 180.76.76.76:53]  file=main.go function=main.testBootstrap line=26
INFO[0000] Found 0 SRV records:                          file=main.go function=main.printAddress line=12

trekianov · April 9, 2021, 9:50am

I still don’t understand if I have to forward specific ports to my PC in the router: can you give me an idea or point me to the right documentation? Thanks!

fabrice · April 9, 2021, 4:11pm

I have the impression that something on your network is intercepting DNS queries and creating issues. It might be a firewall, a router, or your ISP. Can you try to directly connect to your modem? Can you try to connect through a hotspot or at another place?

The normal response to dig @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv should look like:

; <<>> DiG 9.16.1-Ubuntu <<>> @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20510
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_algobootstrap._tcp.testnet.algorand.network. IN SRV

;; ANSWER SECTION:
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 relay-washington-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 relay-singaporeg1-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 relay-singaporeg2-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 relay-dublin-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 r19.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 relay-london-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 80 relay-switzerland1-testnet-algorand.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 r20.algorand-testnet.network.
_algobootstrap._tcp.testnet.algorand.network. 299 IN SRV 1 1 4161 relay-virginia-testnet-algorand.algorand-testnet.network.

;; Query time: 20 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 09 12:09:46 EDT 2021
;; MSG SIZE  rcvd: 709

Yours has additional warnings such as:

;; Warning: Message parser reports malformed message packet.
;; WARNING: Message has 5 extra bytes at end

or

;; Warning: Message parser reports malformed message packet.
;; WARNING: Message has 36 extra bytes at end

Regarding opening ports on the firewall, Algorand nodes do not require to open any ports from the firewall. Only relays need to.

fabrice · April 9, 2021, 4:25pm

Can you try to run:

dig @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv +trace
dig @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv +trace +dnssec

trekianov · April 13, 2021, 4:27pm

I don’t know what has changed but currently everything is working properly:

goal node status -d /var/lib/algorand
Last committed block: 953277
Time since last block: 0.2s
Sync Time: 12593.9s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0
Round for next consensus protocol: 953278
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: mainnet-v1.0
Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=

trekianov · April 13, 2021, 4:29pm

dig @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv +trace

; <<>> DiG 9.16.1-Ubuntu <<>> @8.8.8.8 _algobootstrap._tcp.testnet.algorand.network srv +trace
; (1 server found)
;; global options: +cmd
.			478003	IN	NS	i.root-servers.net.
.			478003	IN	NS	e.root-servers.net.
.			478003	IN	NS	m.root-servers.net.
.			478003	IN	NS	c.root-servers.net.
.			478003	IN	NS	b.root-servers.net.
.			478003	IN	NS	h.root-servers.net.
.			478003	IN	NS	a.root-servers.net.
.			478003	IN	NS	f.root-servers.net.
.			478003	IN	NS	j.root-servers.net.
.			478003	IN	NS	l.root-servers.net.
.			478003	IN	NS	g.root-servers.net.
.			478003	IN	NS	k.root-servers.net.
.			478003	IN	NS	d.root-servers.net.
;; Received 239 bytes from 8.8.8.8#53(8.8.8.8) in 3 ms

Vytek · April 16, 2021, 10:47am

For my problem I think that there is a problem with my local ISP. Using cloud system like Vultr, Google Cloud, Azure or DigitalOcean all works fine!

Thank for all the suggests

trekianov · April 16, 2021, 12:37pm

I am located in Italy with Vodafone as ISP, maybe is related to this.

pilesofgravitas · February 15, 2022, 9:41pm

I had this same problem. The issue was the system clock was wrong. Not sure but my guess is that DNSSEC will/can fail if the clock is to far of.

Just leving this here if others run into the same thing.

Topic		Replies	Views
Algorand Node not Syncing - my node version 2.1.4 General	3	542	April 28, 2021
Algod syncing very slowly General	13	1020	January 27, 2022
Algo Node stopped syncing at 2.5.6 and resolved General	1	358	July 15, 2021
Algorand node not syncing with mainnet - a firewall issue? General noderunner	4	664	October 25, 2021
MainNet and TestNet Update: Go-Algorand 3.17.0 Announcements testnet , mainnet , releases	0	360	August 9, 2023

DNS problem with Algorand Server on Ubuntu 20.04

Related topics