Node is not processing blocks that lookback block doesn't exist, cannot authenticate new block

Node is not processing blocks

The file algod.token was modified and algod was restarted 1 hour ago. The node is not processing blocks. Repeated attempts to reboot algod were unsuccessful

{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"error","line":379,"msg":"fetchAndWrite(27017265): ledger write failed: transaction X36ZGQ34LTKRDMDFKAO4USRG3DZHUYE5G6DZGS767PVH7JO7ZBLQ: transaction rejected by ApprovalProgram","name":"","time":"2023-02-16T16:56:09.718592+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017265): did not fetch or write the block","name":"","time":"2023-02-16T16:56:09.718612+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017267): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:09.781616+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017267): did not fetch or write the block","name":"","time":"2023-02-16T16:56:09.781674+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":387,"msg":"fetchAndWrite(27017266): previous block doesn't exist (perhaps fetching block 27017265 failed)","name":"","time":"2023-02-16T16:56:09.953574+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017266): did not fetch or write the block","name":"","time":"2023-02-16T16:56:09.953630+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017268): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.135588+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017268): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.135661+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017270): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.135693+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017270): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.135720+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017272): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.135755+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017272): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.135775+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017274): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322277+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017274): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322344+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017276): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322375+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017276): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322399+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017278): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322430+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017278): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322454+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017269): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322458+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017269): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322507+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017280): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322482+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017280): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322541+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017271): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322531+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017271): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322577+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017273): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322613+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017273): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322659+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017275): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322689+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017275): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322720+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017277): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322748+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017277): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322782+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite](http://github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite)","level":"warning","line":301,"msg":"fetchAndWrite(27017279): lookback block doesn't exist, cannot authenticate new block","name":"","time":"2023-02-16T16:56:10.322803+08:00"}
{"Context":"sync","file":"service.go","function":"[github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1](http://github.com/algorand/go-algorand/catchup.(*Service).pipelineCallback.func1)","level":"info","line":405,"msg":"pipelineCallback(27017279): did not fetch or write the block","name":"","time":"2023-02-16T16:56:10.322818+08:00"}
{"Context":"sync","details":{"StartRound":27017264,"EndRound":27017264,"Time":749758403,"InitSync":false},"file":"telemetry.go","function":"[github.com/algorand/go-algorand/logging.(*telemetryState).logTelemetry](http://github.com/algorand/go-algorand/logging.(*telemetryState).logTelemetry)","instanceName":"uYiFu8Cmcef2NGki","level":"info","line":261,"msg":"/ApplicationState/CatchupStop","name":"","session":"","time":"2023-02-16T16:56:10.322851+08:00","v":"3.12.2"}

My environment

algod -v

3.12.2.stable [rel/stable] (commit #181490e3) 
go-algorand is licensed with AGPLv3.0 
source code available at [GitHub - algorand/go-algorand: Algorand's official implementation in Go.](https://github.com/algorand/go-algorand)

goal node status

Last committed block: 27017264
Time since last block: 0.0s
Sync Time: 0.0s
Last consensus protocol: GitHub - algorandfoundation/specs at 44fa607d6051730f5264526bf3c108d51f0eadb6
Next consensus protocol: GitHub - algorandfoundation/specs at 44fa607d6051730f5264526bf3c108d51f0eadb6
Round for next consensus protocol: 27017265
Next consensus protocol supported: true
Last Catchpoint: 27010000#TB54RTUVA53PVCTINDSBF46AWESAVH7XZGNHGKIKJO4GAM6EOM5A
Genesis ID: mainnet-v1.0
Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=

System

Linux 5.10.102-99.473.amzn2.x86_64 #1 SMP Wed Mar 2 19:14:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

How to fix it up? tks

Hey ctry, Please could you try updating your node to the latest version and see if that helps?

1 Like

Had similar issue with 3.13.3 … Issue was solved by restart of the node.

1 Like

Nice. Using version v3.14.2.stable , the nodes are back in sync. tks

1 Like