Fast Catchup not working for our node on MainNet

I’m happy to hear that this is the method you’re using for the update. That would allow the update script to choose the optimal binary for your platform. I don’t know if the recent Raspberry PI OS was validated againt the update.sh, but it definitally should.

The fact that you’re no longer seeing memory issues might be a good indication that it does pick the correct executable. When it does fail, what is the fail reason ? The node.log should be able to tell you that. Try to grep for catchpoint.

@tsachi here is the last part of what I’ve found into node.archive.log, but I need help to interpret it:

{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 24.158404838s","name":"","readonly":false,"time":"2020-08-28T18:06:46.358071+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 25.894136324s","name":"","readonly":false,"time":"2020-08-28T18:07:26.173308+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 36.313175204s","name":"","readonly":false,"time":"2020-08-28T18:08:16.121020+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 32.541202632s","name":"","readonly":false,"time":"2020-08-28T18:09:02.541449+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 37.378767403s","name":"","readonly":false,"time":"2020-08-28T18:09:53.868994+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 46.48052051s","name":"","readonly":false,"time":"2020-08-28T18:10:56.126022+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 50.479435774s","name":"","readonly":false,"time":"2020-08-28T18:12:03.081049+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 737.804647ms","name":"","readonly":false,"time":"2020-08-28T18:12:10.024907+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 39.183075121s","name":"","readonly":false,"time":"2020-08-28T18:12:58.488246+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 43.18968605s","name":"","readonly":false,"time":"2020-08-28T18:13:56.154227+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 4.825808255s","name":"","readonly":false,"time":"2020-08-28T18:14:07.250155+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 44.390137192s","name":"","readonly":false,"time":"2020-08-28T18:15:01.509935+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 43.540042833s","name":"","readonly":false,"time":"2020-08-28T18:15:59.810084+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 55.050510895s","name":"","readonly":false,"time":"2020-08-28T18:17:14.973615+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 4.088145443s","name":"","readonly":false,"time":"2020-08-28T18:17:36.324394+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 55.802440381s","name":"","readonly":false,"time":"2020-08-28T18:18:40.209376+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m0.129440775s","name":"","readonly":false,"time":"2020-08-28T18:20:00.588336+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 195.498571ms","name":"","readonly":false,"time":"2020-08-28T18:20:06.129146+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 2.828967997s","name":"","readonly":false,"time":"2020-08-28T18:20:17.544347+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 45.798204967s","name":"","readonly":false,"time":"2020-08-28T18:21:18.176055+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m6.324626322s","name":"","readonly":false,"time":"2020-08-28T18:22:47.748188+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 58.462052979s","name":"","readonly":false,"time":"2020-08-28T18:24:03.393665+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 57.945454854s","name":"","readonly":false,"time":"2020-08-28T18:25:16.761280+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m2.030750011s","name":"","readonly":false,"time":"2020-08-28T18:26:34.305707+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 51.323908316s","name":"","readonly":false,"time":"2020-08-28T18:27:41.718411+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m7.364659879s","name":"","readonly":false,"time":"2020-08-28T18:29:06.456129+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 55.107420792s","name":"","readonly":false,"time":"2020-08-28T18:30:17.960202+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m1.831749736s","name":"","readonly":false,"time":"2020-08-28T18:31:35.993411+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 52.790472832s","name":"","readonly":false,"time":"2020-08-28T18:32:45.943488+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 41.468215ms","name":"","readonly":false,"time":"2020-08-28T18:32:48.285959+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m1.348392513s","name":"","readonly":false,"time":"2020-08-28T18:34:05.164043+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m3.733541615s","name":"","readonly":false,"time":"2020-08-28T18:35:26.309374+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m3.73249174s","name":"","readonly":false,"time":"2020-08-28T18:36:47.864061+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m0.203957939s","name":"","readonly":false,"time":"2020-08-28T18:38:05.270018+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m5.350465578s","name":"","readonly":false,"time":"2020-08-28T18:39:28.035255+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 76.896063ms","name":"","readonly":false,"time":"2020-08-28T18:39:33.606887+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m4.980213627s","name":"","readonly":false,"time":"2020-08-28T18:40:49.843544+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m8.832519488s","name":"","readonly":false,"time":"2020-08-28T18:42:15.452258+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 78.769431ms","name":"","readonly":false,"time":"2020-08-28T18:42:22.196299+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m6.459185189s","name":"","readonly":false,"time":"2020-08-28T18:43:39.777069+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m13.385987757s","name":"","readonly":false,"time":"2020-08-28T18:45:09.561494+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 2m8.306212157s","name":"","readonly":false,"time":"2020-08-28T18:47:35.677501+02:00"}
{"file":"catchpointService.go","function":"github.com/algorand/go-algorand/catchup.(*CatchpointCatchupService).processStageLedgerDownload","level":"warning","line":293,"msg":"unable to download ledger : context deadline exceeded","name":"","time":"2020-08-28T18:47:35.717078+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).ResetStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:177","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1m50.095259397s","name":"","readonly":false,"time":"2020-08-28T18:49:26.820994+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 1.183327372s","name":"","readonly":false,"time":"2020-08-28T18:49:38.152356+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 3.540351281s","name":"","readonly":false,"time":"2020-08-28T18:49:51.393872+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 5.493225476s","name":"","readonly":false,"time":"2020-08-28T18:50:07.143107+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 7.743167959s","name":"","readonly":false,"time":"2020-08-28T18:50:25.649590+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 9.940710385s","name":"","readonly":false,"time":"2020-08-28T18:50:46.744273+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 14.350936266s","name":"","readonly":false,"time":"2020-08-28T18:51:12.518498+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 17.284236443s","name":"","readonly":false,"time":"2020-08-28T18:51:41.286341+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 22.317491183s","name":"","readonly":false,"time":"2020-08-28T18:52:15.490093+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 22.399006263s","name":"","readonly":false,"time":"2020-08-28T18:52:50.167527+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 36.401643ms","name":"","readonly":false,"time":"2020-08-28T18:52:58.084855+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 21.223299657s","name":"","readonly":false,"time":"2020-08-28T18:53:24.194351+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 28.283678467s","name":"","readonly":false,"time":"2020-08-28T18:54:05.231913+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 33.25351547s","name":"","readonly":false,"time":"2020-08-28T18:54:51.433467+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 30.176835061s","name":"","readonly":false,"time":"2020-08-28T18:55:34.874454+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).processStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:294","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 36.497417191s","name":"","readonly":false,"time":"2020-08-28T18:56:26.479127+02:00"}
{"callee":"github.com/algorand/go-algorand/ledger.(*CatchpointCatchupAccessorImpl).ResetStagingBalances.func1","caller":"/home/ubuntu/go/src/github.com/algorand/go-algorand/ledger/catchupaccessor.go:177","file":"dbutil.go","function":"github.com/algorand/go-algorand/util/db.(*Accessor).atomic","level":"warning","line":355,"msg":"dbatomic: tx surpassed expected deadline by 16.487308757s","name":"","readonly":false,"time":"2020-08-28T18:56:43.967612+02:00"}
{"file":"catchpointService.go","function":"github.com/algorand/go-algorand/catchup.(*CatchpointCatchupService).run","level":"warning","line":201,"msg":"catchpoint catchup stage error : context canceled","name":"","time":"2020-08-28T18:56:43.967876+02:00"}
{"file":"node.go","function":"github.com/algorand/go-algorand/node.(*AlgorandFullNode).SetCatchpointCatchupMode.func1","level":"info","line":935,"msg":"Indexer is not available - indexer is not active","name":"","time":"2020-08-28T18:56:44.044601+02:00"}

Hi,

The only entry here that could be adjusted is ‘unable to download ledger : context deadline exceeded’. It means that the accumulated time it was processing this catchpoint exceeded the time limits, and the node decided to terminate the processing. I think that there is a config option to extend that, but can’t recall which one was it. I’ll try to look into that next week.

1 Like

Just for sake of completeness I add few more errors referred to the last fast catchup trial, maybe could be useful to identify what is going on:

{"file":"catchpointService.go","function":"github.com/algorand/go-algorand/catchup.(*CatchpointCatchupService).processStageLedgerDownload","level":"warning","line":293,"msg":"unable to download ledger : context deadline exceeded","name":"","time":"2020-08-29T21:22:28.409267+02:00"}
{"file":"catchpointService.go","function":"github.com/algorand/go-algorand/catchup.(*CatchpointCatchupService).processStageLedgerDownload","level":"warning","line":293,"msg":"unable to download ledger : getPeerLedger : http ledger fetcher response has an invalid content type : text/html; charset=UTF-8","name":"","time":"2020-08-29T21:24:40.184665+02:00"}
{"file":"catchpointService.go","function":"github.com/algorand/go-algorand/catchup.(*CatchpointCatchupService).run","level":"warning","line":201,"msg":"catchpoint catchup stage error : context canceled","name":"","time":"2020-08-29T21:27:39.876930+02:00"}

The getPeerLedger : http ledger fetcher response has an invalid content type : text/html; charset=UTF-8 is clearly something that need to be worked on; I suspect ( but don’t have enough information to be sure of ) that there is a relay out there that wasn’t upgraded. While this is not an issue for the protocol and/or the operation of the catchpoint catchup, it should get addressed.

Is this problem solved? I also encountered this problem here, Catchpoint accounts processed has been processing in a loop, I hope to get help

1 Like

Just to verify, since this is your first appearance here - you’ve ran into this issue while trying to run this on a Raspberry PI 4 with 8GB memory, correct ?

( i.e. I’m able to run the fast catchup from mainnet on my MacBook with no issues. I’d like to verify the above issue is limited to Raspberry PI )

No, mine is centos 64 8h 16g. I have no problem catching up quickly on the test network, but it is always unsuccessful on the mainnet. The number of Catchpoint accounts processed is always in the loop. I have modified the number of retries and the timeout period, but no Works, and I can’t get any error logs
image

Exactly the same experience.

The Fast catchup was successful on the TestNet, both on a Raspberry Pi 4 with 4GB and with 8GB.

The Fast catchup failed on MainNet instead, it keeps looping forever.

I will see if I could repeat the testing for that tomorrow on centos as well. You haven’t specified which version of centos you’re running, so I’ll be using version 8.

also, just to confirm @shixiaofeia - your centos machine is connected to the internet using a broadband connection with a bandwidth of 10mbps or more, right ? ( I’m just trying to make sure it’s not a spotty cellular connection or a dial up… )

image

@shixiaofeia - that’s only telling me that the centos machine is connected via it’s ethernet port, and that the ethernet adapter speed is super-awesome.

Could you also confirm that the router that connect this computer to the internet ( i.e. cable/dsl/fiber/other ) has ~10mbps ? ( i.e. mega bit per second ).

I’m not sure, how should I confirm?

I’m not affiliated in any way with the following site, but I used to find it useful:

As long as you have at least 10mbps, it shouldn’t be an issue.

Alternatively, you can try to use the command line to get these metrics :

According to the above, you appear to have an excellent internet connection.
I will double check the reported behavior on my end using a centos machine.

Thank you, I noticed that many people encountered the same problem as me, but I am curious that testnet and mainnet shouldn’t be the same?

The main difference from the perspective of the fast catchup is the number of accounts.
Testnet has far fewer accounts.

Following up on @shixiaofeia experiment -
I created a centos 7.7.1 docker container, grabbed version 2.1.4 using the rpm installer, and attempted to catchup to 8980000#KRIDEMKJI3SVXAMV5RKUBUEFQWZLYFMPJKLSA7LKLG4EOWZQDYXQ.

following this experiment, I have two more questions:

  1. What type of storage do you use on the centos machine ? is it a nvme, ssd, magnetic, or network storage disk ?
  2. Could you provide some metrics regarding the duration after which it “restarted” ?

I know that in the past, we’ve seen issues with people running their nodes on network storage disks ( such as EBS volumes ).

I should be using a SATA hard drive. There are no corresponding indicators. I have tried directly using the default configuration or modifying the number of synchronizations and the timeout period, but they all failed. I want to know that the total blocks and total blocks will appear after the accounts processed. Are these two indicators down blocks? I have always been accounts processed and restarted from 0 after the processing is completed. I have never seen blocks related parameters. One thing may be related to this. I use the Installing with Other Linux Distros method Installed, i.e. wget https://raw.githubusercontent.com/algorand/go-algorand-doc/master/downloads/installers/update.sh, will it be related to this?