Our indexer has gotten stuck with this error:
{“error”:“AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415”,“level”:“error”,“msg”:“adding block 20537415 to database failed”,“time”:“2022-04-23T20:09:44+02:00”}
We may have inadvertently caused this by running a background reindex (REINDEX CONCURRENTLY) while the indexer was running.
Any tips on how to get it unstuck?
First check that you have the latest version of the indexer.
If you have the latest version, the simplest solution is to clear the database and resync from the beginning but it takes a lot of time.
We are on 2.10.0, which seems to be the latest version according to github releases.
The simplest solution you mention is the last resort, as it took us 77 days to sync from block 6M to current (no 2TB SSD servers available in our budget). We will likely abandon the endeavor of running our own indexer if we are unable to fix this instance.
Any ideas about the nature of the error? We couldn’t find it in the source code.
We found the indexer logs around the time the error was first encountered but nothing seems amiss:
Apr 21 10:06:51 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537413","time":"2022-04-21T10:06:51+02:00"}
Apr 21 10:06:52 k44 algorand-indexer[85573]: {"level":"info","msg":"round r=20537413 (41 txn) imported in 818.390509ms","time":"2022-04-21T10:06:52+02:00"}
Apr 21 10:06:56 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537414","time":"2022-04-21T10:06:56+02:00"}
Apr 21 10:06:56 k44 algorand-indexer[85573]: {"level":"info","msg":"round r=20537414 (30 txn) imported in 714.184434ms","time":"2022-04-21T10:06:56+02:00"}
Apr 21 10:07:00 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:00+02:00"}
Apr 21 10:07:01 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:01+02:00"}
Apr 21 10:07:02 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:02+02:00"}
Apr 21 10:07:02 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:02+02:00"}
Apr 21 10:07:03 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:03+02:00"}
Apr 21 10:07:04 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:03+02:00"}
Apr 21 10:07:04 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:04+02:00"}
Apr 21 10:07:04 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:04+02:00"}
The algod that feeds the indexer seems normal for those rounds as well.
Previous round header - processed OK:
{
"block": {
"earn": 218250,
"fees": "Y76M3MSY6DKBRHBL7C3NNDXGS5IIMQVQVUAB6MP4XEMMGVF2QWNPL226CA",
"frac": 332612560,
"gen": "mainnet-v1.0",
"gh": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
"prev": "blk-XXDSGYHLHZZDXVVIKADQBE4SN2QRXWVFIXTBCYPENPILQ4RQ3LPA",
"proto": "https://github.com/algorandfoundation/specs/tree/d5ac876d7ede07367dbaa26e149aa42589aac1f7",
"rate": 600000,
"rnd": 20537414,
"rwcalr": 21000000,
"rwd": "737777777777777777777777777777777777777777777777777UFEJ2CI",
"seed": "FT3SeQnJSkZ0qmWcFCZl6P7x7HVYRtLIphqxPm4gJxk=",
"tc": 710532082,
"ts": 1650528411,
"txn": "wNvbGfCyVSfjCMyIadbN1Wm0QxI28FFA1rwZCxMn1IA=",
"txns": [
Round we are stuck on:
{
"block": {
"earn": 218250,
"fees": "Y76M3MSY6DKBRHBL7C3NNDXGS5IIMQVQVUAB6MP4XEMMGVF2QWNPL226CA",
"frac": 333212560,
"gen": "mainnet-v1.0",
"gh": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
"prev": "blk-6FDIWYYAY7PH3FJQFRDHJNEFFYHUEJONEGIJDVNSXQUA7VVR3S3A",
"proto": "https://github.com/algorandfoundation/specs/tree/d5ac876d7ede07367dbaa26e149aa42589aac1f7",
"rate": 600000,
"rnd": 20537415,
"rwcalr": 21000000,
"rwd": "737777777777777777777777777777777777777777777777777UFEJ2CI",
"seed": "WyCWA2l9/bD9IGR5NB4dNkUwPKp9jc/EPKoHuHot8bA=",
"tc": 710532112,
"ts": 1650528416,
"txn": "KItphoDC0K4CpYaJounQFKIDb3f6EcDWCwbUXTYzUic=",
"txns": [
{
Found the source of the error in the source code - it is in go-algorand repo:
Hmm, unfortunately I think this is broken, because the database seems to be in an inconsistent state. For data integrity I’d unfortunately recommend re-indexing, because the block evaluator is getting a different answer on the indexer side.
If your project is small, there’s a free indexer API service available at AlgoNode | Free Algorand API provider. I think they have a good reputation.
Also, the website provides a blockchain snapshot for a quick, full sync-up for non-production environment - i.e., use it on your own risk.
This particular error can happen if you upgraded Indexer (I think it was 2.6 → 2.8) without doing a re-sync of the DB. There was a pretty significant DB change and I don’t think Indexer could update the DB. I vaguely recall it throwing this error.
Also, from 2.9 to 2.10 there was a DB update that takes about 1 hour on good hardware to update.
It’s unfortunate, but like others said, you may be in for a re-index from block 0
Thanks all. This was very disappointing and we will likely not run an indexer going forward. We’ll make do with the free/community options.
We may attempt to fix it by deleting the last few blocks entered but not holding our breaths about doing this correctly.
There was no indexer upgrade involved BTW.