I am trying to understand the inner workings of Algorand, from the codebase it looks like the data is being written to SQL Lite and you can read the data using a standalone database tool, Indexer, that uses PostgresSQL. Is that correct?
If I were to run a private node what databases could I use in a node? Why was SQL Lite chosen?
The algorand node ( algod ) is using sqlite for its internal store.
Separately from that, the indexer, a completely independent process, knows how to communicate with the algorand node ( algod ) over REST API, query the block, and store it’s content to a Postgres database.
Sqlite was chosen as it provides excellent storage performance, embedded, lightweight and fault tolerant. Other databases might provide better indexing features - but these are not required for an algorand node.
I have a question regarding this topic. Running an (archival) node, this database is already pretty large on disk. What if Algorand grows 10x? Will we be required to have a 10 TB disk to store it? That doesn’t sound lightweight.
The real solution here is not to make an archival node “lighter” but rather find ways to avoid requiring the archival node at all.
The indexer, for instance, require the archival node only during bootstapping. After that, quering the latests blocks would suffice.
Is that answer your question ?
That’s interesting and indeed would help. If it weren’t for updates like the 2.8.0 where I need a full archival node again, I would do that. Is it foreseeable how often indexer does need to be rebuild?