Hi,
I started the mainnet data of Algorand to be loaded in archival mode on a 3 TB disk of a VM machine which has 8 CPU and 32 GB RAM. In 2 days it became 165GB. But after 13 days more it can only reach 318 GB. How can I make this process faster? The indexer also works in parallel.
Is there a variable that should be set to a different value in config.json file to make this process faster?
Hope to hear from the support soon.
The outputs of the commands to check the status and settings in config.json are here:
$ goal node status
Last committed block: 13401207
Time since last block: 2.9s
Sync Time: 1322034.2s
Last consensus protocol: GitHub - algorandfoundation/specs at 3a83c4c743f8b17adfd73944b4319c25722a6782
Next consensus protocol: GitHub - algorandfoundation/specs at ac2255d586c4474d4ebcf3809acccb59b7ef34ff
Round for next consensus protocol: 13409876
Next consensus protocol supported: true
Last Catchpoint:
Genesis ID: mainnet-v1.0
Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=
$ goal version -v
Version: [v1 v2]
GenesisID: mainnet-v1.0
Build: 3.5.1.stable [rel/stable] (commit #aa2fb0ee)
ubuntu@algorand-node:~$ du -hs $ALGORAND_DATA/*
577M /mnt/algorand/agreement.cdv
1.1G /mnt/algorand/agreement.cdv.archive
4.0K /mnt/algorand/algod.admin.token
0 /mnt/algorand/algod.lock
4.0K /mnt/algorand/algod.net
4.0K /mnt/algorand/algod.pid
4.0K /mnt/algorand/algod.token
4.0K /mnt/algorand/config.json
28K /mnt/algorand/genesis.json
4.0K /mnt/algorand/goal.cache
4.0K /mnt/algorand/logging.config
du: cannot read directory ‘/mnt/algorand/mainnet-v1.0’: Permission denied
4.0K /mnt/algorand/mainnet-v1.0
1.1G /mnt/algorand/node.archive.log
806M /mnt/algorand/node.log
$ sudo du -hs /mnt/algorand
313G /mnt/algorand
$ cat $ALGORAND_DATA/config.json
{
“Version”: 19,
“AccountUpdatesStatsInterval”: 5000000000,
“AccountsRebuildSynchronousMode”: 1,
“AnnounceParticipationKey”: true,
“Archival”: true,
“BaseLoggerDebugLevel”: 4,
“BlockServiceCustomFallbackEndpoints”: “”,
“BroadcastConnectionsLimit”: -1,
“CadaverSizeTarget”: 1073741824,
“CatchpointFileHistoryLength”: 365,
“CatchpointInterval”: 10000,
“CatchpointTracking”: 0,
“CatchupBlockDownloadRetryAttempts”: 1000,
“CatchupBlockValidateMode”: 0,
“CatchupFailurePeerRefreshRate”: 10,
“CatchupGossipBlockFetchTimeoutSec”: 4,
“CatchupHTTPBlockFetchTimeoutSec”: 4,
“CatchupLedgerDownloadRetryAttempts”: 50,
“CatchupParallelBlocks”: 4,
“ConnectionsRateLimitingCount”: 60,
“ConnectionsRateLimitingWindowSeconds”: 1,
“DNSBootstrapID”: “.algorand.network”,
“DNSSecurityFlags”: 1,
“DeadlockDetection”: 0,
“DisableLocalhostConnectionRateLimit”: true,
“DisableNetworking”: false,
“DisableOutgoingConnectionThrottling”: false,
“EnableAccountUpdatesStats”: false,
“EnableAgreementReporting”: false,
“EnableAgreementTimeMetrics”: false,
“EnableAssembleStats”: false,
“EnableBlockService”: false,
“EnableBlockServiceFallbackToArchiver”: true,
“EnableCatchupFromArchiveServers”: false,
“EnableDeveloperAPI”: false,
“EnableGossipBlockService”: true,
“EnableIncomingMessageFilter”: false,
“EnableLedgerService”: false,
“EnableMetricReporting”: false,
“EnableOutgoingNetworkMessageFiltering”: true,
“EnablePingHandler”: true,
“EnableProcessBlockStats”: false,
“EnableProfiler”: false,
“EnableRequestLogger”: false,
“EnableTopAccountsReporting”: false,
“EnableVerbosedTransactionSyncLogging”: false,
“EndpointAddress”: “172.23.149.211:4001”,
“FallbackDNSResolverAddress”: “”,
“ForceFetchTransactions”: false,
“ForceRelayMessages”: false,
“GossipFanout”: 4,
“IncomingConnectionsLimit”: 800,
“IncomingMessageFilterBucketCount”: 5,
“IncomingMessageFilterBucketSize”: 512,
“IsIndexerActive”: true,
“LedgerSynchronousMode”: 2,
“LogArchiveMaxAge”: “”,
“LogArchiveName”: “node.archive.log”,
“LogSizeLimit”: 1073741824,
“MaxCatchpointDownloadDuration”: 7200000000000,
“MaxConnectionsPerIP”: 30,
“MinCatchpointFileDownloadBytesPerSecond”: 20480,
“NetAddress”: “”,
“NetworkMessageTraceServer”: “”,
“NetworkProtocolVersion”: “”,
“NodeExporterListenAddress”: “:9100”,
“NodeExporterPath”: “./node_exporter”,
“OptimizeAccountsDatabaseOnStartup”: false,
“OutgoingMessageFilterBucketCount”: 3,
“OutgoingMessageFilterBucketSize”: 128,
“ParticipationKeysRefreshInterval”: 60000000000,
“PeerConnectionsUpdateInterval”: 3600,
“PeerPingPeriodSeconds”: 0,
“PriorityPeers”: {},
“ProposalAssemblyTime”: 250000000,
“PublicAddress”: “”,
“ReconnectTime”: 60000000000,
“ReservedFDs”: 256,
“RestReadTimeoutSeconds”: 15,
“RestWriteTimeoutSeconds”: 120,
“RunHosted”: false,
“SuggestedFeeBlockHistory”: 3,
“SuggestedFeeSlidingWindowSize”: 50,
“TLSCertFile”: “”,
“TLSKeyFile”: “”,
“TelemetryToLog”: true,
“TransactionSyncDataExchangeRate”: 0,
“TransactionSyncSignificantMessageThreshold”: 0,
“TxPoolExponentialIncreaseFactor”: 2,
“TxPoolSize”: 15000,
“TxSyncIntervalSeconds”: 60,
“TxSyncServeResponseSize”: 1000000,
“TxSyncTimeoutSeconds”: 30,
“UseXForwardedForAddressField”: “”,
“VerifiedTranscationsCacheSize”: 30000
}
$ iostat
Linux 5.4.0-105-generic (algorand-node-and-kafka) 04/20/22 x86_64 (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
8.41 0.00 2.71 8.86 0.13 79.89
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop0 0.01 0.01 0.00 0.00 32466 0 0
loop1 0.00 0.00 0.00 0.00 3604 0 0
loop2 0.00 0.00 0.00 0.00 3129 0 0
loop3 0.00 0.00 0.00 0.00 3645 0 0
loop4 0.01 0.01 0.00 0.00 13905 0 0
loop5 0.03 0.03 0.00 0.00 67308 0 0
loop6 0.00 0.00 0.00 0.00 3650 0 0
loop7 0.00 0.00 0.00 0.00 13 0 0
vda 12.87 13.42 327.12 0.00 32053649 781264694 0
vdc 1039.83 45.77 9275.26 0.00 109312677 22152265092 0