Sandbox fails to complete initialization on testnet -- it doesn't even try to "fast-catchup." Can anybody help?

General
1

When I launch sandbox with ./sandbox up testnet, instead of getting the expected “Indexer disabled for this configuration,” I get this:

OCI runtime exec failed: exec failed: container_linux.go:380: starting container
 process caused: exec: "cmd/algorand-indexer/algorand-indexer": stat cmd/algorand-indexer/algoran
d-indexer: no such file or directory: unknown

and there is no "fast-catchup."

The version information returned is

Version: [v1 v2] 
GenesisID: testnet-v1.0 
Build: 3.2.3.stable [rel/stable] (commit #d2289a52)

I looked at the config.testnet file, and it shows:

export ALGOD_CHANNEL="stable"
export ALGOD_URL=""
export ALGOD_BRANCH=""
export ALGOD_SHA=""
export NETWORK="testnet"
export NETWORK_BOOTSTRAP_URL="<network>.algorand.network"
export NETWORK_GENESIS_FILE="genesis/testnet/genesis.json"
export INDEXER_URL=""
export INDEXER_BRANCH=""
export INDEXER_SHA=""
export INDEXER_DISABLED="true"

It continues to do this after doing “./sandbox reset” and “./sandbox clean” repeatedly.

2
  • What is your OS? (Ubuntu/macOS/Windows? version? if Windows how do you run things?)
  • What is your Docker version? Have you tried to reset Docker to factory defaults?
  • What is your sandbox version? git status and git rev-parse HEAD
  • Can you provide the full output of ./sandbox up testnet and the full sandbox.log (you may want to upload it on https://gist.github.com/ and link it there)?
3

Thanks.

OS is Mac OS Catalina.

Docker was just installed last week. Preferences/Software Updates shows

You’re up to date
Docker Desktop 4.4.2 (73305) is currently the newest version available.

“git status” in ~/sandbox shows

On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	.DS_Store
	genesis/.DS_Store

nothing added to commit but untracked files present (use "git add" to track)

and git rev-parse HEAD returns

872ef63256f77d47455d1e004f0035dadcea9d8d

I put the “.sandbox up testnet” output in the previous thread. For convenience, I repeat it here:

$ ./sandbox up testnet

Starting sandbox for: testnet
see sandbox.log for detailed progress, or use -v.
* started!

algod version
12885032963
3.2.3.stable [rel/stable] (commit #d2289a52)
go-algorand is licensed with AGPLv3.0
source code available at https://github.com/algorand/go-algorand

Indexer version
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "cmd/algorand-indexer/algorand-indexer": stat cmd/algorand-indexer/algorand-indexer: no such file or directory: unknown

Complete log file is at Algorand sandbox log file · GitHub
(I don’t see anything in there that needs to be redacted, and I trust that you would have told me if there were.)

4

I have a very similar configuration and here is what I see:

$ ./sandbox up testnet

Starting sandbox for: testnet
see sandbox.log for detailed progress, or use -v.
* started!

algod version
12885032963
3.2.3.stable [rel/stable] (commit #d2289a52)
go-algorand is licensed with AGPLv3.0
source code available at https://github.com/algorand/go-algorand

Indexer version
Indexer disabled for this configuration.

Postgres version
postgres (PostgreSQL) 13.3

algod - goal node status
Last committed block: 396
Time since last block: 0.0s
Sync Time: 5.1s
Last consensus protocol: https://github.com/algorand/spec/tree/a26ed78ed8f834e2b9ccb6eb7d3ee9f629a6e622
Next consensus protocol: https://github.com/algorand/spec/tree/a26ed78ed8f834e2b9ccb6eb7d3ee9f629a6e622
Round for next consensus protocol: 397
Next consensus protocol supported: true
Last Catchpoint:
Genesis ID: testnet-v1.0
Genesis hash: SGO1GKSzyE7IEPItTxCByw9x8FmnrCDexi9/cOUJOiI=

indexer - health
Indexer disabled for this configuration.

Starting fast-catchup with catchpoint: 19250000#J7XQD3K62ZAL2VWNKVGQNLWFXLS7ZO5X2FTKFONUKEFOOXXB4RXQ
* Account processing complete.
* Blocks downloaded.

Fast-catchup complete! Printing status...

algod - goal node status
Last committed block: 19250217
Time since last block: 0.0s
Sync Time: 1.8s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19250218
Next consensus protocol supported: true
Last Catchpoint:
Genesis ID: testnet-v1.0
Genesis hash: SGO1GKSzyE7IEPItTxCByw9x8FmnrCDexi9/cOUJOiI=

indexer - health
Indexer disabled for this configuration.

The only difference I see between my configuration and yours is that I’m on macOS BigSur. Are you able to try to update to BigSur?

The other thing I would try is to reset completely Docker: “reset to factory defaults” - Logs and troubleshooting | Docker Documentation
Doing so will remove all your Docker containers.

5

Tried it. Still no joy.

Big Sur is not an option at this time, and it shouldn’t have to be: there’s an old 2014 Mac Mini in the next office (we call it the “Project Mini” because it’s used for projects that might contaminate a more mission-critical system), also on Catalina (just one build different from my 2017 iMac), and some weeks ago, I brought up Sandbox there, as a quick test. On that machine, Sandbox consistently comes up just fine.

On the “Project Mini,” I Sandbox shows algod version 12885032962 (vs. …63 on my iMac), commit #97e80680 (vs. commit #d2289a52 on my iMac).

Curiouser and curiouser.

Update
I just compared the ~/sandbox directories between my iMac and the Project Mini. Except for datestamps, they look identical: nothing present on one box but missing on the other, and the sizes appear to match.

6

Do you have the latest update of XCode (if installed) and XCode Command Line Tools on the mac?

If yes, the only thing I can think that may be an issue is the use of a non-traditional shell or non-traditional environment variables.
Can you try to run everything in the basic shell with the profile disabled, that is within:

/bin/zsh -f -d
1 Like
7

Yes, XCode is present on my iMac.

Doing a “./sandbox up testnet” under “/bin/zsh -f -d” does get me to fast-catchup. I don’t get a working progress bar, the way I do on the Project Mini, but it seems to be doing something.

8

So this means you have an issue with your shell.
Either you’re not using the proper zsh, or you have something bad in your ~/.zshrc

1 Like
9

First of all, when I launched Sandbox under “/bin/zsh -f -d,” fast-catchup started, but it looked like it never stopped. It never did go back to a command prompt; rather, it just kept spitting out another update report (with the progress bar locked at zero) every second or so.

Second of all, I did a ps -p $$ on my iMac and on the Project Mini. I have Bash set as my shell-of-choice on my iMac, whereas the Project Mini shows zsh.

Third, guessing that the Sandbox was caught up even if it didn’t look like it, I went ahead and ran the “Getting Started” application again, and this time, it worked.

Just for the sake of empiricism, I also went over to the Project Mini, shut down the Sandbox that I’d started on it, and switched shells to Bash.

It behaves exactly the same under Bash as under Zsh. So it’s not the shell itself.

So now, we need to figure out what’s fighting with the Sandbox.

11

If I do a printenv on my iMac, in a default terminal session, I get:

$ printenv
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/cn/vsn822092lqdlp8g6psy47380000gn/T/
TERM_PROGRAM_VERSION=433
TERM_SESSION_ID=19553C23-52E7-4FF7-94DD-1CB7A4044607
USER=jameslampert
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.Ntl3FrYsJh/Listeners
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
PWD=/Users/jameslampert
LANG=en_US.UTF-8
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/jameslampert
LOGNAME=jameslampert
DISPLAY=/private/tmp/com.apple.launchd.az2lAJ6Q0G/org.macosforge.xquartz:0
_=/usr/bin/printenv

If I do it in “/bin/zsh -f -d,” I get:

% printenv
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/cn/vsn822092lqdlp8g6psy47380000gn/T/
TERM_PROGRAM_VERSION=433
TERM_SESSION_ID=19553C23-52E7-4FF7-94DD-1CB7A4044607
USER=jameslampert
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.Ntl3FrYsJh/Listeners
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
PWD=/Users/jameslampert
LANG=en_US.UTF-8
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=2
HOME=/Users/jameslampert
LOGNAME=jameslampert
DISPLAY=/private/tmp/com.apple.launchd.az2lAJ6Q0G/org.macosforge.xquartz:0
OLDPWD=/Users/jameslampert
_=/usr/bin/printenv

On the “Project Mini,” printenv in a default terminal session (zsh here) gets (user name redacted):

% printenv
TMPDIR=/var/folders/dr/qngm024x5112b02k3srg7hp40000gn/T/
XPC_FLAGS=0x0
TERM_PROGRAM_VERSION=433
LANG=en_US.UTF-8
TERM_PROGRAM=Apple_Terminal
XPC_SERVICE_NAME=0
TERM_SESSION_ID=EF6A72A4-2F95-4423-891B-DBC6AFBE7D2C
TERM=xterm-256color
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.xzvjexmKqL/Listeners
SHELL=/bin/zsh
HOME=/Users/<REDACTED>
LOGNAME=<REDACTED>
USER=<REDACTED>
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
SHLVL=1
PWD=/Users/<REDACTED>
OLDPWD=/Users/<REDACTED>
_=/usr/bin/printenv

while in Bash, it returns:

$ printenv
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/zsh
TERM=xterm-256color
TMPDIR=/var/folders/dr/qngm024x5112b02k3srg7hp40000gn/T/
TERM_PROGRAM_VERSION=433
TERM_SESSION_ID=EF6A72A4-2F95-4423-891B-DBC6AFBE7D2C
USER=<REDACTED>
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.xzvjexmKqL/Listeners
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
_=/usr/bin/printenv
PWD=/Users/<REDACTED>
LANG=en_US.UTF-8
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=2
HOME=/Users/<REDACTED>
LOGNAME=<REDACTED>

Update
I hadn’t noticed that “sandbox/sandbox” was a shell script. That means it can be tinkered with.

The line where it’s blowing up is

INDEXER_VERSION=$(dc exec indexer cmd/algorand-indexer/algorand-indexer -v) && echo ${INDEXER_VERSION} || curl -s "localhost:8980/health?pretty"

Can somebody explain the line?

If I comment out the line, then I get (using my default of Bash):

$ ./sandbox up testnet

Starting sandbox for: testnet
see sandbox.log for detailed progress, or use -v.
* started!

algod version
12885032963
3.2.3.stable [rel/stable] (commit #d2289a52)
go-algorand is licensed with AGPLv3.0
source code available at https://github.com/algorand/go-algorand

Postgres version
postgres (PostgreSQL) 13.5

algod - goal node status
Last committed block: 19268274
Time since last block: 3.6s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Genesis hash: SGO1GKSzyE7IEPItTxCByw9x8FmnrCDexi9/cOUJOiI=

indexer - health
Indexer disabled for this configuration.

Starting fast-catchup with catchpoint: 19280000#VRGPXH3UQ2HI7TLNMHVBEIWFXKBP4PMK63K34HEQDCX47RKI7WEQ

Last committed block: 19268274
Time since last block: 5.4s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19268274
Time since last block: 6.0s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19268274
Time since last block: 6.5s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19268274
Time since last block: 7.0s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19268274
Time since last block: 7.6s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19268274
Time since last block: 8.1s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19268274
Time since last block: 8.6s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19268275
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%

and so on ad nauseum, with no status bar, and with the individual reports scrolling up the terminal window, rather than overwriting each other in the terminal window, which is what I was getting before in “/bin/zsh -f -d,” which also looks very much like what “lawlesskane” is seeing in another thread.

Eventually, the “Sync Time” stops incrementing, and resets to zero:

. . .
Last committed block: 19283709
Time since last block: 3.8s
Sync Time: 759.4s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19283710
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19283710
Time since last block: 0.1s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19283711
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
Last committed block: 19283710
Time since last block: 0.6s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/bc36005dbd776e6d1eaf0c560619bb183215645c
Round for next consensus protocol: 19283711
Next consensus protocol supported: true
Last Catchpoint: 
Genesis ID: testnet-v1.0
Processing accounts: : [                                        ] [0/1000] 0%
. . .

but the reports continue to scroll up the terminal window. Based on my experiences of yesterday, I’m guessing that when that happens, I can ctrl-c out, and use the sandbox node.

12

Different behavior today:

The first time I tried to bring up Sandbox (with the Indexer Version lines still commented out), it got as far as “algod - goal node status” and then crashed with

Cannot contact Algorand node: Get "http://[::]:4001/v2/status": dial tcp [::]:4001: connect: cannot assign requested address

I saw that once last week as well, but it started, and went to fast-catchup (albeit anomalous fast-catchup) the second time.

Today, it took three tries to get Sandbox to start.

Also, last week, the “Sync Time” value got up to 759.4, then reset to zero, and stopped incrementing, which I took to mean I could ctrl-C out of the anomalous fast-catchup. Today, it continued to count up; I finally hit ctrl-C at Sync Time: 3111.3s (and it seemed to work, although I didn’t dispense any new “Monopoly Algos”).