Very interesting to see libsodium fork in tree. Forking libsodium is not entirely unusual but I have never seen it in tree before
Is there an easy way to see what you guys have changed? How about security audit reports?
Yeah, we’re not happy with it just being in tree either. Suggestions welcome.
Previously we had crypto/libsodium-fork as a git subtree – so you could e.g. split it out using git subtree split. This got messed up when we squashed commits. We could bring that back at the expense of massively cluttering the commit history.
We considered git submodules, but git submodules are quite awkward to use – anyone who clones our repo would have to run git submodule update --init. We could maybe put that in the makefile but submodules can still get pretty clumsy when switching branches.
We also considered having the build scripts call git clone directly, but that seems even uglier than submodules.
Another thought we had was to package libsodium-fork separately from go-algorand as (for instance) a separate .deb to be installed systemwide and then have go-algorand dynamically link against it (and have go-algorand’s .deb list libsodium-fork as a dependency). We’d need to change the name of libsodium-fork to avoid conflicting with existing libsodium installations, which would mean adding more differences from upstream libsodium.
None of these options seem great. Any ideas?
PS: For now, to check that the contents of go-algorand/crypto/libsodium-fork match github.com/algorand/libsodium, you can use e.g. diff -r. The only differences should be autotools build artifacts that we removed in go-algorand.
Also: for comparing github.com/algorand/libsodium to upstream libsodium, you’ll want to compare to the stable branch: https://github.com/jedisct1/libsodium/compare/stable...algorand:draft-irtf-cfrg-vrf-03
Cool! I love that you guys added in the VRF primitives. Do you think it will ever get merged upstream? Has this gone through security audit?
Thanks for posting a link to the diff.
Regarding having the fork in-tree, submodules is actually a pretty common approach depending on language. Normally, the git submodule init is handled by make but using make is unusual for a go project. Many rust projects these days are using submodules to maintain a similar thing but the rust build system allows for custom build scripts.
One more unusual option that I have seen is code inclusion via symlinks. You checkout the other repo in some other place in the filesystem and you symlink it into your git repo folder using some kind of script. That is even clunkier than the submodule init though.
Created a github issue for this: https://github.com/algorand/go-algorand/issues/20
As for merging upstream, libsodium isn’t interested unless VRFs become widely-used primitives enough to justify adding them to the library, and even then they wouldn’t want to add this VRF construction until it finishes the standardization process.
Using git submodules is usually a preferred solution. It will not obfuscate the git history and keep a clear project tree. Then, as already suggested you can use a make or bash script for installing. This is just one more very easy step for developers, which is no widely accepted.