This error means the smart contract rejected your transaction.
The reason may depend on the exact smart contract.
But many smart contracts nowadays check that the size of the group is a specific number (in your case = 4 transactions). With such smart contracts you cannot combine two sets of transactions like you did in a single atomic group.
See Transaction Group for multiple LogicSig signed transactions - #6 by fabrice for the same issue with TinyMan.
Note: Ideally smart contracts should not enforce the size of the group to allow such uses. This will be the recommendation in the ABI. https://github.com/algorandfoundation/ARCs/issues/44