Basically. the wallets are not what holds the bitcoins per say... Technically all of your bitcoins are stored and dictated by the ledger. but its heavily encrypted so as a general thumb of rule only you who know the decryption key can use a wallet client to make changes with your wallet within the ledger.
now this is not entirely accurate but it should dumb it down a little to make the concept easier to grasp. hope that helped