Even if you don't really own files called bitcoins but rather a sort of "ownership certificate" that allows you to set these "digital coins" in motion, we can simplify by saying that bitcoins are stored in an electronic wallet. The wallet can be represented by an application on your smartphone or software on your computer. This application contains your public addresses, that are alphanumeric strings, through which you can "receive" bitcoins (analogous to the well-known banking IBAN), and private keys, to be kept hidden (similar to your bank credentials).
The private key certifies your property, so if you lose that you cannot recover your bitcoins.
Through the private key, your wallet can calculate the spendable budget and can have new transactions verified.
The vast majority of wallets do not show the user the private keys to prevent the latter from sharing them, but if they are non-custodial wallets they allow the owner to export them and save them elsewhere.
Some users prefer to keep their funds on wallets that do not allow this private key backup: these are called custodial wallets and are centralized systems, analogous to classic banks but often with a much less efficient security system than the latters.