There are a number of physical formats you can consider when buying gold:
If you're not buying an huge amount of gold, you should aim to maintain direct control over it at your home or other property.
If you're investing heavily, to the point where you need a safe alternative location to store it, be sure to go somewhere like Switzerland with strong property rights. It's key to make sure they can't lend or hedge using your stash.
Governments around the world have seized noble metals from time to time, so keeping your entire stash locally could be a bad idea depending on where you live.
Safe jurisdictions like Switzerland should be considered when you get to the point of holding a significant (tens of thousands of dollars worth generally) amount.
It's important to comply with the law when it comes to buying gold, and knowing what they are goes a long way.
Under a certain dollar amount, there's no need to declare (which may be attractive to some), but once you hit a threshold you will need to (and should) declare your holdings.
Buying gold on credit or speculating on the future price can lead to getting yourself in a significant financial jam.
Instead, put some money to the side that you're sure you're not going to need in a short time horizon, and use that to invest.