There are a number of physical formats you can consider when buying gold:
If you're not buying an huge amount of gold, you should aim to maintain direct control over it at your home or other property.
If you're investing heavily, to the point where you need a safe alternative location to store it, be sure to go somewhere like Switzerland with strong property rights. It's key to make sure they can't lend or hedge using your stash.
The Swiss government is unique, in that it's highly decentralized. They simply don't have the structure that would allow for a central authority to come in and confiscate property.
Because of this, they've long been considered a safe haven for metals like gold, and the reason we recommend considering sending some there once you've accumulated a large amount.
Governments around the world have seized noble metals from time to time, so keeping your entire stash locally could be a bad idea depending on where you live.
Safe jurisdictions like Switzerland should be considered when you get to the point of holding a significant (tens of thousands of dollars worth generally) amount.
As a general rule, you should look at buying legal tender gold coins wherever possible. This will almost certainly include less overhead and premium than other formats, and be closer to the true spot price of gold.
Purchasing rare gold coins should be avoided unless you really know what you're doing. Some subjective measures of value are easy to misjudge, and may result in you getting taken advantage of.