There are a number of physical formats you can consider when buying gold:
Buying gold on credit or speculating on the future price can lead to getting yourself in a significant financial jam.
Instead, put some money to the side that you're sure you're not going to need in a short time horizon, and use that to invest.
Gold owned through a commodities exchange, or "on paper" is no substitute for physical bars, coins, or jewelry. Exchanges are heavily leveraged, which means if everyone tried to "cash out" at once, there isn't enough physical metal to go around.
Moreover, precious metals are typically meant as a hedge against the complete collapse of the traditional banking infrastructure. If you wind up needing it, you'll want to have it close at hand!
As a general rule, you should look at buying legal tender gold coins wherever possible. This will almost certainly include less overhead and premium than other formats, and be closer to the true spot price of gold.
Purchasing rare gold coins should be avoided unless you really know what you're doing. Some subjective measures of value are easy to misjudge, and may result in you getting taken advantage of.
The Swiss government is unique, in that it's highly decentralized. They simply don't have the structure that would allow for a central authority to come in and confiscate property.
Because of this, they've long been considered a safe haven for metals like gold, and the reason we recommend considering sending some there once you've accumulated a large amount.