There are a number of physical formats you can consider when buying gold:
Gold owned through a commodities exchange, or "on paper" is no substitute for physical bars, coins, or jewelry. Exchanges are heavily leveraged, which means if everyone tried to "cash out" at once, there isn't enough physical metal to go around.
Moreover, precious metals are typically meant as a hedge against the complete collapse of the traditional banking infrastructure. If you wind up needing it, you'll want to have it close at hand!
Buying gold on credit or speculating on the future price can lead to getting yourself in a significant financial jam.
Instead, put some money to the side that you're sure you're not going to need in a short time horizon, and use that to invest.
Governments around the world have seized noble metals from time to time, so keeping your entire stash locally could be a bad idea depending on where you live.
Safe jurisdictions like Switzerland should be considered when you get to the point of holding a significant (tens of thousands of dollars worth generally) amount.
Physical precious metals like gold are a hedge to the traditional banking system. In a crisis, if the system collapses, it may very well take your property down with it (if it can).
If there is a run on the banks, having your gold under your direct control means they can't confiscate it, block access to it, or have loaned it out in a leveraged fashion.