There are a number of physical formats you can consider when buying gold:
Gold owned through a commodities exchange, or "on paper" is no substitute for physical bars, coins, or jewelry. Exchanges are heavily leveraged, which means if everyone tried to "cash out" at once, there isn't enough physical metal to go around.
Moreover, precious metals are typically meant as a hedge against the complete collapse of the traditional banking infrastructure. If you wind up needing it, you'll want to have it close at hand!
As a general rule, you should look at buying legal tender gold coins wherever possible. This will almost certainly include less overhead and premium than other formats, and be closer to the true spot price of gold.
Purchasing rare gold coins should be avoided unless you really know what you're doing. Some subjective measures of value are easy to misjudge, and may result in you getting taken advantage of.
When investing in any asset, including gold, it's important to keep the time horizon in mind. Prices fluctuate, and over a longer time horizon there's a better chance of your investment appreciating in value.
The last thing anyone would want is to invest money they'll need in the coming months, only to have the market move against them. Talk to your financial adviser, and carefully plan any future purchases.
Physical precious metals like gold are a hedge to the traditional banking system. In a crisis, if the system collapses, it may very well take your property down with it (if it can).
If there is a run on the banks, having your gold under your direct control means they can't confiscate it, block access to it, or have loaned it out in a leveraged fashion.