There are a number of physical formats you can consider when buying gold:
Governments around the world have seized noble metals from time to time, so keeping your entire stash locally could be a bad idea depending on where you live.
Safe jurisdictions like Switzerland should be considered when you get to the point of holding a significant (tens of thousands of dollars worth generally) amount.
Gold owned through a commodities exchange, or "on paper" is no substitute for physical bars, coins, or jewelry. Exchanges are heavily leveraged, which means if everyone tried to "cash out" at once, there isn't enough physical metal to go around.
Moreover, precious metals are typically meant as a hedge against the complete collapse of the traditional banking infrastructure. If you wind up needing it, you'll want to have it close at hand!
It's important to comply with the law when it comes to buying gold, and knowing what they are goes a long way.
Under a certain dollar amount, there's no need to declare (which may be attractive to some), but once you hit a threshold you will need to (and should) declare your holdings.
When investing in any asset, including gold, it's important to keep the time horizon in mind. Prices fluctuate, and over a longer time horizon there's a better chance of your investment appreciating in value.
The last thing anyone would want is to invest money they'll need in the coming months, only to have the market move against them. Talk to your financial adviser, and carefully plan any future purchases.