There are a number of physical formats you can consider when buying silver:
If you're not buying an huge amount of silver, you should aim to maintain direct control over it at your home or other property.
If you're investing heavily, to the point where you need a safe alternative location to store it, be sure to go somewhere like Switzerland with strong property rights. It's key to make sure they can't lend or hedge using your stash.
Physical precious metals like silver are a hedge to the traditional banking system. In a crisis, if the system collapses, it may very well take your property down with it (if it can).
If there is a run on the banks, having your silver under your direct control means they can't confiscate it, block access to it, or have loaned it out in a leveraged fashion.
It's important to comply with the law when it comes to buying silver, and knowing what they are goes a long way.
Under a certain dollar amount, there's no need to declare (which may be attractive to some), but once you hit a threshold you will need to (and should) declare your holdings.
As a general rule, you should look at buying legal tender silver coins wherever possible. This will almost certainly include less overhead and premium than other formats, and be closer to the true spot price of silver.
Purchasing rare silver coins should be avoided unless you really know what you're doing. Some subjective measures of value are easy to misjudge, and may result in you getting taken advantage of.