No. mg/l refers to *TOTAL* amount of silver content by weight in one litre of water.
Colour does not signify particle size alone, it can also indicate density or number of particles in solution. So, just because a solution develops colour doesn't necessarily mean the particles are of an undesirable size.
Unless you got mud in the bottom of the storage vessel or a 'film' on the surface of the water - don't worry about colour up to golden.
Silver has a solubility limit, hence anything over that 20 you may find gravity will pull the larger particles out of solution and end up on the bottom of the storage vessel.
It is not possible to get 400 or whatever without the need to add something to that water to keep the silver suspended in solution - I wouldn't touch that stuff because it's not only silver you are consuming, and more than likely those particles will be larger in size, which is why another substance is added to the water preventing those particles from forming mud on the bottom of the storage vessel.