Bearings should be replaced when they stay rough, grind, wobble, show significant rust or no longer spin consistently after sensible cleaning and lubrication.
Listen and feel
A little noise is not always a failure, but a rough feeling under load, visible damage or repeated sticking is a good reason to stop relying on that bearing.
Replace the set
Replacing all eight bearings together keeps the setup predictable. Keep any genuinely good spares labelled and dry.