My information says: 1975 and later v6 engine require that the valve guides be cut down on camshafts with lifts above .420" And your signature says that you have a camshaft in your v6 that is bigger than that. If the retainer is hitting the valve guide it can cause all the rocker problems you mentioned, broken rocker shafts, bent push rods. I would recommend that you need a cam button. If any of the lifters came out of their bores and the cam did walk forward or even backward, part of a lobe or bearing surface on the camshaft could be in the way of letting the lifter fall back down into it's bore.
Oh sh it war wagon may be right