The pump gear may tend to push the cam to the rear, but if it's not held in front, it has to bounce some. I have heard a couple of regular flat tappet cam motors where the stock button/spring has broken; they chattered and knocked and the one that wasn't fixed died shortly thereafter from cam failure.
With a roller cam that doesn't have lifters pushing it in as well, it's even more critical. If the cam moves more than a few thousandths back and forth, it's gonna chew those rollers up, IMO.