You will have to adjust your sights for sure, but I think there are some benefits to going a bit heavier. It quiets the bow. If you have multiple pins then they get spaced out a bit more, which I feel improves sight picture (Iuse a single pin). I think there a some penetration improvements as well, but I'm just guessing.
Bows are so fast today that I just don't care about changes that impact speed. For me it's all about better arrow flight.
For example, I recently got away from blazer vanes, straight fletched. Went to low profile longer vanes with a strong right helical. It slowed things down a tiny bit, mostly noticeable beyond 40 yards. But so much more forgiving and quiter. Same was true when I added more weight to my arrow. I'd never shoot at animal 60 yards away, but my setup is plenty accurate to do it now.
It's worth experimenting. Small changes can result in big improvements. Speed is over rated. Forgiveness and accuracy are the most important things for me because when that stress gets into me right before the shot, my usual shot routine goes to hell