Your transmission is spinning faster. Which means more friction. Which means more heat. Which means more wear. It's not a huge difference, but it IS wearing things faster.
This should help.
View attachment 124517
All wheel spacers are bad. The thicker the spacer, the worse the are. But they are always a suboptimal choice. The correct solution is wheels with the proper backspacing and offset.