So, if you take a JK above a 2.5 or maybe 3 inch lift you need to change out the control arms and a number of other items in order for the geometry of the steering and suspension components to be correct. I'm a bit surprised nobody mentioned what was going on before me, but that is the short of it.
It's a common problem when people put on big lifts without doing the proper research first.
When the geometry is off, what you're describing is exactly what happens. Best option is to get rid of the spacers and see how it drives and then go from there. As I said you should be fine with a 2.5 inch lift. The only problem you may have is that your shocks may be too long now.