That would certainly work. So does a small screwdriver. I use plastic to avoid damage to the trim, but if you're careful there's no reason metal wouldn't work just as well.
My concern with the metal tool was pinching the wire(s). I pulled a usb C cable along with the GPS and camera cable. My pillar trim was super stiff, my plastic trim popper was bending. Could be the roll cage tubes are causing some of the tension too.
I have a 6 port USB power supply that powers my dash bar items. I used one of the empty slots to power the mirror. The USB power supply has a on/off button to shut all the ports off. I just have to remember to use it.
The rubber duck is sitting on the USB hub. I am new to the duck thing. It has been a few years since I have had a Jeep. Odd but funny.