The whole panel would have to be taken off. What you must realize is that the wires are fed through a plastic tube that you would see once you open the door. Its possible that a wire could've been pinched.
Here is a great write up from a user on ToyotaNation.Com http://www.toyotanation.com/forum/3...313-how-upgrade-your-front-door-speakers.html