So is this an S vs LE problem?
I have a 14 LE and don’t have this problem.
Are all the seals in tact? Are they clean? Is the mirror tight?
Also, I suggest this to everyone, clean the rubber seals regularly. You can use Mothers VLR, Mequires, Armorall, etc.
The idea is to keep the rubber pliable.