Remove the glove box, reach up above it near where the air bag is, there is a 1 inch cube sized relay. You can here it click when you turn the blower fan from off to on. Remove it and inspect for oil, rust, corrosion etc on the terminals.
In my case, it had leaked oil all over the socket causing ac blower fan malfunction. This relay powers up the ac circuit, lights the ac dash led, runs the blower fan, and powers the compressor solenoid to engage....sounds like this is your problem since cold air cuts out yet blower fan still works. It could also be refrig switches under the hood, but since you can play with the dash switch and it resumes cold air, I would begin with the relay.
It's a pricey replacement part 50-$60 but solved my problem 100%. If you have good cold air when it does work, then you prolly don't have a refrig leak which leads me back to the relay being faulty...and the age of the car etc.
Let us know what you find...