I have this add-on but the directions aren't clear. Do I need to keep this running while I fly or does it change the textures semi-permanently until I want to change them again?

Open REX and do a back up of your original textures. Then choose the textures you want to apply to your sim. At the end close REX. Open your sim and see if you like it. Let's say you dont like your clouds, close your sim open REX and change the clouds settings to your liking close REX and see if you like what you changed. If so you don't need to open REX again. If you need to restore your original textures go to REX and restore, you will have the default textures. Hope this helps.

Cheers

Do I need to choose anything in particular in P3D V4? It has different REX weather choices.

I forgot to mention that in REX Settings you have to choose your simulator, so you have to point to your p3d v4 root folder and if you are using any weather engine do the same. I am using AS16.
Regarding p3d itself i don't change anything in his settings. I think the weather rex options are just a simple preset of clouds and weather textures but im not sure.

Btw these are the first steps to do after install REX.