Prize Rules are basically exactly what they sound like: vessels are taken as prizes rather than sunk. The U-Boat would surface near the vessel, and persuade the crew to follow them to a German-held port. Other aspects of the Rule were that passenger ships weren't to be sunk, and crews of merchant ships were to be allowed to man lifeboats before their vessel was destroyed. Only ships that were a posing a danger could be fired on without warning--example, warships and belligerent merchants.
In my mind, it's something like a "only fire if fired upon" Gentleman's rule, and like the quote says, it was abandoned early on in 1939. Something of a carry-over from older days.
There should be some good resources to check up on this, but that's the gist, I'd say.