I use to use ROF in 3d way back, and it worked fine with 4 gigs of ram. As soon as about 2 or 3 updates ago, I started getting the out of memory message you are getting. (You have to turn post effects off to use 3D). I really didnt care, since the 3D gave me headaches and the 3D effect didnt look all that great anyway with ROF. Rise of flight is not made to work with 3D and it was very hard to aim with it too. Things at a distance looked good but stuff up close like the gun sights are completly out of focus. I think something changed in ROF a few patches ago that stops the 3d from working now.
I tried it on my friends PC with 6 gigs or ram and he gets out of memory also when the 3D is enabled. I hate to say it, but the 12gigs of ram probably wont make a difference, but since ram is cheap now, why not.
If it does work, let me know.
Intel Core i7-3770K
GB (4 x 4GB) 240-Pin DDR3 SDRAM DDR3 2000 (PC3 16000)
ASUS GeForce GTX 660 Ti