Pretty much this: Vanish gives you 3 sec stealth improvement, causing dmg not to break stealth. Frost Nova should not get you out with vanish as well, I guess it's the same for HoT and PC (not sure), but it's 100% for Frost Nova (already have a video of that from Retail). However, after 3 sec and if you have dots, it should proc Subterfuge.
So you have a total of 6 sec of stealth (Vanish + Subterfuge)
I hate Subterfuge and stealth mechanic. It might sound easy, but it's complex as hell to fix and to make the devs understand.