That boundary line has to be a bug. There are steps you can take to sorta get rid of it, but it's annoying.
Anytime I try masking and keying (On one clip or stacked) I get that boundary of the mask.
Do you have Motion? Doing this kind of thing can be done easier inside of Motion. It stinks that there's not a 'Send to Motion' option yet, so you'll have to do it manually. **Unless you buy X Send to Motion. I haven't bought it myself because I think it's silly that that feature requires a plugin.