avoid putting yourself in situations where you dont have to be in. if you can. however, the decision is still yours.
consider the kids real father. what are you and them gonna do if the father comes back?
if you enter the relationship i think you're gonna face some tough issues sooner or later. i think a working relationship should not be full of stress. just my 2 cents.