I had an affair with a man at work (we are both married) and I've fallen in love with him. But I think it was just sex for him, so I ended it. Now I want him back, because I miss him more than you can imagine.
Since you're both married, enjoy the sex. Enjoy it immenseley, but don't end your marriages.
My wife fell in love with some dude like that, she ended our marrage, then he dumped her.
Try to stay away from him. Work on whatever problems you have that led you to have an affair in the first place. If you want to fix your marriage, great. If not, walk away and start again. You have to do what's best for you.
Then just make sure it's just for sex.