Not Sure What I Can Do Or How To Handle This

Over the summer my son’s college roommate visited us from out of state.
One morning my son was up early while his roommate was still sleeping so I asked him to run to the grocery store for me.
A few minutes after he left his roommate came down to the kitchen. We chatted for a few minutes and it was obvious he was flirting with me. I was confused how to react because I was actually attracted to him but he leaned over kissing me on the cheek. When I didn’t say anything he turn me kissing me on the mouth which quickly resulted in a more passionate kiss. The next thing I knew he lifted me up sitting me on the kitchen table pulled my skirt up and had sex with me.
As soon as he finished I ran to the bathroom and cleaned up. When I returned to the kitchen my son had gotten back so after that we never had time to talk and he went back home. I’m not on birth control but my period came the next morning after he came in me so I didn’t have to get a morning after pill.
Now I have an issue because he called earlier this week wanting to slip in town to see me again. When I told him we probably shouldn’t do that he threatened to tell my husband. I’m afraid if I continue to see him I’ll end up getting pregnant. My husband has had an operation that prevents us from having any more children so that would be a real problem. I’m not sure how to get out of the situation but will probably have to see him until I can figure something out.