I made the mistake of telling my husbands friend and business partner what I did. Now he holds it over me for sex.

When my husband an I split for a few weeks, I went to a bar. I let a guy take me to his place and we had sex. It was not something I was confortable with after, and I ask my husbands friend about it.
He told me to not tell what I did knowing my husband would NEVER let me live this down, would never trust me again.

My huaband and his friend had an arguement and spilt the business up. When his ex friend came ot get his equipment, I met him. I ask if he would keep my secret? he laughed and said that would be the ultimate killer for him, would serve him right.
I begged him and he kept laughing, I reached in his pants and gave him oral in the office. He took me pants off and did me on the sofa and two more times on the desk. He said he would not tell if i would promise to give my husband a kiss after sucking his, well you know.
When papers and contracts needs to be signed, my husband makes me go to his former pertners Co. When I get there he sends the secretary out to get things and take me in his office. He has sex with me for a long time. He said he likes me sore and satisfied.
When I get back my husband ask what took so long, I say he was out of the office or made me wiat. I am worried he might check my panties some time, he will know for sure what I have done.
I want to stop this and I want no mroe to do with this guy. He ofered me to come live with him and I think he would lvoe that. to get back at my husband and to have me as his sex slave when he wants. But i want it over.
How do I do this and not piss this guy off? and now have him tell my husband. I worry my husband will one day find out.