Should I Or Not

Had a threesome with an older colleague of mine and his wife.
It was a pretty wild night and we both fucked his semi drunk wife over and over again, in her mouth, pussy and arsehole. With her taking us both several times during the sex.
However on more than one occasion he fed me his cock to suck and I did it, more to keep things highly sexed than me actually wanting to at first.
Yet on maybe the third or fourth time I found myself deep throating him and he said I sucked cock better than his wife.
He wanted to fuck me too, but I said no at first.

When I was about to leave around one in the morning with his wife fast asleep, he asked me if I'd like to shower.
We ended up in the shower together where he had me suck him again. And once he was hard he had me face the wall as he slowly slid his cock up my arse. It hurt a little bit, but not as much as I thought it might given he's pretty big.
He fucked me in that one position for about ten minutes before he came up my backside.
Telling me as my taxi arrived that he'd love to fuck me again, I said it'd be a one off as I'm not gay.

This was eight weeks ago. Since then we've all had sex again a few times, but I didn't let him fuck me even though he wanted to in front of his wife.
She couldn't care less if he fucks young men as all she's interested in is being fucked hard herself and the drink really.
But he's definitely wanting to fuck me even without his wife present. He's even tried to seduce me in work, but it's not happened.

The thing is, I actually enjoyed him fucking me in the shower and didn't want to admit to myself, as I don't see myself as gay.
They've invited me to stay over this coming weekend with his wife asking me if I'd let her watch her husband fuck me. I think it's him telling her to ask me, yet I do enjoy fucking her.
I honestly don't know what to do.