Relationship With My Father In-law

So I am in a relationship with my father in law.
For personal reasons I am living with him at the moment and we have suddenly got this crazy connection. He was comforting me one evening and we ended up kissing. Then on a separate night one thing led to another and he gave me oral. My husband knows and now we have an open marriage, he is also dating another girl whilst I date his dad. Me and the father law are thinking about a proper relationship and have already been out on dates. We have a really good emotional connection. Me and my husband still love each other and are together, and he is fine with me being with his dad as well, but I'm worried about his family's reaction to me dating my husbands father.