The man took advantage of you. You were weak and vulnerable. You were lonely and physical needing a body to make love. It happens and there is nothing wrong with that if you are a willing participants. But I think you feel trap in this relationship and there is no solution but stay with the guy. You may want to ask yourself, do I want a physical relationship or emotional or both. I know the outcome maybe disastrous to you. Based on that you have to make the right call.

I think you should leave the relationship cuz you have no future in it.
Good luck