Possibility of husband cheating with his ex wife

My husband and I have been together for 4 years, married for 3 of those years. He has 2 children with his ex. We have custody.

She has had an on and off again boyfriend. People have made comments that my husband and the ex are trying to work things out. This mostly happens when the ex wife and her boy toy are not together.

I have confronted the both of them numerous times and everytime they deny it.

My gut feeling is telling me different. What am I to do?