It's been a few months since you posted your question, so perhaps circumstances have changed.
If your husband is well enough to travel and visit his mistress, he is well enough to find another caregiver.
In all situations of emotional abuse, I counsel people from my own experience—get your own house in order. Find a counselor or psychologist to help you find your own worth and strength. Then you will be able to make a decision about the relationship from a position of power and self-worth. And you'll have a counselor on your team to help you with the emotional fallout.
Let us know how you are doing. Be well.