There are a lot of issues in your plea and this crisis is just sort of "the crowning blow"! I will give you my opinions below. Please understand that I am a guy, a Christian pastor and tempted just as much as any man by the things of this world, especially a pretty woman.
I sense your hurt and wish that I could give you a hug and tell you that it's going to be okay. Remember when your mom or dad could do that?
I can understand your disgust and rejection of his behavior, but you must understand that you are reacting from a position of hurt. If all had been swell in your marriage, I imagine that forgiving him would be a lot easier and quicker. Since you are already hurt, you want to punish him for doing things that hurt you.
I don't think that you will find an answer in leaving him. It might solve this problem but introduce so many more that you haven't even dreamed of! On the other had, staying in the present circumstance is not attractive and probably won't work for long.
So you must find a way through this maze. It is not easy to do and I doubt that many couples can manage it without counseling. [I'm not referring to the informal counseling with friends in the church, but at least a few sessions with a trained counselor.]
Think back to the time when you were first in love, first married, first being intimate. There was a thrill there that you haven't felt in a long while. What will it take to get it back? Because that's where I think you need to go.
I'm a pilot and learned early on that some clouds had hail, ice and bad things in them. When I flew into one of them, I was trained to turn around and get out, to go back to the sunny skies and puffy clouds that I liked! In a way, that's what you need to do. Turn around, go back to the good times, the times when you laughed and made love and didn't spend a lot of time thinking about it. It was a natural thing and gave you both pleasure. That pleasure made physical changes in your hormones, your moods, your muscle tone, your digestion, etc. It carried over into the rest of your life, like riding a bicycle down a hill really fast, so you don't have to pedal very hard going up the next hill.
His passive-aggressive behavior doesn't really make the task easier, but whether he's placid or a roaring vesuvius, you can learn to deal with that and he can learn to act differently. I just think that you need some help to pick your way through a lot of problems.
None of us are perfect and tossing us into a bed together doesn't really improve us, so we have to learn to be together. When we find ourselves in a mess, we have to turn around together and search for the good things that we once shared. They are still there, but are usually covered with a kid's toys, some laundry and a lot of small hurts that sort of ground us down.
Look for the light, seek some help and don't forget that prayer can open us up to answers that we probably can't find any other way.