I believe that your theory is correct and in addition, insoluble fiber can scrape the wall of the intestines irritating them further.
I don’t know if you take any probiotics and enzymes. You could try eating small amounts of yoghurt or kefir. You could also take supplements, acidophilus, bifidus, etc.
Your IBS could be caused, at least partially, by the lack of micro-organism in your digestive system. I have been allergic to many different foods but with the intake of various micro-organisms and enzymes I have overcome some of these allergies. I still don’t eat wheat, beef, pork and nuts. I only drink 1 cup of coffee each day, no alcohol, no sodas, no commercial prepared drinks. Like you I do not gain weight by eating a lot. I find exercise very important to my digestive system and my health.
I cover some of these subjects in my blog titled “My challenge”, link below.