Just the fact that you felt somewhat better when you took the gluten and dairy out of your diet means that you are having some problem with one or both of these foods. The experts advise following a gf diet for at least 3 months before making decisions about whether it helps or not. Before going back to a gf diet make sure that you have been tested for Celiac Disease. Your symptoms suggest that this may be one of your problems. You will need to have a complete Celiac Panel. Look at the information about testing at Celiac.com.
I've found the gluten free Specific Carbohydrate Diet very helpful.
You may find some of the digestive health programs at the CZ Liver Flush Support Forum helpful in curing digestive problems whether you are sensitive to gluten or not.