Flax oil has been touted for its Omega-3 fatty acids. The Omega-3s do have several anti-cancer properties, the most important one being that they are anti-angiogenic. The rub is that Flax oil does not contain Omega-3 fatty acids, instead it contains Alpha-Linolenic Fatty Acid which has the potential in the host animal to be converted to Omega-3s. Radioactive tracer studies seem to indicate that at least in humans anywhere from 0.2% to 15% of Alpha-LNA is actually converted to the longer chain 3-Omega fatty acids in the host. Per the standard Flax oil of 230 mg of Alpha-LNA per gm of oil, that comes out to 4.6 mg – 34.5 mg 3-Omega fatty acids per 1 gm of Flax Oil. The standard fish oil gel cap has in it 300 mg of 3-Omega fatty acids per gm, of which 120 mg is the more potent form DHA.
The other big problem with flax oil is the presence of Linoleic Fatty Acid which is a precursor to 6-Omega Fatty Acids. The Omega-6 fatty acids have been shown to promote cancer and angiogenesis. That's why flax oil could be considered dangerous for dogs with cancer. The conversion rate for the Linoleic Fatty Acid was not given. The ratio of Alpha-LNA to Linoleic Fatty Acid is about 2:1.
The last disadvantage for flax oil is that some dogs become nauseated on it. If you want the benefit of Omega-3s stick with fish oil (e.g. fish oil, salmon oil, cod liver oil). If given in excess fish oil might cause some loose stools.
As for cottage cheese although it has some carbohydrate (i.e. bad for cancer) it's a fairly good source of protein and fat. But there is nothing magic about it. If your dog is not eating due to the cancer, then cottage cheese is a good try to stimulate its appetite. But from a nutrtion point of view vs cancer, any high protein/high fat food is good, not just cottage cheese.
If your dog has been diagnosed with Osteosarcoma, then outside of amputation of the limb, a new treatment approach is with drugs that interfer with angiogenesis (blood vessel growth). You might want to check out the possibility of getting into a trial at the Animal Cancer Institute.