I would see the sweat of the palms as more destructive to the rubber gloves than the castor oil- sweat is often acidic in scale!
But try a google search, you'll find gold nuggets.
Google 'castor oil and latex'
and you'll see laughable, but informative results like:
Why does Simply Slick use Castor Oil? Because it's safe to use with ... https://www.pinterest.com/pin/406168460118952631/
Because it's safe to use with Latex, Polyisoprene, and Polyurethane condoms! Check out our Ingredient Spotlight: Castor Oil blog article for more info.
and part of the text of that webpage is:
Castor Oil, a vegetable oil, is an attractive alternative to petroleum-derived lubricants due to its great lubricity and biodegradeability. Castor Oil has better low temperature viscosity properties than most vegetable oils, making it useful as a lubricant in today’s jet, diesel, and even race car applications. Castor Oil is a preferred lubricant because it does not does not dissolve natural rubber (Latex) seals.  "
Your best test of course is to buy the gloves that will fit your purpose best, squirt some castor oil into the fingers of the gloves, and let them lay in the dark somewhere warm (wrapped in a towel near a radiator would be ideal) for 3-5 days. After the test period, you can pull them out and test if they rip easily, by stretching the materials.
Of course, you'll want powder-free gloves!
In case you were wondering, rubber is vulcanized latex.
Of course, a person more familiar with polyurethanes, latex and the materials gloves can be made of, would be very valuable for answering this question. Nurses especially!