Usually 20 minutes is the recommended amount of time but there's no law that you can't work up to it.
You're right that the oil absorbs bacteria and toxins and should never be swallowed (though accidentally swallowing some has happened to all of us from time to time). Hard to believe that it was very toxic after 2-3 minutes but I guess it's possible.
Olive oil will get the job done but many of us don't use it because of its strong taste and the fact that it can stain the teeth. Keep trying oils til you find one you enjoy pulling with. My favorites are flax seed and coconut oil but a lot of people don't like the taste of flax seed oil either.
In any case congrats on your decision to start pulling. It's the best thing you can do for your mouth and one of the best for your health overall. And keep posting about your results!