Not to scare you unnecessarily, but herpes is not part of standard STD testing--so an "all clear" doesn't rule out herpes unless you've specifically been tested for it. So, unless youd are diagnosed by culture during an outbreak, or have a specific blood test that tests for antibodies to HSV-1 and HSV-2, you will not know for sure.
If you want to be sure, you should go to your doctor and get the blood test done. Explain to them your symptoms, and ask for the blood test.
A caveat: if this is a recent infection (less than 12-16 weeks), the blood test will likely come back negative because your body hasn't had time to develop antibodies. Also--if you get cold sores, the test will come back positive for HSV-1, but it's not site specific... it's possible to have HSV-1 in both locations (orally and genitally).
If you've had the bumps before, either you could have had herpes for some period of time, or it could be something else entirely--like ingrown hairs or something.