A condom can't provide complete protection. If some exposed skin (e.g., on the scrotum) comes in contact with a partner's infected skin, the virus could still be transmitted. Likewise, if the scrotum, legs, or torso of the male wearing the condom have some infected tissue, their contact with a partner can infect him/her. You can also get herpes on the lips, gums, or roof of the mouth by performing oral sex on a partner who has a genital infection. There are rare cases where people get it on the hands or knees after contact with infected tissue.
So, a condom is the best you can do for vaginal, oral, or anal contact with the penis, but it's not perfect. If you provide stimulation of an infected partner with your fingers or hands, it's a good idea to wash your hands with soap immediately after. Don't worry too much about towels or clothes-- once they've dried, any HSV on them will die.