Hi, Paranoid, we're so glad you wrote. Let's see if we can ease your anxiety a little.
A man with HPV would have gotten it from some sort of genital contact -- most often through vaginal or anal sex, but it is possible to pass it to a partner just through skin-to-skin contact in the genital area with no intercourse. Most men won't ever have symptoms, like Anon said above, but they can pass the virus on to women. Some men do get genital warts, but only about 1% of sexually active men in the United States have them at any given time.
So if you and your partner were both virgins, and had not had any genital contact with an infected person, then no, you would not be at risk for the virus.
Here's a page about HPV in men from the Centers for Disease Control (CDC):
As Anon said, there are many kinds of the human papillomavirus. HPV itself carries hundreds of variations, according to the Mayo Clinic, and 30 or 40 of those can affect the genital area. But most HPV infections do not progress to cervical cancer. If abnormal cells develop, routine Pap tests usually can detect them at an early stage so they can be treated. Regular Pap tests are important because women with early cervical cancer generally don't have any symptoms.
Here's a very thorough overview of HPV, how it's transmitted, symptoms and treatment:
How old are you? Have you talked to your doctor about birth control, or had your first Pap smear?