If you and your fiance are having unprotected sex without a condom (regardless if it is 10 seconds or 10 minutes), that would be a reason to use another form of birth control. This is an important topic to discuss, and find a method that works for both of you.
You are saying he wants you to use hormonal birth control (I assume you mean the pill), but it sounds like you do not like this option. There are so many birth control options that fall into two basic categories: hormonal and barrier, and you can each discuss the pros and cons of each method to find out what will work best for you both. If you have any questions about any option, please ask and we provide as much information as you need to make an informed decision.
The most important thing about birth control: it only works if you use it correctly and consistently.
Regarding your last question: anytime you have unprotected sex, you have a risk of pregnancy, and there is no "safe" time during your cycle to have sex. The time during your cycle after your period ends is actually one of the more fertile times for most women. However, if he did not ejaculate and used a condom within 10 seconds, your chance of pregnancy would be very low in this one-time occurrence.