I'm sorry you have to go through this. I have not experienced this personally, however, I know a bit about fibroids. Fibroids most commonly grow in the uterus. It isn't unheard of to have one grow on/near an ovary, but it is less common. It can be hard to be sure what is growing on your ovary is definitely a fibroid. How was it diagnosed? Sometimes ovarian cysts or masses can be similar to a fibroid, so you definitely don't want to leave it there in case it really isn't a fibroid, if that makes sense. You would want a biopsy and it would be ideal to remove the tumor.
I think it might be wise to get a second opinion. Actually, it is always wise to get a second opinion when surgery is in the treatment plan. Always. But particularly in this case, because perhaps a surgeon who has done this particular surgery many times before can find a way to save that ovary. Different surgeons have different ways of doing things. I would find a surgeon who does this all the time. Go to the website of a teaching hospital or medical school near you, and browse through the gynecology dept. Find someone who is particularly interested in situations such as yours, and make an appointment for a second opinion. Or even ask your current doc for a referral to someone they would go to in your circumstance. A doctor should never be upset with you for wanting a second opinion before surgery.
The worst thing that could happen is the second surgeon agrees with yours. But it may open the door to other possibilities for you. As for recovery, typically an endoscopic removal of an ovary is not a long recovery at all. It may even be an outpatient procedure, depending on whether they do it through your abdomen or a little endoscopy slit or even vaginally. You will be sore afterward, you will be limited from climbing a lot of stairs or heavy lifting for a few weeks. Recovery is longer if it is an open abdominal incision.
I wish you luck. Keep us posted.