I'm sorry to hear about your brother and thanks for asking us at EmpowHer for suggestions. I hope we are able to help you through this difficult situation.
As Coach Virginia stated, the hospital is unable to keep your brother there against his will. He would have to sign papers agreeing to leave voluntarily and against medical advise, but he can leave.
It is important for you, your family, friends, or anyone he will listen to to explain to him that he will die if he leaves the hospital prematurely.
He is lucky to still be alive.
It may help explaining to him how lucky is to be alive just from waiting after his appendix ruptured, how he should be dead already and there is a reason he hasn't died yet.
Unfortunately, if he does choose to leave, you would be unable to force him back to the hospital unless he falls unconscious. Without medical supervision, it is likely this will happen rather quickly. For this reason, I would have someone stay with him at all times because he will inevitably need that someone to call 911 when the time presents itself.
This is a very difficult position for you to be faced with, especially having little or no control over the situation. The best thing you can do is be there the only way you know how for someone you love so dearly and consult with others if you are feeling too overwhelmed, stressed out, or depressed.
I hope your brother gets the help he needs. You are a great sister, trying to be there for him the only way you know how, and seeking advice to help him through this difficult time.
Good luck and keep us updated on the situation.