It is good that there is some authority involved in this situation, and I hope that helps you withdraw from it even more. Continue to support your mom emotionally while keeping yourself from getting involved. And if there's a chance that your brother will be violent toward your mother when he gets out of the jail cell, she needs to tell the police that she fears for her safety. And change her locks while she's at it, if he has a key.
Hang in there, Bexiboo. You have a lot of support and understanding here.