It's allowed for them to request it, but I would decline it. They can escalate it to Airbnb but the agent would need to find evidence that the listing promised something you didn't deliver. Also, guests need to bring an issue to your attention and then give you time to resolve it.
It's helpful to know the details of how a guest refund works. Some CS agents will push you to refund a guest when it's not warranted based on their own policies. Note that they didn't follow the process to inform Airbnb and work with you to resolve the issue. By leaving w/o informing either they don't have a leg to stand on. You may need to point this out to a CS agent if they do escalate it.
https://www.airbnb.com/help/article/544/what-is-airbnb-s-guest-refund-policy-for-homes
I could understand their complaint if you advertised a "woman hosted - for female travelers only" experience, but that's not the case at all!
You have a picture of you and your BF as your host pic. Your listing describes 3 other bedrooms occupied by renters and give no guarantee those are all women. If they wanted to ensure a female escort in the home at all times they ought to have made that (ridiculous) request before booking.
I wouldn't refund a cent. When I make stupid decisions I call it my "idiot tax". They made an unfounded assumption that you would be in the home (all the time? just at night?) - they pay the idiot tax so they learn to read or ask questions next time.