Does Handicapped Accessibility Morally Entail Handicapped Exclusivity?

So I went to lunch last week with Aspazia, Kerry, and Raj in the faculty dining hall on campus and when we looked for a table, the only one we saw open was at the front. We sat down and halfway through the meal, we realized why that table was open -- it was designated as handicap accessible. (To be fair, the designation was rather subtle and easy to miss.)

This, of course, sparked conversation. It's a small school and we knew that there happens, at this time, not to be a great demand on such accommodations in the dining room, so we knew that most likely we weren't keeping anyone from a table more comfortable for them.

But in the course of the conversation, I mentioned that when I am out with the kids and they both need to go potty, I use the stalls made accessible for those with disabilities because they are much more convenient -- it would be nearly impossible to fit three people into a single stall and I didn't want to leave the door open or one of the kids outside. At the same time, I would never even consider parking in a handicap reserved parking place (and not because of the legal concern or the possible fine). I argued that there was a difference in the cases. The bathroom stall would be used for a very short time and the person, most likely, could wait. If not, he could voice the issue and I could quickly correct the situation. With the parking place, though, there would be no chance of correction and no sense of how long I would be parked.

I'm not sure I convinced anyone with this line -- even myself. Are these pragmatic issues really enough to hang the argument on? Is there a difference between parking spaces and bathroom stalls? Intuitively, it seems to me, at least, that there is. But if so, what?