Provided you ride at a controlled leisurely pace and anticipate road / pedestrian conflicts , I haven't had any issues. But i notice they get a real bad rap among some people. I myself enjoy the simpleness and fluidity of the ride.
Except in a few rare circumstances, skidding takes longer to stop then having wheels that are spinning but about to lock up. Primarily because the coefficient of static friction is higher than the coefficient of kinetic friction (i.e. you get less braking force when sliding then when rolling). So having a skidding rear tire and a braked front takes longer to stop then when you have two brake wheels. But the differences are probably negligible.
The higher the deceleration rate, the more weight is transferred from the rear to front wheel, and braking power is increased in the front and reduced in the rear. For a minimum distance straight-line stop on a good surface, the rear brake doesnt contribute anything due to weight transfer.
Think more about your knees, all the wear and tear that would normally go into your break pads now goes into your knees. Fixies havn't been in wide use long enough but I would bet that long term fixie use will correlate strongly with knee injuries. Instead of swapping out your break pads you can swap out your knee joints.