>if you point the camera at a person you need their consent
In public spaces (streets, squares, etc.), you may generally photograph people without consent for purely private/non-commercial purposes, as long as it is not intrusive, secretive in a way that invades privacy, or shows them in a state of helplessness, intimacy, or compromise (the latter can also trigger criminal provisions under § 201a StGB).
Casual or incidental street photography is usually fine; highly targeted or harassing photography of individuals can still violate general personality rights.
However - the key rule (Recht am eigenen Bild under §§ 22–23 Kunsturhebergesetz / KUG) mainly restricts publishing or publicly displaying recognizable images of people, not merely taking them in public.
This is a distinction many Germans do not quite comprehend - but it is also the case that it’s not very well understood in the USA, too.
Disclaimer: I am a recordist, I am responsible for a few well-received (even by the original artists themselves) bootleg recordings of famous gigs, I have a strong bias towards recording public environments and a heavy aversion to any censorship, whatsoever. Given that, however, if someone asks me not to take their picture, I first give them the option of receiving a copy for their own purposes, and if they decline, I respectfully delete.
> I have a strong bias towards recording public environments and a heavy aversion to any censorship, whatsoever.
This is a balancing of legally protected interests [Güterabwägung] between privacy and censhorship. Independently of what the law may say, many Germans are strongly biased towards privacy here. Be also aware that in Germany the strong US-American concept of "free speech" does not exist; the laws that do exist are weaker.
> Given that, however, if someone asks me not to take their picture, I first give them the option of receiving a copy for their own purposes, and if they decline, I respectfully delete.
Why not ask for permission instead of objection? (even if, as you claim, the formal written law may be on your side)
> Why not ask for permission instead of objection?
Because in most cases it was never my intention to photograph a stranger - they imposed this into the conversation by contacting me - most of the time I’m taking pictures of a lovely park, or a nice flower arrangement, or the broken beer glasses in the street after Carnival - and it is only after being interrupted that I offer the option. I certainly do not take photographs of random strangers unless it somehow captures the idyl of the scene, usually.
And in many cases actually, the conversation starts off irritable but ends quite friendly, especially if the photo is a good one and the person changes their mind and wants a copy.
It is possible to pivot from annoyance to friendliness. Especially if one understands the laws and the reasoning behind them. Ask for forgiveness, not permission.