@thor @k1tteh
I abandoned my search earlier, cause I thought ffmpeg was giving me buggy audio. Turns out there was something wrong with mpv, and a restart solved it.
This will download the stream, extracts only the audio via ffmpeg, and outputs only the audio to listen to mpv. But for your use, maybe you can find something that can take the audio stream instead and direct it to your iPhone. I have no idea what that could be. I honestly don't know how I'd do it for Android either, but I wouldn't need to do things like this for Android in the first place.
The only positive in all of this, is that the output audio is AAC, so iPhone friendly in theory.
yt-dlp -f 94 "https://www.youtube.com/watch?v=jfKfPfyJRdk" -o - | ffmpeg -i pipe: -c:a copy -map a -f adts pipe: | mpv -