Privileged ports can be used by processes that are running without root permissions.

I guess you mean unprivileged ports?

So if the sshd process would crash or stop for some other reason, any malicious user process could pretend to be the real ssh server without privilege escalation.

Not really, except on the very first connection because you need access to the root-owned and otherwise inaccessible SSH host key, otherwise you’ll get the message a lot of people have probably seen after they reinstalled a system (something like “SOMEONE MIGHT BE DOING SOMETHING VERY NASTY!”).