I had a look through my bookmarks and found this great writeup. This goes into a lot of detail and talks about a lot of services which might be a bit confronting. Honestly, searching for smaller piece by piece guides for installing this service or that service and slowly building out your projects is a good way to go.
I ended up using caddy-docker-proxy because the vanilla Caddyfile method was giving me some trouble with my internal docker hostnames. This container just lets you add some labels to your other containers in your docker compose file and then works out the proxy for you, so long as you’ve made the appropriate DNS records.