I had that thought after I replied when I realized that most of the reputable search crawlers will publish the IPs/ranges they use in addition to the UA. The disreputable ones (cough Bytedance cough Xiaomi cough) will just spoof Chrome on Windows 10 and flood you with requests from AWS datacenters in Shanghai or Singapore.
That said, I may still continue looking into working with one of the actual self-hosted search engines (vs meta search) and see how well that works.
…yeah.
I had that thought after I replied when I realized that most of the reputable search crawlers will publish the IPs/ranges they use in addition to the UA. The disreputable ones (cough Bytedance cough Xiaomi cough) will just spoof Chrome on Windows 10 and flood you with requests from AWS datacenters in Shanghai or Singapore.
That said, I may still continue looking into working with one of the actual self-hosted search engines (vs meta search) and see how well that works.