Different proxy Servers
The ProxyMesh proxy servers accept a few that can be used to modify the default behavior. The proxy server will also return certain when a request is not successful. All requests to the proxy server will return with an additional header, , which contains the IP used to complete the request. Note that to make use of the custom headers.
All HTTP clients should allow for adding custom headers and inspecting response headers. For the Chrome web browser, you can use plugins such as Header-Editor or ModHeader to manipulate headers.
A remote site can return any number of response codes, including the same ones listed below, but for different reasons. It is important to inspect the response body in order to differentiate between whether the response came from the proxy server or the remote site.402 Your account has been de-activated due to cancellation or lack of payment, or you are not authorized on the particular proxy server. If your account is active, then look at modifying your proxy server authorizations. 403 The remote site has been blacklisted and can only be accessed thru the open proxy server. 407 502 The IP requested in the X-ProxyMesh-IP request header cannot be found. This response will include X-ProxyMesh-IP as well as X-ProxyMesh-IP-Not-Found both containing the same IP as the X-ProxyMesh-IP request header. If these headers are not present in the response, then this response came from the remote site, not the proxy server. 503
There are a number of different situations that can generate a 503 error, and each situation produces a different response body. All of these situations are temporary, generally lasting a few minutes at most.no available IP addresses You have excluded all available IPs in the X-ProxyMesh-Not-IP request header. You'll need to wait until the IPs rotate, or reduce the number of excluded IPs. no proxies available There are no outgoing IPs available, due to some error in the IP rotation. This is rare, and is usually caused by an issue with the hosting provider that generally resolves within 1 hour. proxy error The proxy server cannot connect to an outgoing IP. This happens more often with the open proxy server, due to the unreliability of the open proxies. server error Something went wrong in the proxy server while trying to authenticate. This is very rare. service unavailable The proxy server cannot handle requests right now. This is rare, but can happen under high load situations. too many connections You have more than 500 open connections to the proxy server from a single IP. This puts undue load on the proxy server, and is a very inefficient use of resources. It is much better to use a shared connection or connection pool for proxy connections. too many errors Your requests have generated more than 60 response errors (with a status code of 400 or greater) over the past 30 seconds. You need to fix what you're doing to stop producing so many errors.
Any other response body likely indicates a rate limit response from the remote site. If this happens regularly, try using a different proxy server, or slowing down your crawl.