How to avoid server 5xx error and timeout errors


Generally speaking there are two kinds of URL lists:

  1. List is made up of links from different domains. For example list of online radio stations,or list of sites that offer a similar service (dentists,florists,music instruments...)
  2. List is made of links from the same domain (usually sitemaps that list all pages of  a site)

In Fast URL checker there is option to use up to 200 connections at the same time. When checking lists of type 1 it makes sense to use even 200 connections and gain in speed will be obvious and there will be no trouble at all,the limiting factors are only your current ISP connection that may vary due numerous reasons and speed to a particular server.

When checking type 2 lists something else is in play which often causes timeout errors or server internal errors -5xx Error.

What is going on here? The thing is that you Windows computer has limit of maximum parallel connections to the same server,and the server also has limit of parallel connections that come from the same IP (computer). Both of these are meant to take load off the remote server so that it can handle as many different users at the time as possible.

So in the result when number of parallel http requests to the same server (domain) exceeds the maximum (both Windows and Servers limit) only allowed numbers of connections are processed  and other either hang until timeout error occurs or immediatelly return an 500 server internal error.

So if you get a lots of timeouts and 5xx Errors the most likely cause is that you are trying to use too many parallel connections. Solution is simple just lower the number of active connections in settings menu. Choosing 5 or 10 connections will probably be good choice but depending on the  server and your Windows version you can try higher settings and see if it works smoothly. Further more number of connections a server can handle at the same time also depends on server's current load (how many different users are connected at the moment) so the speed may vary during the day or even from minute to minute.

Web Analytics