ping a list of websites to see if they are alive?

MrGuvernment

MrGuvernment

Fully [H]
Joined
Aug 3, 2004
Messages
21,810
Hey all,

I have someone at work who has a list of 2700 websites she is going one by one on to see if they are active or not (affiliate related) and i thought, there has to be an app or something that we could plug the list of sites into to see if they are active or not to at least cut out the site that 404 or have dns error or something.

does such a thing exist... google hasn't helped me much but finding port scanner and such.
 
MrGuvernment said:
Hey all,

I have someone at work who has a list of 2700 websites she is going one by one on to see if they are active or not (affiliate related) and i thought, there has to be an app or something that we could plug the list of sites into to see if they are active or not to at least cut out the site that 404 or have dns error or something.

does such a thing exist... google hasn't helped me much but finding port scanner and such.
Click to expand...

Performance Monitoring tools it what your looking for. HP Sitescope and BAC can both do it but they are not free.

Nagios and cacti are 2 free ones.

This is some serious shit though not a simple script that just checks if the sites exist they are full blown performance monitoring tools that require a significant amount of effort to get established.

That being said your company should already have something like this in place especially if you have 2700 sites that you have to look at contact someone in IT.
 
Xenu seems to be doing the trick, basic and gives enough results to determine if a site is active or not, dont need Nagious i would think (overkill), i am not monitoring the site, just need to see if a URL is alive or dead really...

This was just something i heard them talk about and thought there is an easier way than to go one by one, we haven't ever had to do this before but ramping up for some marketing so they want to reach out to some 3rd parties and go from there, why we dont have something like this already in place
 
MrGuvernment said:
Xenu seems to be doing the trick, basic and gives enough results to determine if a site is active or not, dont need Nagious i would think (overkill), i am not monitoring the site, just need to see if a URL is alive or dead really...

This was just something i heard them talk about and thought there is an easier way than to go one by one, we haven't ever had to do this before but ramping up for some marketing so they want to reach out to some 3rd parties and go from there, why we dont have something like this already in place
Click to expand...

I see 1 time deal I don't know of a good pre-made solution but you could code something pretty quick that will open all the sites at once or do it in like batches of 50.

like if you have this list for example

www.google.com
www.Hardforum.com
www.HiLowThere.com

you can to a find and replace www. with "C:\Program Files\Internet Explorer\iexplore.exe" www.

Which will leave you with

"C:\Program Files\Internet Explorer\iexplore.exe" www.google.com
"C:\Program Files\Internet Explorer\iexplore.exe" www.Hardforum.com
"C:\Program Files\Internet Explorer\iexplore.exe" www.HiLowThere.com

Save this as a .bat file and when you click on it all 3 sites will open at once.........Slow computes will die.
 
Last edited:
use a bat file, and put the list of domains into a txt
then do something like this

Code: 
@ECHO OFF
date /t >> status.log
time /t >> status.log
for /f %%z in (list.txt)do pinghosts.bat %%z

where pinghosts.bat contains:
Code: 
@ECHO OFF
ping -n 1 %1 |findstr "Reply from" >> status.log
echo %ERRORLEVEL%
if errorlevel==1 goto notgood
if errorlevel==0 goto good
:notgood
echo %1 IS UNRESPONSIVE >> status.log
goto end
:good
echo %1 is alive >> status.log
goto end
:end
echo ------------------------------------------------------- >> status.log

these bat files will output:

Code: 
Mon 11/02/2009 
08:17 AM
Reply from ip: bytes=32 time=7ms TTL=124

SERVER1 is alive 
------------------------------------------------------- 
Reply from ip: bytes=32 time=10ms TTL=124

SERVER2 is alive 
------------------------------------------------------- 
Reply from ip: bytes=32 time=2ms TTL=124

SERVER3 is alive 
------------------------------------------------------- 
Reply from ip: bytes=32 time=2ms TTL=124

SERVER4 is alive 
------------------------------------------------------- 
Reply from ip: bytes=32 time=7ms TTL=124

SERVER5 is alive 
------------------------------------------------------- 
Reply from ip: bytes=32 time=7ms TTL=124

SERVER6 is alive 
-------------------------------------------------------

if it times out, it will say "SERVERx is UNRESPONSIVE" instead of "is alive"
for example:
Code: 
Mon 11/23/2009 
07:55 AM
SERVER1 IS UNRESPONSIVE
 
To see if an IP is actually serving pages, you will need to use some sort of monitoring tool. Nagios can do port checks and http status checks, but I don't think there is an off the shelf, up and running within 15 minutes option to automate this.
 
adi said:
To see if an IP is actually serving pages, you will need to use some sort of monitoring tool. Nagios can do port checks and http status checks, but I don't think there is an off the shelf, up and running within 15 minutes option to automate this.
Click to expand...

ah, right.

the bat files will eliminate the domains that are no longer in DNS, at least.
 
The issue you're going to have is if you need to know if the site actually has the content you're expecting: ie: is the site just a domain place holder or is it the actual site of the affiliate? They'll both resolve via DNS and receive data from http get, but the former is pretty much a dead site and probably worthless with regards to what she's trying to do.

Beyond that, writing a simple check in perl would be fairly easy. You'd create a list of hosts that receive data from a get request and diff them from the original list. The logic is easier that way so you don't have to account for the various error conditions you'll see.

If you need to know if it's valid content, well, that's pretty tough.
 
Jay_oasis said:
nagios can do this
Click to expand...
Yup, I was just coming in here to say that. And depending on how fancy you want to get, you can do more advanced tests ( not just a ping test, but maybe a connection test, or page change test ) to test the higher level stuff.

Nagios; is there anything it can't do?
 
Jay_oasis said:
Maybe I should get two other people to help me plug all 3 at the same time, it would be quicker.
Click to expand...

Possibly but then you might be stuck in a situation where it only works with all three plugged in and becomes finicky when only utilizing 1 outlet because not enough juice is being supplied.
 
Last edited:
LOL!

well i am keeping the bat file ideas for future, she got most of it done and alot of sites were just dead, i also just set up a nagios server at work for some monitoring so i now know another use for it!
 
if you use nagios, you might want to also look at centreon. It is a much nicer interface. It gives you a database back end, and will give you a web interface to create all the config files, just create what you want through that, tell it to reload and restart nagios and done. I used nagios for about a year or two before I found that and wouldn't ever go back to the old way. you can create one object, setup everything that you want, select it and tell it to copy and get a exact copy of that with all the services you want. that or create a template, then when creating the new host tell it the template type and to include all the services for that type and you are set.
 
You must log in or register to reply here.
Back
Top