1st time i ran my sitemap crawler at a high rate..finished 800k pages in 38 hours (ning would ip ban if i went above 200k in 48 hours)
i had zero errors
after some mysql tweaks anbd a couple other cjhanges i get a whole lot of 500 errors while running the crawler at 1/4 the speed
[Wed Jan 28 11:18:40.826397 2015] [core:error] [pid 11262:tid 140498851251968] [client 66.249.64.50:57394] End of script output before headers: router.php
[Wed Jan 28 11:19:40.863357 2015] [core:error] [pid 11262:tid 140499019089664] [client 66.249.64.42:36190] End of script output before headers: router.php
[Wed Jan 28 11:19:40.897618 2015] [core:error] [pid 3408:tid 140498977130240] [client 73.188.173.92:56257] End of script output before headers: router.php, $
[Wed Jan 28 11:20:40.820875 2015] [core:error] [pid 3408:tid 140498935170816] [client 73.188.173.92:56861] End of script output before headers: router.php, $
[Wed Jan 28 11:20:40.854989 2015] [core:error] [pid 11262:tid 140498945660672] [client 73.188.173.92:56859] End of script output before headers: router.php,$
[Wed Jan 28 11:21:40.848171 2015] [core:error] [pid 3447:tid 140498956150528] [client 73.188.173.92:57538] End of script output before headers: router.php, $
[Wed Jan 28 11:21:40.914211 2015] [core:error] [pid 3408:tid 140499019089664] [client 73.188.173.92:57540] End of script output before headers: router.php, $
[Wed Jan 28 11:21:40.946438 2015] [core:error] [pid 3446:tid 140499050559232] [client 73.188.173.92:57539] End of script output before headers: router.php, $
ah damn it didnt copy the stuff after $
theres also several not a jpeg errors bot mostly these end of script before output headers errors
any ideas?
--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
updated by @soaringeagle: 03/08/15 08:14:16AM