Forum Activity for @soaringeagle

weird sitemap crawl results after urlscanner update

Installation and Configuration

if you look at the affected pages the youtube descriptions have malformed urls with the last letter of the text added before the httpin the url
the crawlers handeling whats on the page, the only way ti handle it diferently is thriugh exclusions
the problem is theres deformed urls on the page thats causing the issue
and this only began after an update

understanding and debugging the core update process

Installation and Configuration

2017-01-27 11:31:49 (142 KB/s) - “100MB.zip.1” saved [104857600/104857600]

calling colocation place to run a network test between servers

understanding and debugging the core update process

Installation and Configuration

watching it low 60k a sec high slightly under 300k a sec (i restarted it)
average seems about 120-130k a sec will let you know when finished

understanding and debugging the core update process

Installation and Configuration

so far its working fine though the transfer rates lower then expected server to server do you have alot of strain on the server now or a throttle limit per request?
100-200 + k a sec isn't very fast i would expect at least few m a sec

weird sitemap crawl results after urlscanner update

Installation and Configuration

already talked to them but they did find an issue with at least the urls causing the loop and they were youtube descriptions that seem to all include the character before the url in the description
somehow the urls are not being handled corectly

understanding and debugging the core update process

Installation and Configuration

it potentially does! but need a tiny bit of clarification 9it confirms a suspicion)
you say it transfers then unzips to the proper location wheres the zipped file transfered to? (and file size) im thinking either the zip doesnt fully transfer or fails the checksum
finding the location of the zip might help

weird sitemap crawl results after urlscanner update

Installation and Configuration

i dont think it would have ever stopped... it seemed to be stuck in an endless loop

understanding and debugging the core update process

Installation and Configuration

situation: 2 sites on same server, neither would update the core both update everything else
1 site, the big site did after awhile upload some folders and files
upgrading to php 7 may have helped slightly
the other site no files are ever transfered

solution 1 download source code files upload files within core to the core release 606 folder run integrity check..and fixed (on bigsite)
site that never transfered any files i attempted variations of the same
upload core folder after renaming it to core release 606 (with proper caps etc)
fail
download core release 606 from 1 site upload to other fail
rename core 606 to 606 bak to see if a new folder would ever be created..fail
zipped core from source uploaded via file manager extracted to 606 folder
fail

so to fully understand the step by step rocess of updating and debugging the failures
please explain in detail how it works
my assumption is
1 transfer files
2 make any db alterations needed
3 update symlink

why did 1 transfer some files the other none i get a server disconnect notice sometimes after 10 min or so..inconsistetly
on the 1 site no folders or files are transfered (ah wait, is a compresed file sent then decompresed?)

why would uploading then running integrity check work only on the site that partially transfered files but not the 1 that nothing transfered on?

every other module transfers easily
so what are the 1st steps taken during update..then next and next (it appears to fail right away even though it doesnt generate any errors for a long time)

is there any error debug code i can add to any file that will show exactly whats going on during the update (verbose) and identify whats causing the core to not update

and finaly since 1 of you stated that they had that issue till upgrade to php 7 what php settings aee u using that maybe the cause


updated by @soaringeagle: 04/27/17 11:10:15PM

weird sitemap crawl results after urlscanner update

Installation and Configuration

inspyder identified an issue with youtube module very often adding the last charachter before urls in descriptions
this seems to be the cause of the issue
example if someone put ( http://www.somesite.com/) the url code is think spaces are handled ok but i think something like
visit our site!
http://whatever
the ! is included even though its on a diferent line (unverified but because there seemed to be a whole lot of them i can't imagine they were all due to no spaces b4 the http)

exclude photos from askimet spam detect

Suggestions

you guys sure do alot though!