solved Some Migrated Blogs Missing Pictures

I think this may have been caused by being the first guinea pig for migration? It appears that pictures were stored in a folder and referenced, but some no longer exist (or didn't transfer).

I think at one point, the way blogs were handled was changed, which may have led to this? At any rate, I seem to be missing pictures all through my migrated blogs. What's weird is the folder obviously exists and the path is good, because it works for some.

For example, this picture is missing http://indiegospel.net/NingMedia/blogs/1-1000/768-CoverCropped.jpg but this one from the same folder is not http://indiegospel.net/NingMedia/blogs/1-1000/734-May10ConcertJPG.jpg

Any ideas?



--

Ken Rich
indiegospel.net

updated by @ken-rich: 02/23/15 07:49:19PM

Does the missing image exist in the original Ning archive? It will be in the same sub-folders that it is in /NingMedia - /blogs/1-1000/768-CoverCropped.jpg


--
Paul Asher - JR Developer and System Import Specialist

I don't know what caused the problem, but I think I have a solution.

I still have a copy of that folder on my hard drive. It's just called Sept. 15th/blogs/1-1000. After doing some comparisons, files are missing in the server version, but of the ones that are there, the labeling matches.

That tells me it's the right folder and not a mismatch to the JSON.

Therefore, I am going to re-upload the missing media files to the folder and see what happens.


--

Ken Rich
indiegospel.net

updated by @ken-rich: 01/09/15 04:59:58AM

Yeah - If the files were missing, for whatever reason, from the server version of the archive they will not have been copied to the NingMedia folder. Uploading the missing ones to their correct folders in NingMedia should do it.


--
Paul Asher - JR Developer and System Import Specialist

Yup - that worked... but I don't know now what else in other folders might have gone missing. I guess all I can do is check around?

Also, it would be nice to know what went wrong, so it doesn't happen again to someone else...


--

Ken Rich
indiegospel.net

updated by @ken-rich: 01/09/15 05:25:12AM

You say they were missing from the uploaded archive so something must have happened whilst they were being uploaded.
It might be worth checking Filezilla (or whatever is being used) for 'Failed Transfers' when doing this.


--
Paul Asher - JR Developer and System Import Specialist

I was using Filezilla and I do usually watch for failed transfers. I can't be 100% sure, but I don't think it's likely I had a bad upload and didn't realize it. Certainly possible but not likely.

I seem to recall, we had to re-import some stuff after the Jamroom team figured out that groups and forums were not being handled correctly.

Didn't I misunderstand you and upload blogs again by mistake when you were looking for pages? I think I recall something along those lines.

At any rate, my Sept 15th Ning archive (on my hard drive) had the missing blog files, so there only seems to be two possibilities.

1) Either the last import was bad and some of the file transfers failed.
2) The import was successful, but some of the files went missing for an unknown reason.


--

Ken Rich
indiegospel.net

Its likely that we'll never get to the bottom of this!!
I know that the import process doesn't delete any files in the archive and the Manage Media process only copies them.
At least you can re-upload the missing ones. To be sure, you could upload the entire blog section to the NingMedia folder but then you would have unused files taking up server disk space.


--
Paul Asher - JR Developer and System Import Specialist

I checked the 1001-2000 blog folder and it had less files than the archive on my hard drive. So did discussions, so I am re-uploading. Haven't checked the others yet.

The question I have now, is are the Ning Media Folders on the hard drive supposed to have the same files as the Sept. 15th media folders we uploaded.

If so, there are files missing from multiple categories. Filezilla lists the number of files making it an easy check.


--

Ken Rich
indiegospel.net

The 'Manage Media' part of the import searches all imported Ning item texts and comments for referenced images, and if found, copies that image from the archive to the NingMedia folder (and changes the reference in the text accordingly). It does this so that all that is needed is ultimately copied from the archive so the archive can be deleted. (NOTE that this is not the way that the most recent versions of jrNingImport work).
Its possible that some Ning archived items are not imported due to invalid data, archive errors etc. so if these items had referenced images would be copied. For instance, if a Ning member wasn't imported because his archived email address was invalid, say, and that member had lots of blogs with lots of comments that referenced images, all of those images would be missing from the NingMedia folder.


--
Paul Asher - JR Developer and System Import Specialist

OK - if there is a new process, would it be beneficial to do a re-import of my archives utilizing the new method? Or would that possibly mess things up at this late date?


There is something I need to understand about the number of files I should be seeing in the folders. Should these folders be more or less identical to the original Ning archive folder on my hard drive?

In other words, if I find a NingMedia folder with practically nothing in it, and the originating folder had hundreds of files, is the discrepancy in file numbers indicating missing files?

I'm not sure how much I can read into a folder comparison.

Does the import tool also search for referenced file attachments like PDF's, because besides images, these were common in blogs and discussions?


--

Ken Rich
indiegospel.net

updated by @ken-rich: 01/10/15 03:33:29PM

This seems to be related to the same issue, I'm missing some attachments to my forum discussions. What's different is the URL information. I don't understand by looking at it which folder to check on the server ot what the actual file is called.

forum/attachment/download?id=1706554%3AUploadedFile%3A101729

For instance, is that supposed to be in the Ning_Archive or NingMedia folders? What is the actual file number being referenced, I don't see long numbers like that.


--

Ken Rich
indiegospel.net

The missing files (that I've confirmed for sure) are PDF's that were referenced by discussions.

Case in point, this link is from a discussion and is supposed to reference a PDF.

When I open the link in the discussions's editor, I see this http://indiegospel.org/forum/attachment/download?id=1706554%3AUploadedFile%3A100941 which is my old domain, perhaps the long numbers are Ning file references from the old system?

On my JR5 the file is migrated to this location - http://indiegospel.net/Ning_Archive/discussions/28-ChurchFathersonTiming.pdf or here http://indiegospel.net/NingMedia/discussions/28-ChurchFathersonTiming.pdf so why is it not linking properly?

I think it may have to do with being the first Ning migrant and the evolution in the file storage and retrieval process? Perhaps Paul is the only one to know?


--

Ken Rich
indiegospel.net

updated by @ken-rich: 01/15/15 05:26:21AM

The "http://indiegospel.org/forum/attachment/download?id=1706554%3AUploadedFile%3A100941" that is in the < a > tag in the Ning discussion text is a link into the Ning scripts to download the file and I guess there is no way to access from outside the original domain (the same way that you can't just dive in and download anything from a JR site). There is also no evidence of the file in your archive, so there isn't much the import tool can do in situations like this.
There are some referenced PDFs in your archive, eg. http://indiegospel.net/indiegospel/forum/christian-teachings/62/pauls-haircut-a-lesson-in-freedom that have been referenced in the discussion text the same way as images etc., so download ok.
I think this might be another Ning archiving failing.


--
Paul Asher - JR Developer and System Import Specialist

Hi Paul,

Given what you are saying, I may have an explanation, or at least a clue.

In the original Ning discussions there was a place for file attachments at the bottom. So one could upload a PDF and reference it, as was the case with "Paul's Haircut". I guess Ning treated it as part of the discussion, and included it in the archive.

There was a second way to include a file, and that was an editor tool (tinyMCE box) that uploaded the file and you referenced it as a link. It was "hidden" except for the link, the first technique produced a visible and separate file attachment view at the bottom of the post.

Perhaps what we are seeing, is that the "hidden" PDF's referenced by the second technique, were as you say "inaccessible".

The only problem I have with that explanation, is the missing files were found in the Ning_Archive discussion folder. That tells me they were accessed, and in fact downloaded then uploaded with the archive.

So, that leaves the question as to why the "a" links were not pointing to them, but instead looking like this:
http://indiegospel.org/forum/attachment/download?id=1706554%3AUploadedFile%3A100941#sthash.1YW3vAIF.dpuf

I can manually relink the files on my own posts, but if you bring over a big site, this may become an issue. I don't understand it at this point, because the files transferred but the new "a" links to them were not good.

It seems to me, whatever is forming the new "a" links must be the culprit but this stuff is way out of my league. I can only tell you what I am seeing.


--

Ken Rich
indiegospel.net

Can you give me an example from your site where there is a URL like "http://indiegospel.org/forum/attachment/download?id=1706554%3AUploadedFile%3A100941" and the archived file it relates to?


--
Paul Asher - JR Developer and System Import Specialist

Hi Paul,

I fixed that link and all the ones in that discussion by manually changing the "A" links to point to the Ning_Archive discussion folder, where the PDF's had apparently transferred to.

Let me see if I can find another example, there is another related discussion I have to check when I get a chance.


--

Ken Rich
indiegospel.net

Hi Paul,

Most of my stuff seems good now, but just found this one missing. http://indiegospel.ning.com/forum/attachment/download?id=1706554%3AUploadedFi58%3A10281

That was supposed to be referencing a PDF just like the "Paul's Haircut" scenario above, where it was uploaded by the Ning file attachment tool (not the Ning tinyMCE) and then referenced.

In this case it didn't work, and there is no PDF in the Ning_Archive folder.

There is in the NingMedia folder, but most likely because I reuploaded that complete archive several days ago to fix the missing pictures problem.

I'm going to change the "a" link to match the PDF in the Ning_Media folder, but above was what I found initially, and it generated a missing page error,

There doesn't seem to be a discernible pattern after all, so you are probably right about random Ning Archive corruptions.

Thanks...


--

Ken Rich
indiegospel.net