blog documentation FAQ followers forum marketplace services youtube

solved Ooops Backups

The Jamroom Network » Forum » Using Jamroom » Page » Ooops Backups
Dazed
Dazed
@dazed
11 years ago
1,022 posts
So after a slight server disaster, I decided I better take a look at my backups on Amazon S3. Much to my dismay, it looks like backups stopped being done automatically some time in Feb. I kicked off an update on Saturday and I do not think it completed so I just ran another. There was no backup on Sunday.

Is anyone else seeing any issues? Nothing in the logs to say there is a problem.
updated by @dazed: 06/19/14 08:12:51AM
brian
@brian
11 years ago
10,149 posts
Dazed:
So after a slight server disaster, I decided I better take a look at my backups on Amazon S3. Much to my dismay, it looks like backups stopped being done automatically some time in Feb. I kicked off an update on Saturday and I do not think it completed so I just ran another. There was no backup on Sunday.

Is anyone else seeing any issues? Nothing in the logs to say there is a problem.

Double check your Amazon S3 credentials. You should see messages such as "successfully backed up ## to S3" in your activity log - if you don't see those then it could be that the process is being killed while it is working for some reason.

Hope this helps!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Thanks Brian. What is odd is here is what I see for today. I do not see anything that looks like a success message.

When I look in my bucket, the only "newer" files are from May 3rd and 5th from when I ran manual backups. So yesterday nothing and nothing so far for today.

Any ideas?


ReadVersioningProps	Requests-Tier2		05/07/14 02:00	05/07/14 03:00	1
ReadLocation	Requests-Tier2		05/07/14 02:00	05/07/14 03:00	2
ListBucket	Requests-Tier1		05/07/14 02:00	05/07/14 03:00	4
ListAllMyBuckets	Requests-Tier1		05/07/14 02:00	05/07/14 03:00	1
ReadVersioningProps	DataTransfer-Out-Bytes		05/07/14 02:00	05/07/14 03:00	113
ReadLocation	DataTransfer-Out-Bytes		05/07/14 02:00	05/07/14 03:00	216
ListBucket	DataTransfer-Out-Bytes		05/07/14 02:00	05/07/14 03:00	144813
ListAllMyBuckets	DataTransfer-Out-Bytes		05/07/14 02:00	05/07/14 03:00	367
ListBucket	Requests-Tier1		05/07/14 03:00	05/07/14 04:00	16
ListBucket	DataTransfer-Out-Bytes		05/07/14 03:00	05/07/14 04:00	558149
ReducedRedundancyStorage	StorageObjectCount		05/07/14 09:00	05/07/14 10:00	69835
ReducedRedundancyStorage	TimedStorage-RRS-ByteHrs		05/07/14 09:00	05/07/14 10:00	6131821544448
ReadLocation	Requests-Tier2		05/07/14 19:00	05/07/14 20:00	2
ListBucket	Requests-Tier1		05/07/14 19:00	05/07/14 20:00	15
ListAllMyBuckets	Requests-Tier1		05/07/14 19:00	05/07/14 20:00	5
ReadVersioningProps	DataTransfer-Out-Bytes		05/07/14 19:00	05/07/14 20:00	113
ReadLocation	DataTransfer-Out-Bytes		05/07/14 19:00	05/07/14 20:00	216
ListBucket	DataTransfer-Out-Bytes		05/07/14 19:00	05/07/14 20:00	521991
ListAllMyBuckets	DataTransfer-Out-Bytes		05/07/14 19:00	05/07/14 20:00	1835
ReadVersioningProps	Requests-Tier2		05/07/14 19:00	05/07/14 20:00	1
brian
@brian
11 years ago
10,149 posts
Do you have automatic backups enabled in the module global config?


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
That was the first thing I looked at :)
brian
@brian
11 years ago
10,149 posts
So when you look in your activity log, and find last nights daily maintenance kick off, you don't see anything from the backup module with messages about copying to S3?


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Brian - That activity log, and there are many versions, look like the one a few posts up. I just looked in my bucket and there is nothing from yesterday so no backups were done at all.
brian
@brian
11 years ago
10,149 posts
Dazed:
Brian - That activity log, and there are many versions, look like the one a few posts up. I just looked in my bucket and there is nothing from yesterday so no backups were done at all.

What you posted above is not from Jamroom's activity log - where do you see that log at?


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
That was S3 activity log.The only thing in JR Logs is the manual backups and the update from 1.0.3 to 1.1.0.
brian
@brian
11 years ago
10,149 posts
Dazed:
That was S3 activity log.The only thing in JR Logs is the manual backups and the update from 1.0.3 to 1.1.0.

Then for whatever reason it is not being kicked off by the daily maintenance. Do you see daily maintenance log entries in your activity log?


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Yes I see those.
brian
@brian
11 years ago
10,149 posts
Dazed:
Yes I see those.

Hmm.. so you see nothing in the activity log for the Backup? I would reload the backup module just in case something is corrupted. If daily maintenance is running, but the S3 backup shows nothing in your activity log, then something is broken - that means the daily maintenance "listener" is not responding, which really should never happen.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Reloaded and will see what happens I guess.
brian
@brian
11 years ago
10,149 posts
Dazed:
Reloaded and will see what happens I guess.

OK - let me know if it still doesn't work - I will have to come on to your system to debug it, as I see it working here on jamroom.net nightly.

Thanks!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Will do. You have logins so feel free to check it out. As of right now, no backups are appearing in the activity log.
Dazed
Dazed
@dazed
11 years ago
1,022 posts
That worked Brian. Thanks so much!!
brian
@brian
11 years ago
10,149 posts
Dazed:
That worked Brian. Thanks so much!!

No that's me ;)

The problem is that the daily maintenance is never completing - if you look in your activity log you will see entries for it starting, but you never get a completion. It appears that there is a triggered event that is causing the listener to run out of memory:

[09-May-2014 00:39:28 UTC] PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 48040238 bytes) in (path)/modules/jrCore-release-5.1.40/lib/misc.php on line 165

That's coming from a serialization line to generate a unique key for the event - it makes me think there is a recursive issue here. I've got a small change in for it on your system, so let's see how it goes tonight - hopefully it fixes it.

Hope this helps!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
updated by @brian: 05/10/14 07:25:04AM
Dazed
Dazed
@dazed
11 years ago
1,022 posts
I saw that but since it did not say backup module I thought that may be attributed to large image files. That looked like the error we used to see in jr4.

Thanks for looking at it.
updated by @dazed: 05/10/14 09:31:22AM
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Hey Brian!

Just a heads up. There has been no activity since you ran it manually on the 10th.
brian
@brian
11 years ago
10,149 posts
Dazed:
Hey Brian!

Just a heads up. There has been no activity since you ran it manually on the 10th.

You can see your daily maintenance is still not finishing. Search your activity log for "daily" and you will see you only see the START message - NOT the finish log entry. So something is bombing out during daily maintenance.

This could take some time to investigate - I'll do the best I can but I have a lot going on here at the moment.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
brian
@brian
11 years ago
10,149 posts
I've got some debug entries in on your system to see where we are getting to. I'll check it out in the AM.

Hope this helps!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Much appreciated Brian.
brian
@brian
11 years ago
10,149 posts
This is working for you now - root cause was a custom module bombing during daily maintenance. That's been fixed, and your S3 backups are completing.

Hope this helps!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
Dazed
Dazed
@dazed
11 years ago
1,022 posts
Just found a few minutes to reply on the forum. I wanted to thank you guys again for the help on this issue. I also wanted to say thanks for mentioning the Activity Log Module. This one completely missed my radar and since it is free, I would suggest making it mandatory in the core download. That module will be a life saver for someone and I would have also noticed this issue some time ago had I loaded it!!