solved DB Backups now always stuck in Queue

For some time now off and on, but happening almost constantly now, when i go to my dashboard's Queue Viewer I keep seeing the Daily Backup and hourly Backups seemingly 'stuck' or sleeping for long periods of time. Please see these two recent screenshots as examples.
There have been no log errors, and System Check has been all green lights. Pausing and resuming the queues doesn't seem to have any effect in getting them going again. Deleting the entries deletes them all, but when I check back again the next day they are there again and sleeping or 'working' endlessly.

When I check the versions of backups available in the DB Backup 'restore' tools for both hourly and daily backups, they seem to show the expected versions there.

My (much smaller) pennywhistle site doesn't seem to have this issue with the Daily Backup, but I often find the hourly back there in 'sleeping' mode.
However my main larger FOTMD.com site is now almost always showing the Daily Backup stuck in time in the queue. I remember this happening about 6 months ago for a few weeks, but then it seemed to fix itself, maybe after some update or another. I never mentioned it because it stopped happening.

Not sure if this is related or not, but since it has to do with my site's backups:
For the past two months my AWS S3 backup bucket (I use AWS/S3 only for my two jamroom sites) has gone from about $3 per month first to $15 and now to over $20 this month...! (yikes, I'll need to do something about that) Anyway, I'm wondering if these 'stuck' backups have anything to do with my new gigantic AWS S3 bills. They started happening around the same time period as far as I can tell.
But before I figure that out, I just want to get this DB backup to quit clogging up or going to sleep without finishing its task.

Thanks for any help!



--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 06/24/18 07:52:57AM

Oh, here are the now-typical screenshots I'm seeing all the time:


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

So this can be normal (at least your second screenshot). The DB backup queue automatically sleeps for ~15 minutes (for the daily backup) and 10 minutes for the hourly backup. This is done since the backups tend to be a bit of "heavy hitters" in that they can put a larger load on the system while they are running. So we sleep for a bit so all the other queues that kick off hourly and daily can finish first.

So "sleeping" is totally normal - just let it sleep.

In your first screenshot however, the daily backup that started on the 18th is clearly "hung" - it's been running for 20 hours - and there is now a SECOND backup that is waiting to be worked (for the 19th). So delete the first one - it's not running any more.

Let me know if that helps.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

Oh wow... I just now did a selective search in my activity log for "backup" on my two sites, and you can see that on the pennywhistle site the backups start and complete relatively quickly. But look at the timestamps for the FOTMD site backups and you can see a big problem there... here are the two comparison screenshots:


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

I've been deleting these things regularly but they keep re-happening daily... mostly on the FOTMD site. I just posted two new screenshots- showing how this keeps happening and the Daily Backup always now runs forEVER, stuck like glue. Or it starts/starts/starts and completes sometimes hours later.
I remember previously it used to be just very tidy and did its thing normally and then switched off again. It's not like my fotmd site has grown all that huge in that time.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 03/19/18 05:31:22PM

Yeah something is up then - if it's dying each night then it's hitting something and breaking. Check in your Activity Log and Error Log for any entries that might give a clue to what's up.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

Ok so on the fotmd site, I just now deleted the Daily Backup processes for both March 18 and 19 (both of which were still 'working' or pending). It's 11:30 pm here, March 19.... soon to be March 20th. My queue viewer is cleared of them and is working other things normally. The daily backup should start again soon on schedule. I just ran an integrity check. I need to go to bed now and I'll check back here in about 9 hours to see if the Daily Backup got stuck again.
Thanks.

P.S. come to think of it, I haven't seen any entries in the activity Error Log in a long time, on either site. Maybe that's a little odd there, not sure.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 03/19/18 08:32:48PM

I can see the queue entry is still in your queue, but the process is no longer running, so something is up. Please open a ticket for me with your admin login so I can check it out.

Thanks!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

Thanks for checking that Brian!
I have to go out for a few hours now, but will open a ticket as soon as I get back... thank you!

p.s. I also think this is why my monthly bill for AWS S3 backup is zooming over $20 for the past two months... previously was always about $3 and I've changed nothing...i only use it for the JR routine backups.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

see the AWS bill details here showing where the usage is happening... one 'normal' one from January, then the suddenly changed to the giant one from February...and continues on now even higher than that...


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015

That's because it is exiting before it finishes, so it never makes it to the "clean up" part where it deletes old backups, so you're just increasing in space right now.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

Just in case anyone else finds themselves in the same situation-
Brian discovered that I had set my S3 buckets enabled for 'versioning'... which prevented the older versions from being removed daily, thus just accumulating more and more content in my AWS S3 backup buckets. $$$
Obviously I didn't quite understand what 'versioning' was...I thought it meant kind of the opposite!
I feel dumb, but happily Brian found the problem (YAY!!!) and this issue I'll now mark 'solved'.


--
...just another satisfied Jamroom customer.
Migrated from Ning to Jamroom June 2015
updated by @strumelia: 03/23/18 12:43:23PM