Queue depth status

DannyA
@dannya
9 years ago
584 posts

Is there any way to get a notification or get the status of the queue depth?

I would like to initiate some process if the queue is getting backed up e.g. spin up additional servers, or re-direct tasks using a different process. Is there a way to monitor queue depth?
updated by @dannya: 10/14/16 10:45:37AM

@michael
9 years ago
7,788 posts

Might be able to use the jrTrace module add an event.
https://www.jamroom.net/the-jamroom-network/networkmarket/92/event-tracer

might need a custom event though if one doesnt exist.

@brian
9 years ago
10,149 posts

You can use the jrCore_get_active_queue_info() function - i.e.

$_result = jrCore_get_active_queue_info();

That will give you detailed information about your queues, including depth.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

great. Thanks!

DannyA
@dannya
9 years ago
584 posts

Using this ($_result = jrCore_get_active_queue_info()) function to get queue info we are getting below mentioned error:

"event listener function does not exist: jrCloudQueueClient_get_queue_info_listener"

Does this function work with the cloud queue?

@brian
9 years ago
10,149 posts

That's an error you really should never see - it means the CloudQueueClient registered a listener function, but when the core tried to call that function, the function does not exist. If you have customized anything, double check you've not changes the queue functions.

Are you trying to get the queue counts from the cloud queue server while ON the cloud queue server or from another server in the cluster?

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

No. From the main Jamroom server. The queue server is a standalone server (based on a previous recommendation from you).

DannyA
@dannya
9 years ago
584 posts

jrCloudQueueClient_get_queue_info_listener function does not exist in jrCloudQueueClient module

@brian
9 years ago
10,149 posts

OK let me check this out and let you know - I'm not sure at this time so will need to look at the code.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

@brian
9 years ago
10,149 posts

You can do this from the master to the Queue Server with the following code:

$url = "http://yourqueueserver.com/cloudqueueserver/info";
$_rs = jrCloudCore_send_encrypted_request($url, array());

$_rs will contain an array of queue info just like jrCore_get_active_queue_info() would produce.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

Using this, we are getting the following error
Array
(
[code] => 400
[text] => Bad Request
[note] => invalid payload
)

@brian
9 years ago
10,149 posts

Looks like it needs a dummy value - i.e.:

jrCloudCore_send_encrypted_request($url, array('temp' => 1));

Let me know if that works.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

Error now 404 instead of 400

Array
(
[code] => 404
[text] => Not Found
)

@brian
9 years ago
10,149 posts

That means the URL is incorrect - double check the URL.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

@sbhadauria
9 years ago
129 posts

I have double checked the URL and I am using the same URL as before when I got 400 error.

@brian
9 years ago
10,149 posts

Make sure the Cloud Core module is installed on the server you are making the call FROM, and that it is configured with your cluster name and password.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

Yes. Cloud core is configured. The queue server is otherwise working and doling out tasks to conversion servers.

DannyA
@dannya
9 years ago
584 posts

To be clear cloud queue server is working, but we are still getting the error for the get active queue info function. Also still hoping for answer for other thread about queue prioritization.

@brian
9 years ago
10,149 posts

@paul answered your other post here:

https://www.jamroom.net/the-jamroom-network/forum/new_posts/41667/prioritizing-queue-jobs

and I will check out this one in more depth on Monday - I see 404 here as well and am not sure why.

Thanks!

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

Ok. Thanks. Just saw the other one too. I was hoping Pauls answer was not definitive. May have to do something custom unless you have any plans to do something. It would be great to set a PRIORITY parameter on a job and have the client execute the highest priority job first.
updated by @dannya: 07/09/16 04:28:29PM

@brian
9 years ago
10,149 posts

DannyA:
Ok. Thanks. Just saw the other one too. I was hoping Pauls answer was not definitive. May have to do something custom unless you have any plans to do something. It would be great to set a PRIORITY parameter on a job and have the client execute the highest priority job first.

Yeah I could see this being useful as well - mainly to ensure email is sent first (or attempted to). I'll check it out and see if it is something that could be supported easily.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

Email is was not a big of a deal for me, but now that you mention it I think it became much more relevent.

My use case was more for encoding priority. I encode to multiple formats; however only one is needed before the file is available to the player. I wanted to make sure that one gets processed right away so the user is not waiting.

I didn't think email was an issue, because I wasn't using the conversion server to send emails. But now that you mention it, I didn't realize emails were getting held up by conversions. I thought the email would get picked up by the email server regardless of the conversion jobs in the queue. I hope that is the case and maybe you can confirm.

If that's still true, one approach to prioritization is to dedicate servers to certain types of queue jobs; so that they don't get held up by conversions. However, I'd still like to assign a priority to the conversions.

@brian
9 years ago
10,149 posts

None of this is really an issue on a busy system. Queues are never held up in any way - there just may not be enough active processes on a system that has no one hitting web pages that a queue may sit there for bit. You'll really only ever see this on a development system. On a "live" system there are enough web processes calling up workers that everything gets worked through very quickly with almost no delay.

I've got something in place for the next Core that will let you set the priority of a queue worker if you want - i.e.

jrCore_register_queue_worker('jrCore', 'send_email', 'jrCore_send_email_queue_worker', 0, 4, 3600, HIGH_PRIORITY_QUEUE);

Note the new HIGH_PRIORITY_QUEUE flag on the register worker call. There is a corresponding LOW_PRIORITY_QUEUE flag if you want the worker to run after all other queues are checked.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
updated by @brian: 07/10/16 09:09:10AM

DannyA
@dannya
9 years ago
584 posts

Sounds promising. A couple of questions:

1. Can different types of encoding jobs have different priorities? Or Is it just by job TYPE.
2. Is there a reason you went with just high/low vs a numerical value? It seems a numerical value would give you much more control.
3. When is this release expected?

FYI, the use case for the queue status is for managing queues as well. I am trying to use the queue status to autoscale a cluster of conversion servers OR send overflow to AWS elastic transcoder if the queue length gets too long. It seems like a conversion server can only process a few jobs concurrently (depending on memory and number of cpu cores), so if more than a couple of users are uploading at a time, the queue can really get backed up quickly.
updated by @dannya: 07/10/16 09:57:20AM

@brian
9 years ago
10,149 posts

DannyA:
Sounds promising. A couple of questions:
1. Can different types of encoding jobs have different priorities? Or Is it just by job TYPE.

For each job you want to be able to adjust the priority on, just make it a unique worker.

Quote:
2. Is there a reason you went with just high/low vs a numerical value? It seems a numerical value would give you much more control.

It actually is numerical - the default priority for a queue worker is 5.

HIGH_PRIORITY_QUEUE = 1 and LOW_PRIORITY_QUEUE = 9. So if you need to you can use any value between 1 and 9. "HIGH_PRIORITY_QUEUE" is just a named constant, which makes it easier to understand what it means rather than "1".

Quote:
3. When is this release expected?

Next version of the core - hopefully soon - it's in testing right now so I would anticipate sometime in the next week or so.

Let me know if that helps.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

That does. Thanks. (I don't know how you managed to get the answers out before I even hit send!)

DannyA
@dannya
9 years ago
584 posts

Hi Brian,

Just wanted to check in and see if you figured out the issue with the 404 error. It's a blocker for us now.

@brian
9 years ago
10,149 posts

I have just pushed new versions of the Queue Client and Queue Server that will help with this. The 404 response is actually correct - it means "no queue entries found". What I've done with the latest update though is have it return a 200 response with "queues are empty" as the data response - this way you know the request was actually successful, even if there are no queue entries.

I also added a new helper function in the Queue Client module:

$_queues = jrCloudQueueClient_get_active_queue_info();

That will return an array of active queue entries on the Queue Server.

Let me know if that works for you.

Thanks!

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

DannyA
@dannya
9 years ago
584 posts

Awesome. Will try it now. Does this include the update you previously mentioned with the queue priority function? Or is that in a future core release?
updated by @dannya: 07/12/16 09:28:44AM

@brian
9 years ago
10,149 posts

DannyA:
Awesome. Will try it now. Does this include the update you previously mentioned with the queue priority function? Or is that in a future core release?

The actual queue functionality is part of the Core, so that will be in the next core release.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

solved Queue depth status

Tags