Conversion server details

Glad to hear you have a conversion server in the works. Would love to get more details as to the requirements, what it does, and especially, how it works.


So we are not just triggering conversion on upload, we also trigger additional conversions through other forms. We also trigger the waveform generation when a new preview file is created. Soon I will be adding the ability to do bpm detection and key detection on the original audio file as well (using some additional command line tools such as SOX and FFmpeg).

My near term goal was to offload all these tasks to a "media processing" server that would off load all these cpu intensive tasks; very similar to your conversion server. I assume i will be able to run these tools on the conversion server as well.

My question is more about how this works. The ideal scenario would allow me to monitor performance on server and spin up additional conversion servers as required by load. This can be done pretty easily with AWS. The question is how you address these servers. Ideally Jamroom would have a cue of pending media processing tasks (e.g conversion). Any number of servers could execute the task as it became available. In other words, the conversion server would pull tasks rather than jamroom push the task.

Can you let me know if this is possible? I thought maybe worker queues could be used for this but I'm not sure.

If this is not possible or not the way it currently works, can you provide any details as to how it works.

updated by @dannya: 11/10/14 10:26:33PM

So right now the conversion server as it is designed, basically works the same as the audio module:

1) when a new audio file is uploaded a "queue entry" is created

2) the conversion server CLIENT (running on the JR site) sees the new queue entry and grabs it first - it then submits a new job to the configured conversion server (configured in the client)

3) the conversion server "talks" to the JR client and requests a secure download of the original media file

4) it does the conversion work (audio or video)

5) it notifies the client JR site that the work is complete and that the client can "pickup" the converted media files

6) the client requests a secure download from the conversion server and catalogs the new files and updates the item

7) the client tells the conversion server it's "complete" and the conversion server cleans the job up.

There is a nice queue viewer on both the client and server side, and failed jobs can be resubmitted, etc.

For your customizations you're going to want to hook in on the server side and client side and do additional processing, etc. That shouldn't be hard to do as those triggers work the same as all other JR events.

I'm hoping to release it in the next couple weeks, so check it out at that time and let me know if you have any followup questions.

Hope this helps!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

I should add the process is identical for video files.


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

Ok. Just wanted to get clarification on a couple of things.

1. Is it dependent on the audio module? We have created our own media modules (3) from the audio module. Unfortunately, this has made it difficult to update every time there is a new update to the audio module. Also, this has caused a number of problems when another module is triggered by the audio module.

2. Can you clarify the question on multiple servers? I.e. can we spin up additional conversion servers and have the client on the jr site submit the job to whichever server responds first?

1) yes - it is dependent on both the audio and video modules, as it utilizes functionality found in those modules to perform the conversion work.

2) No - there's no "auto scaling" as that is beyond what 99.9% of our customers would ever need. Pick up a $100 a month 32 core server from OVH and you can convert 32 video files and 32 audio files at the same time.

Hope this helps!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

I'm not worried about autoscaling. That is a function for aws. However I am looking to run multiple conversion servers.If I get to a point where there are more that 32 people uploading at once,i need to be able to scale. (sorry, but i am building something different than your typical customers. additionally i will be developing api's that may increase volume significantly). Its also may be cheaper to spin up servers as needed than to run a 32 core server all the time.

I'm happy to do the work myself, (which was the original plan). I thought the conversion server would be a great foundation. I'll have to see if its possible to do some development on the the client to support multiple conversion servers or if we need to start from scratch.
updated by @dannya: 10/09/14 07:02:59PM

I don't have the client 100% complete yet, but you'll be able to run multiple conversion servers, but you'll need to configure the JR client to know that they are there. So you could have 10 different conversion servers if you want.

However, you'd have to create a custom bit of code to automate the addition/removal of servers from the group of servers - that's not something it's going to handle for you.

Hope this helps!


--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

Great. We'll look into it when its ready. But yes, we'll have to see how we handle addition/removal. Not sure how you "register" each server (ID, ip, hostname, etc) and how you secure the connection between the two. Ideally, if it can use any ID, and you can use the same key to accesses the server, then we can get the servers to self register themselves in the client.

Alternatively, we create a bunch in the client in advance and pre-configure the servers with the a varying identifier. Then we just turn them on and off as needed based on load.

We can figure that part out, as long as JR can use multiple servers its fine. The harder part is making sure the load is distributed to multiple conversion servers evenly. This is difficult if you push jobs to the conversion server (e.g. round robin), but fairly easy is you just cue up the job and let the first available conversion server grab it.

Looking forward to digging in.