Forum Activity for @ken-rich

Ken Rich
@ken-rich
02/01/15 02:47:07PM
926 posts

Large Video Conversion - Time Out
Using Jamroom

I set it to high and it's still not utilizing the CPU. I put in a small MP4 video 3 minutes long, 15.4 MB, 360P from youtube.

The host sat and watched his panel and I'm registering 0-2% (see attached).

When he converts a video his goes up to over 50% for a few seconds on a small video, and it's done. I'm at over 8,000 seconds on this one so far, and still going.

Our server settings are the same, the only difference is the type of conversion script - ergo he suspects a misconfiguration in the script's (JR conversion) settings.

He said if there is something the script needs on the server, tell him where it is and he will give it, but there is nothing on his end that would prevent the CPU from working. The resources are there, it's waiting for the script to call on it, but the script isn't placing a demand.
Scree.jpg Scree.jpg - 94KB

updated by @ken-rich: 02/01/15 02:49:41PM
Ken Rich
@ken-rich
02/01/15 11:33:20AM
926 posts

Large Video Conversion - Time Out
Using Jamroom

Yes - it's at highest which may be the issue in terms of not seeing it "pegged". Plus we did an upgrade recently which would allow it to spread out more. Another upgrade from S3 to S4 storage is imminent which will probably give another speed increase.

I'll try it on high and see what happens - thanks.
Ken Rich
@ken-rich
02/01/15 11:07:20AM
926 posts

Large Video Conversion - Time Out
Using Jamroom

derrickhand300:
I didn't really understand what he did but he explained everything comes at a sacrifice somewhere else.

I work from a 32 inch monitor here and after the changes I can tell no difference in video quality.
The biggest thing I noticed is that a (Im guessing now) 15-20 mb file uploaded- when it hit the conversion phases it only lasted "seconds" and not minutes"- where it was taking several minutes before it only seems to take 15-20 seconds ( it really popped)
After looking at the videos on my monitor and thinking 70% of my site visitors will be using much smaller mobile devices I decided it was a sacrifice worth making, at least until I get the site up and get some feedback.

Wow - that's an impressive performance increase. Also, if you can't see or hear the difference - why not?
Ken Rich
@ken-rich
02/01/15 10:58:57AM
926 posts

Large Video Conversion - Time Out
Using Jamroom

brian:
However, there's still an issue here with your server - our conversions appear to run much, much faster (over 4 times faster) than on your server, so something is limiting you on your end.

Hi Brian,

On your other points, I have no problem.

However, as you saw (attached above), according to Jamroom's own performance tests - my speed is good. So in order for what you say to be true, your conversion server would have to be a "monster" built 4 times faster than the baseline for your performance tests (and beyond what most people can afford).

Also, the host checked my settings and I have no performance limits, caps, or bottlenecks of any sort. At least nothing imposed by the server. He says the conversion settings are all controlled by the script, so how can the issue be on "my end".

When converting, my maximum CPU usage is only 15.7 % of what the server is capable of giving me, and sometimes it's "idle", so why isn't the converter utilizing what's available?

To my mind, that is the real issue here. What limits the CPU usage? The host says that's all controlled by the script.
updated by @ken-rich: 02/01/15 10:59:36AM
Ken Rich
@ken-rich
02/01/15 09:03:10AM
926 posts

Large Video Conversion - Time Out
Using Jamroom

Hi Derrick,

I looked at this and compared it to the original in modules/jrVideo-release-1.2.3/plugins/flv.php. This snippet is the important part.
flv -acodec libmp3lame -vf scale=640:-2 -ar 22050 -ab 128k -vcodec libx264 -preset ultrafast -qp 0

I'm certainly no expert, but from what I can gather this is what the changes do.

-vf scale=640:-2 was added - which is probably what solved the widescreen to 4:3 problem.

-preset veryfast to -preset ultrafast - probably gave most of your speed boost. It increases speed by sacrificing compression. So your files take up more space on the server.

-ar 44,100 to -ar 22050 halves the audio sample frequency. Probably most noticeable on the "crispness" of the hi hat and such, but I agree most people wouldn't tell the difference. However, I don't see that as being the big speed boost.

-qp 0 was added. A write up I found says constant quantizer mode. Generally should not be used, since CRF gives better quality at the same bitrate. CRF defaults to 23 when not specified which is a reasonable balance between speed/compression/quality.

===================================

I'm wondering if you are not making a sacrifice in quality to increase speed, due to a separate issue. If I am understanding correctly, video conversion is extremely resource intensive, so a core being used for it should be registering high usage on a performance monitor.

However, my sever guy tells me when I am converting, my CPU usage is minimal, and that is controlled by the script. So perhaps we should be looking for commands that allow the CPU core to work harder to decrease time, rather than sacrificing quality (or compression) to decrease time.

I see there exists a command called -threads and another called -cpu which would have an effect on how the CPU is utilized, but I'm not competent enough to suggest settings.

===================================

Another method to approach this (according to some write ups), might be to simply copy over some files (when they are compatible with the container). Not sure if or how that would work in JR.
Quote:
An MP4 file normally contains H.264 video and AAC audio, both of which are compatible with the FLV container. You could simply copy over the streams:
ffmpeg -i input.mp4 -c copy output.flv
Since you're copying the streams, this will be as close to instantaneous as you're going to get and will not result in any loss of quality.

updated by @ken-rich: 02/01/15 09:09:12AM
Ken Rich
@ken-rich
02/01/15 05:30:08AM
926 posts

Large Video Conversion - Time Out
Using Jamroom

derrickhand300:
I hired someone to try and fix this- I don't completely understand what they did but it does seem they convert 3X faster now and I can upload widescreen and have them output as widescreen
Still having a problem with large videos failing though ( 400mb)
The changes he made were in the flv.php file-I will try to post it below ( he did something to do with "ultrafast" and changed (lowered the audio quality) Hope this helps

// make sure we are not being called directly
defined('APP_DIR') or exit();
function jrVideo_flv_decode($input_file,$_options,$error_file)
{
return $input_file;
}
function jrVideo_flv_encode($input_file,$_options,$error_file)
{
$ffmpeg = jrVideo_get_ffmpeg_command();

ob_start();
system("{$ffmpeg} -y -i \"{$input_file}\" -threads ". jrVideo_get_ffmpeg_thread() ." -f flv -acodec libmp3lame -vf scale=640:-2 -ar 22050 -ab 128k -vcodec libx264 -preset ultrafast -qp 0 \"{$input_file}.flv\" >/dev/null 2>{$error_file}",$ret);
ob_end_clean();
return "{$input_file}.flv";
}

Wow Derrick - thanks a lot. First of all, for confirmation that yes there does appear to be a speed issue here. Also, for supplying the fix you paid for - mighty nice of you.

I'm not sure what all that "gobble-de-goop" means, but I'm willing to bet Brian does, and can probably implement it for everyone in an update.
updated by @ken-rich: 02/01/15 05:43:57AM
Ken Rich
@ken-rich
01/31/15 11:47:17PM
926 posts

Large Video Conversion - Time Out
Using Jamroom

It's not actually 5 times faster. I did an objective test, by getting him to convert a video that took me 5 minutes 20 seconds (at highest setting). He did it in 1 minute 45 seconds. So using the same file, it's actually about 3 times faster.

I don't know if his converter is FFMPEG, he said he had these packages (below) installed. I'm just wondering if there is a way to let the JR converter run faster. The script controls the settings, and according to his observations, the server (core) is not working hard during conversion.

Lighttpd Streaming
Lighttpd web server with mod_flv_streaming installed.
Apache mod_flvx Video Streaming
Apache web server with mod_flvx installed.
Apache mod_h264 Video Streaming (HD)
Apache web server with mod_h264 installed.
RTMP Video Streaming
Adobe Flash Media Server installed with the streaming directory pointed to the script video directory.
Ken Rich
@ken-rich
01/31/15 01:44:43PM
926 posts

Large Video Conversion - Time Out
Using Jamroom

Hi Brian,

Just a quick note cause I'm late to be somewhere. I noticed today that the lowest and highest setting on the video conversion don't make much of a discernible difference, at least on a small file.

That may be the issue, yesterday when I had it converting a large video on highest setting, the host was watching the sever and it wasn't working. It wasn't "grabbing" a core and working it hard, even though at the server level it's set "full throttle". It wasn't even breaking a sweat.

If the script's settings are choking it, is there a way to "open it up". Maybe after highest, another level. Perhaps called "insane - full throttle - use at own risk."
updated by @ken-rich: 01/31/15 01:45:32PM
Ken Rich
@ken-rich
01/30/15 09:49:37PM
926 posts

Large Video Conversion - Time Out
Using Jamroom

brian:
That sound about right for a 15.4 mb file, so I don't think that is bad. One thing that is important to know is that your hosting provider is likely just converting the video file ONE time. The JR conversion process:
- converts the video to FLV (H264) for desktop and some android phones
- converts the video to an MP4 for mobile
- grabs a screenshot from the video

The host's converter does the same - FLV (H264) and MP4, plus it grabs 8 thumbnails. It just does it approximately 5 times faster (by current estimates) - on the same hardware.

Whatever configuration settings exist regarding JR's conversion process, are set by the script itself. I have it set to highest, that's all I have control over.

So at this point, I don't know what to think. Something has to account for the huge discrepancy in speed. I'm going to test his converter with a know quantity, for an exact comparison.

Also, I managed to convert an 11 minute long 98.4 MB MP4. However, it didn't leave a log entry and it took over an hour.
Ken Rich
@ken-rich
01/30/15 04:01:39PM
926 posts

Large Video Conversion - Time Out
Using Jamroom

Hi Brian,

That's useful information - thanks.

I don't know why the site loaded slow for you, unless there is a bad node between you and it, the performance test Jamroom provides says the server is fast (see attached).
Capture2.JPG.jpg Capture2.JPG.jpg - 68KB

updated by @ken-rich: 01/30/15 04:01:57PM
  41