ffmpeg issue revisited (nearer a solution)

@soaringeagle
8 years ago
3,304 posts

well finally got the ffmpeg issue diagnosed
8 meg vids convert
1 gig vids do not
the diagnosis i gt from a tech is
This is happening because php-cli has post_max_size value is 8M, the ffmpeg is using php exec command inside the application code

now that i know what is causing this..how should it be fixed?

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities
updated by @soaringeagle: 09/06/17 03:05:09PM

@michael
8 years ago
7,772 posts

maybe not use php-cli (cgi) and run it as mod_ruid2 as is outlined in the guide:

Guide: "Configuring apache with mod ruid2"
https://www.jamroom.net/brian/documentation/guides/1202/configuring-apache-with-mod-ruid2

@brian
8 years ago
10,148 posts

post_max_size has nothing to do with running a system() call - post_max_size is used when a form is posted to enforce a max upload size.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

@soaringeagle
8 years ago
3,304 posts

i run litespeed not apache (90% of images wouldnt load with apache)

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@soaringeagle
8 years ago
3,304 posts

can mod_ruid be used with ls

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@brian
8 years ago
10,148 posts

soaringeagle:
can mod_ruid be used with ls

Probably not.

Your ffmpeg issues are unique to your server, and unfortunately we've provided all the feedback we can - figuring out the root cause of why ffmpeg does not function properly on your server is something you and your hosting provider and going to have to figure out.

I know what Lightspeed is, but have never used it. The fact you're having image issues with Apache makes me believe your server was never setup properly and is still misconfigured. Since we don't do server support here in the forum it's something you'll want to work with your hosting provider on. If they cannot help you, you should consider alternate hosting and find a hosting provider that really knows what they are doing.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

@soaringeagle
8 years ago
3,304 posts

i am my hosting company
and the message i posted above was provided by a server management expert after they diagnosed the ffmpg
i had many other experts try as well these were the 1st ones that were able to pinpoint an issue
hell i had ffmpeg supoort themselves try to figure it out

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@brian
8 years ago
10,148 posts

I apologize - I'm not trying to make you frustrated, but if the post above about post_max_size is indicative of the 'expert' opinion, then you've got the wrong experts.

the ffmpeg bundled with Jamroom should run on your server without issue.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

@soaringeagle
8 years ago
3,304 posts

it does but only for small files for large files it runs and runs then simply stops (i think with sigkill)
it seems like a timeout or something
n 8 meg vid converts fast..1 gig not at all

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@brian
8 years ago
10,148 posts

soaringeagle:
it does but only for small files for large files it runs and runs then simply stops (i think with sigkill)
it seems like a timeout or something
n 8 meg vid converts fast..1 gig not at all

a 1 gig video is going to use about 4-5 gigs of RAM, and could take many hours to convert - like 2-3 hours. Make sure you've allowed up to say 6gigs of RAM to be used per process in your php.ini (memory_limit).

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

@soaringeagle
8 years ago
3,304 posts

i alowed 24 gigs for a max upload size of 6.4 or 8.4 somewhere around here
it pins the 1 core at 100% for awhile..guessing 1/2 an hour then simply stops the queue items stay in queue (it used to disaper but of queue after a day or 2 now they simply stay in and i guess retry the conversion from time to time

i did have to increase max request body size as well just so it would upload

what timeouts or other factors could be affecting it
i had assummed it was a conflicting dependency from when ffmpeg was installed on the system
that is till i got the 8 meg vid to convert
i don't think even a 100 meg vid will convert but i'lll try that

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@brian
8 years ago
10,148 posts

You can also try increasing the max_execution_time:

http://us2.php.net/manual/en/info.configuration.php#ini.max-execution-time

Jamroom already does this internally, but you could try setting that really high (it's in seconds, so set it to like 86400 for 24 hours) and see if that helps.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

@soaringeagle
8 years ago
3,304 posts

i think i already have that high but will try thnx
ok i set that higher the 8 m vid worked
although in edge i only see the player bar at bottom i see the whole vid in ff
also in edge clicking download vid the file wont play even though i downloaded it prior to the conversion finishing and it played then
i hope the larger vids convert now
but can you see if this works in edge (oh you only use linux dont you)
https://www.dreadlockssite.com/soaring-eagle/uploaded_video/368/testing-small

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@soaringeagle
8 years ago
3,304 posts

it seems the video (uploaded) module might not work in edge browser at all
i get an exception thrown but not caught error
(and the 1.16 gig vid failed upload, it was doing that b4 cause of max_responce_body_size)

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@michael
8 years ago
7,772 posts

That video:
https://www.dreadlockssite.com/soaring-eagle/uploaded_video/368/testing-small

plays for me on firefox on osx, but is only 4 seconds long. Don't have edge browser sorry. It does seem like the video failed to fully convert though, so that could be why its not playing on edge.

@soaringeagle
8 years ago
3,304 posts

nah it was only 4 sec long just an 8 meg file cause i knew an 8 meg file would convert.
i'm trying a 300 meg now and uploading a 1.1g
large uploads were failing but i think it was the response body size (i know for a fact that caused a 513, but think i had to increase both static and dynamic)
the uploads would fail at the very end ..
its encouraging that at least small files are converting
i'll test that link on my laptop tomorow ..on here all i get isthe strip at the bottom with the play button

i'm closer to figuring out the conversion thing..if i can figure out what size file it starts to fail at i think it will give me a clue

btw 2 of the vids that were sitting in queue for awhile have nearly 2000 retries

question
is there a connection made right when the conversion finishes.. it seems to run a good while then just stops i keep wondering if its thee last step in the process
i do believe the logs had a connection reset by peer right within a few seconds of the procesS ENDING (before or after i'm not sure)

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@soaringeagle
8 years ago
3,304 posts

ok hate to bother you with this

but the large file upload fails in around this block of errors

2017-06-05 22:57:43	Error	71.225.6.155	503	POST /core/upload_file/?orderid=0&upload_name=video_file&field_name=pm_video_file&token=3be0ba0147aa212c07da397774a58166&upload_token=8360f0a81f70272cf0f678856bb0627e&extensions=3g2%2C3gp%2Cavi%2Cf4v%2Cflv%2Cm4v%2Cmkv%2Cmov%2Cmp4%2Cmpg%2Cogv%2Cwebm%2Cwmv&multiple=false&pm_video_file=DSCN3422.MOV HTTP/1.1	https://www.dreadlockssite.com/uploaded_video/create	Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0	646 	Apache SSL access
2017-06-05 23:57:44	Error			.485 [INFO] [71.225.6.155:56612:HTTP2-79] connection to [/tmp/lshttpd/APVH_dreadlockssite_Suphp70.sock] on request #2, confirmed, 1, associated process: 120965, running: 1, error: Connection reset by peer!				Apache error
2017-06-05 23:57:44	Error			.486 [NOTICE] [71.225.6.155:56612:HTTP2-79] oops! 503 Service Unavailable				Apache error
2017-06-05 23:57:44	Error			.486 [NOTICE] [71.225.6.155:56612:HTTP2-79] Content len: 1139015680, Request line: 'POST /core/upload_file/?orderid=0&upload_name=video_file&field_name=pm_video_file&token=3be0ba0147aa212c07da397774a58166&upload_token=8360f0a81f70272cf0f678856bb0627e&extensions=3g2%2C3gp%2Cavi%2Cf4v%2Cflv%2Cm4v%2Cmkv%2Cmov%2Cmp4%2Cmpg%2Cogv%2Cwebm%2Cwmv&multiple=false&pm_video_file=DSCN3422.MOV HTTP/1.1'				Apache error
2017-06-05 23:57:44	Error			.486 [INFO] [71.225.6.155:56612:HTTP2-79] Cookie len: 467, _ga=GA1.2.62b44f77-b288-4fcc-bed8-919f3631c221; __unam=1142a75-1592e3184eb-101847da-455; _gid=GA1.2.465482449.1496297152; dash_reload=on; jrcore_pager_rows=100; PHPSESSID=5f12b10ujies8onb4ta199qkh3; sessfa5a8cdcbcc8=5lu0vv836c1gok7kcsc8n3jv05; imgsizg="256"; fcsid=gik1caho2r7mhudgm71r6ni2f0; jr_location_url=https://www.dreadlockssite.com/core/queue_entry_reset/id=5578279; _bcf_tracking_expired=1; autofa5a8cdcbcc8=83-8793325bf740d6c8463dd6f01b8002da; cc_loggedin=1				Apache error
2017-06-05 23:57:44	Error			.486 [NOTICE] [71.225.6.155:56612:HTTP2-79] Redirect: #1, URL: /modules/jrCore/router.php				Apache error
2017-06-05 23:57:44	Error			.486 [INFO] [71.225.6.155:56612:HTTP2-79] abort request..., code: 4				Apache error

correct me if i am wrong..but its the 503 on the socket that triggers the failure? i believe i saw that same message right before each failure

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@michael
8 years ago
7,772 posts

I'll expand my dev server to allow for 1GB uploads to see if I can reproduce this.

In the mean time:
https://www.lifewire.com/503-service-unavailable-explained-2622940

might have something useful in it.

@michael
8 years ago
7,772 posts

been an hour since I uploaded the 1GB file to my server, conversion is still going. If your server cuts processes off that take a while, this takes a while.

Recommend setting it up using the guide.

Guide: "High performance Jamroom Server Setup"
https://www.jamroom.net/brian/documentation/guides/1188/high-performance-jamroom-on-digitalocean

@soaringeagle
8 years ago
3,304 posts

its not getting to the conversion the upload fails just before it finishes

it goes to 99% then says upload failed
that guids how i originaly set it up using ubuntu
but certain admin pages..dashboard, system check etc resulted in blank pages
we ended up switching to centos and plesk and litespeed (and mariadb)
also i had to use your guide then tweak it setting some of the db settings higher

my cpu power took a huge hit when i bought my own server
but the db for dreadlockssite alones a few gigs

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@michael
8 years ago
7,772 posts

hard to know the cause. another guess is, is the /tmp directory limited perhaps. Im not really a server guy.

Yet another guess is is the file able to be copied from the /tmp location to its final destination.

@soaringeagle
8 years ago
3,304 posts

tmp i believes unrestricted (not a set volume size)
smaller files indeed copy over 300 meg files do (but do not convert)
i'll be out in the wilderness for the next month
so will need to fix it when i get back
i'll post the fix when i find it

--
soaringeagle
head dreadhead at dreadlocks site
glider pilot student and member/volunteer coordinator with freedoms wings international soaring for people with disabilities

@brian
8 years ago
10,148 posts

I've done some fairly in depth testing to day on really large video files (2G+) and have encountered some issues when running on a slower system (with a hard disk instead of SSD). Basically when JR goes to copy the file from the tmp upload cache out to the media directory, it could take longer than 30 seconds and time out on PHP's side.

I didn't see it on a 1G video file, but on a 2.8G video file it would fail every time. I'm not sure if this is related to your issue, but I have fixed this in the next Core (6.1.0b4) and Video module (2.0.0b2) releases which will be out soon.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

ffmpeg issue revisited (nearer a solution)

Tags