Best way to transcode 4 movies simultaneously

Hi,

Sorry to bother you all.

I'm in the process of transcoding hundreds of 1080p movies to h.265 using Handbrake CLI, using a watercooled Linux server with 24 cores of Xeon with AVX2.

I basically want to have multiple x265 transcodes running to make best use of the hardware, would I simply run multiple GNU screens with handbrake, or use maybe vSphere with multiple VM's running handbrake?

Thanks for taking the time to read my post, looking forward to your input.

Richard

Just run multiple handbrakes.

As a bump, when you figure out how to run multiple terminals, can you let us know what settings you are using and how many FPS per handbrake you are getting?

Or at least let me know, i'm very interested. The HW 265 encoders are absolute trash(especially the AMD one, it is garbage.)

I would really be interested in knowing the specs of the encoding performance myself as well, the x265 encoder is still going through development but the actual encoding performance in terms of FPS is still quite low even on high end hardware and from what I remember the performance sorta tapers off after about 6-8 cores so doing multiple ones with even some processor affinity per instance of HandBrake to really spread it around well and thorough might make it work even better.

But I'd really love to get some hard data on the kind of FPS encoding speeds you're achieving once you figure out your working encoding methodology, absolutely.

And you're not bothering us, that's what this forum is all about: [H]ardware enthusiasts that really give a shit about such things as how many cores you've got, FPS encoding speeds, etc. If you've ever seen the movie "xXx" (the first one, not the super terribad shit sequels) there's the line where Xander Cage is about to be yanked out the back of a military cargo plane by parachute and he looks at the camera and says "I live for this shit!"

Yeah, that's about how most of us feel about computers and related technologies so, no worries, you'll fit right in soon enough (even though you joined what, 6 years ago, come on).

Check https://www.videohelp.com/ and let us know what you managed to do.

They have e-x-t-e-n-s-i-v-e info and may provide you with a solution that you didnt think of.

Also try, https://www.pugetsystems.com/solutions/content_creation/index.php

They make workstation type machines specifically for the video editing crowd. Go there not to buy a machine, but to check their forum. You will find answers to a lot of specific questions like yours and see what comes out of it. I first thought about running vm's. Maybe you will find a way to make it work efficiently.

Please keep us updated. Have never run a xeon, good to know the abilities especially since threadripper and new i9 came out. I have always been trying to figure out the exact encoding difference between a cpu/igp vs max threads. Quick sync and comparables vs max threads. Whats better if you want to encode a lot of videos? Yeah, im interested.

On a different note, I have seen firsthand the differences between .265 and .264 video. Played it on VLC and was amazed. In my case less than half the size and better quality than .264 mp4.

Its crazy good, but I didnt encode the file.

hope you find the answer that works best for you.

Khahhblaab said:

Check https://www.videohelp.com/ and let us know what you managed to do.

They have e-x-t-e-n-s-i-v-e info and may provide you with a solution that you didnt think of.

Also try, https://www.pugetsystems.com/solutions/content_creation/index.php

They make workstation type machines specifically for the video editing crowd. Go there not to buy a machine, but to check their forum. You will find answers to a lot of specific questions like yours and see what comes out of it. I first thought about running vm's. Maybe you will find a way to make it work efficiently.

Please keep us updated. Have never run a xeon, good to know the abilities especially since threadripper and new i9 came out. I have always been trying to figure out the exact encoding difference between a cpu/igp vs max threads. Quick sync and comparables vs max threads. Whats better if you want to encode a lot of videos? Yeah, im interested.

On a different note, I have seen firsthand the differences between .265 and .264 video. Played it on VLC and was amazed. In my case less than half the size and better quality than .264 mp4.

Its crazy good, but I didnt encode the file.

hope you find the answer that works best for you.

Click to expand...

It's getting better, especially at starved bitrates.

Unfortunately to do it properly you get about 2fps on an i7. Yeesh

My findings on 5 simultaneous Handbrake CLI encodes from blu-ray 1080p to H.265 with source framerate and audio passthrough at CRF 17 using 24 x Xeon cores with HT at 2.5GHz (AVX2), 192GB DDR4 RAM.

OS: Ubuntu 17.04, Handbrake installed with apt using John Stebbins PPA, all packages up to date with apt.

[John Wick Chapter 2] 30GB down to 8GB took 20760 secs for 175680 frames so 8.5fps

[Don't Breathe] 18GB down to 4GB took 12660 seconds for 128160 frames so 10fps

[Get Out] 24GB down to 5GB took 15360 seconds for 124800 frames so 8fps

[The Autopsy of Jane Doe] 18GB down to 4GB took 15180 seconds for 123840 frames so 8fps

[T2 Trainspotting] 23GB down to 8GB took 19380 seconds for 168480 frames so 8.5fps

Combined 43fps.

The watercooling was keeping the processors at 40degC so the CPU's are probably not throttling.

Very happy with the quality. Handbrake JSON renamed to TXT attached.

Thank you sir... 43fps combined is crazy! Much better than trying on a dual/quad.

That's nice, it really is, good job on the setup and thanks for the detailed info, very nice indeed. Wish I had that kind of crunching power, got this old Dell Latitude E6420 with a Sandy Bridge i7 dual core and I can only imagine being able to crunch with that kind of power.

But then again, there are Xeon-powered laptops nowadays from Dell in the Precision line of mobile workstations - no they don't come with 24 freakin' core CPUs but even so, it would be lights years beyond what I'm capable of. Now if I could only win a lottery or some relative I don't even know I have would die and drop me a big load of cash I'd be good to go.

Later that same day edit...

Figured I'd do a simple test encode for shits and giggles. TRON:Legacy Blu-ray source, chapter 8 (the Lightcycle battle with CLU and Sam and the others), 6 minutes 11 seconds of material, using the JSON preset that was posted by the OP on this Latitude E6420 with the Core i7-2640m dual core i7 (2 cores/4threads), 8GB DDR3 1333, etc.

Estimated time to completion at the 5:30 second mark (meaning it's been encoding for 5 mins 30 second so far) is another 39 mins, Avg FPS 3.5.

Well it works, but it's got my damned CPU at ~95C on both cores give or take a bit, of course it's 108F outside my apartment here in downtown Las Vegas at this moment and the AC is set for 80F inside (trust me, that's the best it can do).

Slowly but surely, of course, it gets the job done but for a full movie (TRON:Legacy is 2 hours 5 mins 9 seconds end to end) I'm not gonna do that.

The Sandy Bridge CPU has QuickSync but it's just shit, really. It's fast as fuck when it works right, can encode even 1080p content at like 300 FPS (seriously) but the resulting quality no matter what you do is just fucking horrid compared to what x264 and x265 can do but you pay for the higher visual quality by increasing the encoding time drastically.

I really wish truly exceptional and very fast hardware encoding with outstanding video quality was possible these days, not even Kaby Lake's QuickSync can get anywhere near what x264/x265 are still capable of. Can't wait to see how slow AVC1 stuff will be when it's finalized into a proper spec soon.

No, seriously, this is the last edit I swear...

Final results: encoded 8920 frames in 2330.02s (3.83 fps), 5265.33 kb/s, Avg QP:19.97
Final file size: 463MB (which is pretty fucking crazy large for just 6 minutes and 11 seconds of footage for 1080p using h.265 which is supposed to provide smaller files at the same visual quality as h.264, it really seriously is, geez.

ACK... I made a booboo too, chapter 8 is "The Real Kevin Flynn" where Sam meets his Father after so long, boo, now going to do the proper chapter 7 encode with more high motion action scenes and see what happens, might bump the CR up a bit, 17 is pretty low (great visual quality of course but a massive hit to encoding performance).

Anyway, something to do on a Saturday afternoon I suppose.

I've used the AMD hardware encoder and it's garbage. The baby lake QSV I don't think is that bad. I've used it before on dumb crap like comedies and TV episodes that I usually play for background noise when I'm cooking dinner or something. Needless to say, I'm so far away from the living room TV you can't tell the difference.

Thanks everyone, what amazes me is the compression, a file size decrease average of 74%

You might consider the following (no meme jokes, seriously):

Have you done any test encodes and changed the parameters you're using - I've seen encodes by some people that are using h.265 (encoded with the x265 encoder, sometimes using HandBrake but more often using the standalone x265.exe encoder executable) and they are incredibly small in terms of the overall file size but they still look absolutely fantastic. I mean I'm not an audio or video mastering engineer, and I don't have the very best playback hardware in the industry of course but even so, those encodes look stunning to me. II saw recently of Avatar from Blu-ray source material - I think the original m2ts file on the Blu-ray itself (the extended version of the movie) was around 36GB or something, and the resulting h.265 encoded result (keeping the resolution/frame rate of the source material and passthrough of the audio content) was around 3.6GB total and I personally could not tell the difference when played on a color calibrated 1920x1080 IPS display panel for perfect 1:1 pixel playback.

Just something to consider, as even with big ass hard drives these days and knowing the change in the encoding parameters will speed up the resulting encode while not seriously affecting the visual quality (but resulting in a pretty substantial cut in the final total file size) might be something to try out. Test a few chapter encodes, not full movies, and see if the visual quality is up to snuff for you, might get all the encodes done faster and it'll still look just as good as the original source material, you never know till you try.

Going from 30GB to 8GB is no easy feat, sure, but if you can get the same or very very similar visual quality faster and with a result that's 2-4GB in size, wham, everything improves in terms of efficiency and you get almost twice the movies in the same amount of storage space.

I remember H.265 encoding an old B&W film "Death by Hanging", the encode was larger than the Bluray, H.265 doesn't seem to like excessive grain, I remember there is a grain tuning option on x265 but I didn't try it.

H.265 seems to love clean digital camera video and 3D!

rthorntn said:

My findings on 5 simultaneous Handbrake CLI encodes from blu-ray 1080p to H.265 with source framerate and audio passthrough at CRF 17 using 24 x Xeon cores with HT at 2.5GHz (AVX2), 192GB DDR4 RAM).

OS: Ubuntu 17.04, Handbrake installed with apt using John Stebbins PPA, all packages up to date with apt.

[John Wick Chapter 2] 30GB down to 8GB took 20760 secs for 175680 frames so 8.5fps

[Don't Breathe] 18GB down to 4GB took 12660 seconds for 128160 frames so 10fps

[Get Out] 24GB down to 5GB took 15360 seconds for 124800 frames so 8fps

[The Autopsy of Jane Doe] 18GB down to 4GB took 15180 seconds for 123840 frames so 8fps

[T2 Trainspotting] 23GB down to 8GB took 19380 seconds for 168480 frames so 8.5fps

Combined 43fps.

The watercooling was keeping the processors at 40degC so it's probably not throttling.

Very happy with the quality. Handbrake JSON renamed to TXT attached.

Click to expand...

Guess the question is answered, concerning max threads vs igp.

What do you feel your fps would be with one encode? Will the additionlal cores working on one video provide any additional speed? Since handbrake is able to use quicksync and VNenc, what are the chances that your xeon is one of the ones with an igp on its die?

My reads are seriously pointing to a possibility that encoding assistance through igp's arent worth the "quality". Good for game recording, some on the fly recording, but not if you want a archival type video and you are quality conscious. I will lose interest in virtually any film - the one exception are old scifi films - if the quality is garbage. A 2K movie that looks like VHS fits this description. Sucks, because one reason for the interest is to transcode old video to digital. Guess it wont be a quick as it could but a better quality experience.

Max threads it is, unless some opensource ware figures out a way to get the speed of quicksync/nvenc and the quality of x264/x265. Maybe something on videolans list of future things to do.

rthorntn said:

Thanks everyone, what amazes me is the compression, a file size decrease average of 74%

Click to expand...

Have you tried splitting a video into 5 parts, slightly overlapping each section so you have a little slack at each video section then encoding each part seperately, and rejoining them? Should get you the 43 fps on a full video. Could be a "cheat" that would allow a faster encode by using max cores/threads on a single video.

maybe something else to try

thx again.

Khahhblaab said:

What do you feel your fps would be with one encode?

Click to expand...

Trying this in the past 8-10fps average is the max, with 1080p it only uses around 6 cores.

Khahhblaab said:

Have you tried splitting a video into 5 parts.

Click to expand...

I'm pretty sure MulticoreWare Inc have a commercial product that does this, RipBot on Windows has hacked something like this together but I don't think MulticoreWare are very happy about it.

Anyone have a way to split a mkv in to four, feed that in to handbrake-cli and then join the outputted mkv's together, I should be able to convert an hour of video in just over 30 mins?

Update: reading up on mkvmerge split functionality but it doesn't look like a foolproof way to seamlessly join and keep audio and subs in sync (to do with having to cut on key (IDR) frames).

Further update (from sneaker on videohelp): The "proper" solution would be to cut in the lossless domain. I.e. write a VapourSynth script with e.g. ffms2 source filter and trimming/slicing. Then pipe that into e.g. x265cli binary. Encode audio separately and as last step merge everything using mkvmerge.

I doubt I will bother