Remastering "Star Trek: Deep Space Nine" with Machine Learning

Megalith

24-bit/48kHz
Staff member
Joined
Aug 20, 2006
Messages
13,003
As significant portions of the show were shot on video (and not film, which can be easily scanned to higher resolutions), Star Trek: Deep Space Nine may never get a legitimate HD remaster, but that hasn’t stopped one fan from experimenting with other technologies to see what it could look like: Stefan Rumen (“CaptRobau”), who has been upscaling Final Fantasy VII using AI Gigapixel, has applied that same neural-networking technique to bump DVD footage of DS9 to 1080p quality. While the results are arguably mild, Rumen hopes CBS and other studios take note of how machine learning can be used for improving video quality.

Since I do not own DS9, I cannot just do what I want with it. While I would love to release full episodes, this is just not legally possible. These videos serve more as a proof of concept for CBS to look into machine learning and neural networks to help remaster DS9 and move it a bit closer to the HD era. Imagine what a real team could do, with more powerful equipment, custom trained neural networks (perhaps training the network on TNG vs. TNG Remastered images) and access to the original SD files instead of a DVDRip like me.
 

KD5ZXG

Gawd
Joined
Mar 24, 2017
Messages
689
More interested if AI could restore garage band audio recordings from 25 years ago at the sea of mud.
 

nutzo

Supreme [H]ardness
Joined
Feb 15, 2004
Messages
7,380
I'm hope someday soon there is a way this can be done real time to improve home movies I recorded early last decade.

I'd like to see what it could do with old 8mm family films I have from 50+ years ago.
Faded, low resolution, 18 frames per second.... DVD quality would be a huge improvement.
 

TexasTea

n00b
Joined
Jun 5, 2015
Messages
12
Seems like only a minor improvement but I like the idea. There are a lot of old "classics" that would be a lot more watchable if they were remastered.
 

Tak Ne

[H]ard|Gawd
Joined
Jan 28, 2008
Messages
1,233
There was only about 3 seconds of the clip where there was any noticable difference and even that was miniscule. It was filmed on 35mm so I'm sure they have a better source than the DVD.
 

GotNoRice

[H]F Junkie
Joined
Jul 11, 2001
Messages
10,455
It was filmed on 35mm so I'm sure they have a better source than the DVD.

Unfortunately they don't. Here is a fantastic article regarding the situation for anyone curious about if/when we will ever see an HD version of DS9 or Voyager. Unfortunately the outlook is not good.

http://www.treknews.net/2017/02/02/why-ds9-voyager-not-on-blu-ray-hd/
In late 1986 and early 1987, during the development of Star Trek: The Next Generation, it was decided early on the only way to produce the series on time and on budget, with all of the VFX demands Trek required, would be to shoot on 35mm film, then finish on videotape. Even the motion control model photography, with all of the individual passes required, would be shot on 35mm and then, in an unprecedented move, actually composited on videotape using multiple, daisy-chained VTRs to minimize the reduction of resolution even further.

Unfortunately, this meant, unlike TOS and The Animated Series, there would be no 35mm finished negative of TNG… and the series would only ever exist on videotape at NTSC resolution. The same would hold true of DS9 and Voyager.

TrekNews.net: Would remastering DS9 and Voyager be more difficult than the TOS and TNG projects were? If so, could you explain?

Robert Meyer Burnett: They absolutely would. Unlike TNG, which shot both all of their live-action and all of their model photography on 35mm film, which made scanning the original elements possible, both DS9 and Voyager made extensive use of CGI for their visual effects, especially in the later seasons. Those visual effects were rendered in standard NTSC resolution, with a maximum of 525 scan lines of resolution per second, split between two interlaced video fields of 262.5 scan lines running at 60 fields per second. So, the original resolution remains far, far below what audiences used to today’s HD, and now UHD resolutions, are accustomed to. These VFX could be upscaled 5x, but they’d have no detail. The Starship Defiant would look like a fuzzy, grey blob.

If the VFX assets originally created for the shows could be acquired, which is a HUGE if, they could be reworked and re-rerendered in 2K resolution for Blu-ray, but even then, VFX artists would have to go in and add all kinds of upgrades to the original shots to make the ships, planets, weapons fire and explosions. look like they fit in with the gorgeous live-action photography. This would entail a number of artists working many, many long hours at considerable expense.

During the latter seasons of the TNG restoration, Mojo, one of the original, Emmy-Winning VFX artists on Voyager, who, at the time, was still in possession of many of the original DS9 and Voyager VFX assets, did a re-rendering VFX test on footage from “The Sacrifice of Angels.” The test really looked spectacular, and proved it could be done. But again, it would still take very talented VFX artists working long hours to accomplish the number of shots required for the episodes at great cost.

However, since then, I’ve heard many of these assets have been lost, either through drive failure, or simply the dumping of all the original data.

The only alternative would be to re-create all of the CG VFX shots from scratch, much the same way CBS Digital re-created TOS’ visual effects. But with the number of elements needed during DS9’s Dominion War arc, with sometimes hundreds of starships in combat, this could cost in the millions, if not tens of millions of dollars, depending on who was doing the VFX.
 

BlackManx

n00b
Joined
Dec 1, 2016
Messages
48
Been looking for a reason to re-watch DS9. I won't hold my breath, but I will remain hopeful that the finances work in terms of remastering the series and syndication on a network such as SYFY.
 

trparky

Gawd
Joined
Jul 23, 2009
Messages
971
Id rather cbs spend 21 million per episode to restore ds9 to hd.. than to film another episode of std.

the sad part is... i cant 'see' any difference in the demo footage
I can't really tell the difference either.
 

TexasTea

n00b
Joined
Jun 5, 2015
Messages
12
Unfortunately they don't. Here is a fantastic article regarding the situation for anyone curious about if/when we will ever see an HD version of DS9 or Voyager. Unfortunately the outlook is not good.

Am I missing something? I thought the idea was to use AI to interpolate and refine lower quality images on the fly. If true, and the methodology improves, wouldn't that mean it wouldn't matter rather they have film to scan?
 

SamuraiInBlack

Supreme [H]ardness
Joined
Oct 10, 2003
Messages
5,744
Some stuff looks better but some actually looks worse to me. I dunno if it's Youtube compression or what, but to me it seemed like the faces of people seemed less "real'. I mean I get they're wearing makeup and mask/prosthetic materials, but some of them didn't seem to be as detailed.
 

Snowdog

[H]F Junkie
Joined
Apr 22, 2006
Messages
11,267
It looks a little better to me. I doubt this technique will give something good enough to satisfy most Blu Ray buyers, but it could be a little upgrade in quality for showing it on streaming services.
 

BloodyIron

2[H]4U
Joined
Jul 11, 2005
Messages
3,439
As a long time ds9 fan I can't tell the difference in this video. The shrinking lowers the fidelity a lot!
 

Derangel

Fully [H]
Joined
Jan 31, 2008
Messages
19,749
Id rather cbs spend 21 million per episode to restore ds9 to hd.. than to film another episode of std.

the sad part is... i cant 'see' any difference in the demo footage

And I'm totally sure you'd be willing to rush out and pay $200+ per season to make it worth the cost, right?
 

chameleoneel

Supreme [H]ardness
Joined
Aug 15, 2005
Messages
4,892
the difference was easily noticeable on my shit quality 720p laptop screen. Color saturation, balance, and contrast are all way better. And clarity is noticeably improved. The clarity improvement is not DRASTIC. It doesn't look like footage filmed in HD. But, it is better and does not look over baked.
 

daglesj

Supreme [H]ardness
Joined
May 7, 2005
Messages
5,437
Oh god the days of watching TNG in fuzzy blurry old NTSC brownovision on UK PAL setups.
 

Burticus

Supreme [H]ardness
Joined
Nov 7, 2005
Messages
4,543
Mild indeed. Looks like a tiny amount of noise cleanup and that's it.
 

beemanit

n00b
Joined
Apr 23, 2016
Messages
33
Throw HD on anything, it makes good marketing even to resell an old series.

Watch this "HD Cup".... See what I did. A cup is not just a cup now it's HD. I bought an HD workbench off Amazon 2 months ago. Granted in that case it stood for Heavy Duty. Still HD.

Honestly I still prefer SD over HD.
 

termite

Supreme [H]ardness
Joined
Aug 27, 2004
Messages
5,642
I am currently re-watching DS9 on Netfix. It is still watchable in SD so do not let the fact it is not HD keep you from watching it.

The majority of the show takes place on the station, or ship (Defiant, shuttle etc) interior so their are not too many scenes where the show's age is glaringly evident.

It was a bit jarring going right from the remastered ST:NG though.
 

M76

[H]F Junkie
Joined
Jun 12, 2012
Messages
12,293
I'm hope someday soon there is a way this can be done real time to improve home movies I recorded early last decade.
It doesn't have to be real time. As long as it can be processed at least @1 fps on reasonable hardware I'm game.
 

M76

[H]F Junkie
Joined
Jun 12, 2012
Messages
12,293
Throw HD on anything, it makes good marketing even to resell an old series.

Watch this "HD Cup".... See what I did. A cup is not just a cup now it's HD. I bought an HD workbench off Amazon 2 months ago. Granted in that case it stood for Heavy Duty. Still HD.

Honestly I still prefer SD over HD.
Like there is no actual difference between SD video and HD video. If you're going to be a luddite at least be an honest one.
 

TexasTea

n00b
Joined
Jun 5, 2015
Messages
12
Throw HD on anything, it makes good marketing even to resell an old series.

Watch this "HD Cup".... See what I did. A cup is not just a cup now it's HD. I bought an HD workbench off Amazon 2 months ago. Granted in that case it stood for Heavy Duty. Still HD.

Honestly I still prefer SD over HD.

Yea, like turbo was or RGB now.
 

AlphaAtlas

[H]ard|Gawd
Staff member
Joined
Mar 3, 2018
Messages
1,713
It doesn't have to be real time. As long as it can be processed at least @1 fps on reasonable hardware I'm game.

Stuff like this can run at a couple of FPS on my 980 TI/4670k. With the right algorithm, and a big RTX card, you can probably upscale a 480p DVD at 24 FPS.

BTW, this guy seemingly did it with the relatively hacky method of converting the video to images, batch processing them, and re-encoding the finished images into a video, but there are other ways to do it.
 

M76

[H]F Junkie
Joined
Jun 12, 2012
Messages
12,293
Stuff like this can run at a couple of FPS on my 980 TI/4670k. With the right algorithm, and a big RTX card, you can probably upscale a 480p DVD at 24 FPS.

BTW, this guy seemingly did it with the relatively hacky method of converting the video to images, batch processing them, and re-encoding the finished images into a video, but there are other ways to do it.
That's actually a terrible way to do it. As it doesn't use temporal information. Adjacent frames could be used to improve the quality of up scaling. However I haven't seen any publicly available machine learning based tools yet to upscale video.
 

AlphaAtlas

[H]ard|Gawd
Staff member
Joined
Mar 3, 2018
Messages
1,713
That's actually a terrible way to do it. As it doesn't use temporal information. Adjacent frames could be used to improve the quality of up scaling. However I haven't seen any publicly available machine learning based tools yet to upscale video.

https://github.com/kice/vs_mxnet

https://github.com/WolframRhodium/muvsfunc/wiki/OpenCV-Python-for-VapourSynth

To be fair, those use neural nets designed for images too, but you can pre (or post) process them with temporal filters. Using temporal information is a mixed bag anyway, as you can get all kinds of artifacts depending on the content.


I've seen a paper or two on AI-based video upscaling filters that use temporal information, but no public code yet.
 

Saturn_V

[H]ard|Gawd
Joined
Sep 24, 2008
Messages
1,823
Tonite only at a theater near you. Probably the only chance DS9 will see a HD remaster.

 

Dr. Righteous

2[H]4U
Joined
Aug 1, 2007
Messages
3,163
Not impressed at all. There are lots of image detail "peaking" filters that can be used that would yield results like that.
 

sfsuphysics

[H]F Junkie
Joined
Jan 14, 2007
Messages
14,960
Not impressed at all. There are lots of image detail "peaking" filters that can be used that would yield results like that.
I thought the same thing, but upon thinking about it, you have a "480" vs a "1080" image, however if you view on a 1080 monitor (I am), then the 1080 squished down to less than half the width of the screen (in side by side) and the 480 fitting about half the width of the screen the resolution difference will not really be that noticeable. It's kind of like saying "4k doesn't look that much better" when you're viewing on a 1080 monitor.
 
Joined
Jun 1, 2018
Messages
691
the absolute irony of being the age to survive the star wars drought after each trilogy to only be bombarded with shitty disney wars and then learn that you are never going to get a TNG/DS9 4k or even real HD release. I took star trek for granted and got the opposite of what I didnt even know I had. someone go back and change the timeline.

and yes the prequel trilogy is unironically a full letter grade better than anything post 2015.
 

TurboGLH

Gawd
Joined
Dec 19, 2002
Messages
676
Not impressed at all. There are lots of image detail "peaking" filters that can be used that would yield results like that.

I thought the same thing, but upon thinking about it, you have a "480" vs a "1080" image, however if you view on a 1080 monitor (I am), then the 1080 squished down to less than half the width of the screen (in side by side) and the 480 fitting about half the width of the screen the resolution difference will not really be that noticeable. It's kind of like saying "4k doesn't look that much better" when you're viewing on a 1080 monitor.

There's an updated comparison video, shows a much better improvement and a a better comparison between original and upgraded. It's uploaded in 4k, and it was much easier to see the difference (on my 4k monitor at least)

 

SvenBent

2[H]4U
Joined
Sep 13, 2008
Messages
3,308
Some stuff looks better but some actually looks worse to me. I dunno if it's Youtube compression or what, but to me it seemed like the faces of people seemed less "real'. I mean I get they're wearing makeup and mask/prosthetic materials, but some of them didn't seem to be as detailed.
I jsut quickly glansed and notice lack of shadow details on face as well. they look like they got polished somehow.

and most of it was still a blurry mesh. not really any noticeable improvements
 

pendragon1

Extremely [H]
Joined
Oct 7, 2000
Messages
32,909
I jsut quickly glansed and notice lack of shadow details on face as well. they look like they got polished somehow.

and most of it was still a blurry mesh. not really any noticeable improvements
like that stupid Instagram filter. cgi looks good though.
 
Top