Remastering "Star Trek: Deep Space Nine" with Machine Learning

Discussion in 'HardForum Tech News' started by Megalith, Mar 16, 2019.

  1. Megalith

    Megalith 24-bit/48kHz Staff Member

    Messages:
    13,004
    Joined:
    Aug 20, 2006
    As significant portions of the show were shot on video (and not film, which can be easily scanned to higher resolutions), Star Trek: Deep Space Nine may never get a legitimate HD remaster, but that hasn’t stopped one fan from experimenting with other technologies to see what it could look like: Stefan Rumen (“CaptRobau”), who has been upscaling Final Fantasy VII using AI Gigapixel, has applied that same neural-networking technique to bump DVD footage of DS9 to 1080p quality. While the results are arguably mild, Rumen hopes CBS and other studios take note of how machine learning can be used for improving video quality.

    Since I do not own DS9, I cannot just do what I want with it. While I would love to release full episodes, this is just not legally possible. These videos serve more as a proof of concept for CBS to look into machine learning and neural networks to help remaster DS9 and move it a bit closer to the HD era. Imagine what a real team could do, with more powerful equipment, custom trained neural networks (perhaps training the network on TNG vs. TNG Remastered images) and access to the original SD files instead of a DVDRip like me.
     
  2. TangledThornz

    TangledThornz Gawd

    Messages:
    623
    Joined:
    Jun 12, 2018
    I'm hope someday soon there is a way this can be done real time to improve home movies I recorded early last decade.
     
  3. KD5ZXG

    KD5ZXG Limp Gawd

    Messages:
    400
    Joined:
    Mar 24, 2017
    More interested if AI could restore garage band audio recordings from 25 years ago at the sea of mud.
     
  4. nutzo

    nutzo [H]ardness Supreme

    Messages:
    7,377
    Joined:
    Feb 15, 2004
    I'd like to see what it could do with old 8mm family films I have from 50+ years ago.
    Faded, low resolution, 18 frames per second.... DVD quality would be a huge improvement.
     
    MatthewK likes this.
  5. TexasTea

    TexasTea n00b

    Messages:
    12
    Joined:
    Jun 5, 2015
    Seems like only a minor improvement but I like the idea. There are a lot of old "classics" that would be a lot more watchable if they were remastered.
     
    Tak Ne likes this.
  6. Tak Ne

    Tak Ne [H]ard|Gawd

    Messages:
    1,233
    Joined:
    Jan 28, 2008
    There was only about 3 seconds of the clip where there was any noticable difference and even that was miniscule. It was filmed on 35mm so I'm sure they have a better source than the DVD.
     
  7. Meeho

    Meeho [H]ardness Supreme

    Messages:
    4,272
    Joined:
    Aug 16, 2010
    Just play it through madVR :)
     
  8. GotNoRice

    GotNoRice [H]ardForum Junkie

    Messages:
    9,138
    Joined:
    Jul 11, 2001
    Unfortunately they don't. Here is a fantastic article regarding the situation for anyone curious about if/when we will ever see an HD version of DS9 or Voyager. Unfortunately the outlook is not good.

    http://www.treknews.net/2017/02/02/why-ds9-voyager-not-on-blu-ray-hd/
     
    The Mad Atheist likes this.
  9. Hisshadow

    Hisshadow n00b

    Messages:
    55
    Joined:
    Mar 11, 2017
    Id rather cbs spend 21 million per episode to restore ds9 to hd.. than to film another episode of std.

    the sad part is... i cant 'see' any difference in the demo footage
     
  10. BlackManx

    BlackManx n00b

    Messages:
    48
    Joined:
    Dec 1, 2016
    Been looking for a reason to re-watch DS9. I won't hold my breath, but I will remain hopeful that the finances work in terms of remastering the series and syndication on a network such as SYFY.
     
  11. trparky

    trparky Gawd

    Messages:
    975
    Joined:
    Jul 23, 2009
    I can't really tell the difference either.
     
  12. TexasTea

    TexasTea n00b

    Messages:
    12
    Joined:
    Jun 5, 2015
    Am I missing something? I thought the idea was to use AI to interpolate and refine lower quality images on the fly. If true, and the methodology improves, wouldn't that mean it wouldn't matter rather they have film to scan?
     
  13. ecktt

    ecktt Limp Gawd

    Messages:
    413
    Joined:
    Oct 22, 2004
    <_<
    >_>
    MadVR
    >_<
     
    Meeho likes this.
  14. SamuraiInBlack

    SamuraiInBlack [H]ardness Supreme

    Messages:
    5,677
    Joined:
    Oct 10, 2003
    Some stuff looks better but some actually looks worse to me. I dunno if it's Youtube compression or what, but to me it seemed like the faces of people seemed less "real'. I mean I get they're wearing makeup and mask/prosthetic materials, but some of them didn't seem to be as detailed.
     
  15. Snowdog

    Snowdog [H]ardForum Junkie

    Messages:
    8,610
    Joined:
    Apr 22, 2006
    It looks a little better to me. I doubt this technique will give something good enough to satisfy most Blu Ray buyers, but it could be a little upgrade in quality for showing it on streaming services.
     
    Shadowed likes this.
  16. BloodyIron

    BloodyIron 2[H]4U

    Messages:
    3,441
    Joined:
    Jul 11, 2005
    As a long time ds9 fan I can't tell the difference in this video. The shrinking lowers the fidelity a lot!
     
  17. Krenum

    Krenum [H]ardForum Junkie

    Messages:
    15,374
    Joined:
    Apr 29, 2005
    DS9 was such a good show. Man I miss old Star Trek.
     
  18. Derangel

    Derangel [H]ard as it Gets

    Messages:
    17,369
    Joined:
    Jan 31, 2008
    And I'm totally sure you'd be willing to rush out and pay $200+ per season to make it worth the cost, right?
     
  19. chameleoneel

    chameleoneel 2[H]4U

    Messages:
    2,838
    Joined:
    Aug 15, 2005
    the difference was easily noticeable on my shit quality 720p laptop screen. Color saturation, balance, and contrast are all way better. And clarity is noticeably improved. The clarity improvement is not DRASTIC. It doesn't look like footage filmed in HD. But, it is better and does not look over baked.
     
  20. daglesj

    daglesj [H]ardness Supreme

    Messages:
    5,031
    Joined:
    May 7, 2005
    Oh god the days of watching TNG in fuzzy blurry old NTSC brownovision on UK PAL setups.
     
  21. Burticus

    Burticus 2[H]4U

    Messages:
    3,835
    Joined:
    Nov 7, 2005
    Mild indeed. Looks like a tiny amount of noise cleanup and that's it.
     
  22. beemanit

    beemanit n00b

    Messages:
    33
    Joined:
    Apr 23, 2016
    Throw HD on anything, it makes good marketing even to resell an old series.

    Watch this "HD Cup".... See what I did. A cup is not just a cup now it's HD. I bought an HD workbench off Amazon 2 months ago. Granted in that case it stood for Heavy Duty. Still HD.

    Honestly I still prefer SD over HD.
     
  23. termite

    termite [H]ardness Supreme

    Messages:
    4,643
    Joined:
    Aug 27, 2004
    I am currently re-watching DS9 on Netfix. It is still watchable in SD so do not let the fact it is not HD keep you from watching it.

    The majority of the show takes place on the station, or ship (Defiant, shuttle etc) interior so their are not too many scenes where the show's age is glaringly evident.

    It was a bit jarring going right from the remastered ST:NG though.
     
  24. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    It doesn't have to be real time. As long as it can be processed at least @1 fps on reasonable hardware I'm game.
     
  25. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    Like there is no actual difference between SD video and HD video. If you're going to be a luddite at least be an honest one.
     
  26. TexasTea

    TexasTea n00b

    Messages:
    12
    Joined:
    Jun 5, 2015
    Yea, like turbo was or RGB now.
     
  27. alxlwson

    alxlwson You Know Where I Live

    Messages:
    5,793
    Joined:
    Aug 25, 2013
    But the turbo button did actually do some things
     
  28. TexasTea

    TexasTea n00b

    Messages:
    12
    Joined:
    Jun 5, 2015
    alxlwson likes this.
  29. AlphaAtlas

    AlphaAtlas [H]ard|Gawd Staff Member

    Messages:
    1,713
    Joined:
    Mar 3, 2018
    Stuff like this can run at a couple of FPS on my 980 TI/4670k. With the right algorithm, and a big RTX card, you can probably upscale a 480p DVD at 24 FPS.

    BTW, this guy seemingly did it with the relatively hacky method of converting the video to images, batch processing them, and re-encoding the finished images into a video, but there are other ways to do it.
     
  30. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    That's actually a terrible way to do it. As it doesn't use temporal information. Adjacent frames could be used to improve the quality of up scaling. However I haven't seen any publicly available machine learning based tools yet to upscale video.
     
  31. AlphaAtlas

    AlphaAtlas [H]ard|Gawd Staff Member

    Messages:
    1,713
    Joined:
    Mar 3, 2018
    https://github.com/kice/vs_mxnet

    https://github.com/WolframRhodium/muvsfunc/wiki/OpenCV-Python-for-VapourSynth

    To be fair, those use neural nets designed for images too, but you can pre (or post) process them with temporal filters. Using temporal information is a mixed bag anyway, as you can get all kinds of artifacts depending on the content.


    I've seen a paper or two on AI-based video upscaling filters that use temporal information, but no public code yet.
     
  32. PaulP

    PaulP Gawd

    Messages:
    776
    Joined:
    Oct 31, 2016
    CBS doesn't give a damn.
     
  33. Saturn_V

    Saturn_V [H]ard|Gawd

    Messages:
    1,560
    Joined:
    Sep 24, 2008
    Tonite only at a theater near you. Probably the only chance DS9 will see a HD remaster.

     
  34. Dr. Righteous

    Dr. Righteous 2[H]4U

    Messages:
    3,120
    Joined:
    Aug 1, 2007
    Not impressed at all. There are lots of image detail "peaking" filters that can be used that would yield results like that.
     
  35. sfsuphysics

    sfsuphysics I don't get it

    Messages:
    13,677
    Joined:
    Jan 14, 2007
    I thought the same thing, but upon thinking about it, you have a "480" vs a "1080" image, however if you view on a 1080 monitor (I am), then the 1080 squished down to less than half the width of the screen (in side by side) and the 480 fitting about half the width of the screen the resolution difference will not really be that noticeable. It's kind of like saying "4k doesn't look that much better" when you're viewing on a 1080 monitor.
     
  36. purple_monster

    purple_monster Limp Gawd

    Messages:
    377
    Joined:
    Jun 1, 2018
    the absolute irony of being the age to survive the star wars drought after each trilogy to only be bombarded with shitty disney wars and then learn that you are never going to get a TNG/DS9 4k or even real HD release. I took star trek for granted and got the opposite of what I didnt even know I had. someone go back and change the timeline.

    and yes the prequel trilogy is unironically a full letter grade better than anything post 2015.
     
  37. TurboGLH

    TurboGLH Gawd

    Messages:
    547
    Joined:
    Dec 19, 2002
    There's an updated comparison video, shows a much better improvement and a a better comparison between original and upgraded. It's uploaded in 4k, and it was much easier to see the difference (on my 4k monitor at least)

     
  38. SvenBent

    SvenBent 2[H]4U

    Messages:
    2,850
    Joined:
    Sep 13, 2008
    I jsut quickly glansed and notice lack of shadow details on face as well. they look like they got polished somehow.

    and most of it was still a blurry mesh. not really any noticeable improvements
     
  39. pendragon1

    pendragon1 [H]ardForum Junkie

    Messages:
    12,516
    Joined:
    Oct 7, 2000
    like that stupid Instagram filter. cgi looks good though.
     
  40. twonunpackmule

    twonunpackmule [H]ard|Gawd

    Messages:
    1,414
    Joined:
    Sep 27, 2005
    To get actual quality footage/audio of live Brianiac would make my genitals tingle.