Color Correcting the Dragon Box - 3 Part Spectacular

[crabshank1]

interesting work, i hope that your settings will be ok soon

Thanks man, nearly there

[crabshank1]

The power of crushing (0.2), on the right:

[lansing]

Spoiler:

1. Download AviSynth+, my .zip file containing AvsPmod and the required shaders, and the contents of my relevant GIthub repo.
   
   First download and install Avisynth-Plus-r1576.exe http://avs-plus.net/ (Homepage link), then download and extract this .zip file somewhere https://1drv.ms/u/s!AjHPdv1_75reiQCdDWp ... M?e=B9T9PJ.
   
   Then all you need to do is change the cd and input variables' paths to where the files are on your system (from "To apply Remap README.txt"), paste it into the main console and it should work. I suggest that you create a new remap.hlsl file, with a different name, for each scene (/time interval) then encode.
   
   Spoiler:

   Image histogram.html + jquery 3.3.1 from my Github: https://github.com/crabshank/HLSL-Resha ... -remappers

2. Import your video into AviSynth.
3. Input the relevant code required to use shaders.
   
   Spoiler:

   Code: Select all

   cd="[*BEGINNING OF PATH*]\AvsPmod"
   LoadPlugin(cd+"\plugins\AviSynthShader-1.6.5\Shader.dll")
   Import(cd+"\plugins\AviSynthShader-1.6.5\Shader.avsi")
   AviSource("[*VIDEO PATH HERE*]",audio=false,pixel_type="RGB32")
   
   ConvertToRGB32(matrix="PC.709") #CONVERT TO RGB 0-255 WHICH IS THE COLOUR SPACE WE NEED TO WORK IN
   
   Input=BicubicResize(2560,1920,b=-1,c=1) #4X UPSCALE; OPTIONAL!
   
   Input.Shader("HSV dither.hlsl") #.Shader(...   #CHAINS SHADERS TOGETHER
   
   ExecuteShader(last, Input)
   
   BicubicResize(640,480,b=-1,c=1) #DOWNSCALE
   
   ...
   

4. (OPTIONAL){Upscale your image. I recommend 2x or 4x the size of the final output (640x480 in this case).}
   Spoiler:

   Code: Select all

   Input=BicubicResize(2560,1920,b=-1,c=1) #4x upscale (kind of)
   

5. Make a copy of remap.hlsl, with which you will adjust the hues. For this I find the easiest way is to find out the current hues you wish to change, I just did it manually using hues from Hikari TV as reference and changing the hues in the image to them, but if any of these are less than 360° and you want to make them more than 0° or they are more than 0° and you want to make them less than 360°, you will need to rotate the hues and add the new hue after rotation as a fixed point.
   
   For example if you have a hue of 357° and you want to make it 7°, you will need to write

   Code: Select all

   static float rotate_hues = 10;

   and then add "7,7," as a point.
   
   If you have rotated, will you need to get the hues you wish to change again, as they will have changed.
6. Using Shader("HSV dither.hlsl"), increase the standard deviation (by whole numbers works fine), for each RGB channel until the macroblocks disappear.
   
   Spoiler:

   • Firstly load up your video into AvsPmod and apply the oneChannel shader to look at each RGB channel separately.
     Skip through the video to find blocky areas:
     
     
     And when you do, increase the standard deviation for the dithering in each channel (change in oneChannel i.e. c0.rgb=c0.r/c0.g/c0.b) until the blocks have been completely dithered away:
     
     
     Keep skipping through to see if the dithering needs to be increase anywhere. You can use the "Balance almost white.hlsl" whose default settings make blocky areas easier to see:
     

7. Create an image made only of areas that are supposed to be greyscale, the more the better (I shall refer to this as the 'white balance calibration image')
   Spoiler:

   • Next, for the white balance I will be using GIMP 2.8, if you are too put the scripts included in the AvsPmod.zip file into C:\Program Files\GIMP 2\share\gimp\2.0\scripts, if that's where you've installed it.
     
     Grab some frames with areas you think should be greyscale and open them as layers:
     
     
     
     
     Then add an alpha channel to all layers:
     
     
     Go through each layer and select the areas (fuzzy select should work OK but it might overdo it so you might need to manually de-select a few areas) you think should be greyscale, Ctrl+I (Invert selection) and Delete, which will leave only the parts you want.
     
     When you've done that, you can tile all the layers into one image:
     

8. Load your image into AvsPmod and apply the same dithering shader that you made to make the macroblocks disappear, then downscale your image if necessary.
9. To white balance, match all the RGB channels to one channel using the image histogram web page from my Github and the remap.hlsl shader. For example, the green channel in my white balance calibration image had no big dark patches in it whereas all the others did, so I chose to map the Red and Blue channels to the Green channel.
   
   Spoiler:

   • Look at each channel and decide which channel you want to match to, I chose green because it had no big artefacts and looked dithered.
     
     Now, load up the image remapping page in your browser and load up the dithers greys image twice, matching both reds and blues to greens in this case:
     
     
     
     
     And copy the recommended values to your remapping shader:
     

10. To fix the value channel, open a frame (with blown out highlights) in GIMP and do Colors > Auto > Stretch Contrast. Which will create a reference image for the values, but most of the time this image will be basically the same. You then match your image to the one you just made in GIMP and paste the remapping points into the same file in which you white balanced (adjusted RGB channels).
11. (OPTIONAL){Downscale your video.}
    Spoiler:

    Code: Select all

    ExecuteShader(last, Input) #RUN DITHERING AND WHITE BALANCE SHADERS THAT YOU'VE DONE SO FAR
    
    Input=BicubicResize(640,480,b=-1,c=1)

12. (OPTIONAL){On this now downscaled image,} using Shader("De-saturate dither.hlsl"), change the value of '#define crushing [value here]' until you're happy that the coloured noise you added from the dithering has disappeared from the blacks and that the blacks haven't been overly crushed.
    
    Credit to ect5150 and jasonspidey71 for helping me get this far.
    
    This guide is nearly complete, the last hurdle is adjusting the saturation channel.

I just spent 2 hours reading and trying out your guide and I'm going to say, 90% of this is pointless. For one, your instruction is terrible, so many confusing steps that nobody can follow. For example, in step 1,

...paste it into the main console

What console? Paste what?

And then the code in step 3 is wrong

Code: Select all

ConvertToRGB32(matrix="PC.709")

#it should be like this 
ConvertToRGB32()

And then in the code here you haven't declare the file path

Code: Select all

Input.Shader("HSV dither.hlsl")

Any regular user that tried to follow would had given up right here because the script is returning an error.

Reason 2, like I said earlier, your usage of dithering is wrong, stop doing it. You're just sharpening the grain, which is a terrible idea. Purists are talking about keeping the grain, not stacking 3 layers of artificial grain on top of the original like what you're doing here.

Reason 3, 90% of your guide have nothing to do with color correction. What you're doing with the grain is irrelevant to the main topic, you're just adding more confusion to people who are trying to read it. The only steps that you really need are 1, 2, 3 and 5 where you talk about changing color. But then again, your instruction on step 5 is very bad as well.

Spoiler:

Make a copy of remap.hlsl, with which you will adjust the hues. For this I find the easiest way is to find out the current hues you wish to change, I just did it manually using hues from Hikari TV as reference and changing the hues in the image to them, but if any of these are less than 360° and you want to make them more than 0° or they are more than 0° and you want to make them less than 360°, you will need to rotate the hues and add the new hue after rotation as a fixed point.

For example if you have a hue of 357° and you want to make it 7°, you will need to write

Code: Select all

static float rotate_hues = 10;

and then add "7,7," as a point.

If you have rotated, will you need to get the hues you wish to change again, as they will have changed.

Where? How?? What??? What software are you even using?

And then there's the accuracy issue, judging from your post here where you try to match to the kai color, your result is not even close.

[crabshank1]

Sorry, posted new reply instead of editing.

[crabshank1]

Well, I'm happy you tried.

What console? Paste what?

The console of AvsPmod.

(from "To apply Remap README.txt"),

I'll admit, there should be no parentheses here and I will correct this in the next update to the guide. This file is in the AvsPmod.zip file, paste the contents of this file into the console of AvsPmod.

And then the code in step 3 is wrong

Code: Select all

ConvertToRGB32(matrix="PC.709")

#it should be like this 
ConvertToRGB32()

I need 0-255, so I must specify PC.601/PC.709. I am now using PC.601 instead of 709.

And then in the code here you haven't declare the file path

Code: Select all

Input.Shader("HSV dither.hlsl")

Any regular user that tried to follow would had given up right here because the script is returning an error.

Works for me without doing so, paste from the from "To apply Remap README.txt" file and change file paths appropriately.

Reason 2, like I said earlier, your usage of dithering is wrong, stop doing it. You're just sharpening the grain, which is a terrible idea. Purists are talking about keeping the grain, not stacking 3 layers of artificial grain on top of the original like what you're doing here.

I do it to deblock. And I consider it the most purist method of doing so, removing blocks without removing detail. I will not leave the blocks in, they're disgusting. What's your deblocking solution?

Reason 3, 90% of your guide have nothing to do with color correction. What you're doing with the grain is irrelevant to the main topic, you're just adding more confusion to people who are trying to read it. The only steps that you really need are 1, 2, 3 and 5 where you talk about changing color. But then again, your instruction on step 5 is very bad as well.

Lol, without 7, 8 and 9 you can't even correct The Final Chapters.

Spoiler:

Make a copy of remap.hlsl, with which you will adjust the hues. For this I find the easiest way is to find out the current hues you wish to change, I just did it manually using hues from Hikari TV as reference and changing the hues in the image to them, but if any of these are less than 360° and you want to make them more than 0° or they are more than 0° and you want to make them less than 360°, you will need to rotate the hues and add the new hue after rotation as a fixed point.

For example if you have a hue of 357° and you want to make it 7°, you will need to write

Code: Select all

static float rotate_hues = 10;

and then add "7,7," as a point.

If you have rotated, will you need to get the hues you wish to change again, as they will have changed.

Where? How?? What??? What software are you even using?

Hlsl files are edited with a text editor (you might need to open it as an administrator). You change the variables, usually written like "#define variable value" or "variable = value".

And then there's the accuracy issue, judging from your post here where you try to match to the kai color, your result is not even close.

I did Kai hues, dither, white balance and crushing. The number of possible colours for one hue is a lot.

I will probably delete all Avisynth and do a fresh install, just to check it works OK for newbies. Also, increase the verbiage of my explanations for those who know nothing of shaders or Avisynth where appropriate.

There is also the option of running the shaders in a stack on your video player and editing them in a text editor then turning the shaders on and off again to see the difference. This is the backup method, to using Avisynth.

[crabshank1]

https://github.com/crabshank/HLSL-Resha ... ches.hlsl/

I saw lots of yellow splotches in the skin when I was working on DBZ ep 268 and I wanted a shader to reduce them. After decomposing colour channels in G'MIC (in GIMP), I saw that the h channel of Lch made those splotches a different colour to the rest of the image. So, using this value, the saturation and a threshold it isolated the splotches (I will add a debug option so you can see) and I reduced their saturation, so now they are greenish. Not perfectly blended but much less obnoxious to the eye.

N.B. the default settings are what I'm using for Z ep 268. There will probably be amendments to this shader.

[lansing]

I do it to deblock. And I consider it the most purist method of doing so, removing blocks without removing detail. I will not leave the blocks in, they're disgusting. What's your deblocking solution?

There is no blocking in all the images, I see only grain or noise, so there is no need to deblock. Furthermore it's pointless to include deblock or noise removal stuff in here because it has nothing to do with color correction.

Hlsl files are edited with a text editor (you might need to open it as an administrator). You change the variables, usually written like "#define variable value" or "variable = value".

Dude there are literally 200-300 lines of codes in each of those files, what are you expecting any regular person to do? Where do they put the code? Where is the "hue" coming from? Where is the common sense?

And then there's the accuracy issue, judging from your post here where you try to match to the kai color, your result is not even close.

I did Kai hues, dither, white balance and crushing. The number of possible colours for one hue is a lot.

You shouldn't be doing any guide if your color matching method wasn't even accurate.

[crabshank1]

There is no blocking in all the images, I see only grain or noise, so there is no need to deblock. Furthermore it's pointless to include deblock or noise removal stuff in here because it has nothing to do with color correction.

viewtopic.php?f=7&t=19448&p=1053303&hil ... s#p1053303

have tried version 2, it worked on some extent, but the problem is that it easily create artifacts.

I'd bet that it didn't create the artefacts, it just made them show up clearer.

Dude there are literally 200-300 lines of codes in each of those files, what are you expecting any regular person to do? Where do they put the code? Where is the "hue" coming from? Where is the common sense?

I'm willing to write a list of what each variable does.

You shouldn't be doing any guide if your color matching method wasn't even accurate.

In HSV, the hue tells you the colour. What else can I match? I can have one reference hue for each thing e.g. Goku's gi, Super Saiyan hair, Supreme Kai's skin etc. and match to that in all episodes.

P.S. I hate digging up old comments, but I felt I had no choice this time.

[Kuwabara]

There is no blocking in all the images, I see only grain or noise, so there is no need to deblock. Furthermore it's pointless to include deblock or noise removal stuff in here because it has nothing to do with color correction.

I'll grant that deblocking doesn't necessarily relate to this thread, but to say that the Dragon Boxes don't have this issue at all is just flat out wrong. You're quick to jump down crabshank1's throat, yet you're either unknowledgeable of, or being very disingenuous about, something incredibly basic that's endemic to compressed video. Even with the very best compression/encoding techniques employed, DVD footage will always be bottlenecked to some degree by various factors, from time, to disc space, to maximum bitrate allowed by commercial players. Even Blu-ray footage falls victim to similar factors, albeit to a much less noticeable degree on average.

I can see compression issues in the Dragon Box footage, it's impossible for there not to be. The NA release exacerbates these issues, since there's an extra audio track involved.

So, even if crabshank1's methods are still a work in progress, I find what they're doing interesting enough to not just hand wave away. Perhaps a separate thread is in order?

If I had any criticism of what you're doing, I'd agree that the grain does look noticeably sharpened to some extent. Also, in the example you did with Vegeta, the darker shade of his clothes looks so crushed that it's hard to see the linework of the folds in his shirt.

[crabshank1]

If I had any criticism of what you're doing, I'd agree that the grain does look noticeably sharpened to some extent. Also, in the example you did with Vegeta, the darker shade of his clothes looks so crushed that it's hard to see the linework of the folds in his shirt.

I think that if the whole image looks sharper, the grain will too. I haven't got a shader that leaves out the grain, yet.

I can lower the amount of crushing, no problem.

Also, the blocks were coloured and messing up my white balancing which is why I dithered them away.

[crabshank1]

https://github.com/crabshank/HLSL-Resha ... ches.hlsl/

I saw lots of yellow splotches in the skin when I was working on DBZ ep 268 and I wanted a shader to reduce them. After decomposing colour channels in G'MIC (in GIMP), I saw that the h channel of Lch made those splotches a different colour to the rest of the image. So, using this value, the saturation and a threshold it isolated the splotches (I will add a debug option so you can see) and I reduced their saturation, so now they are greenish. Not perfectly blended but much less obnoxious to the eye.

N.B. the default settings are what I'm using for Z ep 268. There will probably be amendments to this shader.

Latest version isolates these splotches and reduces the hues of them. Much more purist IMO.

[lansing]

There is no blocking in all the images, I see only grain or noise, so there is no need to deblock. Furthermore it's pointless to include deblock or noise removal stuff in here because it has nothing to do with color correction.

viewtopic.php?f=7&t=19448&p=1053303&hil ... s#p1053303

have tried version 2, it worked on some extent, but the problem is that it easily create artifacts.

I'd bet that it didn't create the artefacts, it just made them show up clearer.

...

P.S. I hate digging up old comments, but I felt I had no choice this time.

What are you even smoking? The "artifact" in that comment was referring to the artifact created by the color correcting software, it stays within the topic of this thread. It wasn't talking about grain removal or what ever you were doing.

You shouldn't be doing any guide if your color matching method wasn't even accurate.

In HSV, the hue tells you the colour. What else can I match? I can have one reference hue for each thing e.g. Goku's gi, Super Saiyan hair, Supreme Kai's skin etc. and match to that in all episodes.

I don't know. I'm just basing my judgement on the results.

[lansing]

There is no blocking in all the images, I see only grain or noise, so there is no need to deblock. Furthermore it's pointless to include deblock or noise removal stuff in here because it has nothing to do with color correction.

I'll grant that deblocking doesn't necessarily relate to this thread, but to say that the Dragon Boxes don't have this issue at all is just flat out wrong. You're quick to jump down crabshank1's throat, yet you're either unknowledgeable of, or being very disingenuous about, something incredibly basic that's endemic to compressed video. Even with the very best compression/encoding techniques employed, DVD footage will always be bottlenecked to some degree by various factors, from time, to disc space, to maximum bitrate allowed by commercial players. Even Blu-ray footage falls victim to similar factors, albeit to a much less noticeable degree on average.

I can see compression issues in the Dragon Box footage, it's impossible for there not to be. The NA release exacerbates these issues, since there's an extra audio track involved.

He has been spamming this thread like no others' business for the past month with the same stuff over and over again. And nobody responded because nobody knows what he's talking about. I think I have waited long enough to speak up until now.

So, even if crabshank1's methods are still a work in progress, I find what they're doing interesting enough to not just hand wave away. Perhaps a separate thread is in order?

Yeah, anything that helps stop the spamming of irrelevant stuff in here is good for me, less confusion for everybody.

[Kuwabara]

He has been spamming this thread like no others' business for the past month with the same stuff over and over again. And nobody responded because nobody knows what he's talking about. I think I have waited long enough to speak up until now.

The fact that you had to wait is pretty telling if you ask me. Even if a ton of their posts were left unanswered, the few other pieces of feedback I have seen have only been positive/constructive.

[crabshank1]

What are you even smoking? The "artifact" in that comment was referring to the artifact created by the color correcting software, it stays within the topic of this thread. It wasn't talking about grain removal or what ever you were doing.

Am I removing grain or sharpening it? Read it again and you might see that the point I was making was that I didn't think the software created the artefact.

I don't know. I'm just basing my judgement on the results.

You want to match colours, and yet when I match the only source of colour in the colour space you moan. What's your colour matching procedure so we can compare notes?

He has been spamming this thread like no others' business for the past month with the same stuff over and over again. And nobody responded because nobody knows what he's talking about. I think I have waited long enough to speak up until now.

I've made plenty of new additions, maybe too many, and I repeated the guide BUT with new additions marked in GREEN so everyone knew it was an addition. Clarifications too. Plus, plenty of people have some idea of what I mean but not exactly because they're not actually doing it, they're not writing the code, and that's totally understandable. However, they have the respect to enquire further, to ask me.

Yeah, anything that helps stop the spamming of irrelevant stuff in here is good for me, less confusion for everybody.

You thought you don't need to white balance; you said there was no macroblocking in the Dbox, just grain (like every frame is a lossless scan); you couldn't see the value in matching hues, the only source of colour in the HSV colour space. You are no authority on pertinence.

Plus I will clarify further, so moving my work to another thread will keep the same amount of confusion in this one. Obviously.

This thread was almost dead before I filled it with my ideas, I said I thought I was necrobumping it. I'm close, DB E109 proves it and I'm currently seeing if I can reduce the yellow spots in the skin. The last real step is how to adjust the saturation channel.

The blocks are there, 100%, I remove them by randomly adding values to each channel without changing the average and then crushing and re-expanding the output so that it reverts close to the mean through the loss in precision of the floating point values. This is the most purist deblocking I can think of. For further proof of blocking, see anyone's comments about how if the CC is too extreme, they "get artifacts". Note: They didn't create them, they were already there.

[lansing]

What are you even smoking? The "artifact" in that comment was referring to the artifact created by the color correcting software, it stays within the topic of this thread. It wasn't talking about grain removal or what ever you were doing.

Am I removing grain or sharpening it? Read it again and you might see that the point I was making was that I didn't think the software created the artefact.

The point here is that the "artifact" I was referring to stays on topic as it has to do with color correction, while the "artifact" that you spent a month trying to fix has nothing to do with it.

You want to match colours, and yet when I match the only source of colour in the colour space you moan. What's your colour matching procedure so we can compare notes?

I thought that should be the whole point of your guide. Why are you blaming me when it's giving poor result?

You thought you don't need to white balance; you said there was no macroblocking in the Dbox, just grain (like every frame is a lossless scan); you couldn't see the value in matching hues, the only source of colour in the HSV colour space. You are no authority on pertinence.

Plus I will clarify further, so moving my work to another thread will keep the same amount of confusion in this one. Obviously.

This thread was almost dead before I filled it with my ideas, I said I thought I was necrobumping it. I'm close, DB E109 proves it and I'm currently seeing if I can reduce the yellow spots in the skin. The last real step is how to adjust the saturation channel.

The blocks are there, 100%, I remove them by randomly adding values to each channel without changing the average and then crushing and re-expanding the output so that it reverts close to the mean through the loss in precision of the floating point values. This is the most purist deblocking I can think of. For further proof of blocking, see anyone's comments about how if the CC is too extreme, they "get artifacts". Note: They didn't create them, they were already there.

I don't want to go into discussion about if there're macroblocking in the Dragon box, I don't care, I just want the contributions here to stay on topic.

[HakkaiBills93]

I don't want to go into discussion about if there're macroblocking in the Dragon box, I don't care, I just want the contributions here to stay on topic.

You are really rude with him, he is the only one that make this topic alive since months, all people with genius idea mostly left their project out of anything or just don't talk about them anymore, i am okay that his methods is not yet ready and it's mostly why i didn't answered for my part except for some kind of encouragement. it's really nice that unlike 99% of all people that show their results only (like a "yes i am strong, i can do that but you can't" and never helped others) he kindly shared what he are doing and how he can do it. the only point where i can agree with you is that he shouldn't have made his tutorial until he reach his final goal as we can loose ourself in all the steps (and it's not a way i usually follow, i am a noob using his way i don't really understand lol) but there's others way to said him

as long as he work about color corrections it's in the topic, if some step don't fit you then just don't follow them that's all. i remember that you said nothing with people that was showing ugly waifux2 things that are farrrrrrrrrr worst than any issue i have seen here

What final result did you managed to get? show yourself where he is wrong with example, screenshot if you really think he is wrong, i think it's the best way to give evidence and make things go forwards instead of just a useless and stupid opinion fight and why not helping him too?

[crabshank1]

I don't want to go into discussion about if there're macroblocking in the Dragon box, I don't care, I just want the contributions here to stay on topic.

You are really rude with him, he is the only one that make this topic alive since months, all people with genius idea mostly left their project out of anything or just don't talk about them anymore, i am okay that his methods is not yet ready and it's mostly why i didn't answered for my part except for some kind of encouragement. it's really nice that unlike 99% of all people that show their results only (like a "yes i am strong, i can do that but you can't" and never helped others) he kindly shared what he are doing and how he can do it. the only point where i can agree with you is that he shouldn't have made his tutorial until he reach his final goal as we can loose ourself in all the steps (and it's not a way i usually follow, i am a noob using his way i don't really understand lol) but there's others way to said him

as long as he work about color corrections it's in the topic, if some step don't fit you then just don't follow them that's all. i remember that you said nothing with people that was showing ugly waifux2 things that are farrrrrrrrrr worst than any issue i have seen here

What final result did you managed to get? show yourself where he is wrong with example, screenshot if you really think he is wrong, i think it's the best way to give evidence and make things go forwards instead of just a useless and stupid opinion fight and why not helping him too?

Thanks for this sensible post. Only reason I updated as I went along was that if something happened to me and I was unable to continue, there would be enough info here in case anyone else wanted to/could figure it out. Then they could see if they could finish it. Also, if anyone was following along they'd get the latest ideas.

[crabshank1]

You want to match colours, and yet when I match the only source of colour in the colour space you moan. What's your colour matching procedure so we can compare notes?

I thought that should be the whole point of your guide. Why are you blaming me when it's giving poor result?

You don't seem to understand HSV. yoU sHoULdn't bE dOINg anY GuiDe If YOu doN't unDErStaND Hsv.

I don't want to go into discussion about if there're macroblocking in the Dragon box, I don't care, I just want the contributions here to stay on topic.

For the (n+1)th time: the macroblocks were coloured and fucking up my white balance (my CC), so I had to lose them even if I didn't hate them. But seeing as you said white balance isn't needed, I finally realised earlier why you keep saying this.

[HakkaiBills93]

For the (n+1)th time: the macroblocks were coloured and fucking up my white balance (my CC), so I had to lose them even if I didn't hate them. But seeing as you said white balance isn't needed, I finally realised earlier why you keep saying this.

you talk about coloured pixels in the white colors? i used to see some when i tried to cc, the only way i managed to find to removed them was DNR, i was thinking that it was caused of the grain lol