Help! Attack of the diagonal lines!

T
Posted By
themosttogain
Dec 12, 2003
Views
1751
Replies
13
Status
Closed
I’m hoping theres a Photoshop genius out there that can help me with some images. I took about 6 photos at a gig last night and they all have the same diagonal line noise running across them. Its the first time i’d used my new digicam (a Toshiba PDR-M25) in the dark and i’m assuming it’s just a problem with the camera. I have uploaded one of the photo’s at http://myweb.tiscali.co.uk/alsplace/ in the hope that someone can either tell me how to get rid of them, or at least improve on them. I have tried all the obvious noise reduction filters, but with no joy. I’m using Photoshop 7 if that’s relevant.

Thanks in advance,

Alan.

Must-have mockup pack for every graphic designer 🔥🔥🔥

Easy-to-use drag-n-drop Photoshop scene creator with more than 2800 items.

T
tacitr
Dec 12, 2003
I’m hoping theres a Photoshop genius out there that can help me with some images. I took about 6 photos at a gig last night and they all have the same diagonal line noise running across them.

This is characteristic of the sort of noise you get from a CCD in low light levels, and it’s one of the reasons I still prefer film to digital. Film offers better dynamic range.

Digital cameras (and CCD-based scanners) have trouble with shadows; the signal gets buried in noise. There’s not a lot you can really do about it, short of experimenting with the Dust and Scratches filter (it’s sometimes helpful to select the dark areas, run Dust & Scratches, undo it, and then fade it in Darken mode) and looking through the channels to see if some judicious channel mixing in the shadows might help.


Rude T-shirts for a rude age: http://www.villaintees.com Art, literature, shareware, polyamory, kink, and more:
http://www.xeromag.com/franklin.html
MR
Milo Rambaldi
Dec 13, 2003
This is "noise" from the CCD.
Photoshop can reduce it with the noise filter but to clean it up you really need to get a plugin like Applied Science Fiction’s ‘GEM’, noise reduction. http://www.asf.com
There is a trial version too.
MR
————————–
"Big Al" wrote in message
I’m hoping theres a Photoshop genius out there that can help me with some images. I took about 6 photos at a gig last night and they all have the same diagonal line noise running across them. Its the first time i’d used my new digicam (a Toshiba PDR-M25) in the dark and i’m assuming it’s just a problem with the camera. I have uploaded one of the photo’s at http://myweb.tiscali.co.uk/alsplace/ in the hope that someone can either tell me how to get rid of them, or at least improve on them. I have tried all the obvious noise reduction filters, but with no joy. I’m using Photoshop 7 if that’s relevant.

Thanks in advance,

Alan.
PR
Peter Reid
Dec 13, 2003
Hi Alan —

A noise-reduction filter I recently purchased is NeatImage (occasionally referenced elsewhere in this group). It uses a process known as deconvolution (part of a magical set of mathematical tools in Fourier optics), in which repeating noise can be sampled, then subtracted from the image, without interfering with other high-frequency components in the picture (high-contrast edges and fine detail). [Alternatively, pre-built noise filters are available for a variety of cameras, each with different speed settings (though not for your Toshiba, alas).] The filter is intended for the removal of low-light CCD noise, but (if handled sensitively) also works wonders in removing grain from old photographs.

To give you an idea, I quickly ran your image through the filter (sampling on the upper-right part of the picture, and otherwise using default settings (there are lots of things you can tweak, so this is only a first-draft attempt!). The result is at
http://www.scifun.ed.ac.uk/tmp/nodiag.jpg
(The file is somewhat compressed, so don’t expect perfection; I just wanted you to see what could be done.)
[BTW, there are other packages which can try to cope with motion/shake blurring, again by using a Fourier convolution approach; they might help with the camera shake on the central figure…]

I _thoroughly_ recommend buying this package, at http://www.neatimage.com (there will be similar ones around, of course; this is just my thoroughly biased view :-).

Cheers,
Peter
PR
Peter Reid
Dec 13, 2003
As an addendum to the last message:
You can vary the amount of noise removed, and how the rest of the image is sharpened or smoothed. In the version I posted, I set the filter to aggressively go after the noise; the effect can be a slightly unnatural ‘plasticising’ of some features (such as skin tones). At 200 % it’s quite obvious on the players’ faces that a) the noise is gone, but b) they’re looking a wee bit like waxworks… Of course, you can always run the filter at two different settings to produce two separate layers, then merge the appropriate bits of each layer.

Anyway, just a thought.
Cheers,
p
T
themosttogain
Dec 13, 2003
Cool…thanks people. I’ll have a go with the plugin you suggested, and failing that, assume i’ll never get them looking right.

Thanks again,

Alan.
S
Sean
Dec 14, 2003
On 12 Dec 2003 22:27:19 GMT, (Tacit) reverently intoned up the aether:

I’m hoping theres a Photoshop genius out there that can help me with some images. I took about 6 photos at a gig last night and they all have the same diagonal line noise running across them.

As for diagonal lines, where are they in the image. All I can see is the flash reflecting off of surfaces and the characteristic noise of out of focus (OoF) underexposed image areas. Mix that together with the long ambient exposure and this image looks perfectly normal. Can you elaborate on where you are seeing these features. Please also take note that all elements of the image further from the lens than the bassist are clearly out of focus and have defocus induced noise in them.

Other than that I can see a few lens flares caused by atmospheric dust close to the lens hit by the flash and a few bad pixels reporting pure blue above and slightly to the right of the guitarist who is not singing.

This is characteristic of the sort of noise you get from a CCD in low light levels, and it’s one of the reasons I still prefer film to digital. Film offers better dynamic range.

Not it is not. This is what underexposure looks like. This occurs on film too. When you mix films wider latitude with the automated color correction and exposure correction done by your normal minilab much of this is not visible. Shooting RAW digital files corrects much of this and the 12-bit exposure latitude is roughly comparable to shooting slide film.

Digital cameras (and CCD-based scanners) have trouble with shadows; the signal gets buried in noise.

This can be corrected to some degree. But unless you are shooting RAW files jpeg compression tends to induce even more noise. Also such noise occurs in OoF and underexposed film shots. The only way to reduce such effects of bad bokeh (the character of the rendering of OoF elements in an image) is buy expensive lenses with neutral bokeh or defocus control. The larger exposure latitude of print film and the development process will obscure much of this.

There’s not a lot you can really do about it, short of
experimenting with the Dust and Scratches filter (it’s sometimes helpful to select the dark areas, run Dust & Scratches, undo it, and then fade it in Darken mode) and looking through the channels to see if some judicious channel mixing in the shadows might help.

This is not a good idea. Put your lens cap on the camera and take several exposures of identical length as the exposure with the noise in it in an uncompressed format (RAW or TIFF) and average these exposures (100% opacity bottom layer, 50% second layer, 33% third layer, 25% fourth layer, 20% fifth layer, 17% …). If these were taken at roughly the same temperature as the original exposure then this average should yield a reasonable map of the dark current noise in your CCD.

Take this image and use calculation and apply image to add together the values in the RGB channels to create an alpha channel. Apply levels to this channels to get a reasonable representation of where you have noise issues in your image. Use this alpha channel to generate a selection to which to apply the Dust and Scratches filter. Then you can fade the application of the filter if needed.

This will correct for CCD induced noise in an image. But if you were shooting jpegs there is an additional class of noise induced by the quantization of the DCT coefficients that far exceeds dark current noise unless you are doing very long exposures (bulb exposure exceeding 8 seconds).

If you wish to be really accurate you should actually be subtracting this dark current noise from the original images on a per channel basis and only using Dust and Scratches to correct for bad pixels in the CCD.

I could go on, but most of the noise in this image is optically induced and are simply a fact of life in low light photography. Please feel free to ask for clarification or if you would like me to mark up an image and put it on the web so you can see what I am talking about more clearly.

my $0.02,

Sean

"In the End, we will remember not the words of our enemies, but the silence of our friends."

– Martin Luther King Jr. (1929-1968)

Redesigned – Last Updated 15 October 2003
Photo Archive @ http://www.tearnet.com/Sean
S
Sean
Dec 14, 2003
On Sat, 13 Dec 2003 00:38:29 GMT, "Milo Rambaldi" reverently intoned up the aether:

This is "noise" from the CCD.

No it is not. It is interference patterns generated by the rings rather than blur circles created by spherical aberrations in the lens in out of focus areas in the image. The best way to correct this is to shoot in full manual in low light situations using a technique called dragging the shutter. By using full manual you can ensure that you use a small enough aperture to achieve acceptable focus over a large enough depth of field balanced with your flash and a long enough exposure to get a feeling of the natural coloration of the scene.

enjoy,

Sean

"In the End, we will remember not the words of our enemies, but the silence of our friends."

– Martin Luther King Jr. (1929-1968)

Redesigned – Last Updated 15 October 2003
Photo Archive @ http://www.tearnet.com/Sean
S
Sean
Dec 14, 2003
On Sat, 13 Dec 2003 18:45:07 -0000, "Peter Reid" reverently intoned up the aether:

As an addendum to the last message:
You can vary the amount of noise removed, and how the rest of the image is sharpened or smoothed. In the version I posted, I set the filter to aggressively go after the noise; the effect can be a slightly unnatural ‘plasticising’ of some features (such as skin tones). At 200 % it’s quite obvious on the players’ faces that a) the noise is gone, but b) they’re looking a wee bit like waxworks… Of course, you can always run the filter at two different settings to produce two separate layers, then merge the appropriate bits of each layer.

Anyway, just a thought.
Cheers,
p

It worked well on the background but I found it rather excessive on the skin as you noted. Another nice filter package for dealing with noise is Grain Surgery 2. That said, the biggest problem with this image is a shallow depth of field and the poor character f the rendering of out of focus elements by the lens. That said, I have never seen a sub $1000 US zoom lens that rendered out of focus elements to my satisfaction. There are some nice fixed focal length SLR lenses starting around $400 US.

But in the end, getting the photo right in the first place is far less work than correcting it digitally. Albeit, I take my photography rather seriously and had a very strong technical introduction to the subject.

enjoy,

Sean

"In the End, we will remember not the words of our enemies, but the silence of our friends."

– Martin Luther King Jr. (1929-1968)

Redesigned – Last Updated 15 October 2003
Photo Archive @ http://www.tearnet.com/Sean
PR
Peter Reid
Dec 15, 2003
Another nice filter package for dealing with
noise is Grain Surgery 2.

Ah!! How wonderful — almost worth buying for the name alone 🙂 Cheers,
p
RP
Richard Peterson
Dec 15, 2003
Although the actual diagonal pattern is not a function of the CCD noise, I don’t see what you mean here. BTW, Sean is right in stating that it is not CCD noise from low light. Low light does lead to CCD noise – especially in the preamplifier. The thing is, Whatever the noise source in a CCD image, it is still noise. Noise, by definition, does not have a pattern to it.

But Sean, I don’t get how dragging the shutter has anything to do with that pattern in the image. It seems to me to be some non-noise function of the electronics, not something optical. I mean, I have never seen something like that on film.
Could you explain?

-Richard

"Sean" wrote in message
On Sat, 13 Dec 2003 00:38:29 GMT, "Milo Rambaldi" reverently intoned up the aether:

This is "noise" from the CCD.

No it is not. It is interference patterns generated by the rings rather than blur circles created by spherical aberrations in the lens in out of focus areas in the image. The best way to correct this is to shoot in full manual in low light situations using a technique called dragging the shutter. By using full manual you can ensure that you use a small enough aperture to achieve acceptable focus over a large enough depth of field balanced with your flash and a long enough exposure to get a feeling of the natural coloration of the scene.
enjoy,

Sean

"In the End, we will remember not the words of our enemies, but the silence of our friends."

– Martin Luther King Jr. (1929-1968)

Redesigned – Last Updated 15 October 2003
Photo Archive @ http://www.tearnet.com/Sean
H
hoffmann
Dec 15, 2003
"Richard Peterson" …
Although the actual diagonal pattern is not a function of the CCD noise, I don’t see what you mean here. BTW, Sean is right in stating that it is not CCD noise from low light. Low light does lead to CCD noise – especially in the preamplifier. The thing is, Whatever the noise source in a CCD image, it is still noise. Noise, by definition, does not have a pattern to it.
But Sean, I don’t get how dragging the shutter has anything to do with that pattern in the image. It seems to me to be some non-noise function of the electronics, not something optical. I mean, I have never seen something like that on film.
Could you explain?

-Richard

It looks like random noise plus a periodical noise with a phase shift from line to line. The periodical noise can be caused by a disturbance at the pre-amplifier (high gain in this situation) by the internal clocked digital circuits.
The phase shift can be a result of the short delay for each line switching.

Best regards –Gernot Hoffmann
S
Sean
Dec 16, 2003
On Mon, 15 Dec 2003 01:49:24 +0000 (UTC), "Peter Reid" reverently intoned up the aether:

Another nice filter package for dealing with
noise is Grain Surgery 2.

Ah!! How wonderful — almost worth buying for the name alone 🙂 Cheers,
p

It also works in 16-bit mode which many may appreciate.

enjoy,

Sean

"In the End, we will remember not the words of our enemies, but the silence of our friends."

– Martin Luther King Jr. (1929-1968)

Redesigned – Last Updated 15 October 2003
Photo Archive @ http://www.tearnet.com/Sean
S
Sean
Dec 16, 2003
On Mon, 15 Dec 2003 09:48:09 GMT, "Richard Peterson" reverently intoned up the aether:

Although the actual diagonal pattern is not a function of the CCD noise, I don’t see what you mean here. BTW, Sean is right in stating that it is not CCD noise from low light. Low light does lead to CCD noise – especially in the preamplifier.

Everything I have read references dark current noise as a function of temperature and not as a function of incident light. Hence CCD noise will be more prevalent at a gig due to the high ambient temperature created by all those closely packed bodies, but the low light will not cause.

I do agree that dark current noise (CCD noise) will be more notable in low light as it is relatively larger in magnitude in comparison to the desired signal.

The thing is, Whatever the noise source in a CCD image,
it is still noise. Noise, by definition, does not have a pattern to it.
But Sean, I don’t get how dragging the shutter has anything to do with that pattern in the image.

The only "diagonal lines" I can see in the posted example are causes by musical equipment and by the ghosting effect caused by dragging the shutter (sharp image plus a ghostlike image added to it from the longer exposure). There is a highly visible noise pattern in the distant out of focus (OoF) areas but

It seems to me to be some non-noise function of the
electronics, not something optical. I mean, I have never seen something like that on film.
Could you explain?

I will try. The starting point for this is to begin with an understanding of image formation on the film plane (CCD) and how focus and optical defects affect the blur circles of points of light. These ideas then need to be extended to more complex images.

When we focus upon a point of light we expect the camera to record this as a point of light on the film plane. Pretend you are 100 miles from the nearest electric light on a moonless night with no wind and you are using a 50 mm lens to focus on a small candle flame 20 meters away to get a point of light. When this point is no longer on the plane of critical focus we expect the point to be rendered as a blur circle. The size of this blur circle corresponds to how out of focus the point of light is. In an idealized optical system the shape of a blur circle should be a uniformly distributed disk of light.

Because of the physical limitations that must be overcome in engineering lenses this idealized blur circle is never achieved. Take a look at figure 6 in the following document to see some reasonable examples of what occurs with most real world lenses.

http://luminous-landscape.com/essays/bokeh.shtml

The term bokeh (boke with a sharp e in Roman alphabet Japanese spelling) is used to describe the character of the blur circles in an image. If you wish to read more on this subject then I would recommend taking at look at the following link.

http://www.trenholm.org/hmmerk/ATVB.pdf

The section where he spouts idiocy and tries to explain convolution should be skipped over as it makes no sense and shows that the author may not completely understand it. Sadly, the only clear explanations of what convolution is are in introductory mathematical (not engineering) texts on Fourier Analysis and that is not the most accessible material.

Taking a look at the specs for the Toshiba PDR-M25 I see that it is a point and shoot consumer digicam so I would expect to have the most common optical characteristic of having good bokeh for objects nearer than the plane of focus an bad bokeh for object beyond the plane of focus. This indicates that the blur circles for out of focus points of light beyond the plane of focus are highly likely to be bright circles with diffusely lit center like the two leftmost blur circle in figure 6 at http://luminous-landscape.com/essays/bokeh.shtml.

It turns out that in this case the blur circles actually appear to be a bright ring with a diffuse fill and a bright center. This type of optical distortion of the blur circle will render fairly similarly whether the OoF object is closer or further from the lens than the plane of critical focus. The determination of this is based upon the shape of out of focus specs of airborne dust near the lens that were struck by the on camera flash. Take a look at
http://www.tearnet.com/sean/tmp/BlurCircle.jpg and you can see how there are actually several of these dust induced lens flares that can be seen from if the contrast of shadow detail is strongly enhanced.

Taking these ideas and expanding them to cover image formation we get the idea that in focus points of light in the image should be rendered as points of light. This occurs on the two heads of hair near the lens and on all objects as close as or closer than the bassist is to the lens. In complete honestly the bassist is only acceptable and not critical focus.

Next we begin looking at the portions of the image which are further away from the lens than the bassist. These areas are out of focus. So each point of light becomes rendered as bright circle of light and bright centered point of light connected by a diffuse and dimly lit circle of light. Now this can create interference patterns. A nice extreme example of this can be seen by looking at the background of the image on the first page of http://www.trenholm.org/hmmerk/ATVB.pdf where a single spire of temple is rendered as two overlapping diffuse spires.

Now if we begin considering how each point in the background there is rendered this way we get a large collection of overlapping circles and points. These interference patterns are often moderately noisy and somewhat aesthetically lacking. But being out of focus such noisy blurs are usually moderately soft at the same time.

This leaves us with the OoF areas being a bumpy but smooth blur (not terribly aesthetically pleasing, but blurred nonetheless). This type of noise can also be found in film shots. If you would like I can hunt some down and scan a few to explore this topic further if you would so like.

Having taken a look at the specs of the Toshiba PDR-M25 I have noted that this camera also only records jpegs. Now jpegs tend to do a very poor job of rendering smooth textures (i.e. blurs) as it is normally high frequency detail lost in the quantizer that makes it smooth. This tends to add a significant amount of noise to image regions where there are smooth textures unless very little compression is used. This specific image was compressed by a factor of 6 so the introduction of such noise is highly likely. Also, by looking at the EXIF data I can see that this was a 1/4 second exposure at f/3. A 1/4 second exposure is not long enough for dark current noise to become significant. With my digicam dark current noise is not visible to the naked eye in an 8 second exposure at 68 degrees fahrenheit. I have found dark current noise to become significant in 30 second and longer exposures at lower temperatures than that.

Hence in a final analysis of this image I would conclude that the noise seen is mostly compression induced noise on noisy blurs. By overlaying this compression induced noise over dark regions it becomes relatively stronger. Zooming in to view the OoF areas at 400% plus zoom levels you can see jpeg compression artifacts introducing small amounts of color noise on top of the noisy induced by defocus.

Whereas, if this were CCD induce noise, then I would expect the amount of noise to be roughly uniform across in focus and out of focus elements in this image. But this is not so.

It is also very common for people to malign digital capture methods while claiming that phenomena they are seeing never happened on film when it was actually the wider exposure latitude of film compressing artifacts into a smaller percentage of the total visualization space that prevented such issues from becoming quite as prominently.

If any of this is unclear, please feel free to ask for clarification as this is Usenet and you are only getting rough draft quality writing from me here.

enjoy your day,

Sean

"In the End, we will remember not the words of our enemies, but the silence of our friends."

– Martin Luther King Jr. (1929-1968)

Redesigned – Last Updated 15 October 2003
Photo Archive @ http://www.tearnet.com/Sean

How to Improve Photoshop Performance

Learn how to optimize Photoshop for maximum speed, troubleshoot common issues, and keep your projects organized so that you can work faster than ever before!

Related Discussion Topics

Nice and short text about related topics in discussion sections