Agree with Funky, a lot of memory acts very finicky when used with a very low memory divider. Try a higher mem divider.
IMO, when using a very low memory divider is the memory timings tend to get set for very tight/aggressive settings. Hence when OC'ing the HTT, the memory gets OC'ed simultaneously with these tight timings - so even though it is well under its DDR400 limit, it tends to hit a wall.
One way to get around this is to manually key in the timings (I see you seem to have set it for 3-3-3-8 , but there are a myriad other timings that you cannot play with in your A8N-E BIOS that might be affecting things).
- I'm assuming you're using Clockgen to OC? If so try this - do not place any memory divider within the BIOS, let the memory run at its default speed. Key in the required settings for the rest of the components - i.e. lower CPU multi, HTT bus multi (again 3x HTT multi is good enough, 2x is a bit too low and common rule is to not keep the HTT too high or too low).
- Next load up A64 tweaker from within Windows and drop MEMCLK Frequency down to 166Mhz, and start up Clockgen and increase the HTT. Keep increasing the HTT till memory clock as reported by Clockgen is close to 200Mhz. Drop the memory divider from within A64Tweaker a notch down once again to 150Mhz, 133Mhz, and so on each time the mem speed nears 200Mhz. Do note, you will have to close ClockGen and work with A64Tweaker and open it again after that for ClockGen to recognise the newer mem divider speeds.