I saw the SSAO artifact you mention, but I don’t find it much distracting.
About the other artifacts, this is something AF can barely (if at all) fix. In fact, hardware texture filtering (interpolation/mipmap generation/AF…) is linear, and it works well when textures contain just colors (or linearly filterable data). Normals cannot be linearly filtered and on top of that, the real problem, colors fetched from those normals don’t have a linear behavior at all. These are the cause of aliasing/moiré/etc.
I have found a nice presentation (Spectacular Specular) that talks about this typical issue, with nice examples, and with all of the theory I used to make a demo program.
To sum up, the basic idea is instead of working with normals or means of normals (a normal fetched from a coarse mipmap is the mean of those who are on bigger mipmaps, this is already very different from the mean of the colors computed from each top-level normal. Anisotropic filtering only helps in this case for tilted surfaces, those who are parallel to the screen are not affected by GPU AF algorithms), we work with local distributions of normals. A gaussian distribution for example only need the mean value (same as before) and a variance (or variance matrix). We can tweak this representation to have data that are linearly filterable, and with some approximations, we only need 3 floats to represent a local distribution of normals, which fits in one FP16 texture. Replacing one normal by a distribution of normals allows us to work as if we have a cone of directions, instead of just one averaged direction. We can work directly with gaussian formulas, or simpler, derive a radius and estimate the footprint of the cone in the environment map, then we only have to fetch a texel that match approximately this footprint (we just select the appropriate mipmap level). The final result is close to the average of the colors computed from each individual local normal, and thus, aliasing and “surface flattening” disappear almost completely.
I know this may not be easy to fully understand what happens, and it may seems difficult to implement or require heavy computations. In facts, the final code is really straightforward, only requiring small modifications to the existing. The biggest penalty is that instead of using a 3 components INT8 texture for normals, you need a 3 components FP16 texture, so twice the memory footprint (the conversion is easy to do, it can be done on load time so you don’t have to change your existing normal maps. And it is fast, it can easily be done at each frame). But you don’t rely on AF anymore to reduce artifacts, you only need it to get more details.
Here is the links to the demo program, and source code. I hope you don’t have an ATI card because I don’t, so I can’t check if it works also on Radeons. This demo shows a very difficult surface to render properly, and you can play with mipmaps, anisotropic filtering, brute force oversampling, and this anti-aliasing “trick”, and see precisely the performance implications.
For the source code, I’m sorry I know that this is not what you are using, it’s a Code::Blocks workspace and use SDL and OpenGL (with GLSL shaders). With what I give you only have to install Code::Blocks and TDM-GCC if you want to compile this though (but I know I shouldn’t ask you to spend time on stuff like that). I made the effort to make things the clearest possible, and the most important things are on the two pixel shaders genleanmap.frag (conversion of classic normal map to a lean map) and bump.frag (the LEAN mapping).
Demo
Sources
I hope you will find it interesting, because it can have a great impact on Automation actually, and it really is a minor modification to do.
And OK for the windows, I would be glad to see the rest of the game