I don't use "hands off" setup capture programs so where an installer is so crappy I can't reliably wrap it, I generally try reverse engineering the installation process. You generally put enough thought into this process to eliminate the most likely setup capture related issues.
A large number of products are simply a bunch of files plus some registry and so are simple to handle. Java applications are examples where little use of "fancy windows features" are likely.
These are the general steps:
The "SysInternals" site is one of my favourite sites and has many free tools (such as "FileMon.exe" and "RegMon.exe") which can help identify install activity and diagnose issues such as missing files and registry.
Monitoring without any tools other than a comparison tool ("WinMerge.EXE" or perhaps your eyes!) is possible. Registry can be exported in text format (its sorted) with many tools including "REGEDIT.EXE" and a sorted file listing can be created with "dir c:\ /s /on /a >somefile.txt". My "IisDumpMetaBase.VBS" can produce a text file representing all or part of an IIS metabase. Do not ignore the possibilites of text comparison! In some situations this is actually the easiest approach!
While this is similar to "setup capture" the real difference is that you fully understand the requirements of the install and so chances of problems are minimised and you don't have to take the full risk of recreating a full setup on a new release of a product (the installation details don't tend to change much between releases).