Good point. As pointed out on the rest of this list, you should be very careful about "data mining", where simply finding a strategy that works on historical data may not be replicable. Also, some strategies work well in some markets (like high volatility) and poorly in others. I think a next level of analysis is trying to figure out when to turn particular strategies on or off.
As for the attacks on technical analysis, many practitioners probably are deceiving themselves that they've found some secret sauce. However, since the market is made of 100% of the people who transact in it, the presence of technical traders means that there must be some price movements as a result of these trends. One ideal would be to find a strategy that beats other technical analysts to the punch. A lot of people use the 12/26 MACD, but maybe the 11/25 MACD would help you eat their lunch before they entered the market...
It is all very tricky and takes a lot of discipline to avoid fooling yourself and getting into real trouble. Also, people need to make sure to focus on both sides of the trade - you only book the profit once you've actually sold the position (most people seem to be much more focused on entries than exits)
If anyone is interested in this kind of software but aren't so into learning a new language (perhaps less applicable to this group), you should check out QuantRunner Software http://www.quantrunner.com. In full disclosure, I am the CEO of the company, but we really focus on making this kind of back testing analysis easier for people to do without programming experience. We also have some novel tools for helping people actually improve their strategies, rather than simply iterate tests. If I had the true answers on the best strategies, I probably wouldn't run a software company. All we hope for is to make it easy enough for people to do their homework without having to worry about coding errors as well as poor strategies.
As for the attacks on technical analysis, many practitioners probably are deceiving themselves that they've found some secret sauce. However, since the market is made of 100% of the people who transact in it, the presence of technical traders means that there must be some price movements as a result of these trends. One ideal would be to find a strategy that beats other technical analysts to the punch. A lot of people use the 12/26 MACD, but maybe the 11/25 MACD would help you eat their lunch before they entered the market...
It is all very tricky and takes a lot of discipline to avoid fooling yourself and getting into real trouble. Also, people need to make sure to focus on both sides of the trade - you only book the profit once you've actually sold the position (most people seem to be much more focused on entries than exits)
If anyone is interested in this kind of software but aren't so into learning a new language (perhaps less applicable to this group), you should check out QuantRunner Software http://www.quantrunner.com. In full disclosure, I am the CEO of the company, but we really focus on making this kind of back testing analysis easier for people to do without programming experience. We also have some novel tools for helping people actually improve their strategies, rather than simply iterate tests. If I had the true answers on the best strategies, I probably wouldn't run a software company. All we hope for is to make it easy enough for people to do their homework without having to worry about coding errors as well as poor strategies.