Sorry to be slow, but I'm not sure that I'm following the proposed approach. Are we trying to avoid the stocks that are likely to underperform or buy the stocks that are likely to outperform? Or some combination of both? Why not just buy the top 5% by ROE (or sales growth or earnings yield or something positively correlated with future return)?
...
I wonder if this is [yet another] case of a strategy, that once used by enough people, ceases to provide the "edge" (the higher returns).

These two points are fairly related.

First, a trivial answer to the first notion: the top 5% by ROE doesn't outperform the top 30% by ROE, so that's why I don't suggest it. This isn't a weakness of the correlation--the "top 30%" thing is extremely robust. It's just that it isn't a linear function. Stock performance, especially long term, does not have linear functions with the quant criteria. A nice ramping of performance among deciles is usually a sign of a vriable that won't get you good performance, more often a sign of a cute coincidence in the historical data. The investment world is more complicated than that.

The observation here is that the statistical correlation of ROE, though extremely reliable, is quite weak. If you only want to pick up a percent or two, it's easy: this half does better than that half, and always will. So stop there.

But if you need marketing to sell a new fund, you need more than that, so you look for the extremes, and claim an extreme advantage. If the top 1/3 is good, then the top 4% is even better, right!? No. More concentrated, more sure of yourself, more subject to front-running, and destined to failure.

The first approach is the one I was suggesting: a very small tilt, and because of its smallness and modest goals, we know it will always work. The half of firms with horrible ROE will always underperform the half of firms with good ROE, for sound economic reasons that will never end. To push a bit further, sure, I've found the third of firms with high ROE reliably outperform the 2/3 that are worse on that metric. But much beyond that, such claims weaken: it comes down to much more sophisticated stock picking, or more elaborate quant criteria...which can and will fail.

Quant investing isn't like combining a lot of linear equations to get the best on every axis. Markets are too heterogeneous for that. Rather, the really good quant performance is (to the extent they exist at all) in a large number of small pockets each bounded by a few variables. For example, rather counterintuitively, low P/E firms with SLOW growth do better than the average firm, even though slow growth by itself is inversely correlated with business results (quite separate from stock price performance--always know which one you're trying to predict).

The good news is that, to the second point quoted above, if you're willing to settle for the small edge of (say) the top 1/3, it won't be competed away. It's a simple fundamental of business that better businesses will (ON AVERAGE) have better results from their assets than poor businesses. Over the long run, that will show up in slightly better average stock returns. Emphasis on "slightly".

Jim