Say you've been running an MI portfolio in a separate account for a while. What's a good definition and metric of success?
Anyone ever thought about that?

It's a tough question to quantify. Outperforming a reasonably chosen benchmark over a long period of time is the acid test, but it isn't a black and white one. How long do you wait? The longer you wait, the less likely the outperformance is due to chance, so it's more of a statistical measure of true value added. Outperforming for ten years is a lot more certainly due to skill than outperforming for one year. But there are still some people who say Warren Buffett has just been lucky. Nobody doubts Elan's 6/3 option portfolio though, it's older than some board members : )

One metric I have toyed with is this:
First, if the end-to-end performance lags the chosen benchmark, it's a fail. Sorry.
Second, the time period has to be long enough to include a bear market. And some bull times too. If not, don't try to see if it working, the answer won't be reliable. Wait a while.
Third, if and only if it looks better than the benchmark from inception to date:
Calculate the performance of the portfolio in all the rolling 6 month periods, as well as the performance of the benchmark in those same period. (Each period might be a month or quarter or half-year or year or two-year stretch...it doesn't really matter, but there should be at least 10 of them). Feel free to pick the interval that makes things look best.*

Some sort of metric of MI success could be seen in the following two numbers:
* The end-to-end CAGR advantage versus the benchmark, the measure of monetary success without regards to whether it was luck or skill;
* The percentage of those rolling subperiods that the portfolio outperformed the benchmark: an attempt to create a metric of how certain you are that the performance was non-random, not just a few lucky picks.

That notion is really only suited to a long-only portfolio. It wouldn't work with a portfolio doing long/cash timing, for which you'd want something more like measuring the improvement in the rolling year downside deviation with a hurdle rate (MAR). I usually test my screens based on two numbers, being "CAGR improvement" and "risk fraction", i.e., what the DDD3 risk metric is when divided by the DDD3 risk metric of the benchmark in the same interval.

Does anyone have any better thoughts? At what point can one be reasonably sure that good performance wasn't just the luck of the draw?

Jim

* why is this OK?
If you pick really short intervals the "beat the index" rate will tend towards 50%, making things look bad...though you do get lots of statistical support. If you pick really long intervals, getting towards the length of the whole history, the statistical support is terrible because of the high overlap but it's such a long period that it's really starting to measure your end to end CAGR...which is, in the end, the only thing that matters, so one can let it slide.