The bigger Berkshire becomes and average returns drift lower will it become a self fulfilling prophecy that the slower trajectory of growth will cause it to trade at lower valuation multiples of for example PBV over the next 10 / 20 years.

Sure.

But remember: their capital allocation skills will presumably fade gradually from excellent towards typical, not from excellent to useless.

The average US stock in the average year has historically returned around inflation + 6.5% on purchase price.
i.e., that's what a monkey with a dartboard might expect in a year that the stock market is not overvalued.

So, I see Berkshire's future rate of growth in value per share converging slowly down towards that number--let's be conservative and round down to 6%.
When they reach that number they will be no better than an average pick.
But inflation + 6% is not so bad.

At that point, the only reason Berkshire would be a good pick is that it's frequently cheaper than the broad market.
And it doesn't contain as many odious business units, depending on your tastes.

None of this suggests that the fair P/B for Berkshire will drop.
That will be determined primarily by how they allocate their incremental capital.
Their stocks and bonds will deserve a multiple of 1.
Wholly owned businesses units doing capex will deserve a higher multiple.
(median P/B among S&P 500 firms is 3.5 these days. Maybe a bit rich, but they generally deserve much more than 1)
Whatever mix Berkshire ends up with will give fair value somewhere in the middle.
These days, somewhere in the 1.5 range seems to work.

The simple way to know what the "fair" P/B should be? Watch the trajectory of ROE.
If ROE is sliding on trend over time, the fair P/B should be lower. And vice versa.

Jim