I'm not a cosmologist, but that is almost a restatement of the dark matter hypothesis, except concentrating it into black holes. One problem is that black holes evaporate over time (Hawking). Another is that we've learned how to see them if they are anywhere near matter because the matter accelerating towards the black hole emits radiation (detectable).

The latest scholarship on galaxy formation is that the first galaxies appeared after about 100-200M years.

And then there's dark energy...

Lots to discover. We appear to be living at a time when that is still possible. In the far future, things may have expanded so much that we wouldn't be able to see much because it will be too far away, moving faster than it's light can ever get to us. We see that phenomenon now, distant objects winking out of our view.