What do think about the Department of Education? If we cut their budget and staff by say 90%, do you think the governors of the various states would run amok for lack of federal interference?

Of course not - but that's another example of what I'm talking about. Because virtually none of the DoE's money goes towards "federal interference."

Most of the Department of Education's budget (about 60%) is income-based student loans (FDSLE) and grants (Pell) for college. Nearly all the rest is giving money to poor schools (Title I grants) and paying for special education needs for disabled kids (about 31%). Not on oversight of the states. The DoE is a favorite target of budget hawks because they can create the perception that it's primarily federal bureaucrats messing around with matters of state control, but in reality nearly all of their budget is just transferring money to poor school districts, special education programs, or college loans/grants. The entirety of everything else they do is just 9% of their budget - which itself is the smallest of any federal agency.

en.wikipedia.org - United States Department of Education

So things like NCLB, Common Core, "Race to the Top," and other efforts to influence local school policies draw a lot of attention - but again, that's not where the money goes. So you can't get any material savings - and certainly not 90% - by getting them to stop "interfering" with governors. Almost all the DoE dollars are just transferred to state and local schools and colleges. As noted above, only about 9% of the DoE budget goes to everything else.

Objections to the DoE are political objections, not budgetary objections - the exercise of DoE's regulatory power and influence over state government educational system constitutes an insignificant amount of their budget.

Albaby