But the 4th says nothing about things crossing the border. Is this simply how it's been interpreted by legal decisions over the years?

Yes.

That's because the Fourth Amendment doesn't substantively require a warrant, or probable cause. The text of the Amendment is:

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

I've bolded the operative language. The Fourth Amendment protects you against searches and seizures that are unreasonable. That's it. Reasonable searches and seizures are allowed, unreasonable one are not.

What is "reasonable"? There's broad categories that are pretty obvious. If law enforcement has a warrant, they can search or seize pursuant to that warrant; if a police officer directly observes you committing a crime, they can seize you. There are other categories that are not blatantly obvious but are still obviously reasonable; if a police officer walking down the street sees a pile of drugs through the window of your car, or hears someone crying for help from a house that's on fire, they'll be able to seize the drugs or enter the premises to rescue the person ("plain view doctrine" and "emergency aid/exigent circumstances exception").

The border search scenario is one where the Court has pretty consistently held that it's reasonable for the government to require everyone crossing the border to be subject to possible search of their person and property. The government gets to/has to control the border, so if you want to bring in a suitcase or shipping container or a car, the government gets to search it for contraband or smuggled items or interdicted agricultural products or what have you. And you can't make them run and get a warrant for every such instance.

My understanding is that this position has never really been seriously disputed at the Court. There have been lots of disagreements about the extent that searches are reasonable (can they search you again fifty miles after you've gone through a border checkpoint? can they draw blood just because you cross the border?). But I don't think there's really been any serious question that the physical objects you are literally carrying with you across the border - your suitcase and personal effects - can be searched by Customs as you enter the country.