How Do I Upgrade My PS4 Games to PS5?
We're going to refer to the official PS5 page again for this. If you have a PS4 game as a physical disc, you will need to insert the game disc into the PS5 to see an upgrade path. You will then have to download the PS5 version and keep the PS4 disc to use and insert every time you play the game. Presumably, the PS4 disc will still work on your PS4 as well.

To take part in this upgrade, you will obviously need the PS5 console with a disc drive. That means no what version of a game you're upgrading, digital or physical, it will always result in a digital PS5 copy of the game.

If you own a digital copy of a game that can be upgraded, you can simply download the game on your new console, as the console and publishers will automatically know you've already purchased a license for the game you're looking to play, and provide you with a PS5 copy of the game. So, as an example, you could preorder Spider-Man Miles Morales on PS4 digitally, and then download the PS5 version onto your new console at no added cost at launch.  

PS4 Games with a Paid PS5 Upgrade Path
Unfortunately, not all games will be including a free upgrade path. Marvel's Spider-Man is available to play as a remastered version on PS5, but there isn't an upgrade version from the PS4 copy, and at the time of writing, it can only be accessed via the PS5 copy of Spider-Man: Miles Morales Ultimate Edition that will cost $69.99.

Moreover, Call of Duty: Black Ops Cold War has a slightly confusing upgrade path. But put simply, you'll need to pay at least $70 to get access to the PS5 version. Right now you have three options, the cross-gen bundle for $70, the PS4 version for $60, and the PS5 version for $70. If you buy the PS4 version, you'll be required to pay an extra $10 to get access to the game on PS5. To be fair, however, this is also the case with Xbox One to Xbox Series X.

NBA 2K21 is another game that isn't receiving much of a free upgrade path, as you'll only be getting an upgrade to the PS5 copy if you own the $96 special edition copy of the game, the Mama Forever Edition. You'd probably be better off waiting for the $69.99 PS5 copy if you're looking to play next-gen.

Edit: We've decided to move Control here, considering you'll be required to buy all the content again with Control Ultimate Edition in order to upgrade for free.
Paid Upgrades
CoD: Black Ops Cold War
Control Ultimate Edition
Marvel's Spider-Man Remastered
NBA 2K21