I don't think you miss anything ultra critical if you miss S5 except of course for Sleeping in Light which is mandatory. S5 contains the beginning of the telepath war, the fall of the Centauri empire, and Penn and Teller.
Season 5 was the "winding down" of the series and it showed the characters embarking on some new paths in life, while other characters would take over such as Vir becoming the Ambassador for the Centauri to B5, G'kar's replacement, Captain Lockley taking over the station, etc.
It also had the beginning of the Telepath War (they REALLY needed to make a movie about that), the ascension of Londo to Emperor of Centauri Prime and the start of the fate that would await him, and more.
I highly encourage the reading of the following B5 novels as JMS has stated they are canon and are based on some of his notes:
Babylon 5 novel #7: this novel shows the backstory of Sheridan's wife, Mr. Morden and how the Icarus found the planet of the Shadows and what happened.
Babylon 5 novel #9: this is the story of Sinclair as he settles into his position as Ambassador of Earth on Mimbar and how he becomes Ranger 1 and starts to rebuild the Rangers. It also brings back his fiance from Season 1 of the show.
The Passing of the Technomages Trilogy of novels: want to know the origins of the Technomages? Want to know how deep a game Elric was playing with Londo in Season 2? Want to know about the origins of Galen the mage that gathered everyone to fight the Drakh super weapon in the tv movie and who would appear in the Crusade spin off? This is the trilogy for that.
The Telepath Trilogy: Want to know the origins of Psi Corps and why it was created? Want to know the tragic origin of Bester? Want to know Bester's life story a bit? The first two novels take care of that and the third is after the Telepath War and Bester's fate is revealed.
Legions of Fire: this trilogy covers the reign of puppet Emperor Molari and how Vir must save his world.
Yes, that's right. I said VIR must save his world.
However this scene from Season 5 shows that perhaps Vir does have a bit of anger in him after all....
https://www.youtube.com/watch?v=t8MjQ5Z7ZNo