Test with a known-good H.264 file (e.g., from sample-videos.com ). If only your file fails:
Go to Help > Check for Updates on Windows, or VLC > Check for Updates on macOS. vlc could not decode the format h264