Regrettably, Google does not provide a simple way to directly download fonts. If you're looking for a specific font file, you can search through the git repository, but keep in mind that only TTF files are available on GitHub.
It's important to note that you should avoid linking directly to GitHub hosted fonts in your CSS. This is because GitHub serves the files with an incorrect mime type, causing potential issues in certain browsers.
Although there isn't a widely-used CDN that supports all formats, you can utilize to download the font files and host them on your own servers.