After some experimentation, I managed to figure out the solution on my own.
style={{outlineColor: 'red', outlineWidth: '1px', outlineStyle: 'solid', margin: '2px'}}
Instead of using style={{borderColor: 'red'}}
, I opted for outlined CSS styles which displayed the borders perfectly.
By applying the above style to every ToggleButton, all borders became visible.
<ToggleButtonGroup value={formats} orientation="vertical" onChange={handleFormat} aria-label="text formatting">
<ToggleButton value="bold" aria-label="bold" style={{outlineColor: 'red', outlineWidth: '1px', outlineStyle: 'solid', margin: '2px'}}>
<FormatBoldIcon />
</ToggleButton>
<ToggleButton value="italic" aria-label="italic" style={{outlineColor: 'red', outlineWidth: '1px', outlineStyle: 'solid', margin: '2px'}}>
<FormatItalicIcon />
</ToggleButton>
<ToggleButton value="underlined" aria-label="underlined" style={{outlineColor: 'red', outlineWidth: '1px', outlineStyle: 'solid', margin: '2px'}}>
<FormatUnderlinedIcon />
</ToggleButton>
<ToggleButton value="color" aria-label="color" disabled style={{outlineColor: 'red', outlineWidth: '1px', outlineStyle: 'solid', margin: '2px'}}>
<FormatColorFillIcon />
<ArrowDropDownIcon />
</ToggleButton>
</ToggleButtonGroup>
https://i.sstatic.net/3dYdl.png
I appreciate everyone's assistance in this matter!