We report a transmission electron microscopy (TEM) study of the impacts of phosphorus and boron passivation processes at 4H-SiC/SiO$_\sf{2}$ interfaces. The chemical and electronic structures at these interfaces have been analyzed using high-resolution TEM and spatially-resolved electron energy-loss spectroscopy (EELS), uncovering a range of phenomena caused by the presence of B and P within their respective boro- and phosphosilicate glass (BSG/PSG) layers. The phosphorus passivation process was observed to induce roughness at the SiC/PSG interface on the order of 100s of nm. Within the PSG layer, phosphorus was found to segregate into nanometer-scale P-rich clusters, contradicting previous reports that it is distributed uniformly throughout the PSG. Similar to N in nitric oxide annealed devices, boron was determined to accumulate in a thin layer (sub-3nm in thickness) at the SiC/BSG interface, with a much narrower distribution than previously reported. EELS measurements indicated boron incorporates in a trigonal bonding configuration, supporting the assertion that it softens the oxide and causes significant stress reduction at the interface with 4H-SiC. These results supply further insight into the sources of mobility enhancement in PSG and BSG-gated devices that could be extended into additional improvement in the channel response of SiC MOSFETs.