A decade after their first detections, we have entered into an era of detailed studies of brown dwarf atmospheres and exploratory studies of extrasolar giant planet (EGP) atmospheres. Aided by large spectral datasets for the former and, as yet, sparse photometry for a few of the latter, we have applied atmosphere models to characterize the properties of these objects. We find that while many processes affect their atmospheres and emergent spectra, clouds in brown dwarf and extrasolar giant planet atmospheres exert perhaps the greatest influence over their spectra. In fact at low spectral resolution, such as will likely be obtained by early space- or ground-based direct detections, the spectra of most EGPs are dominated by their cloud structure. However, clouds are simultaneously the most challenging component to model. Fortunately clouds also play a dominant role in the atmospheres of L-type brown dwarfs and cloud dynamics controls the transition from L- to T-type objects, thus affording an opportunity to gain insight from the comparison of models to data. In my talk I will discuss the role of atmospheric models in the direct detection and interpretation of extrasolar giant planets, survey my own group's modeling efforts, and discuss what we have learned from our application of cloudy atmosphere models to the observed spectra of brown dwarfs. I will conclude with a short survey of our work on the atmospheres of the hot Jupiters and a look to the future.