The structure of marine phytoplankton communities in the North Atlantic Ocean varies considerably on seasonal, interannual, and longer timescales in response to environmental change. However, the causes of ecological variability on interannual and longer timescales remain uncertain. Here, using a half-century of observations, we compare changes in atmospheric forcing (surface wind speed and heat fluxes) and ocean surface properties (sea surface temperature, mixed layer depth, thermal stratification, and turbulent kinetic energy) with variability in total phytoplankton biomass and the abundances of diatoms and dinoflagellates, as measured by the Continuous Plankton Recorder survey. On seasonal timescales, there is a clear connection between observed changes in the physical environment and the phytoplankton assemblages. Strong turbulence, deep mixed layers, and weak stratification decrease diatom abundance in the subpolar gyre, but increase diatoms in the subtropical gyre, a pattern broadly consistent with growth limitation of phytoplankton in high and low latitudes by light and nutrients, respectively. In contrast, dinoflagellates prosper in stratified, weakly turbulent conditions in sampled portions of the subpolar and subtropical gyres. On interannual to multidecadal timescales, however, the links between observed ecological and physical changes are much weaker. The physical mechanisms that differentiate the fates of diatoms and dinoflagellates on seasonal timescales do not appear to control their longer-term variability, perhaps because year-to-year variability in the phytoplankton assemblages is greater than in the physical drivers. This suggests that other biological (e.g., zooplankton grazing, chaos in the plankton) or physical mechanisms (e.g., changes in ocean circulation) may play important regulatory roles.