Motivated by the properties of the iron chalcogenides, we study the phase diagram of a generalized Heisenberg model with frustrated bilinear-biquadratic interactions on a square lattice. We identify zero-temperature phases with antiferroquadrupolar and Ising-nematic orders. The effects of quantum fluctuations and interlayer couplings are analyzed. We propose the Ising-nematic order as underlying the structural phase transition observed in the normal state of FeSe, and discuss the role of the Goldstone modes of the antiferroquadrupolar order for the dipolar magnetic fluctuations in this system. Our results provide a considerably broadened perspective on the overall magnetic phase diagram of the iron chalcogenides and pnictides, and are amenable to tests by new experiments.