We study the $L^2$-convergence of two types of ergodic averages. The first is the average of a product of functions evaluated at return times along arithmetic progressions, such as the expressions appearing in Furstenberg’s proof of Szemerédi’s theorem. The second average is taken along cubes whose sizes tend to $+\infty$. For each average, we show that it is sufficient to prove the convergence for special systems, the characteristic factors. We build these factors in a general way, independent of the type of the average. To each of these factors we associate a natural group of transformations and give them the structure of a nilmanifold. From the second convergence result we derive a combinatorial interpretation for the arithmetic structure inside a set of integers of positive upper density.