Abstract
Let $G$ be a separable locally compact group, $Z$ a closed subgroup contained in the center of $G$ such that center $(G)/Z$ is compact. An irreducible unitary representation $\pi$ of $G$ is said to be square integrable mod $Z$ (or a member of the discrete series of $G$) if there exist non-zero vectors $\varphi$, $\psi$ in the representation space $\mathfrak{G}(\pi)$ such that $\int_{G/Z}|(\pi(g)\varphi:\psi)|^2 dg < \infty$, where $dg$ is the right Haar measure of $G/Z$.
The problem of classifying semi-simple Lie groups and connected, simply connected nilpotent Lie groups having discrete series has been solved by Harish-Chandra, and J. Wolf and C. C. Moore. In this paper we shall consider the problem for another class of Lie groups whose radicals are connected, simply connected nilpotent Lie groups whose radicals are connected, simply connected nilpotent Lie groups (these are called $U$-groups). Thus let $G$ be a connected $U$-group, $N$ its radical, and $S$ a maximal connected semi-simple subgroup of $G$. Apart from some technical requirements on $S$ it will be proved that $G$ has discrete series if and only if:
(A) The center of $N$ is the connected component of the identity in the center of $G$, and
(B) Both $N$ and $S$ have discrete series. Furthermore every member of the discrete series of $G$ may be written as the “tensor product” of a member of the discrete series of $S$ extended trivially to $N$ with an irreducible representation of $G$ whose restriction to $N$ is a member of the discrete series of $N$. Finaly an algorithm to determine $U$-groups having discrete series as successive extensions of simpler $U$-groups is also described.
Roughly speaking, Mackey’s theory of group extensions ([10]) will be applied to solve the problem by induction. Thus let $H$ be a closed normal abelian subgroup of $G$, $\pi$ an irreducible representation of $G$. If $\pi|_H$ is defined by a transitive quasi-orbit with respect to the action of $G$ in $\hat{H}$, then $\pi$ is induced by some irreducible representation of the stability subgroup $G_0$ determined by this orbit. Here the assumption that $H$ is abelian is essential so that the Fourier analysis on $H$ can be successfully applied to relate the existence of the discrete series of $G$ to that of $G_0$. Moreover to carry out an induction argument, the class of Lie groups to which $G$ belongs must be chosen so that $G_0$ remains in the same class. Fortunately, thanks to the description of the structure of stability subgroups corresponding to open orbits of some semi-simple linear algebraic groups in [1], it will be proved that the class of $U$-groups satisfies all the above requirements. Therefore our program may be carried through completely.
The organization is as follows: in Section 1 some facts about square integrable representations and representation theory of group extensions are recalled. Theorem 1.3 is crucial for the proof of the main result. The structure of $U$-groups and stability subgroups corresponding to the open orbits of some semi-simple linear groups is described in Section 2, and Section 3 is reserved for the determination of square integrable representations of some special Lie groups whose radicals are isomorphic to the Heisenberg groups. Theorems 4.4 and 4.5 of Section 4 give the solution of the problem under consideration. Finally an example of a Lie group with discrete series in which the conditions of Theorems 4.4 and 4.5 do not hold is presented in Section 5.
The small underlined letters $\underline{g}, \underline{h},\underline{z},\cdots$ denote the Lie algebras of the Lie groups $G$, $H$, $Z,\cdots$ . The character group of an abelian locally compact $H$ is denoted by $\hat H$ and the subgroup of $\hat H$ consisting of all characters trivial on the closed subgroup $Z$ of $H$ is denoted by $(H/Z)^\ast$. Similarly the dual of a real vector space $V$ is denoted by $V^\ast$, and the subspace of $V^\ast$ consisting of all linear forms on $V$ vanishing on some subspace $W$ of $V$ is denoted by $(V/W)^\ast$.
The modular function of a locally compact group $G$ is denoted by $\Delta_G(\cdot)$. The representation space of a representation $\pi$ of $G$ is denoted by $\mathfrak{G}(\pi)$. Finally all Hilbert space scalar products except the natural scalar products of Euclidean spaces are denoted by $(\cdot \colon \cdot)$ without any indices. The context will make clear in which Hilbert space the scalar product is taken.
The author would like to express his gratitude to the Department of Mathematics of the Institute of Polytechnics in Hanoi for helping to prepare the article. He would also like to thank the referees for many helpful suggestions, especially for the shorter proof of Proposition 2.3.
