Optimization on the Quaternion Stiefel Manifold with Applications—Part I: Basic Geometry

The quaternion Stiefel manifold, denoted by $${{\,\textrm{St}\,}}_{\mathbb {Q}}(n,p)$$ St Q ( n , p ) , is the set of $$n\times p$$ n × p partially unitary quaternion matrices. Optimization problems on $${{\,\textrm{St}\,}}_{\mathbb {Q}}(n,p)$$ St Q ( n , p ) arise in several areas, including color image processing, numerical quaternion linear algebra, and airborne direct georeferencing. This work, with its two parts, is aimed at developing a Riemannian optimization approach to problems over $${{\,\textrm{St}\,}}_{\mathbb {Q}}(n,p)$$ St Q ( n , p ) and presenting its application to robust dimension reduction of quaternion data. In this part, we focus on the relevant geometric tools. We first study the basic geometry of $${{\,\textrm{St}\,}}_{\mathbb {Q}}(n,p)$$ St Q ( n , p ) ; in particular, its relations to the complex and real Stiefel manifolds are established. Then, formulas for tangent space, Riemannian gradient, Riemannian Hessian, various retractions, and three types of vector transports are derived. With these tools, Riemannian optimization algorithms can be adapted to the quaternion domain. Application to a novel model of quaternion principal component analysis based on $$\ell _1$$ ℓ 1 -norm is given in the second part.