Our publication describing ADMIXTURE's algorithm is
D.H. Alexander, J. Novembre, and K. Lange. Fast model-based estimation of ancestry in unrelated individuals. Genome Research, 19:1655–1664, 2009.
and we encourage you to read the preprint pdf. If you use ADMIXTURE, please cite this publication in your paper.

The quasi-Newton convergence acceleration method is described in more detail in

H. Zhou, D. H. Alexander, and K.  Lange. A quasi-Newton method for accelerating the convergence of iterative optimization algorithms. Statistics and Computing, 2009.
for which a preprint is also available.