The $A_\infty$ de Rham theorem and higher holonomies