Why this dark night of the soul, in particular this dark night of the spirit? We are told, by many contemplative saints and writers, it is because the soul needs to be purified completely before it can attain to the Spiritual Marriage, and the ego, the "false self" which has been layered on year by year, must "die". The Lord does this through his Light, through His Divine purifying fire of Love. The soul is overwhelmed by this Light; it is blinded, and can only move forward in faith. The soul cannot perceive the Lord's presence, but He is there, very close by.
"This is the reason the soul's suffering for God at this time is so intense: she is drawing nearer to Him, and so she has greater experience within herself of the void of God, of very heavy darkness, and of spiritual fire which dries up and purges her, so that thus purified she may be united with Him....He is intolerable darkness to her when He is spiritually near her, for the supernatural light darkens with its excess the natural light."
St. John of the Cross
(The Spiritual Canticle)
"We have to go through this painful experience to come to our true self. It is a harrowing journey, a death to self - the false self - and no one wants to die. But it is the only path to life, to freedom, to peace, to true love."
M. Basil Pennington, O.C.S.O.
(A Place Apart. Monastic Prayer and Practice for Everyone)
"The Night of Spirit feels like dying. But it isn't death. It is a liberation from the tyranny of the false self. It is the necessary preparation for the full transmission of divine light, life, and love. Lazarus as a symbol of Christian transformation is very close to us, and indeed is us in moments of deep purification, especially when that experience is prolonged and we feel as though there is no hope that the night will ever end. The Night of Spirit in particular is extremely searching. The Divine Therapist lovingly moderates the intensity of self-knowledge according to our state of life and capacity. Mary sitting at the feet of Jesus is only beginning the contemplative journey. Contemplation is not the reward of virtue. It is a necessity for virtue. It leads to the experience of the presence of God in pure faith. God then withdraws the divine presence seeming to abandon us in the tomb, as it were. God returns at the appropriate time to call us forth from our darkness, confinement, loneliness, dereliction, and grief."
Thomas Keating
(The Better Part. Stages of Contemplative Living)