Zerge was born in Oppmanna, Kristianstad in 1894. He grew up at the smallholding Kullesta in Arkelstorp, surrounded by a vast and untouched garden. His father, Nils Svensson, was a cantor, bell-ringer, and portrait painter. His father's passion for portrait painting inspired Zerge to move to Stockholm after his studies to open his own studio in conjunction with his art studies at the Royal Academy of Fine Arts.

After his study period and artistic career in Stockholm, Zerge spent some time in Italy where he cultivated an interest in gardening and converted to Catholicism. He also lived in France for a while, where he met the variety singer and actor Maurice Chevalier. As Zerge was an excellent folk singer, they quickly found common ground and often sang together in social settings.

In 1931, Zerge bid on his childhood home, Kullesta, at an auction. He moved back home and lived collectively with his two older sisters. He designed and created a beautiful garden inspired by his time in Italy. His sisters ran the local bank, and many banking transactions also took place at Kullesta. Zerge was both a Freemason and a frequently engaged master of ceremonies. It is said in the local community that he was popular both as a person and as an artist. His older sisters are said to have spoiled their much younger brother. Zerge made a good living from portrait paintings but also painted floral motifs. Zerge was closely associated with his cousin, the sculptor Nils Möllerberg, three years his senior, whose family the portrait from the auction originates from.