Richards was born in 1903 in the village of Dunvant, near Swansea, the son of Thomas Coslett Richards and Sarah Richards (born Jones). He and his younger brother and sister, Owen and Esther, were brought up in a highly cultured, working-class environment. His mother came from a family of craftsmen; his father, an employee of a tinplate foundry in Gowerton, was active in the local church, wrote poetry in Welsh and English and for many years conducted the Dunvant Excelsior Male Voice Choir. All three children were taught to play the piano, and became familiar with the works of Bach and Handel in the cycle of Christian celebration. In later years music would be an important stimulus to Richards's painting - as would his youthful sensitivity to the landscapes of Gower and the cycles of nature.At Gowerton Intermediate School he drew constantly and won local competitions. When he left school to become apprenticed to a firm of electricians in Swansea, he devoted his evenings to studying engineering draughtsmanship at Swansea College of Technology and drawing at the Swansea College of Art (now part of Swansea Metropolitan University). In 1921, at the age of 18, Richards enrolled full-time at the Swansea College of Art, then under the direction of William Grant Murray. During his time at the College he spent less time in painting than in drawing from classical casts and studying industrial design and graphics. The strongest impact on him during these years appears to have been the week's summer school in 1923, which he spent under the direction of Hugh Blaker at Gregynog Hall, the country house of Gwendoline and Margaret Davies, where he first saw the canvases of Renoir, Van Gogh, Monet, Cézanne, Corot and Daumier, the sculpture of Rodin and sheets of old-master and modern drawings. The experience confirmed him in his vocation; and in the same year he applied for, and won, a scholarship to study in London at the Royal College of Art. Richards entered the Royal College of Art in 1924.Afterwards Richards spent most of his life in London, apart from a period teaching art in Cardiff, where he was head of painting at Cardiff School of Art during World War II. In 1929 he married Frances Clayton, a fellow artist. They had two daughters - Rachel (born 1932) and Rhiannon (born 1945). Rachel married the paleontologist Colin Patterson.