To what extent was self-publishing a rejection of and challenge to traditional publishing models, and to what extent was it an alternative when no other options were available?
In the earliest period (pre-1840), self-publishing was practically the only option available to Black authors: Olaudah Equiano published by subscription; Phillis Wheatley had a patron; William Grimes, Boyrereau Brinch, George White, David Walker, and Venture Smith all self-published.
The rise of the abolition movement opened new doors, but it also made self-publishing more attractive for authors like Frederick Douglass and Sojourner Truth. Michaël Roy talks about this in Fugitive Texts: Douglass self-published his 1845 Narrative to reap the profits from his authoring and lecturing. And, though he had My Bondage and My Freedom published by a commercial press in 1855, Douglass made more money by self-publishing Narrative.
By the late nineteenth century, it was once again difficult to find a publisher for a Black-authored book, so self-publishing became the norm. This was true for autobiographers and for belletristic authors. In general, self-publishing was less a rejection of a mainstream option than it was a creative way to access the public sphere without the support of a publisher.
Are there examples of authors publishing work of a social or political nature that a traditional publisher would not have accepted?
The clearest example of this would be David Walker’s Appeal (1829), which never could have found a commercial publisher even if it were marketable since the political arguments in the pamphlet were too incendiary and radical. Ditto for someone like Norvel Blair, who had to leave his hometown to publish his narrative because he believed that no one would have even printed the pamphlet (let alone published it). Blair claimed that Republican politicians in his hometown paid off the newspapers so that they would not let him share his story.
Self-publishers produced texts that would never have been acceptable to publishers for other reasons, too, especially with regards to quality and audience. Though many self-published books sold well within limited geographic or temporal spaces, they were not the kinds of publications that would have been good bets for a publisher seeking a national or even a regional readership.