Papa John's founder John Schnatter is punching back after resigning from his company's board last week in disgrace over racially insensitive comments he made during a "diversity media training" conference call with marketing firm Laundry Service.

In a Saturday letter to the board of directors, Schnatter claims that Laundry Service tried to blackmail the pizza chain for $6 million to keep quiet about his use of the N-word during an answer to a question over whether he's a racist. He also says that he refused to work with Kanye West over the artist's use of the N-word. 

Schnatter resigned his post last week at the behest of Papa John's board after confirming the Forbes' report. However, he said the comments were taken out of context and that he was provoked into using the N-word after Laundry Service executives on the call suggested the pizza chain bring on performer Kanye West as a co-spokesman for television spots and promotions.

Schnatter said he refused to work with West because "he uses the 'N' word in his lyrics," according to the letter. It was later on the call that Schnatter said he used the actual word when pressed whether or not he was a racist. -CNBC

"I then said something on the order of, Colonel Sanders used the word 'N,' (I actually used the word,) that I would never use that word and Papa John’s doesn’t use that word," Schnatter told the board. "Let me be very clear: I never used the 'N' word in that meeting as a racial epithet, nor would I ever."

During the market call, Schnatter also reflected on his early life in Indiana, where he said people used to drag African-Americans from trucks until they died. He apparently intended for the remarks to convey his antipathy to racism, but multiple individuals on the call found them to be offensive, the source said.

Schnatter reiterated his side of events in the letter, stating that Laundry Service was brought in to conduct a "diversity media training" session ahead of a conference he was attending.

He said Papa John's fired Laundry Service the day after the call took place, but still owed the media agency about $1.3 million. Schnatter said the agency requested $6 million "because they claimed some of their people had been offended by what I had said," he wrote.

One their attorneys threatened "a smear campaign" if Papa John's didn't pay up, Schnatter said, adding that the company ultimately paid Laundry Service $2.5 million. -CNBC

"I will not allow either my good name or the good name of the company I founded and love to be unfairly tainted,” Schnatter told the board - which he criticized for forcing his resignation as chairman before conducting a detailed investigation into a Wednesday Forbes article detailing his use of the N-word.

“The board asked me to step down as chairman without apparently doing any investigation," Schnatter said in the letter obtained by CNBC. "I agreed, though today I believe it was a mistake to do so. I have checked with corporate governance experts who tell me that this was not a proper action by the Board."

Glaser, Schnatter's lawyer, warned the board in a separate letter sent Sunday against removing him altogether after several directors questioned whether he should give up his seat, she said. Glaser told CNBC the board doesn't have authority to remove him without shareholder approval.

That would be a tough vote to win since Schnatter owns almost 30 percent of the outstanding shares, according to FactSet. Glaser asked the board to open an investigation into the call and subsequent events. -CNBC

On Sunday the pizza chain announce that Schnatter was prohibited from speaking to the press, and that in addition to being removed from promotional materials, he had his office space revoked at its Louisville, Kentucky, headquarters.