Ms Dorries was sacked as culture secretary by Mr Johnson’s successor, Liz Truss.
But in a blunder she was later given the payment, the equivalent of three months’ salary in lieu of notice.
She was not eligible for the payout, however, which according to the rules should only be given those under 65. Ms Dorries had turned 65 several months earlier.
Ms Dorries promised to pay the money back "on Monday morning" as she said she was only made aware of the mistake on Friday.
And she joked that as a result “everybody knows I'm not 49".
Government accounts show she received £16,876 as an exit payout as she departed the cabinet.
Ms Dorries later quit as an MP in protest at her exclusion from Mr Johnson's resignation honours list.
She blamed Rishi Sunak for her lack of a gong and triggered a by-election in her Mid Bedfordshire seat.
Wiithin weeks Labour had overturned a huge Tory majority to win the constituency.
The Mirror revealed earlier this week that disgraced ex-MP Peter Bone had also wrongly been paid almost £5,600 after he spent just 82 days as the deputy leader of the Commons under Mr Johnson. He was 69 at the time.
Baroness Stedman-Scott also got £17,442 when she stood down as a Department of Work and Pensions minister in 2022. She was 67. And Sir David Evennett also received £4,479 when he left his role as a Government whip in October 2022, at the age of 73.
Labour has pledged to reform the rules on ministerial severance pay if it wins the next election this year after it emerged that last year's political turmoil cost the taxpayer nearly £1 million.
Cabinet Office minister Alex Burghart said earlier this month that the payments “should not apply where a person has attained the age of 65.
"If a former minister was incorrectly given a severance payment, the relevant department will contact the individual to recover the over-payment."