The messages included racist and sexist comments and apparent antisemitic remarks, reports suggest.
Gwynne has also been suspended by the Labour Party as it investigates the statements, which were leaked to the Mail on Sunday newspaper.
A spokesperson for the Government said: 'The prime minister is determined to uphold standards of conduct in public office and lead a government in the service of working people.
'He will not hesitate to take action against any minister who fails to meet these standards, as he has in this case.'
In response, Gwynne said on social media: 'I deeply regret my badly misjudged comments and apologise for any offense I've caused. I've served the Labour Party all my life and it was a huge honour to be appointed a minister by Keir Starmer.
'I entirely understand the decisions the PM and the party have taken and, while very sad to have been suspended, will support them in any way I can.'