The result of the vote was 42-98.
Prime Minister Julia Gillard and Opposition Leader Tony Abbott sat on the same side of the chamber to vote against the private bill, proposed by Labor backbencher Stephen Jones.
Labor allowed its members a conscience vote on the issue - meaning they did not have to all vote in the same way - but the coalition did not.
Former prime minister Kevin Rudd also voted to defeat the legislation.
Ministers who voted for the bill included Peter Garrett, Nicola Roxon, Jenny Macklin, Anthony Albanese, Greg Combet, Tanya Plibersek, Kate Ellis, Bill Shorten, and Gary Gray.
Independent MP Tony Windsor voted against the bill while Rob Oakeshott and Andrew Wilkie voted for it. Earlier, Australian Greens MP Adam Bandt tried to stop the parliament taking the vote.
He said there should be no decision on the matter until coalition members were given a conscience vote.
"I fear that we are about to witness a very cynical move to put this vote quickly in an attempt to clear it off the political agenda," he told parliament.
A failed vote on Wednesday would not deter the Greens from continuing attempts to remove discrimination from the Marriage Act.
Leader of the House Anthony Albanese rejected the suggestion the vote was somehow a surprise to members. He said that opposing even having a vote did nothing to advance the cause.
The House of Representatives is also considering another bill to legalise same sex marriage, sponsored by Greens MP Adam Bandt and independent Andrew Wilkie.
A sad day in Australia....... the so called "Lucky Country" ..... I don't feel very lucky today...........