Diesel trains and cabs were blamed after Friends of the Earth highlighted the rank as one of the worst places for harmful pollutants - above four London locations.
It came second in a top ten of sites with the highest annual average nitrogen dioxide levels.
Ibrar Hussain, GMB branch secretary for the taxi trade and a driver himself, says taxis aren't the only problem.
He said: "The layout at the station is a major contributor to the emissions. It's a very poor design and is chaotic.
"The vast majority of times traffic is at a standstill, especially at busy periods. The emissions are from traffic chaos but there are other vehicles such as passenger cars entering the rank area so you cannot blame taxis alone.
"Drivers are very conscious of the problems of working at the station, but no one listens. It cost millions of pounds to design the area, but no one thought of traffic layout coming in and out, especially the location of the rank."
Mr Hussain said Sheffield Council needed to "think outside the box".
"I have been working on the station for more than 30 years and the traffic situation has never been so bad. The whole infrastructure and layout is wrong and needs a rethink.
"All around Suffolk Road, Sheaf Street and inside the station area needs looking at and changing."
Coun Bob Johnson, Cabinet member for transport at Sheffield Council, said one of the most significant ways of improving the air quality at the station was the electrification of the Midland Main Line and the replacement of the diesel trains.
He added: "We are also planning for the introduction of a Clean Air Zone, which would also go towards reducing pollution throughout the city by incentivising drivers to buy cleaner vehicles.
"The zone would charge the most polluting diesel vehicles, while offering grants and support for drivers who chose cleaner vehicles, such as electric taxis."