It pledged to reduce emissions from areas like refrigeration and transport.
Critics said the pledge did not extend to the supermarket's supply network, which accounts for most of its emissions. However, Sainsbury's said it would be contacting its suppliers.
Chief executive Mike Coupe said the retailer would be writing to its suppliers to ensure "they are playing their part".
He added: "We need to understand further up our supply chains how we impact the environment and make sure our suppliers are working towards eliminating their carbon emissions as well."
Asked whether it would help carbon emissions if Sainsbury's sold more UK produce instead of food from overseas, he said: "We have to find a balance regarding 'food miles'.
"We have to bring our customers on the journey with us. It is more environmentally-friendly to grow products in environments that are suitable for those products.
"But equally there is a trade-off between doing that, our customers having availability of those products, and the way that we bring them to the market."
Currently, Sainsbury's produces one million tonnes of carbon each year, although, the supermarket said that figure had fallen by a third in the past 15 years.
Over the next two decades, Sainsbury's will spend an average of £50m a year on measures such as converting some of its vehicles to use alternative fuel and redesigning stores to be more energy efficient in order to achieve the goal.
The plan is one of Mr Coupe's final actions as the boss of Sainsbury's. Last week, he announced he would step down in May.
"We are committing to reduce our own carbon emissions and become net zero by 2040, 10 years ahead of the government's own targets, because 2050 isn't soon enough," he said in a statement.
He said the chain had spent £260m on measures such as energy-efficient lighting and refrigeration over the last decade.
The 2040 deadline targets emissions that Sainsbury's directly controls, such as heating and lighting.
"Direct operations is just the tip of iceberg," Dexter Galvin from charity, the Carbon Disclosure Project, told the BBC. "[Sainsbury's is] too late to the party. Tesco and Co-op have approved targets already including supply chain," he said.
While governments struggle to meet their ambitions on climate change, many businesses are putting words into action. In fact, there's something of a climate "arms race" between consumer-facing firms like supermarkets.
Sainsbury's promise of net zero emissions by 2040 is a full decade earlier than Tesco's target.
But the plan's not quite as impressive as it seems because so far Sainsbury's target applies only to its own emissions - NOT the much higher emissions of its suppliers. The full accounting will come later, the firm says.
There's a target on water too - Sainsbury's says it's reducing water usage by capturing rain on roofs, and recycling water from car washes and ice on fish counters.
But most radical is its initiative to halve plastic packaging by 2025. This is said to be very ambitious.
Experts warn, though, that in their rush to ban plastic, firms shouldn't use substitutes that are even worse for the planet.
Mike Childs, head of policy for Friends of the Earth, said: "The influence supermarket chains have over suppliers is also huge - they must use that to encourage better environmental standards while still ensuring a fair deal for farmers."
However, he added: "It's encouraging to see Sainsbury's stepping up to the plate on the climate emergency - the rapid transition to a net zero economy is urgently required.
"Supermarkets have a huge influence on our personal carbon footprints, so the more they can do to embrace and encourage greener lifestyles the better for us all."
Rival Tesco has said it plans to achieve net zero by 2050, 10 years later than Sainsbury's. However, it has also set targets for its suppliers demanding they reduce carbon emissions by 7% this year.
