The Marmaray tunnel is the world’s first connecting two continents, and is designed to withstand earthquakes.
It was inaugurated on the 90th anniversary of the Republic of Turkey.
Turkish Prime Minister Recep Tayyip Erdogan has for years championed the undersea engineering project, first conceived by an Ottoman sultan in 1860.
Work began in 2004, but archaeological excavations delayed construction.
The underwater section runs for 0.8 miles (1.4 km), but in total the tunnel is 8.5 miles (13.6 km) long.
Japan invested $1bn of the $4bn (£3.4bn) total cost of the project, named Marmaray, which is a conflation of the nearby Sea of Marmara with “ray”, the Turkish word for rail.
The BBC’s James Reynolds in Istanbul says the Turkish government hopes the new route under the Bosphorus will eventually develop into an important trading route.
In theory it brings closer the day when it will be possible to travel from London to Beijing via Istanbul by train.
The Marmaray project will upgrade existing suburban train lines to create a direct link joining the southern part of the city across the Bosphorus Strait.
Istanbul is one of the world’s biggest cities, with about 16 million people. Some two million, according to the AFP news agency, cross the Bosphorus every day via just two bridges, causing severe traffic congestion.
The Marmaray has been dubbed “the project of the century” by Turkey
The rail service will be capable of carrying 75,000 people per hour in either direction.
“While creating a transport axis between the east and west points of the city, I believe it will soothe the problem” of congestion, said Istanbul’s mayor Kadir Topbas.
But critics of Prime Minister Erdogan have seen the tunnel as one of his grandiose construction projects for the city where he used to be mayor.
Detractors of his proposals, including a third airport, a parallel canal, a third bridge over the Bosphorus and a second tunnel – for cars, south of Marmaray – say they illustrate Mr Erdogan’s “pharaonic” ambitions.
Authorities came under fire earlier this year when protesters opposed plans to redevelop a park in Istanbul. Widespread violence between anti-government demonstrators and security forces ensued.
The rail tunnel is still not fully operational after Tuesday’s opening, AFP reports.
“The part that is in service is very limited. All that has been delayed until much later,” said Tayfun Kahraman, president of the Istanbul Chamber of Urban Planners.
“We are wondering why this inauguration is happening so soon.”