Modi also made an unscheduled trip to a village, Chango, and said he was "deeply touched by the impromptu reception and their joy".
Dressed in olive green and donning a local cap with a green flap adorning dry flowers, the Prime Minister interacted with men from the Indo-Tibetan Border Police (ITBP), the Dogra Scouts and the Army at Sumdoh.
Sumdoh, some 330 km from Shimla, is located on the border of Kinnaur and Lahaul-Spiti districts.
Official sources said the Prime Minister reached around 11 am and spent more than three hours at Sumdoh and Chango.
Mingling freely with the assembled soldiers and paratroopers, Modi went up to them holding a plate of sweets. At least one soldier offered him a piece of sweets in return, much to the joy of everyone.
He was accompanied by National Security Adviser Ajit Doval and the army chief, General Dalbir Singh Suhag.
The Prime Minister also met personnel of the General Reserve Engineering Force (GREF), a wing of the Border Roads Organisation (BRO) that maintains highways, at Sumdoh, an official said.
Modi then spent some time with the locals in Chango village in Kinnaur district, known for its delicious apples.
Villagers were taken aback by the Prime Minister's arrival, and then raised slogans hailing him. Dressed in warm clothes, Modi spent time with the residents of the village, including women and children.
He also posed with everyone for a group photograph, against the majestic background of the mountains.
Bharatiya Janata Party's member of parliament Ram Swaroop Sharma told IANS: "It's a historic moment for the locals of the remote village adjoining China to interact with the Prime Minister."
"This is the first visit of any Prime Minister to this area and this would ensure development in the area," an elated Sharma said.
This was Modi's second visit to the Mandi parliamentary constituency this month. Earlier, he visited Mandi town on October 18 to inaugurate three hydropower projects and address a public rally.
The Lahaul-Spiti district, spread over 13,835 sq km, is a place of remote, untouched beauty with just 31,528 people. The climatic conditions are harsh as much of the land falls under a cold desert where during winters the mercury drops to below minus 20 degrees Celsius below the freezing point.
The Prime Minister celebrated his first Diwali after assuming the office in 2014 with soldiers posted in Siachen. In 2015, he was at the India-Pakistan border in Punjab.