Mongodb - hostname/IP changed for all hosts in sharding with replica set configuration

The issue is solved now.

The final piece of puzzle was to find where was the config server connection info saved in replica set mongod. It's in system.version collection under admin db. I had to follow the following steps

1. Start the mongod on all replicaset members with security authorization, replication and sharding disabled. Made necessary changed on config file.
2. Under admin db, the following two documents in system.version had the config server connection string.

db.system.version.find( {"_id" : { $in : [ "shardIdentity" , "minOpTimeRecovery" ]} })

3. Updated both the documents with new config server connection string via db.system.version.update command.
4. Shut down the mongod processes and enabled security authorization, replication and sharding in the mongod config file.
5. Successfully started replica set mongod instances.

Note : I am new to mongo and not sure if we should be making changes to internal system collections. Since it was a test setup, I took the risk and did these experiments which paid off. Its on a production environment a resolution can't be guaranteed.

I run this procedure on my local machine. It seems to work, but I cannot guarantee anything.

• Stop all mongod/mongos services on all nodes

Config ReplicaSet

• Start one mongod config server in maintenance mode
• Drop local database
• Update config.shards
• Shutdown mongod
• Delete dbPath of all config servers
• Start all mongod config servers
• Connect to first mongod config server
• Initiate ReplicaSet

Example (Windows style):

SET MAINTENANCE_LOG=--logpath C:\MongoDB\log\mongo_maintenance.log --logappend
SET MAINTENANCE_NET=--bind_ip localhost --port 55555
SET MAINTENANCE_MISC=--setParameter skipShardingConfigurationChecks=true --setParameter disableLogicalSessionCacheRefresh=true


start mongod --dbpath C:\MongoDB\data\mongocfg_1 %MAINTENANCE_LOG% %MAINTENANCE_MISC% %MAINTENANCE_NET%
mongo --norc localhost:55555/admin 
db.getSiblingDB('local').dropDatabase()
db.getSiblingDB('config').getCollection("shards").updateOne(
   {_id : "shard_01"}, 
   {$set: {host: "shard_01/," }}
)
db.getSiblingDB('config').getCollection("shards").updateOne(
   {_id : "shard_02"}, 
   {$set: {host: "shard_02/," }}
)
db.getSiblingDB('config').getCollection("shards").updateOne(
   {_id : "shard_03"}, 
   {$set: {host: "shard_03/," }}
)
db.getSiblingDB('admin').shutdownServer()
exit

rmdir C:\MongoDB\data\mongocfg_2
rmdir C:\MongoDB\data\mongocfg_3

net start MongoDB_Config_1
net start MongoDB_Config_2
net start MongoDB_Config_3

mongo "mongodb://user:password@localhost:27029/admin?authSource=admin"
rs.initiate(
  {
    _id: "configRepSet",
    configsvr: true,
    members: [
      { _id: 0, host: "", priority: 10 },
      { _id: 1, host: "", priority: 5 },
      { _id: 2, host: "", priority: 5 }
    ]
  }
)
rs.status()
while (! db.hello().isWritablePrimary ) { sleep(1000) }
exit

Shard ReplicaSet

Repeat below for each shard

• Start one mongod shard server (preferable the former PRIMARY) in maintenance mode
• Drop local database
• Update admin.system.version
• Shutdown mongod
• Delete dbPath of all shard servers
• Start all mongod shard servers
• Connect to first mongod shard server
• Initiate ReplicaSet

Example (Windows style):

SET MAINTENANCE_LOG=--logpath C:\MongoDB\log\mongo_maintenance.log --logappend
SET MAINTENANCE_NET=--bind_ip localhost --port 55555
SET MAINTENANCE_MISC=--setParameter skipShardingConfigurationChecks=true --setParameter disableLogicalSessionCacheRefresh=true

start mongod --dbpath C:\MongoDB\data\mongoshard_1prim %MAINTENANCE_LOG% %MAINTENANCE_MISC% %MAINTENANCE_NET%
mongo --norc localhost:55555/admin 
db.getSiblingDB('local').dropDatabase()
db.getSiblingDB('admin').getCollection("system.version").updateOne(
   {_id : "shardIdentity"}, 
   {$set: { configsvrConnectionString: "configRepSet/,," }}
)
db.getSiblingDB('admin').shutdownServer()
exit

rmdir C:\MongoDB\data\mongoshard_1sec\*
rmdir C:\MongoDB\data\mongoshard_1arb\*

net start MongoDB_Shard_1prim
net start MongoDB_Shard_1sec
net start MongoDB_Shard_1arb


mongo "mongodb://user:password@localhost:37028/admin?authSource=admin"
rs.initiate(
  {
    _id: "shard_01",
    members: [
      { _id: 0, host: "", priority: 10 },
      { _id: 1, host: "", priority: 5 },
      { _id: 2, host: "", arbiterOnly: true }
    ]
  }
)
rs.status()
while (! db.hello().isWritablePrimary ) { sleep(1000) }
exit

Router

This one the the simplest part.

• Edit mongos config file and put new sharging.configDB string
• Start mongos