You need to check Reg-info and Sec-info settings. Most of the cases this is the troublemaker(!) Please pay special attention to this phase! Make sure that they are set as per the Notes:
Note 1425765 - Generating sec_info reg_info
Note 1947412 - MDM Memory increase and RFC connection error
- Note 1434117 - Bypassing sec_info without reg_info
If you have maintained here not only MDM but other systems as well, I recommend you to backup and inactivate those entries and focus only on MDS. When the MDM RFC registration is finished successfully, you can add the other entries again. Then you can be 100% sure that the issue is not with the incorrect secinfo/reginfo parameters.
In case MDM Server still doesnt appear in SMGW -> Logged on Clients list, please follow the note below, even if you do not use NW 7.4. In rare cases it helps.
Note 1889010 - MDM Server failed to register to gateway of Netweaver 7.40
For NW 7.4
Set it according to the SAP Note 1889010 - MDM Server failed to register to gateway of Netweaver 7.40