To prevent unwanted connecting to your host from people and  only allow connections to specific IP addresses, We have simple way to allow it in your server. Just try to follow the below steps,

For example would you like to allow All connection from address

Method 1:

Any changes in your hosts.allow and hosts.deny file takes immediate effect.

Open the file hosts.deny and add the below line,

# vim /etc/hosts.deny

To allow multiple ip address :

Method 2:

Another method is use of a Firewall,

Allow IP Address to SSH (port 1234) and check your ethernet name using ifconfig command
#  iptables -A INPUT -s -i eth0 -p tcp -m tcp --dport 1234  -m state --state NEW,ESTABLISHED -j ACCEPT

Deny all other SSH connections:
#   iptables -A INPUT -p tcp --dport 1234 -j DROP

Save the iptables,
#  /etc/init.d/iptables save

SSH Access to Specific IP for User:

Open your ssh configure file and update it
# vim /etc/ssh/sshd_config

AllowUsers username@ username@ user@