To prevent unwanted connecting to your host from people and  only allow connections to specific IP addresses, We have simple way to allow it in your server. Just try to follow the below steps,

For example would you like to allow All connection from address 192.168.0.100.

Method 1:

Any changes in your hosts.allow and hosts.deny file takes immediate effect.

Open the file hosts.deny and add the below line,

# vim /etc/hosts.deny
sshd: ALL EXCEPT  192.168.0.100

To allow multiple ip address :
sshd: ALL EXCEPT  192.168.0.100 192.168.0.101

Method 2:

Another method is use of a Firewall,

Allow IP Address 192.168.0.100 to SSH (port 1234) and check your ethernet name using ifconfig command
 
#  iptables -A INPUT -s 192.168.0.100/24 -i eth0 -p tcp -m tcp --dport 1234  -m state --state NEW,ESTABLISHED -j ACCEPT

Deny all other SSH connections:
 
#   iptables -A INPUT -p tcp --dport 1234 -j DROP

Save the iptables,
 
#  /etc/init.d/iptables save

SSH Access to Specific IP for User:

Open your ssh configure file and update it
 
# vim /etc/ssh/sshd_config


AllowUsers username@192.168.0.100 username@192.168.0.101 user@192.168.0.100