The Pentagon is probing how extremist groups are recruiting new members among the military, with a seemingly “aggressive” effort to attract members who are about the leave the service.