Black Hat Briefings, Usa 2007 [audio] Presentations From The Security Conference.

A growing number of proprietary protocols are using end-to-end encryption to avoid being detected via network-based systems performing Intrusion Detection/Prevention and Application Rate Shaping. Attackers frequently use well known ports that are open through most firewalls to tunnel commands for controlling zombie systems. This presentation shows that a framework is indeed possible to identify encrypted protocols or anomalous usage of well known ports. The framework relies on performing statistical analysis on protocol packets and flows, and uniquely maps each protocol in a 10-dimensional space. Clustering algorithms are applied to accurately identify a wide variety of protocols. This novel approach provides network and security administrators a powerful tool to use in enforcing traffic policy, even when users are actively attempting to evade these policies. An open-source implementation will be released during the presentation.