Memory-efficient distribution of regular expressions for fast deep packet inspection

Abstract

Current trends in network security force network intrusion detection systems (NIDS) to scan network traffic at wirespeed beyond 10 Gbps against increasingly complex patterns, often specified using regular expressions. As a result, dedicated regular-expression accelerators have recently received considerable attention. The storage efficiency of the compiled patterns is a key factor in the overall performance and critically depends on the distribution of the patterns to a limited number of parallel pattern-matching engines. In this work, we first present a formal definition and complexity analysis of the pattern distribution problem and then introduce optimal and heuristic methods to solve it. Our experiments with five sets of regular expressions from both public and proprietary NIDS result in an up to 8.8x better storage efficiency than the state of the art. The average improvement is 2.3x. Copyright 2009 ACM.