NIST defines intrusion detection to be:

Intrusion prevention goes beyond intrusion detection, although “prevention” is strictly speaking an exaggeration:

In Definition 2, an “attempt” can be sending an alarm to the administrator(s), resetting a network connection, reconfiguring a firewall to block traffic from the source address, etc.

Thus, “intrusion detection and prevention system” (IDPS) is synonymous with “intrusion prevention system” (IPS) [SM07, Appendix A], and is a bit of a redundant term.

Fig. 1 shows an example of a high-end IPS appliance from Cisco.

In the 1970s and the early 1980s, administrators had to print out system logs on papers and manually audit the printout [KV02]. The process was clearly 1️⃣ reliant on the auditors’ expertise, 2️⃣ time-consuming, and 3️⃣ too slow to detect attacks in progress.

In the 1980s, storage became cheaper, and intrusion detection programs became available for analysing audit logs online, but the programs could only be run at night when the system’s user load was low [KV02]. Thus, detecting attacks in time remained a challenge.

In the early 1990s, real-time intrusion detection systems (IDSs) that analysed audit logs as the logs were produced became available [KV02].

Since the inception of IDSs, the quality and quantity of audit logs have always been a challenge [KV02]:

Quality

The accuracy (how often the data are correct) and precision (how close the reported values are to the true values) of the data collected are crucial.

Inaccurate or imprecise data could lead to false negatives (illegitimate events misdiagnosed as legitimate) or false positives (legitimate events misdiagnosed as illegitimate).

Quantity

On one hand, not collecting enough data ⇒ an attack could evade detection. For example, network-related data alone cannot help with detecting a malware that does not access any network.

On the other, collecting too much data (from too many sources and/or too frequently) ⇒ storage could run out and processing could take too long.

Effective detection of attacks depends on capturing all relevant data at sufficiently fine time scales.

An IDS is meant to detect [KV02]:

Misuses

These are abuses and attacks of known patterns that can be encoded in computer-interpretable rules or signatures.

An abuse is an intentional or unintentional violation of organisational policies.

Anomalies

An anomaly, also called outlier, is a “significant” deviation from some model of normalcy, where “significance” is contextual and can be fluid.

A model of normalcy is called a baseline model and is often challenging to build for a dynamic environment.

Accordingly, intrusion detection algorithms can be classified as [SM07; YT11, p. 2; BK14, p. 4; Led22; Pal22]:

Rule-based / signature-based

A rule specifies the conditions of legitimacy of an event, typically with the help of signatures.

Signatures can be

• exploit-facing, e.g., malicious byte sequences, email subject headings associated with phishing, email attachments containing executable binaries, traffic going to known malicious domains, scanning of file hashes; or
• vulnerability-facing, e.g., SSH requests specifying a vulnerable version number, system log entries indicating disablement of auditing.

Signatures can encode violations of organisational policies, e.g., remote login attempts as “root”.

Stateful protocol analysis or deep packet inspection analyses protocols at the application layer to compare vendor-developed profiles of benign protocol activity against observed events to identify deviations [SH09, Jun16]. This is an example of rule-based detection, contrary to the classification in [LRLT13].

As threats grow, the dictionary of signatures necessarily grows, demanding more computational and storage resources accordingly.

Rule-based detection is effective against known threats that can be expressed using a set of rules and signatures.

Rule-based detection is ineffective against known threats that are hard to capture using a finite set of rules and signatures, e.g., multi-stage attacks; as well as previously unknown threats.

Snort is an industry-standard open-source rule-based intrusion prevention software (hence, also an intrusion detection software). It can for example be found in the appliance in Fig. 1.

Anomaly-based / behaviour-based

This is the application of machine learning techniques to determining whether a user or component behaviour is anomalous.

Depends on the prior establishment of a baseline behavioural model.

Can detect previously unknown attacks as long as the attacks manifest as anomalies relative to the baseline model.

Effective at detection but tends to produce excessive false positives.

Behaviour-based detection, when applied to network behaviour, falls into the area of network behaviour analysis.

Definition 3: Network behaviour analysis [SM07, Xu22]

End-to-end process of collecting, extracting, analysing, modelling, and interpreting network behaviour (e.g., distributed denial of service, worm, backdoor, policy violation) of end systems and network applications from a large volume of network traffic data such as TCP/IP data packets and network flows.

A network behaviour analysis pipeline typically consists of these steps [Xu22, Fig. 2.1]: 1️⃣ network traffic data collection, 2️⃣ data storage and preprocessing, 3️⃣ behavioural feature selection and exploration, 4️⃣ analysis and modeling, 5️⃣ behavioural insights and applications.

Depending on its type, an IDS can comprise several types of components, as shown in Fig. 2: sensors/agents (which monitor and analyse network activities and may also perform preventive actions), management servers, database servers, user and administrator consoles, and management networks [SM07].

There is more than one way to classify IDSs. See the attachment for different classifications of IDSs.

Watch the following LinkedIn Learning video for a quick summary of IDS:

What is an IDS? from Protecting Your Network with Open Source Software by Jungwoo Ryoo