With businesses increasing the volume of transactions, processes, and operations they conduct over the Internet, security is always a major concern. As your data traverses across the Internet, it’ll certainly face various threats. These threats are capable of stealing, tampering with, or performing other malicious acts on your data. To address these threats, businesses, government agencies, and various organizations constantly work together or independently to come up with solutions to secure Internet traffic. One of the most successfully developed solutions is the Internet Protocol Security or IPsec suite. 

In this post, I’m going to introduce you to the basic concepts of IPsec, explain how it works, go over the key IPsec protocols, and discuss other relevant information that’ll help you get a good grasp on this important bundle of technology.

First, let me explain what IPsec is. 

Developed by the Internet Engineering Task Force (IETF) in the 1990s and used in VPNs, IPsec is a suite of protocols that secures Internet connections at the network layer. It does this by providing encryption, authentication, data integrity, and replay protection. Let me explain each of these security functions first. This will give you a better appreciation of how IPsec works and its importance in business operations.  

Encryption is a way of rendering text unreadable to people who don’t have the required decryption key. In effect, it preserves data confidentiality. This is important because, without it, a threat actor can intercept your packets. For instance, because your packets go through the Internet, a cyberattacker can initiate a man-in-the-middle attack. This means they can extract any confidential data you might have.

Another example is that a cyberattacker can also extract usernames and passwords whenever remote users log in to your system. They can then use those stolen credentials to perform unauthorized logins themselves. Threat actors can’t do this if you encrypt your packets, as this restricts them from making out the contents of those packets.

Authentication refers to the process of verifying the authenticity of a particular set of data—i.e., whether the information is true. In the context of IPsec, you’re verifying the source IP or the origin of the packet.

It’s important to ascertain the packet’s origin. If it doesn’t match what you were expecting, it could mean the packet you just received might actually be coming from a malicious site spoofing a legitimate source. 

A data integrity check is a process that verifies whether the data in question hasn’t gotten tampered with. In the context of IPsec, you’re checking the integrity of the packet. The part of the packet that’s checked depends on the IPsec header, AH, or ESP (more on these two later). But you can always check the payload or the transmitted data.

You need to conduct an integrity check on the data you receive. That’s because a threat actor always can intercept your packets, alter them, and then send them along. Data integrity checks will help you detect any tampering and prevent you from processing maliciously modified data. 

Replay protection is a security mechanism that thwarts replay attacks. In this context, a replay attack is a cyber-attack where a threat actor intercepts an IPsec packet, captures its contents, and then resends or replays it to the same destination at a later time. Note that the replay attack will still work even if you encrypt the IPsec packets. This happens because the attacker doesn’t have to alter or even obtain the packet’s contents.

So why is this a cause of concern? Well, if that packet contains a valid request that prompts a valid response, the replayed packet will also elicit a valid response. For example, if the replayed packet requests access to your system, your system won’t notice anything suspicious and will simply grant access. IPsec’s replay protection mechanism detects these replay attacks and generates an error message, thereby preventing the malicious packet from getting a response. 

The combination of all of these features makes IPsec suitable for Internet connections that require a high level of security. It’s the reason why IPsec gets used in securing VPNs, which are then called IPsec VPNs; more on those in the next section!

Although you do have other ways to secure a VPN (e.g., by using SSL/TLS), IPsec is arguably the most popular choice. That’s why IPsec VPNs are likewise the most widely used VPNs. Before I define what an IPsec VPN is, it might be best to review what a VPN is first. 

A virtual private network, or VPN, is a service that creates a private network within a public network like the Internet. VPN solutions achieve this through a process known as tunneling. It’s a technique where packets originally intended for a private network get encapsulated so that they can traverse a public network. 

Although tunneling alone can already create a VPN and provide some form of privacy, it doesn’t offer enough protection against today’s threats such as man-in-the-middle attacks. You need a VPN with security mechanisms such as encryption and authentication for that. Today, most VPN solutions support these security capabilities. And of these solutions, IPsec VPNs are arguably the most popular. 

As the name suggests, an IPsec VPN is a VPN that leverages IPsec protocols for security. IPsec VPNs use cryptographic algorithms to provide confidentiality, authentication, and data integrity, as well as a counter and number sequencing for detecting replayed packets. 

You can implement IPSec VPNs using the following VPN architectures:

A site-to-site VPN connects two or more networks, or sites, using IPsec VPN gateways. Each network should have one VPN gateway. The gateway may be a router or a firewall that supports an IPsec VPN . It can also be a separate device specifically designed for IPsec VPN operations. You’d typically use a site-to-site VPN to establish private communications and data exchanges between a central office and a branch office, or between two different companies. 

A remote access (a.k.a. client-to-site) VPN enables users to connect from VPN clients to a central location (e.g., a data center) and access resources hosted there. This type of VPN can be particularly useful in companies that need to support remote workers. Note that if you have multiple devices that need to connect to your VPN gateway, then each device must have a VPN client on it. 

This architecture is the least popular among the three. It usually comes into play when an IT administrator wants to perform certain tasks on a remote server. In this case, the server itself provides VPN services (notice you have no VPN gateway here) and the IT administrator’s device acts as a VPN client.

IPsec VPNs fully function based on the rules of various IPsec protocols. Let’s talk about the most important ones now.

As stated earlier, IPsec isn’t just one protocol. Rather, it contains several protocols. The four that I’ll talk about in this next section are IP Authentication Header (AH) protocol, Encapsulation Security Payload (ESP) protocol, Internet Key Exchange (IKE) protocol, and Internet Security Association and Key Management Protocol (ISAKMP). Let’s start with the IP AH protocol.

Defined in RFC 4302, the IP Authentication Header, or IP AH, protocol provides origin authentication, data integrity, and replay protection services to IPsec traffic. Because this IPsec protocol doesn’t support encryption, many organizations consider it obsolete and use ESP instead. However, because AH performs data integrity checks on certain parts of a packet (specifically the outer IP header) that ESP does not, you do have some organizations that still use both AH and ESP.

Specified in RFC 4303, the Encapsulation Security Payload (ESP) protocol provides packet payload encryption in addition to authentication, data integrity, and replay protection services. Even if ESP doesn’t provide integrity protection to the outer IP header, many organizations don’t perceive this as a critical deficiency. Most organizations worry more about the confidentiality of their traffic. Hence, they prefer this IPsec protocol over AH because of its packet payload encryption capabilities.

Defined in RFC 7296, the Internet Key Exchange Protocol (IKE) is primarily responsible for the negotiation, creation, and management of security associations. A security association (SA) is a set of parameters that define IPsec security and other functionality pertinent to two parties trying to establish a common IPsec connection. The entire IKE process consists of two phases: Phase 1 and Phase 2.

The goal of Phase 1 is for the two parties to establish a secure channel for management-related functions (e.g., health status of the IPsec tunnel, renegotiation of cryptographic keys, etc). This channel is often called IKE SA. To build this channel, the two parties negotiate security parameters such as the encryption algorithm, authentication method, integrity protection algorithm, etc. 

The goal of Phase 2, on the other hand, is to establish the actual IPsec tunnel where the data packets will go through. This is the IPsec SA. So, later in this article, when I say IKE SA, know that I’m referring to the management channel. And when I say IPsec SA, I’m referring to the IPsec VPN tunnel itself. 

Originally specified in RFC 2408, Internet Security Association and Key Management Protocol, or ISAKMP, is a framework for authentication and key exchange. It defines the procedures as well as the packet formats involved in building, negotiating, changing, and terminating SAs. A lot of what you see in IKE (e.g., syntax and attributes of SAs) is based on ISAKMP.

When you use either the AH or ESP IPsec protocol, you can configure them to encapsulate packets using one of two modes: Tunnel Mode or Transport Mode. Let’s get into those now.

AH and ESP can encapsulate packets using either tunnel mode or transport mode. It’s important to get familiar with these two modes because each one works best with certain architectures. Let’s start by talking about tunnel mode.

Tunnel mode is the more widely used between the two and is the default mode. In this mode, AH or ESP creates a new IP header when they encapsulate a packet. This new IP header contains the source and destination IP addresses of the endpoints involved, whether they’re gateways, clients, or hosts. For this reason, tunnel mode can work with all three VPN architectures mentioned previously.  

Unlike in tunnel mode, AH and ESP don’t have to create a new IP header when they encapsulate a packet in transport mode. Transport mode simply uses the original IP header. For this reason, this mode is best for implementations of the host-to-host architecture because you have direct communication between the two endpoints, and you don’t have to change the IP addresses in the IP header.

That just about covers it regarding IPsec tunnel and transport mode. Now, I’ll talk about how IPsec actually works.

Let’s now put everything you’ve learned so far together and talk about how IPsec works. I’ll go over the steps of a site-to-site scenario, as it’s the more popular architecture or use case. Let me bring up the site-to-site diagram I showed you earlier so you can use it as a point of reference. 

Now, without further ado, here’s a brief overview of the steps involved in a site-to-site scenario: 

Before we end, let me briefly touch upon one of the most contentious topics about VPNs. 

Whenever you have a discussion on IPSec VPN, a comparison with SSL VPN is usually not far behind. That’s because SSL VPN is slowly replacing IPSec VPN in some organizations. You can find a more detailed discussion comparing these two types of VPNs in the works here at TechGenix. It should be out very soon, so check it out when it’s available. In the meantime, here are my two cents on the subject.

An SSL VPN is a VPN that uses Transport Layer Security (TLS) for security. It’s called SSL VPN because Secure Socket Layer (SSL) is the predecessor of TLS. While SSL is already obsolete, it was originally used for this type of VPN and the name stuck. You can find SSL/TLS built into every modern web browser, which is why, in most cases, users only need a web browser to use an SSL VPN. 

One major advantage of SSL VPNs over IPsec VPNs is their ease of deployment and management in client-to-site scenarios. Compared to IPsec VPN clients, which require complex processes and technical know-how, SSL VPN clients are much easier to deploy, configure, and manage. In fact, SSL VPN users only need to install a plugin on any modern web browser and they’ll be ready to go. 

From a security standpoint, IPsec VPNs have certain advantages over SSL VPNs. I won’t dive deep into the details here (that’s for another article), but here are a couple of points to consider:

So does that mean that IPSec VPNs are better than SSL VPNs? Not necessarily. Because IPsec VPN clients are quite difficult to configure and maintain, IPSec VPNs are more suitable for site-to-site use cases. In a site-to-site scenario, you only need to deploy a VPN gateway on each site. Thus, it’s much easier from a management perspective compared to client-to-site scenarios. Your IT team should be easily able to handle that. 

That was a lot of information, wasn’t it? Look at the bright side, you’ve gained tons of knowledge in a short span of time! Let’s wrap up now, as we always do.

Confidentiality, data integrity, and authenticity are three qualities that are crucial to every business transaction. As many of these transactions are now getting conducted over the Internet, solutions that support these qualities have become integral to business operations. Among the most widely used of this breed of solutions are IPsec VPNs. 

In this article, I introduced you to the basic concepts of IPsec. You learned about the qualities of IPsec, VPN architectures, IPsec protocols, IPsec modes, and other related concepts. I also briefly compared IPsec VPNs with SSL VPNs. While this is by no means a deep dive into the inner workings of IPsec, I’m hoping I’ve covered enough to give you a good grasp on the subject. 

I’m sure some of these discussions opened up a couple of questions regarding IPsec in your mind. See if you can find some answers in the FAQ and Resources sections below. 

Yes, Layer 2 Tunneling Protocol, or L2TP,  which is a tunneling protocol used for VPNs, doesn’t encrypt data on its own. Rather, most implementations of L2TP get paired with IPsec, which then provides encryption functionality. If you’re using, or about to use a VPN solution that uses L2TP, check first if it’s actually paired with IPsec. 

SSL/TLS stands for Secure Sockets Layer or Transport Layer Security (TLS), which are technologies that protect various application layer protocols such as HTTP and FTP using encryption. Encryption protects data by making them unreadable. SSL is the older technology and TLS is the current one, but most users keep using the acronym SSL as it’s still more widely known. 

Encryption algorithms are the rules and instructions needed to transform cleartext (non-encrypted text) to ciphertext (encrypted text). Some of the most commonly used algorithms include AES, 3DES, RC6, IDEA, and Blowfish. Most cryptographic algorithms use complex math such as modular arithmetic, number theory, prime factorization, elliptic curves, and others. 

Padding Oracle on Downgraded Legacy Encryption, or POODLE, is a vulnerability that can get exploited when clients perform a protocol downgrade from TLS to SSLv3. As explained earlier in the article, SSL is already obsolete. One of the reasons is that it has several vulnerabilities. Unfortunately, endpoints sometimes perform protocol downgrades to work around server-side interoperability issues. The best way to counter a POODLE exploit is by disabling SSLv3 altogether. The method varies from one application to another, so you’ll have to contact your software vendor for the proper steps.

Heartbleed was a bug in OpenSSL (which is an implementation of SSL/TLS) that allowed a threat actor to send a malformed heartbeat request in order to elicit a response that might contain more data than usual. A heartbeat request is an OpenSSL feature that allows one endpoint to check if its link with another endpoint is still alive. Note that this bug only affects systems that use OpenSSL. Assuming you do use OpenSSL, you can address Heartbleed by simply applying the latest patch. This is such an old bug (appeared in 2012), so chances are, you’ve already addressed it a long time ago. 

Subscribe to our newsletters for more quality content.

Understand the importance, challenges, and types of cloud network security.

Weigh the pros and cons of using KerioControl.

Learn how to deploy a VPN in a Windows environment.

Discover the infosec policy-enforcement benefits of a SIEM.

Discover the top 6 trends in cloud computing in 2022. 

OPSEC is how you secure your information in an company through continual risk management based on some form of governance strategy. Read on to discover…

GUI-based Windows management tools are specifically designed to avoid exposing passwords. Even so, it’s sometimes possible to use PowerShell to help reveal a password. This…

If you’re searching for the top cloud security standards and frameworks, this is the article for you. These standards can be leveraged to improve your…

In this article, you’ll learn what account takeover fraud (ATO) is and how you can protect yourself from it. We discuss how cybercriminals get your…

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Learn about the latest security threats, system optimization tricks, and the hottest new technologies in the industry.

TechGenix reaches millions of IT Professionals every month, empowering them with the answers and tools they need to set up, configure, maintain and enhance their networks.