Do Not Block ICMP: How Blocking ICMP Leads to MTU Problems and Connectivity Issues

Do Not Block ICMP: How Blocking ICMP Leads to MTU Problems and Connectivity Issues

Real-Life Scenario: ICMP Blocking Causing API Failures

We received a ticket about some API requests failing when developers tried to reach third-party services. To investigate the issue, we decided to use a specific pod to reproduce it. We began sending requests via curl from the pod to the third-party service. Some requests failed, and when we checked the network traffic, we saw ICMP packets. We were curious about why curl was sending these packets and began looking into the problem.

We soon discovered that the ICMP packets were of type 3, meaning “Destination Unreachable.” To get to the bottom of it, we called in a colleague from the network team to investigate the router level. It turned out that the edge router of the third-party service was blocking ICMP packets, and the returned MTU size was abnormal. This blockage was causing the MTU issue. After informing the network team that ICMP packets should not be blocked, they removed the block, and the issue was fixed.

The ICMP Type 3 message is part of Path MTU Discovery (PMTUD). PMTUD helps determine the best MTU size along the network path. It works by sending packets with the “Don’t Fragment” (DF) flag, so the sender can figure out the right MTU size. If a packet is too large, the router sends an ICMP “Fragmentation Needed” message back, telling the sender to reduce the packet size. If ICMP packets are blocked, the sender never receives this message and keeps sending large packets, which leads to packet loss and connection issues.

Understanding ICMP: The Backbone of Network Communication

The Internet Control Message Protocol (ICMP) is a network protocol used to send error messages and operational information between network devices. ICMP is an essential part of the Internet Protocol (IP) suite and is used for various network operations, including diagnosing network issues, troubleshooting connectivity problems, and performing network management tasks.

The Importance of Allowing ICMP Packets

ICMP packets play a crucial role in network operations, especially in MTU discovery and identifying network issues. Blocking ICMP can negatively impact network performance and cause serious connectivity problems.

Common Reasons for Blocking ICMP Packets

ICMP packets are sometimes blocked by firewalls and network security devices to prevent certain types of attacks, such as ping flooding and denial-of-service (DoS) attacks. Blocking ICMP can help protect network resources and prevent malicious activities that exploit ICMP vulnerabilities.

How Firewall Blocking of ICMP Leads to MTU Problems

To avoid MTU-related issues, it is important to ensure that ICMP packets are not blocked by firewalls. Allowing ICMP traffic helps maintain proper network functionality and prevents connectivity problems caused by MTU mismatches.When ICMP is blocked, these “Fragmentation Needed” messages do not reach the sender. As a result, the sender continues to send packets that are too large, leading to packet loss and connectivity issues. This can cause significant problems, especially in networks with varying MTU sizes.ICMP is used for various network operations, including the Path MTU Discovery (PMTUD) process. PMTUD helps determine the optimal MTU size for a network path by sending packets with the “Don’t Fragment” (DF) flag set. If a packet is too large to be forwarded without fragmentation, the router sends an ICMP “Fragmentation Needed” message back to the sender, indicating the need to reduce the packet size.When a firewall blocks ICMP packets, it can cause issues with the Maximum Transmission Unit (MTU). MTU is the largest size of a packet that can be sent over a network. If a packet is larger than the MTU, it needs to be fragmented into smaller packets.