Skip to content

URL encoding and web scraping

URL encoding is a mechanism for encoding information in a Uniform Resource Locator (URI) under specific constraints. It is also frequently utilized in the context of [[web scraping]] to bypass text-based filtering and security controls.

Overview

While URL encoding is a standard part of URI syntax, it can be repurposed as an evasion technique.^[400-devops__02-OS-and-Linux-Basics__ssl__certificate__curl_(60)_SSL_certificate_problem.md]

This method works by converting a string (such as a filter keyword or malicious payload) into its URL-encoded format (e.g., percent-encoding). Because the encoded string differs from the plain text, security filters or pattern matchers that only scan for the raw text may fail to identify the blocked content, allowing the request to proceed.^[400-devops__02-OS-and-Linux-Basics__ssl__certificate__curl_(60)_SSL_certificate_problem.md]

  • [[web scraping]]

Sources