Koan 18: The Loose Bundle

Exploring the pitfalls of raw string literals in Python

In Python, a raw string literal is a string marked by the r prefix. It instructs the Python interpreter to treat the string “as it is written”, ignoring any escape sequences (such as \n for newline, or \t for tab).

Last week in Koan 17 we explored how raw string literals can be used to prevent the “plague of backslashes” needed to search for strings containing escape sequences.

While raw string literals are certainly handy, there is one pitfall to watch out for. If a raw string ends in an odd number of backslashes, the last backslash escapes the closing quote. The string never ends, and the parser wanders into the void.

Part 1: The Broken Promise of Rawness

In a raw string literal one might assume that the backslash loses all of its power. However, this is not true. The backslash is stripped of its ability to create special characters (escape sequences), but it is not stripped of its ability to escape the next character.

Consider what a parser might do for the following:

s = r"The path is \"

When the parser encounters \”, it follows a strict rule: A backslash before a quote means the quote is part of the data, and not the end of the container. Even though this is a raw string, the parser still lets you include quotes:

The backslash prevented the string from closing, but it also remained in the string.

Part 2: The Unbound String

This leads us to the weaver's bundle. If you want a raw string that ends with a single backslash, you create a problem for the parser. With the following string, your intent might be to create a string containing just a backslash, but instead:

1. The parser sees the opening ”

2. It enters raw mode

3. It sees \

4. It sees ” immediately after

5. Because of the backslash, it treats the “ as a literal character inside the string

6. It continues looking for the closing quote, encounters the end of the line and raises a SyntaxError

This is why you can’t create a raw string literal that ends in an odd number of backslashes. The sharp edges of the bamboo pierce the binding rope.

Part 3: Securing the Bundle

If you absolutely must end a string with a backslash, you cannot use a raw string literal in isolation. You must separate the content from the termination.

There are several ways to tie this knot.

1. Concatenate the raw string literal with a regular string with a single escaped backslash

2. Use an fstring, and either include the backslash as a variable, or escape it outside a variable

3. Same as the first method, but concatenate the string by using join on an array of strings containing the escaped backslash

The Master’s Reflection

The raw string is a tool of convenience, but it is still a servant to the parser. While it treats backslashes as literals for the sake of the string's value, it still respects backslashes for the sake of the string's syntax. A diligent developer separates the boundary from the content when these conflicts arise.