Reading and Writing Files in Python

Why File I/O Matters

Here's the thing: without files, your programs are just calculators. They compute, they print, and then everything vanishes. Real-world applications need to:

• Persist data across restarts (user settings, game saves, databases)
• Process incoming information (CSV reports, configuration files, logs)
• Store output for later analysis (results, generated documents, telemetry)
• Build on previous work (reading what you wrote yesterday)

Python's file handling is intentionally simple, but it has depth. Once you understand the fundamentals, you'll recognize the patterns everywhere, web frameworks, data science libraries, DevOps tools. They all lean on the same core concepts you're about to learn.

Opening Files: The Foundation

Everything starts with the open() function. Think of it as a bridge between your Python code and the filesystem. You hand open() a filename and a mode, and it gives you a file object you can work with.

The signature looks deceptively simple, but this single function call triggers a whole chain of system-level events that most tutorials gloss over:

file = open("data.txt", "r")

Simple enough, right? But here's what actually happens: the operating system locates the file, grants your program access, and returns a file object. That object tracks your current position in the file, remembers what mode you're using, and knows how to read or write data.

The catch? If you don't close the file properly, bad things happen. Your program might lock the file, preventing other processes from accessing it. Memory leaks. Resource exhaustion. This is where mode comes in, and more importantly, why the with statement exists, but we'll get there. For now, understand that every open() call is a promise to the operating system, and you need to honor it with a matching close().

File Modes: Understanding the Three Pillars

Python's file modes control how the file operates. You specify them as the second argument to open(). Understanding them deeply prevents entire categories of bugs, corrupted files, permission errors, and silent data loss.

open("filename.txt", "mode")

The three fundamental modes are:

Read Mode: "r"

file = open("message.txt", "r")

What it does: Opens the file for reading. The file must exist. You can read from it, but you cannot write to it. If the file doesn't exist, Python raises FileNotFoundError.

When to use it: Loading configuration files, reading input data, processing logs.

# This works
content = file.read()
print(content)
 
# This fails, PermissionError
file.write("new text")  # Error: not writable

Write Mode: "w"

file = open("output.txt", "w")

What it does: Opens the file for writing. If the file exists, it gets truncated (emptied). You can write to it, but you cannot read from it.

Critical warning: Truncation is permanent. If you open an existing file in write mode and don't write anything, you've deleted its contents.

# Dangerous! If the file has content, it's gone.
file = open("data.txt", "w")
file.close()
# data.txt is now empty

When to use it: Generating new files, overwriting old versions, saving results.

Append Mode: "a"

file = open("log.txt", "a")

What it does: Opens the file for writing, but at the end. If the file exists, it preserves existing content and adds to it. If the file doesn't exist, it creates it. You can write but not read.

When to use it: Logging events, accumulating data, adding records without destroying history.

file = open("log.txt", "a")
file.write("New log entry\n")
file.close()

Text vs Binary Mode

This is the split that trips up beginners more than any other concept in file I/O, and most tutorials treat it as a footnote. It deserves real attention. When you open a file in text mode, Python doesn't just read raw bytes off the disk, it actively interprets those bytes as characters using an encoding (usually UTF-8) and also translates line endings. On Windows, \r\n becomes \n transparently when reading; on Mac and Linux, \n stays \n. This is usually what you want for human-readable files, but it means you're not seeing the raw bytes on disk.

By default, Python treats files as text (mode "r" is really "rt", mode "w" is really "wt"). But you can work with binary data too:

• Text modes ("r", "w", "a"): Python handles encoding/decoding automatically
• Binary modes ("rb", "wb", "ab"): Raw bytes, no interpretation

# Text mode, Python converts strings to/from bytes
with open("greeting.txt", "w") as f:
    f.write("Hello, World!")  # String → bytes (UTF-8)
 
# Binary mode, you work directly with bytes
with open("image.jpg", "rb") as f:
    data = f.read()  # Raw bytes
    print(type(data))  # <class 'bytes'>

Why does this matter? Text files are human-readable and encoding-aware. Binary files are for anything that isn't text: images, executables, serialized Python objects. You'll use binary mode when dealing with non-text file types.

The distinction becomes critical when you're working with files that contain binary data but have a .txt extension, or when you need to precisely control byte-level behavior. If you open a JPEG in text mode on Windows, Python will attempt to decode the binary content as UTF-8 and also mangle any bytes that look like \r\n, corrupting the image data entirely. Always think before you choose a mode: am I working with human-readable text, or raw bytes?

Context Managers and Safety

Here's a truth about software: the most dangerous bugs are not the ones that crash your program immediately. They're the ones that quietly accumulate damage over time. Resource leaks are exactly this kind of bug. An unclosed file handle doesn't cause a crash on the next line, it might cause a crash hours later when the OS runs out of available file descriptors, or it might cause data corruption when another process tries to access the same file.

Here's the problem: if you open a file and forget to close it, resources leak.

# BAD, file never closes
file = open("data.txt", "r")
content = file.read()
# If an error happens here, file.close() never runs
print(content)

The solution is the with statement (a context manager). It guarantees the file closes, even if an error occurs:

# GOOD, file closes automatically
with open("data.txt", "r") as file:
    content = file.read()
    print(content)
# File is closed here, no matter what

Why this matters: The with statement is not optional. It's the standard, idiomatic way to handle files in Python. The file closes automatically when you exit the with block, whether you finished successfully or hit an exception.

# Even if an error occurs, the file closes
with open("data.txt", "r") as file:
    content = file.read()
    result = int(content)  # Might fail
# File is still closed!

Under the hood, the with statement invokes __enter__ when you enter the block and __exit__ when you leave, regardless of how you leave. This is the context manager protocol, and it's one of Python's most elegant features. You'll see this pattern used far beyond files: database connections, network sockets, thread locks, and temporary directories all benefit from the same guarantee. Make with a muscle-memory habit now, and you'll write safer code in every domain. Make this a habit. Always use with. Always.

Reading Files: Four Approaches

Once you've opened a file in read mode, you can access the data in different ways. Your choice depends on what you're trying to do.

Approach 1: read(), Everything at Once

The simplest approach loads the entire file contents into a single string. This works beautifully for small files and is perfectly readable, but it has a critical limitation: it loads the entire file into RAM at once.

with open("story.txt", "r") as file:
    entire_content = file.read()
    print(entire_content)

read() returns the entire file as a single string. This is great for small files, but if you're reading a 10GB log file, you'll run out of memory.

# Example
with open("poem.txt", "r") as file:
    poem = file.read()
    print(f"The file has {len(poem)} characters")

Use read() confidently for configuration files, small data files, and templates. If you're ever unsure whether a file might be large, prefer the iteration approach instead.

Approach 2: readline(), One Line at a Time

When you need precise control over position in the file, readline() gives you that granularity. It's the surgeon's scalpel of file reading, deliberate, careful, and explicit.

with open("data.txt", "r") as file:
    first_line = file.readline()
    second_line = file.readline()
    print(first_line)
    print(second_line)

readline() reads one line and advances the file position. Call it repeatedly to read subsequent lines. Each line includes the \n character at the end.

with open("data.txt", "r") as file:
    line = file.readline()
    while line:  # Empty string when EOF
        print(line.strip())  # Remove the \n
        line = file.readline()

This approach is useful when you need to do something different with the first line versus subsequent lines, for example, treating a CSV header separately from data rows.

Approach 3: readlines(), All Lines as a List

with open("data.txt", "r") as file:
    lines = file.readlines()
    for line in lines:
        print(line.strip())

readlines() returns a list of strings, one per line (including \n). This is convenient when you want random access to lines.

with open("config.txt", "r") as file:
    lines = file.readlines()
    # Access specific lines
    print(lines[0])  # First line
    print(lines[-1])  # Last line

Keep in mind that readlines() loads the entire file into memory as a list, so the same memory caveat that applies to read() applies here. The advantage over read() is that you get a pre-split list of lines, which can simplify code that needs to index into specific positions of the file.

Approach 4: Iteration, The Pythonic Way

with open("data.txt", "r") as file:
    for line in file:
        print(line.strip())

This is my favorite. Iterating directly over the file object is memory-efficient and readable. Python reads lines one at a time, so you never load the entire file.

# Process a massive file without running out of memory
with open("huge_log.txt", "r") as file:
    for line in file:
        if "ERROR" in line:
            print(line)

Why iteration wins: It's Pythonic, memory-efficient, and beautiful. When you iterate over a file object, you're working with its natural data structure. This is the approach you'll see in production code, data science notebooks, and system tools. Internalize this pattern early and it'll serve you for the rest of your Python career.

Writing Files: Practical Patterns

Writing is the mirror image of reading. You open in write or append mode and use the write() or writelines() methods.

Writing Strings: write()

Before you write a single character to a file, understand one critical difference from print(): write() does not add a newline for you. Every \n must be explicit. This surprises developers who come from print() habits, and it causes files that look fine in Python but come out as a single unbroken line when opened in a text editor.

with open("output.txt", "w") as file:
    file.write("Hello, World!\n")
    file.write("This is a test.\n")

write() outputs a single string. If you want a newline, you must include \n. Unlike print(), it doesn't add newlines automatically.

# Common mistake
with open("output.txt", "w") as file:
    file.write("Line 1")
    file.write("Line 2")
# Result: "Line 1Line 2" (no newline between them)
 
# Correct way
with open("output.txt", "w") as file:
    file.write("Line 1\n")
    file.write("Line 2\n")

Writing Multiple Lines: writelines()

lines = ["Apple\n", "Banana\n", "Cherry\n"]
 
with open("fruits.txt", "w") as file:
    file.writelines(lines)

writelines() accepts an iterable (list, tuple, generator) and writes each item. Unlike write(), it doesn't add newlines, they must be in the strings.

# Practical example: saving search results
results = ["Result 1", "Result 2", "Result 3"]
formatted = [f"{r}\n" for r in results]
 
with open("results.txt", "w") as file:
    file.writelines(formatted)

The list comprehension pattern for adding newlines to a list before passing it to writelines() is idiomatic Python. It's slightly more efficient than calling write() in a loop because it reduces the number of Python-to-OS calls, though for most use cases the difference is negligible.

Building Files Progressively

# Start fresh
with open("log.txt", "w") as file:
    file.write("Starting new log\n")
 
# Add to it later
with open("log.txt", "a") as file:
    file.write("Event 1\n")
    file.write("Event 2\n")

The hidden layer: Opening a file in write mode ("w") truncates it immediately. Even if you open and don't write anything, the file gets emptied. This is why append mode ("a") exists, it's the safe way to add to existing files.

Working With File Attributes

File objects have attributes and methods that give you metadata and control:

Position Tracking: tell() and seek()

These two methods give you a superpower: the ability to jump around inside a file instead of reading it sequentially. Think of the file as a tape, and the position as the read head. tell() tells you where the head is; seek() moves it.

with open("data.txt", "r") as file:
    print(file.tell())  # 0 (at start)
    file.read(5)        # Read 5 characters
    print(file.tell())  # 5 (position after read)
    file.seek(0)        # Jump back to start
    print(file.tell())  # 0 again

tell() returns your current position in the file (in bytes). seek() jumps to a position. These are useful when you need random access.

# Real-world example: reading file in chunks
chunk_size = 1024
 
def process(data):
    print(f"Processing {len(data)} bytes")
 
with open("large_file.bin", "rb") as file:
    while True:
        chunk = file.read(chunk_size)
        if not chunk:
            break
        process(chunk)
        print(f"Position: {file.tell()}")

File Metadata: name, mode, closed

with open("data.txt", "r") as file:
    print(file.name)     # "data.txt"
    print(file.mode)     # "r"
    print(file.closed)   # False
 
# After exiting the with block:
print(file.closed)       # True

These attributes let you confirm what you're working with, especially in functions or debugging.

Encoding Pitfalls

If you've never had a file reading job blow up with UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1234, consider yourself lucky, and consider this section insurance. Encoding errors are one of the most common sources of production bugs in data pipelines. They're especially treacherous because they often work fine on your development machine (which uses UTF-8 everywhere), then fail spectacularly on files created on Windows by users who have no idea what an encoding even is.

The problem is that text files don't carry a label saying "I was encoded with Windows-1252." Python has to guess, or use the system default, which varies by platform. On Windows, the default is often cp1252; on Linux and Mac, it's usually UTF-8. A file written on a Windows machine with special characters like curly quotes, em dashes, or accented letters will fail to decode on a Linux server unless you handle encoding explicitly.

By default, Python assumes UTF-8 encoding (the universal standard for text). But files can use other encodings, and mismatches cause UnicodeDecodeError.

# This might fail if the file isn't UTF-8
with open("mystery.txt", "r") as file:
    content = file.read()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0x...

The fix: specify encoding explicitly:

# Explicitly handle encoding
with open("mystery.txt", "r", encoding="latin-1") as file:
    content = file.read()

Common encodings:

• "utf-8" (default): Handles any language, emoji, special characters
• "ascii": English only, strict
• "latin-1": Western European
• "cp1252": Windows Western European

Best practice: Always specify encoding. Write files as UTF-8, read with UTF-8. It prevents surprises across platforms.

# Safe pattern
with open("output.txt", "w", encoding="utf-8") as file:
    file.write("Café ☕ 你好\n")
 
with open("output.txt", "r", encoding="utf-8") as file:
    content = file.read()
    print(content)  # Works perfectly

When you must deal with files of unknown encoding, say, legacy data from a third-party system, use the errors parameter: open("legacy.txt", "r", encoding="utf-8", errors="replace") will substitute a replacement character for any bytes that don't decode cleanly, letting you at least inspect the data. And if you need to auto-detect encoding, the chardet library is your friend. But the best strategy remains write UTF-8 everywhere you control, and document the encoding wherever you don't.

Real-World Patterns

Pattern 1: Counting Lines in a File

def count_lines(filename):
    """Count lines in a file efficiently."""
    count = 0
    with open(filename, "r", encoding="utf-8") as file:
        for line in file:
            count += 1
    return count
 
lines = count_lines("data.txt")
print(f"The file has {lines} lines")

Why this works: Iterating over the file object is memory-efficient. You process lines one at a time, never storing the entire file.

Pattern 2: Processing and Filtering

def filter_errors(input_file, output_file):
    """Extract error lines from a log file."""
    with open(input_file, "r", encoding="utf-8") as infile:
        with open(output_file, "w", encoding="utf-8") as outfile:
            for line in infile:
                if "ERROR" in line:
                    outfile.write(line)
 
filter_errors("app.log", "errors.txt")

The pattern: Read from one file, write to another. This is the foundation of data processing pipelines.

Pattern 3: Configuration Files

def load_config(filename):
    """Load key=value configuration."""
    config = {}
    with open(filename, "r", encoding="utf-8") as file:
        for line in file:
            line = line.strip()
            if line and not line.startswith("#"):
                key, value = line.split("=", 1)
                config[key.strip()] = value.strip()
    return config
 
# config.txt
# debug=true
# api_key=secret123
# timeout=30
 
settings = load_config("config.txt")
print(settings["debug"])  # "true"

Pattern 4: Appending to a Log

import datetime
 
def log_event(event, filename="app.log"):
    """Append an event to a log file with timestamp."""
    timestamp = datetime.datetime.now().isoformat()
    with open(filename, "a", encoding="utf-8") as file:
        file.write(f"[{timestamp}] {event}\n")
 
log_event("Application started")
log_event("User logged in: alice@example.com")
log_event("Error: Division by zero")

This pattern is used everywhere, web frameworks, databases, monitoring systems. It's bulletproof because append mode never truncates.

Common File I/O Mistakes

No tutorial on file I/O is complete without a rogue's gallery of the mistakes that burn people in production. These are not theoretical edge cases, every developer on a serious Python project has encountered at least two of these. Read them carefully. They'll save you hours.

Mistake 1: Opening in write mode when you meant append. This is the silent data-destroyer. You have a log file accumulating days of events, and you accidentally open it with "w" instead of "a". The file empties instantly. Python won't warn you. The data is gone unless you have a backup. Always double-check your mode when you intend to preserve existing content.

Mistake 2: Forgetting that write() doesn't add newlines. You write ten "lines" to a file and open it to find one long unbroken string. Always be explicit about \n. If you're writing structured data, consider using print(text, file=f) which does add newlines automatically, it's a valid style for human-readable output files.

Mistake 3: Opening files inside loops. Every open() call has overhead. If you need to write multiple items, open the file once outside the loop and keep it open. Repeatedly opening and closing a file for each record in a 100,000-row dataset is measurably slower and puts unnecessary pressure on the OS.

Mistake 4: Ignoring encoding until it breaks. The code works on your machine, you ship it, and it crashes for users with non-ASCII characters in their filenames or file contents. Explicitly specify encoding="utf-8" on every open() call for text files, even in internal tools. It takes three seconds and prevents a whole class of bugs.

Mistake 5: Assuming the file pointer is at the beginning. If you call file.read() to get all the content, then call it again, you get an empty string, because the pointer is now at the end. If you need to re-read the content, call file.seek(0) first, or just store the result of the first read in a variable and reuse it.

These five mistakes account for the majority of file I/O bugs you'll encounter in the wild. Know them, and you'll spend far less time debugging and far more time building.

Common Errors and How to Fix Them

Error 1: FileNotFoundError

# This fails if data.txt doesn't exist
with open("data.txt", "r") as file:
    content = file.read()
# FileNotFoundError: [Errno 2] No such file or directory: 'data.txt'

Fix: Check if the file exists before reading:

import os
 
if os.path.exists("data.txt"):
    with open("data.txt", "r") as file:
        content = file.read()
else:
    print("File not found")

Or handle the exception:

try:
    with open("data.txt", "r") as file:
        content = file.read()
except FileNotFoundError:
    print("File not found. Creating empty file.")
    content = ""

Error 2: PermissionError

# This fails if you don't have write permissions
with open("/root/data.txt", "w") as file:
    file.write("test")
# PermissionError: [Errno 13] Permission denied: '/root/data.txt'

Fix: Check permissions or run with appropriate privileges. On Linux/Mac, use chmod:

chmod 644 data.txt  # Make readable/writable

Error 3: UnicodeDecodeError

# File isn't UTF-8
with open("binary_data.bin", "r") as file:
    content = file.read()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0x...

Fixes:

1. If it's a text file in a different encoding:

with open("data.txt", "r", encoding="latin-1") as file:
    content = file.read()

2. If it's actually binary:

with open("data.bin", "rb") as file:
    content = file.read()  # Returns bytes, not str

Error 4: Writing to a Read-Only File

with open("data.txt", "r") as file:
    file.write("test")
# io.UnsupportedOperation: not writable

Fix: Open with the correct mode:

with open("data.txt", "w") as file:  # Use "w" or "a"
    file.write("test")

Error 5: File Locked After Crash

If your program crashes without closing files properly, they might remain locked. This is another reason to always use with, it handles cleanup even during exceptions.

# Safe, file closes even if error occurs
with open("data.txt", "w") as file:
    data = process()  # Might fail
    file.write(data)

Binary Files: Reading and Writing Non-Text Data

Text files are strings. Binary files are everything else: images, audio, pickled objects.

Reading Binary Data

# Read an image file
with open("photo.jpg", "rb") as file:
    image_data = file.read()
    print(f"Read {len(image_data)} bytes")
    print(type(image_data))  # <class 'bytes'>

Binary mode ("rb", "wb") returns bytes objects instead of str. You can inspect them:

with open("photo.jpg", "rb") as file:
    first_bytes = file.read(10)
    print(first_bytes)  # b'\xff\xd8\xff\xe0...'

Those leading bytes, \xff\xd8\xff, are a JPEG magic number, a signature that identifies the file format. Many binary file formats begin with a known byte sequence, and reading binary mode lets you inspect and work with these signatures directly. This is foundational for building file parsers or any tool that needs to validate file types without relying on extensions.

Writing Binary Data

# Copy a file in binary mode (works for any file type)
def copy_file(source, destination):
    with open(source, "rb") as src:
        with open(destination, "wb") as dst:
            dst.write(src.read())
 
copy_file("original.jpg", "backup.jpg")

This pattern works for any file, text, images, executables. Binary mode is the universal approach.

Chunked Processing for Large Files

# Process a large file without loading it all into memory
def process_large_file(filename):
    chunk_size = 8192  # 8KB chunks
 
    with open(filename, "rb") as file:
        while True:
            chunk = file.read(chunk_size)
            if not chunk:
                break
            process_chunk(chunk)
 
def process_chunk(chunk):
    # Do something with the chunk
    print(f"Processing {len(chunk)} bytes")

This is how professional tools handle gigabyte-sized files. They read in chunks, process, and discard rather than loading everything at once. The 8KB chunk size is not magic, it's a common choice because it aligns well with typical filesystem block sizes and CPU cache behavior. You'll see this same pattern in HTTP streaming, database bulk imports, and machine learning data loaders. Mastering chunked I/O now prepares you for working with datasets far too large to fit in RAM.

File Attributes and Introspection

Getting File Size

import os
 
size = os.path.getsize("data.txt")
print(f"File size: {size} bytes")
 
# Human-readable format
def format_size(bytes_size):
    for unit in ["B", "KB", "MB", "GB"]:
        if bytes_size < 1024:
            return f"{bytes_size:.2f} {unit}"
        bytes_size /= 1024
    return f"{bytes_size:.2f} TB"
 
print(format_size(os.path.getsize("data.txt")))

Checking Existence and Type

import os
 
if os.path.exists("data.txt"):
    if os.path.isfile("data.txt"):
        print("It's a file")
    elif os.path.isdir("data.txt"):
        print("It's a directory")
else:
    print("Doesn't exist")

File Modification Time

import os
import time
 
mtime = os.path.getmtime("data.txt")
readable_time = time.ctime(mtime)
print(f"Last modified: {readable_time}")

The Hidden Layer: Why Mode Matters

Here's what you really need to understand: the mode system exists because files are shared resources. Multiple programs might try to access the same file. The mode prevents conflicts:

• Read mode ("r"): "I'm looking but not changing"
• Write mode ("w"): "I own this file, nobody else reads until I'm done"
• Append mode ("a"): "I'm adding to the end, others can read the existing content"

The operating system enforces these rules. If you open in write mode, other processes can't read the file (on most systems). This prevents corruption.

Also, opening files is expensive. Your operating system has to find the file on disk, check permissions, allocate resources. That's why you should open once and read multiple lines, not open repeatedly in a loop:

def process(data):
    print(f"Processing {len(data)} items")
 
# BAD, open 100 times
for i in range(100):
    with open("data.txt", "r") as file:
        lines = file.readlines()
 
# GOOD, open once
with open("data.txt", "r") as file:
    lines = file.readlines()
    for i in range(100):
        process(lines)

Summary

File I/O is fundamental to real programming. You now understand:

• Modes: read ("r"), write ("w"), append ("a"), and binary variants
• Context managers: Always use with to guarantee cleanup
• Reading strategies: read(), readline(), readlines(), iteration
• Writing basics: write() and writelines() with proper newlines
• Encoding: Always specify UTF-8 explicitly
• Patterns: Logging, filtering, configuration, processing
• Error handling: FileNotFoundError, PermissionError, UnicodeDecodeError
• Binary files: Different approach for non-text data
• The why: File modes are resource protection, context managers prevent leaks

From here, you're ready to build real applications that persist data. You'll use these patterns in every project, web frameworks, data analysis, system automation, games. Master this, and you're mastering the bridge between your program and the physical world.

File I/O is not glamorous, but it is foundational. The developers who understand it deeply write code that runs reliably in production, handles edge cases gracefully, and scales to massive datasets without choking. The developers who treat it as an afterthought write code that works in the happy path and breaks mysteriously in the real world. You now have the knowledge to be in the first group. Use context managers every time. Be explicit about encoding every time. Choose your mode deliberately every time. These are small habits with outsized impact on the quality of everything you build from here forward.

The next article dives into pathlib, which makes file paths Pythonic and cross-platform. But you've got the fundamentals locked down now.