The log analyzer script has print() statements everywhere — debug info, progress updates, error messages, the final report. What's the problem with that?
When the ops team runs it normally, they don't want to see the debug output. But when I'm debugging a problem on Ahmad's server, I need all the detail. Right now I comment and uncomment print() statements to switch between modes. It's terrible and I know it.
The logging module is the fix. Think of it as a ship's logbook — each entry has a timestamp, a severity level, and a message. You write entries at whatever level is appropriate, and the logbook's filter decides what's actually recorded. Your code never changes:
import logging
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("log-analyzer")
logger.debug("Parsing 50000 log entries") # DEBUG
logger.info("Processed 1000 entries so far") # INFO
logger.warning("Slow query detected: 1205ms") # WARNING
logger.error("Failed to open log file") # ERROR
logger.critical("Disk full — aborting") # CRITICALSo there are five levels in order of severity: DEBUG, INFO, WARNING, ERROR, CRITICAL. The filter at the top says "only show DEBUG and above" — which means everything. If I change to level=logging.WARNING, I'd only see WARNING, ERROR, and CRITICAL.
Exactly. The level on the handler is the filter. The levels on the individual calls are labels. Set the filter high for quiet operation, low for debugging. Your application code stays the same. The filtering happens at the output layer.
So for the ops team's cron job: level=logging.WARNING in production — they only see problems. For debugging on Ahmad's server: level=logging.DEBUG — they see everything. The script itself doesn't change at all.
And with argparse from yesterday, you can make it a command-line flag:
import argparse, logging
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", action="store_true")
args = parser.parse_args()
level = logging.DEBUG if args.verbose else logging.WARNING
logging.basicConfig(level=level, format="%(asctime)s [%(levelname)s]: %(message)s")--verbose flag sets logging to DEBUG. Normal run sets it to WARNING. That's the tool I was imagining when you introduced argparse. That's a real CLI tool.
One more pattern — logging to a file as well as the console:
import logging
logger = logging.getLogger("log-analyzer")
logger.setLevel(logging.DEBUG)
# Console handler — INFO and above only
console = logging.StreamHandler()
console.setLevel(logging.INFO)
console.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
# File handler — DEBUG and above, full detail
file_handler = logging.FileHandler("analyzer.log")
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
logger.addHandler(console)
logger.addHandler(file_handler)
logger.debug("Starting parse phase") # goes to file only
logger.info("Parsing complete") # goes to console and fileOne logger, two handlers, different levels. The console gets INFO and above — clean for the ops team. The file gets DEBUG and above — full trace for post-mortem debugging. I've been thinking of logging as one output. It's a routing system.
A routing system with levels. The logger checks "is this message at or above my minimum level?" and passes it to each handler, which also checks its own minimum. Multiple destinations, independent filters, zero changes to the calling code.
Today's problem: configure a logger for the analyzer script, write entries at appropriate levels for different stages of processing — start, progress, warnings for malformed entries, errors for missing files — and assert that the right messages appear at the right levels.
Tomorrow: pprint and timeit — the debugging microscope and the stopwatch. After two weeks of building the log analyzer pipeline, you'll have tools to inspect its data structures clearly and measure whether it's fast enough for the ops team's 50,000-entry daily logs.
The logging module codifies a practice that every experienced developer rediscovers independently: differentiate output by severity, filter it by audience, and route it to appropriate destinations — all without changing the calling code. The module provides infrastructure for all three.
Python's logging has five standard levels in ascending severity: DEBUG (10), INFO (20), WARNING (30), ERROR (40), CRITICAL (50). Each level has a corresponding method on logger objects. Setting a handler's level to WARNING means only messages at WARNING (30) or above pass through. The level integers are importable as constants: logging.DEBUG, logging.INFO, etc.
Loggers are organized in a hierarchy by dotted name: logging.getLogger("myapp") and logging.getLogger("myapp.parser") have a parent-child relationship. Log messages propagate up the hierarchy unless logger.propagate = False is set. The root logger (accessed via logging.getLogger() or logging.basicConfig()) is the ancestor of all named loggers. This allows library code to log under its own name, and application code to configure where and how those messages appear.
A handler defines where log records go: StreamHandler writes to a stream (stdout or stderr by default), FileHandler writes to a file, RotatingFileHandler caps file size and rotates, SMTPHandler emails. A formatter defines how records look: %(asctime)s, %(levelname)s, %(name)s, %(message)s, %(filename)s, %(lineno)d. Attach multiple handlers to one logger for simultaneous console + file output with independent level filters.
logging.basicConfig() configures the root logger with a single handler in one call — appropriate for scripts. For applications with multiple modules, create named loggers per module with logging.getLogger(__name__) and configure handlers once at the application entry point. Named loggers are created once and reused — getLogger("foo") always returns the same object.
Each log call creates a LogRecord object with metadata: timestamp, level, logger name, filename, line number, process ID. The formatter extracts fields from the LogRecord using % codes. For machine-readable logs, use logging.Formatter(json.dumps({"time": "%(asctime)s", "level": "%(levelname)s", "msg": "%(message)s"})) or a structured logging library like structlog. Structured logging is essential when your log output will itself be parsed by another log analyzer.
Sign up to write and run code in this lesson.
The log analyzer script has print() statements everywhere — debug info, progress updates, error messages, the final report. What's the problem with that?
When the ops team runs it normally, they don't want to see the debug output. But when I'm debugging a problem on Ahmad's server, I need all the detail. Right now I comment and uncomment print() statements to switch between modes. It's terrible and I know it.
The logging module is the fix. Think of it as a ship's logbook — each entry has a timestamp, a severity level, and a message. You write entries at whatever level is appropriate, and the logbook's filter decides what's actually recorded. Your code never changes:
import logging
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("log-analyzer")
logger.debug("Parsing 50000 log entries") # DEBUG
logger.info("Processed 1000 entries so far") # INFO
logger.warning("Slow query detected: 1205ms") # WARNING
logger.error("Failed to open log file") # ERROR
logger.critical("Disk full — aborting") # CRITICALSo there are five levels in order of severity: DEBUG, INFO, WARNING, ERROR, CRITICAL. The filter at the top says "only show DEBUG and above" — which means everything. If I change to level=logging.WARNING, I'd only see WARNING, ERROR, and CRITICAL.
Exactly. The level on the handler is the filter. The levels on the individual calls are labels. Set the filter high for quiet operation, low for debugging. Your application code stays the same. The filtering happens at the output layer.
So for the ops team's cron job: level=logging.WARNING in production — they only see problems. For debugging on Ahmad's server: level=logging.DEBUG — they see everything. The script itself doesn't change at all.
And with argparse from yesterday, you can make it a command-line flag:
import argparse, logging
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", action="store_true")
args = parser.parse_args()
level = logging.DEBUG if args.verbose else logging.WARNING
logging.basicConfig(level=level, format="%(asctime)s [%(levelname)s]: %(message)s")--verbose flag sets logging to DEBUG. Normal run sets it to WARNING. That's the tool I was imagining when you introduced argparse. That's a real CLI tool.
One more pattern — logging to a file as well as the console:
import logging
logger = logging.getLogger("log-analyzer")
logger.setLevel(logging.DEBUG)
# Console handler — INFO and above only
console = logging.StreamHandler()
console.setLevel(logging.INFO)
console.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
# File handler — DEBUG and above, full detail
file_handler = logging.FileHandler("analyzer.log")
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
logger.addHandler(console)
logger.addHandler(file_handler)
logger.debug("Starting parse phase") # goes to file only
logger.info("Parsing complete") # goes to console and fileOne logger, two handlers, different levels. The console gets INFO and above — clean for the ops team. The file gets DEBUG and above — full trace for post-mortem debugging. I've been thinking of logging as one output. It's a routing system.
A routing system with levels. The logger checks "is this message at or above my minimum level?" and passes it to each handler, which also checks its own minimum. Multiple destinations, independent filters, zero changes to the calling code.
Today's problem: configure a logger for the analyzer script, write entries at appropriate levels for different stages of processing — start, progress, warnings for malformed entries, errors for missing files — and assert that the right messages appear at the right levels.
Tomorrow: pprint and timeit — the debugging microscope and the stopwatch. After two weeks of building the log analyzer pipeline, you'll have tools to inspect its data structures clearly and measure whether it's fast enough for the ops team's 50,000-entry daily logs.
The logging module codifies a practice that every experienced developer rediscovers independently: differentiate output by severity, filter it by audience, and route it to appropriate destinations — all without changing the calling code. The module provides infrastructure for all three.
Python's logging has five standard levels in ascending severity: DEBUG (10), INFO (20), WARNING (30), ERROR (40), CRITICAL (50). Each level has a corresponding method on logger objects. Setting a handler's level to WARNING means only messages at WARNING (30) or above pass through. The level integers are importable as constants: logging.DEBUG, logging.INFO, etc.
Loggers are organized in a hierarchy by dotted name: logging.getLogger("myapp") and logging.getLogger("myapp.parser") have a parent-child relationship. Log messages propagate up the hierarchy unless logger.propagate = False is set. The root logger (accessed via logging.getLogger() or logging.basicConfig()) is the ancestor of all named loggers. This allows library code to log under its own name, and application code to configure where and how those messages appear.
A handler defines where log records go: StreamHandler writes to a stream (stdout or stderr by default), FileHandler writes to a file, RotatingFileHandler caps file size and rotates, SMTPHandler emails. A formatter defines how records look: %(asctime)s, %(levelname)s, %(name)s, %(message)s, %(filename)s, %(lineno)d. Attach multiple handlers to one logger for simultaneous console + file output with independent level filters.
logging.basicConfig() configures the root logger with a single handler in one call — appropriate for scripts. For applications with multiple modules, create named loggers per module with logging.getLogger(__name__) and configure handlers once at the application entry point. Named loggers are created once and reused — getLogger("foo") always returns the same object.
Each log call creates a LogRecord object with metadata: timestamp, level, logger name, filename, line number, process ID. The formatter extracts fields from the LogRecord using % codes. For machine-readable logs, use logging.Formatter(json.dumps({"time": "%(asctime)s", "level": "%(levelname)s", "msg": "%(message)s"})) or a structured logging library like structlog. Structured logging is essential when your log output will itself be parsed by another log analyzer.