Log Rotation and File Splitting in Go: Insights from logrus, zap, and slog

Preface

In existing logging libraries, including Go’s built-in log/slog logging library, they typically support log file rotation and splitting. However, these features are not built in directly—they need to be actively configured by us to enable them.

This article will explore several popular logging libraries, such as logrus, zap, and the official slog. We will analyze the key design elements of these libraries and discuss how they support the configuration of log rotation and splitting.

Brief Analysis of the Designs of logrus, zap, and slog

When comparing the design of logrus, zap, and slog, one prominent commonality is that they all include the crucial property of io.Writer. This property plays a central role in the design of logging frameworks, as it determines the target location for log output.

logrus Logging Library

logrus is a feature-rich logging library for Go, providing structured logging, log level control, and other features.

When using logrus, you can create a Logger instance by calling logrus.New(). With this instance, we can perform many operations, such as customizing the log output location and printing logs. Let’s look at the following code:

logger := logrus.New()
logger.Out = os.Stdout // Standard output
// Or redirect to a file
// out, err := os.OpenFile("file.log", os.O_CREATE|os.O_WRONLY, 0666)
// if err != nil {
//   panic(err)
// }
// logger.Out = out

The definition of the Logger struct is as follows:

type Logger struct {
    Out io.Writer
    Hooks LevelHooks
    Formatter Formatter
    // Other fields...
}

The key property is Out, whose type is io.Writer. This property is used to specify the log output target, whether it’s standard output, a file, or another custom output medium.

zap Logging Library

zap is a highly performant logging library. It provides structured logging, multi-level log control, and flexible configuration options.

Similar to logrus, zap also allows you to decide the log output location via configuration, but the implementation differs slightly. In zap, log output is configured through zapcore.Core. When creating an instance of zapcore.Core, you need to specify an implementation of the zapcore.WriteSyncer interface as a parameter, which directly determines the target for log output. To create a zapcore.WriteSyncer instance, you usually use the zapcore.AddSync() function, which takes an io.Writer type parameter.

Here is a basic example of creating a log instance with zap:

writer := zapcore.AddSync(os.Stdout) // Use standard output as the log target
core := zapcore.NewCore(
    zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
    writer,
    zap.InfoLevel,
)
logger := zap.New(core)
defer logger.Sync() // Flush any buffered log entries

// Use logger to record logs

The key is the zapcore.AddSync() function, which takes an io.Writer type parameter used to specify the log output target, whether it’s standard output, a file, or another custom output medium.

slog Logging Library

slog is an official logging library introduced in Go 1.21.0, providing structured logging. If you want to learn more about the slog logging library, you can check out our previous article.

Similar to logrus and zap, slog also allows users to specify the log output target by providing an io.Writer parameter. This setting is made when creating an implementation of the slog.Handler interface.

textLogger := slog.New(slog.NewTextHandler(os.Stdout, nil))
jsonLogger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

In these two functions, the first parameter of slog.NewTextHandler and slog.NewJSONHandler is of type io.Writer.

Summary of the Analysis

From our analysis of the three mainstream logging libraries—logrus, zap, and slog—we can see a key commonality: when handling log output, all of them rely on the io.Writer interface. These logging libraries use the io.Writer interface as the type of a crucial parameter, allowing you to set the target of the log output.

Implementation Mechanisms and Practices of Log Rotation and Splitting

Implementation Mechanism

After analyzing the design of logrus, zap, and slog, we have discovered their commonalities. Now, let’s dive deeper into the mechanism of log rotation and splitting.

To implement log file rotation and splitting, we usually leverage third-party libraries such as lumberjack. Of course, there are other similar libraries available, but we won’t list them all here.

lumberjack is a library specifically designed for log rotation and splitting. Its function is similar to a pluggable component. By configuring this component and integrating it with your chosen logging library, you can achieve log file rotation and splitting.

Here is the code to initialize a lumberjack component:

log := &lumberjack.Logger{
    Filename:   "/path/file.log", // Location of the log file
    MaxSize:    10, // Maximum file size (in MB)
    MaxBackups: 3, // Maximum number of old files to retain
    MaxAge:     28, // Maximum number of days to retain old files
    Compress:   true, // Whether to compress/archive old files
    LocalTime:  true, // Use local time for timestamps
}

In this example, we create a lumberjack.Logger instance and set the following parameters:

Filename: Specifies the storage path of the log file.
MaxSize: The file will rotate when it reaches this many MB.
MaxBackups: The maximum number of old log files to keep.
MaxAge: The maximum retention period (in days) for old files.
Compress: Whether to compress old files (e.g., convert to .gz).

It is important to note that the Logger struct of lumberjack implements the io.Writer interface. This means all the core logic for log file rotation and splitting is encapsulated within the Write method. This implementation also makes it easy for the Logger struct to be integrated into any logging library that supports an io.Writer parameter.

Once you understand this, you probably already know how to implement log rotation and splitting. Since the logger struct of lumberjack implements the io.Writer interface, passing it into a third-party library allows you to complete the integration and configuration.

Practice

Implementation with logrus Logging Library

log := &lumberjack.Logger{
    Filename:   "/path/file.log", // Location of the log file
    MaxSize:    10,               // Maximum file size (in MB)
    MaxBackups: 3,                // Maximum number of old files to retain
    MaxAge:     28,               // Maximum number of days to retain old files
    Compress:   true,             // Whether to compress/archive old files
    LocalTime:  true,             // Use local time for timestamps
}
logger := logrus.New()
logger.Out = log

Implementation with zap Logging Library

log := &lumberjack.Logger{
    Filename:   "/path/file.log", // Location of the log file
    MaxSize:    10,               // Maximum file size (in MB)
    MaxBackups: 3,                // Maximum number of old files to retain
    MaxAge:     28,               // Maximum number of days to retain old files
    Compress:   true,             // Whether to compress/archive old files
    LocalTime:  true,             // Use local time for timestamps
}
writer := zapcore.AddSync(log)
core := zapcore.NewCore(
    zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
    writer,
    zap.InfoLevel,
)
logger := zap.New(core)
defer logger.Sync() // Flush any buffered log entries

Implementation with slog Logging Library

log := &lumberjack.Logger{
    Filename:   "/path/file.log", // Location of the log file
    MaxSize:    10,               // Maximum file size (in MB)
    MaxBackups: 3,                // Maximum number of old files to retain
    MaxAge:     28,               // Maximum number of days to retain old files
    Compress:   true,             // Whether to compress/archive old files
    LocalTime:  true,             // Use local time for timestamps
}
textLogger := slog.New(slog.NewTextHandler(log, nil))
jsonLogger := slog.New(slog.NewJSONHandler(log, nil))

Conclusion

This article provided a brief analysis of the design elements of three popular logging libraries: logrus, zap, and slog. We found that although they differ in the details of how logging instances are created, they all rely on the io.Writer interface parameter to handle log output. By mastering how to configure the io.Writer parameter and combining it with the lumberjack library, we can achieve log file rotation and splitting.

Even if new logging libraries are introduced in the future, we can quickly integrate log file rotation and splitting using similar methods.