Using the Logger middleware with a database


#1

Hello,

I’m looking to store the results from the Logger middleware into a database (SQLServer for now). I’m wondering how to do this with the middleware pattern being used with echo. Right now I’m logging it to a file using Lumberjack (https://github.com/natefinch/lumberjack), setting the output of the Logger middleware to Lumberjack.

Has anyone done this before and have a patter that works with Echo? I’m familiar with setting up a data access layer, but not so sure how to go about using the data from the Logger in a custom way that lets me populate SQL queries with the information from the logger.

Any ideas or hints are appreciated!

Thanks,
Johann

Code that I’m using right now:

main.go

func main() {
	runWebServer()
}

func runWebServer() {
	// initialize new echo web server
	e := echo.New()

	// initialize logging subsystem
	logLevel := os.Getenv("log_level")
	if logLevel != "DEV" {
		e.Use(middleware.LoggerWithConfig(middleware.LoggerConfig{
			Format: `{"time":"${time_rfc3339_nano}","id":"${id}","remote_ip":"${remote_ip}","host":"${host}",` +
				`"method":"${method}","uri":"${uri}","status":${status}, "latency":${latency},` +
				`"latency_human":"${latency_human}","bytes_in":${bytes_in},` +
				`"bytes_out":${bytes_out}}` + "\r\n",
			Output: &lumberjack.Logger{
				Filename:   filepath.Join("logs", "logfile.log"),
				MaxSize:    500,
				MaxBackups: 3,
				MaxAge:     30,
			},
		}))
	} else {
		e.Use(middleware.Logger())
	}
	// initialize data access layer to database
	dal, err := dal.NewMSSQL(os.Getenv("database_conn"))
	if err != nil {
		panic(err)
	}

	// configure routing tables
	api.Route(e, dal)

	// start echo web server
	e.Logger.Fatal(e.Start(":" + os.Getenv("port")))
}

#2

Hi there,

If you take a look at https://github.com/natefinch/lumberjack/blob/v2.0/lumberjack.go#L135 you can see that it implements io.Writer, which is not surprising considering that it is required for the Output field of LoggerConfig.

You can use a similar pattern: pass in a type (that can set the connection details) which implements io.Writer. The Write() method would then get what the logger gets, but instead of writing it to a file it sends it to a database. However if you are writing this for a production environment, adding time consuming and/or prone to error things to the request/response cycle is not a good practice (for example, what if you cannot temporarily send data to your database? should the request really error out, or should you rather enqueue and retry later; what if it takes too long, etc.).

From that perspective, your logger could be even simpler as it should be decoupled from the DB anyway, and you should instead simply use it to enqueue log entries (and have a separate goroutine read them and send them to the db, possibly batch them, possibly handle retries, etc.).

It could look something like:

type logger struct {
    ch chan []byte
}

func (l logger) Write(p []byte) (n int, err error) {
    l.ch <- p
    return len(p), nil
}

func main() {
    ch, wg := make(chan []byte), new(sync.WaitGroup)
    wg.Add(1)

    go runLogger(ch, wg)
    runWebServer(ch)

    close(ch)
    wg.Wait()
}

func runLogger(ch chan []byte) {
    defer wg.Done()

    for entry := range ch {
        // handle entry - send to SQL, or group them and send them in batches,
        // handle errors, perform retries, etc.
    }

    // handle any cleanup, flush/close db connection, etc.
}

// Note you will need to implement graceful shutdown: https://echo.labstack.com/cookbook/graceful-shutdown
// so that runLogger() can also complete gracefully.
func runWebServer(ch chan []byte) {
    // ...
		e.Use(middleware.LoggerWithConfig(middleware.LoggerConfig{
			Format: `{"time":"${time_rfc3339_nano}","id":"${id}","remote_ip":"${remote_ip}","host":"${host}",` +
				`"method":"${method}","uri":"${uri}","status":${status}, "latency":${latency},` +
				`"latency_human":"${latency_human}","bytes_in":${bytes_in},` +
				`"bytes_out":${bytes_out}}` + "\r\n",
			Output: logger{ch},
		}))
    // ...
}

Hope it helps!

Cheers!
Alex