1. Articles/

How I organize (most of) my Go microservices

·7 mins

librarian Gopher

When first learning Go, my initial instinct was to find a framework that would scaffold the basic folder structure and functionality that would allow me to build my first project. It didn’t take long to realize that Go, thanks to its rich standard library ecosystem, didn’t really need a framework, and often the framework works against you. Since then, I’ve onboarded many developers in my team that was coming from different languages and were also confused by a lack of standardized folder structure from different parts of the company.

One hard-to-overlook advantage of not having a pre-packaged boilerplate project is that it allows your team to structure the project in a way that best fits your very specific needs. On the other hand, some patterns are frequently repeated, and in those cases, I like to create template projects that can be used as a starting point. It should be minimal and built with easy-to-replace components.

First: a few ground rules #

When creating Go packages, I try to follow these rules:

  • keep the package name as simple as possible, if possible in a single word.
  • keep it all lowercase, with no separation.
  • keep the package name singular.
  • For each package, I usually have a .go file with the same name, declaring the main interface or struct of the package.
  • For non-package folders, it is ok to have a multi-word name, but separate them by -, not _.

The minimal folder structure #

.
├── 📁 .local
├── 📁 api
├── 📁 build
├── 📁 deployment
├── 📁 cmd
├── 📁 docs
│   ├── 📁 architecture
│   │   └── 📁 decision-records
│   ├── 📁 develop
│   ├── 📁 release-notes
│   └── 📁 runbooks
├── 📁 scripts
└── 📁 internal
    ├── 📁 app
    ├── 📁 logger
    ├── 📁 metrics
    └── 📁 server

📁 .local #

The .local folder is where I keep all the tools and configurations required to run the application locally, but that is not part of our deployment strategy. Secrets, docker-compose files, debugging libraries, and so on. Some of the files here are not committed to git.

📁 api #

The api folder houses all the API definition files for the supported protocols. That is OpenAPI specifications for REST, .proto files for gRPC, GraphQL schemas, etc.

📁 build #

The build folder contains building scripts and components, used to create images and binaries used for both deployment and local development. Here you will find Dockerfile, package configuration scripts, etc.

deployment #

The deployment folder contains all the deployment-specific files, such as helm charts and templates.

📁 docs #

These are (mostly) markdown files. Some of them are rendered by tools like Backstage or Mkdocs, and some are targeted toward developers working on the project and can be rendered in their IDE of choice.

Usually contains the following subfolders:

  • architecture: This is where I keep diagrams, decision records, etc
  • develop: This is the developer’s target documentation. E.g. how to debug a specific system, how to install a given library locally, and so on…
  • release-notes: here we keep a semantic version file with the release notes, explaining what has been fixed and which new features have been deployed.
  • runbooks: runbooks are a set of instructions on how to debug/solve production problems if we’ve seen them before.

📁 scripts #

Here I keep all the scripts, mostly bash and shell, but sometimes in Go as well. This includes scripts to build the docker image, set up the environment on the developer’s local machine, update libraries, etc.

There are usually invoked the Makefile in the root of the project. Keeping them here helps keep the Makefile small and simple.

📁 cmd #

The cmd folder contains the entry points for the application, it will contain folders that share the same name as the expected binaries, and each will usually contain a single, small, main.go file.

The main.go file can stay small because the business logic is delegated to the multiple packages inside the internal, (or in some cases pkg) directory. They are instantiated and configured by the app package. More on it below.

📁 internal #

The internal directory contains all the private packages and libraries that comprise the service. The code here is not meant to be imported by other applications.

In most of my projects, four internal packages are very often found: app, server, logger, and metrics. The options pattern is often present in these packages. It makes them flexible and re-usable since the variable configuration is coming from without, for example:

└── 📁 package
    ├── 📄 package.go
    └── 📄 option.go

The app package is the only one instantiated and ran directly from the cmd folder. I like to have it with the following signature

type App interface {
  New(context.Context, ...Options)
  Run(context.Context)
}

The app package is responsible for configuring (through respective options) all the other packages and injecting their dependencies. This pattern enables a simple use of Dependency Injection and configurability for each package.

It will also deal with gracefully shutting down all the other components if we receive a signal.

The other internal packages you will typically find in my boilerplate repository are:

  • 📁 logger: usually a wrapper around popular logging libraries, to adapt to my own use case.
  • 📁 metric: all the metrics, observability, and code-level monitoring. Likely prometheus logic, etc.
  • 📁 server: this is the actual server running the application. Because some projects I use REST, some gRPC, some both (and on rare occasions, GraphQL), I like to have my interface as:
 type Server interface {
   Start(context.Context, chan error)
   Stop(context.Context)
 }

Then each file http.go, grpc.go, graphql.go will implement the protocol-specific logic.

📄 Root files #

In the root of the project, you will also typically find the following files (amongst others):

  • 📄 .gitattributes
  • 📄 .gitignore
  • 📄 .golangci.yaml
  • 📄 .editorconfig
  • 📄 CODEOWNERS
  • 📄 CONTRIBUTING.md
  • 📄 LICENSE.md
  • 📄 Makefile
  • 📄 README.md
  • 📄 SECURITY.md
  • 📄 go.mod
  • 📄 go.sum

These also include other files like sonar-project.properties if using Sonarqube, catalog-info.yaml if using Backstage, and so on.

Final project skeleton #

.
├── 📄 .gitattributes
├── 📄 .gitignore
├── 📄 .golangci.yaml
├── 📄 go.mod
├── 📄 go.sum
├── 📄 .editorconfig
├── 📁 .local
│   ├── 📄 docker-compose.yaml
│   └── 📄 docker-compose.debug.yaml
├── 📄 CODEOWNERS
├── 📄 CONTRIBUTING.md
├── 📄 LICENSE.md
├── 📄 Makefile
├── 📄 README.md
├── 📄 SECURITY.md
├── 📁 api
│   └── 📄 openapi.json
├── 📁 build
│   └── 📄 Dockerfile
├── 📁 deployment
│   └── 📁 helm
│       ├── 📁 templates
│       ├── 📄 .helmignore
│       └── 📄 Chart.yaml
├── 📁 cmd
│   └── 📁 myapp
│       └── 📄 main.go
├── 📁 docs
│   ├── 📁 architecture
│   │   └── 📁 decision-records
│   ├── 📁 develop
│   ├── 📁 release-notes
│   │   ├── 📄 template.md
│   │   └── 📄 v1.0.0.md
│   └── 📁 runbooks
├── 📁 scripts
└──📁 internal
    ├── 📁 app
    │   ├── 📄 app.go
    │   └── 📄 option.go
    ├── 📁 logger
    │   ├── 📄 logger.go
    │   └── 📄 option.go
    ├── 📁 metrics
    │   ├── 📄 prometheus.go
    │   └── 📄 option.go
    └── 📁 server
        ├── 📄 grpc.go
        ├── 📄 http.go
        ├── 📄 option.go
        └── 📄 server.go

Next Steps: extracting reusable packages as dependencies #

As you can imagine, some of these folders inside internal end up being very similar from microservice to microservice. In certain cases, they can be extracted to be their own module and imported as a dependency. This is especially relevant if you want to update the package in a single location, and have the individual services control their dependency versions by choosing to upgrade it when they are ready.

The logger and metrics packages are good examples of packages that can be extracted as dependencies, as they are not going to change much from project to project.