Protocol-Agnostic Design for Scalable and Maintainable Backends

When building out backend services, I have a general philosophy of isolating the boundaries of applications. This translates to one of the core principles I use when building applications: Abstract the transport layer from your business logic.

The transport layer is typically responsible for serializing and deserializing data across a network, translating requests, responses, and errors for the business layer, and routing network requests to the business layer. This makes it easy to reuse the business layer across various network protocols, enforces separation of concerns, and improves testability of your application.

I’m being intentionally vague about what happens in the business layer, as that is highly specific to the application you are building and can have its own separate patterns or architecture. However, for transport layer abstraction to work the business layer must consistently use:

Clearly defined request objects
Clearly defined response objects
Well-scoped error types

A Simple HTTP Service

source code

To illustrate the usefulness of transport layer abstraction, we will use a common example of updating a user in a database. Let’s start by implementing an extremely simplified HTTP endpoint in Go using the echo framework.

package main

import (
	"net/http"

	"github.com/labstack/echo/v4"
)

type User struct {
	ID        string `param:"id"`
	FirstName string `json:"first_name"`
	LastName  string `json:"last_name"`
}

func main() {
	userDB := make(map[string]*User)
	e := echo.New()
	e.PUT("/users/:id", updateUser(userDB))
	e.Logger.Fatal(e.Start(":8080"))
}

func updateUser(userDB map[string]*User) echo.HandlerFunc {
	return func(c echo.Context) error {
		var user User
		err := c.Bind(&user)
		if err != nil {
			return c.NoContent(http.StatusBadRequest)
		}
		dbUser, ok := userDB[user.ID]
		if !ok {
			return c.NoContent(http.StatusNotFound)
		}

		dbUser.FirstName = user.FirstName
		dbUser.LastName = user.LastName

		return c.JSON(http.StatusOK, dbUser)
	}
}

Now that we have built out the service let’s add some tests.

package main
...
func TestUpdateUser(t *testing.T) {

	t.Run("should update user when user exists", func(t *testing.T) {

		mockDB := map[string]*User{
			"test_user_id": &User{
				ID:        "test_user_id",
				FirstName: "Test",
				LastName:  "User",
			},
		}
		userJSON := `{"first_name":"UpdatedTest", "last_name":"UpdatedUser"}`

		e := echo.New()
		req := httptest.NewRequest(http.MethodPut, "/", strings.NewReader(userJSON))
		req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON)
		rec := httptest.NewRecorder()
		c := e.NewContext(req, rec)
		c.SetPath("/users/:id")
		c.SetParamNames("id")
		c.SetParamValues("test_user_id")
		h := updateUser(mockDB)

		if assert.NoError(t, h(c)) {
			assert.Equal(t, http.StatusOK, rec.Code)
			var response User
			if !assert.NoError(t, json.Unmarshal(rec.Body.Bytes(), &response)) {
				t.FailNow()
			}
			assert.Equal(t, mockDB["test_user_id"], &response)
			assert.Equal(t, "UpdatedTest", mockDB["test_user_id"].FirstName)
			assert.Equal(t, "UpdatedUser", mockDB["test_user_id"].LastName)
		}
	})

	t.Run("should return 404 when user not found", func(t *testing.T) {

		mockDB := map[string]*User{
			"test_user_id": &User{
				ID:        "test_user_id",
				FirstName: "Test",
				LastName:  "User",
			},
		}
		userJSON := `{"first_name":"UpdatedTest", "last_name":"UpdatedUser"}`

		e := echo.New()
		req := httptest.NewRequest(http.MethodPut, "/", strings.NewReader(userJSON))
		req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON)
		rec := httptest.NewRecorder()
		c := e.NewContext(req, rec)
		c.SetPath("/users/:id")
		c.SetParamNames("id")
		c.SetParamValues("bad_user_id")
		h := updateUser(mockDB)

		if assert.NoError(t, h(c)) {
			assert.Equal(t, http.StatusNotFound, rec.Code)
		}
	})

	t.Run("should return 400 when request is invalid", func(t *testing.T) {

		mockDB := map[string]*User{
			"test_user_id": &User{
				ID:        "test_user_id",
				FirstName: "Test",
				LastName:  "User",
			},
		}
		userJSON := `{"first_name":"UpdatedTest", "last_name":300}`

		e := echo.New()
		req := httptest.NewRequest(http.MethodPut, "/", strings.NewReader(userJSON))
		req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON)
		rec := httptest.NewRecorder()
		c := e.NewContext(req, rec)
		c.SetPath("/users/:id")
		c.SetParamNames("id")
		c.SetParamValues("bad_user_id")
		h := updateUser(mockDB)

		if assert.NoError(t, h(c)) {
			assert.Equal(t, http.StatusBadRequest, rec.Code)
		}
	})
}

We now have a working and tested service that allows clients to update users. Unfortunately, requirements have changed and now we need to also expose a GRPC service for updating users. Now we have to:

Re-implement the update logic in the gRPC server
Recreate tests for the new server
Risk having divergent behavior between the gRPC service and the HTTP service.

What if we wanted to accept updates from a queue like AWS SQS? Let’s implement another handler that listens to an SQS queue and reimplement the update logic there.

There is also something subtle about the test we already wrote. We are testing both the business logic and the transport layer logic for each test case. Notice the tags on the user struct? What if we wanted to use XML instead of JSON? We would have to update the tags on the user struct. There is also the logic around the ID being a path param instead of a apart of the request body. These are all things that only matter to the transport layer. Most projects start off simple, but it’s not uncommon for requirements or an application’s context to change.

Here’s how the architecture evolves when you decouple the transport layer from the business logic:

Abstracting the Transport Layer

source code

The current service example works, but it’s hard to change how the service runs because our business logic is tightly coupled to HTTP. If we ever need or want to change how clients interact with the service we have to duplicate both the business logic and the error handling for any new protocols, and re-test both the protocol layer and the business logic. If we instead abstract the transport layer we can minimize the changes needed to the service to support new protocols.

By decoupling the business layer from the transport layer you enable:

A single source of truth for business logic
Reusable and isolated components for business logic
Protocol-specific handlers that focus only on mapping requests, responses, and errors from the business layer to the transport layer

Let’s look at how to refactor our current example.

package main
...
func main() {
	userDB := make(map[string]*users.User)
	usersService := users.Service{
		DB: userDB,
	}

	httpService := transport.NewHTTPServer(&transport.HTTPConfig{
		Port:         8080,
		UsersService: usersService,
	})

	httpService.Start()
}

Application Errors

Define shared application error types that are mapped to protocol-level errors by the transport layer.

package errors
...
var (
	ErrNotFound   = errors.New("not found")
	ErrBadRequest = errors.New("bad request")
)

Users Business Layer

package users
...
type UpdateUserRequest struct {
	ID        string
	FirstName string
	LastName  string
}

type UpdateUserResponse struct {
	ID        string
	FirstName string
	LastName  string
}

type Service struct {
	DB map[string]*User
}

func (s *Service) UpdateUser(request UpdateUserRequest) (UpdateUserResponse, error) {
	user, ok := s.DB[request.ID]
	if !ok {
		return UpdateUserResponse{}, fmt.Errorf("unable to find user in db: %w", appErr.ErrNotFound)
	}

	if request.FirstName != "" {
		user.FirstName = request.FirstName
	}

	if request.LastName != "" {
		user.LastName = request.LastName
	}

	return UpdateUserResponse{
		ID:        user.ID,
		FirstName: user.FirstName,
		LastName:  user.LastName,
	}, nil
}

This is pure business logic: no HTTP, no network serialization, no JSON binding, no error mapping.

Transport Layer

package transport
...
type HTTPServer struct {
	e    *echo.Echo
	port int
}

func NewHTTPServer(config *HTTPConfig) *HTTPServer {
	e := echo.New()
	registerHTTPRoutes(e, config.UsersService)
	return &HTTPServer{
		e:    e,
		port: config.Port,
	}
}

func (server *HTTPServer) Start() {
	server.e.Logger.Fatal(server.e.Start(fmt.Sprintf(":%d", server.port)))
}

func registerHTTPRoutes(e *echo.Echo, usersService users.Service) {
	e.PUT("/users/:id", updateUser(usersService))
}

type updateUserRequest struct {
	ID        string `param:"id"`
	FirstName string `json:"first_name"`
	LastName  string `json:"last_name"`
}

type updateUserResponse struct {
	ID        string `json:"id"`
	FirstName string `json:"first_name"`
	LastName  string `json:"last_name"`
}

func updateUser(usersService users.Service) echo.HandlerFunc {
	return func(c echo.Context) error {
		var request updateUserRequest
		err := c.Bind(&request)
		if err != nil {
			return c.NoContent(http.StatusBadRequest)
		}

		resp, err := usersService.UpdateUser(users.UpdateUserRequest{
			ID:        request.ID,
			FirstName: request.FirstName,
			LastName:  request.LastName,
		})

		if err != nil {
			return c.NoContent(MapErrorToHTTPStatus(err))
		}
		return c.JSON(http.StatusOK, updateUserResponse{
			ID:        resp.ID,
			FirstName: resp.FirstName,
			LastName:  resp.LastName,
		})
	}
}

func MapErrorToHTTPStatus(err error) int {
	switch {
	case errors.Is(err, appErr.ErrNotFound):
		return http.StatusNotFound
	case errors.Is(err, appErr.ErrBadRequest):
		return http.StatusNotFound
	}
	return http.StatusInternalServerError
}

This layer handles routing, JSON decoding, and error mapping. The business logic doesn’t know or care that it’s being called from HTTP.

If you wanted to expose this over gRPC, you’d just implement another transport layer using the same users.Service interface.

Each transport acts like a plug-in, wiring into the same business logic core.

diagram showing abstractions of the transport layer

Adding support for SQS

The same principles apply if you’re receiving data from a message queue like Amazon SQS. Here’s an example that uses a dedicated request object and reuses the exact same business logic. This separation allows each transport to decode, validate, and forward requests independently, keeping business logic untouched.

package transport
...
type updateUserSQSRequest struct {
	ID        string `json:"id"`
	FirstName string `json:"first_name"`
	LastName  string `json:"last_name"`
}

type SQSServer struct {
...
}

func (s *SQSServer) Start(ctx context.Context) {
	for {
		output, err := s.Client.ReceiveMessage(ctx, &sqs.ReceiveMessageInput{
			QueueUrl:            &s.QueueURL,
			MaxNumberOfMessages: 1,
			WaitTimeSeconds:     10,
		})

		for _, msg := range output.Messages {
			var req updateUserSQSRequest
			if err := json.Unmarshal([]byte(*msg.Body), &req); err != nil {
				log.Printf("failed to unmarshal SQS message: %v", err)
				continue
			}

			resp, err := s.UsersService.UpdateUser(users.UpdateUserRequest{
				ID:        req.ID,
				FirstName: req.FirstName,
				LastName:  req.LastName,
			})
			if err != nil {
				log.Printf("unable to update user: %v", err)
				continue
			}

			log.Printf("updated user from SQS: %+v", resp)
			_, err = s.Client.DeleteMessage(ctx, &sqs.DeleteMessageInput{
				QueueUrl:      &s.QueueURL,
				ReceiptHandle: msg.ReceiptHandle,
			})
		}

		time.Sleep(s.PollInterval)
	}
}

Why This Matters

Yes, this adds boilerplate and might seem like overengineering for simple services, but this abstraction makes the service significantly easier to change and test.

Want to validate first names? Just update the service logic.
Want to expose gRPC? Add a gRPC handler that uses the same service.
Want to restrict HTTP to only update first names? Change only the HTTP handler.

You no longer have to rewrite logic, and you’re removing the coupling between the network and your application.

Final Thoughts

Most services start off small and focused, but business requirements change frequently. Abstracting away the transport layer from business logic helps keep your services maintainable, testable, and extensible. You don’t have to account for every requirement up front, but you make it significantly easier to accommodate changes later. This also gives structure to your applications and makes you intentional about how you build out services.