Microservices

Multi-tenant cloud architectures. How to make a decision?

In this post, I want to show you two different cloud architectures for multi-tenancy. It doesn’t matter if you are planning to have a SaaS product with millions of tenants or just a handful of them, it is important to think about the pros and cons of the different models.

We will discuss the simplified cloud infrastructure architecture, not the code. Should you have separate databases? replicate workloads? a different load-balancer?
Let’s answer all those questions.

Multi-User vs Multi-Tenant. What do we mean by multi-tenancy?

A quick google search shows a lot of different answers for that question, so let’s clarify what we will be covering in this post.
When we say multi-tenant we refer to a software architecture designed in such a manner that it can serve multiple users, grouped in tenants. Not only a multi-user SaaS app.
For example, a software whose customers are companies, and the users are the companies employees.

Multi-User vs Multi-Tenant

In this post, we will be focusing on the multi-tenant architectures, not the multi-user ones. Therefore the analysis is going to be focused on 4 items:

  • Ingress
  • Backend
  • Data
  • Cloud Infra and Security
  • Product compatibility. When to choose each one?

Architecture #1: tenant aware app

Tenant aware app (single database)

When we say tenant aware, we are saying that the application contains the logic for splitting the traffic/data between tenants. So, for example, based on subdomains or using a login form with a “tenant Id” field (any parameter that identifies the tenant) the application contains code that is aware of from which tenant the request is coming from.

Ingress, backend and data layers

For this approach, the ingress layer and the backend layer are the same for all tenants. The important separation has to be done in the code (as we discussed above) and for the data storage we have mainly 2 options:

  • Single database – shared schema or multiple schemas
  • Multiple databases
Tenant aware app (multiple databases)

Cloud Infra and Security

The multi-tenancy is achieved in a hybrid manner, part of the work is done by the infrastructure and the other part is done at the application itself. The cloud part is easily achievable here, you just need a scalable ingress, backend, and database.

It is a good option when your application is going to be built mostly on serverless services, like cloud functions/lambdas or app engine/beanstalk. In this case, the ingress and backend infrastructure is managed by your cloud provider. For the data layer, either option you choose (single DB vs multiple DB) you will have to come up with some automation to create new tenants.

Product compatibility. When to choose it?

The main infrastructure is unique and shared between tenants therefore it is cheap. It allows you to scale down to the lowest amount of resources possible, thus saving money.

It’s a great choice for products that will have thousands, millions of tenants. Generally, those products are not expensive for the tenants. Let me give you an example… imagine a cloud email app. Thousands of companies will pay a little money for it.

When using the “tenant-aware” architecture, making sure your code is good and your tenants are properly isolated becomes a priority. By no means you want to have problems with mixed data or causing cross issues between two different tenants.

Approach #2: replicated workloads

Replicated Workloads (multiple ingresses)

For this approach, the multi-tenancy is resolved completely at the infrastructure level, therefore, the code for each tenant is not aware of any multi-tenant capabilities.

Ingress, backend and data layers

The ingress can be multiple ingresses (above image) or a single one (below image) that reroutes the traffic based on subdomains or paths in the URL, or any other tenant identifier field that a “smart” load balancer can detect, for example, the AWS Application Load Balancer can redirect the traffic based on a header value.

Replicated Workloads (single ingresses)

The decision either to choose one ingress vs multiple ingresses comes with some tradeoffs too.
If you have a single ingress, depending on the type of ingress you design, you may face a scalability issue. For example, if you are using Google Cloud HTTPs Load Balancer as a single entry point, and you have one subdomain per tenant, you will need to double-check your SSL certificates because there is a hard limit of the number of certificates the load balancer can have.
If you go with the multiple ingresses approach, make sure you have everything automated because managing a lot of ingresses will become an issue when the number of tenants raises up.

Cloud Infra and Security

There is a backend and a data layer completely dedicated to each tenant, thus we have completely isolated and therefore more secure environments.

The negative part is that we have more resource usage. Instead of grouping compute resources together, you will have idle resources laying around everywhere. Fortunately, Kubernetes comes to help us here! This is a good approach when you plan to use microservices and/or dockerized workloads. Replicating namespaces and workloads is not “that expensive” with Kubernetes. Of course, you need to think about the data and ingress layers too.

Product compatibility. When to choose it?

The tradeoff is clear, by using this approach, your infrastructure costs more, but probably you do not need to spend a lot of time in making your code “tenant-aware”.

This is a good option when your product is expensive for the customers so you can afford to have a more expensive infrastructure.
Again, if security and isolation are really important, this approach is a must for complying with the standards.

Conclusion

Each architecture has benefits and tradeoffs, but the important takeaway from here is that it is not only a technical decision but it has to be aligned with the expectations from the product side of your SaaS application. How much it will cost to obtain new customers? How much your customers will be willing to pay for your product? is isolation a key?

I hope this post helps you to trigger the right discussions. 🙂

Microservices architecture – orchestrator, choreography, hybrid… which approach to use?

So you want to build microservices, you have identified the properly isolated domains and boundaries for each one of them and now it’s time to find out how to make them interact with each other… Now is when you ask yourself, should we use an orchestrator approach? a choreography approach? mmmm, maybe a hybrid solution?

In this post, I’m going to talk about the different microservices architectures, and I will try to answer those questions, so when you have to make the call you have at least an idea of what are the tradeoffs of each one.

The orchestrator architecture

Orchestrator architecture

On the orchestrator architecture, our users will probably hit an api-gateway, which will then trigger an event on the orchestrator. Now the orchestrator service is in charge of executing the business logic by making requests to the other microservices while keeping track of the event status. It is a centralized controller and the requests are often synchronous.

Let’s dive into the pros and cons of the orchestrator approach:

Pros

  • Business logic is “hardcoded” and tangible
  • Easy request flow and status tracking

Cons

  • Tightly coupled services. Dependencies
  • Single point of failure

The choreography architecture

Choreography architecture

The choreography architecture, often called message broker or reactive architecture is, reactive and loosely coupled. Here, a message/event broker receives the requests and the microservices subscribe to them. Each one of them triggers when a certain message appears in the broker. If needed, after a service is triggered by a message it can also talk to other services directly, to chain different flows. Messaging is asynchronous here, with a publisher/subscriber pattern.

It has some advantages over the orchestrator architecture, but it comes with other problems, let’s dive into them:

Pros

  • Loosely coupled services
  • Isolated, independent packages of services
  • No single point of failure

Cons

  • Monitoring and observability are quite complex
  • Difficult request flow control (timeouts, retries, errors)
  • End-to-end knowledge of the system, where is the business logic?

Benefits and tradeoffs of both approaches

A typical tradeoff of the orchestrator approach is that you have tightly coupled services, on this topic, the choreography approach is the clear winner.

The single-point-of-failure vs no-point-of-failure would score another point for the choreography architecture, yet it is something that can be solved with the underlying cloud architecture and some good programming skills. So, no clear winner here.

Monitoring, observability, and flow control are big benefits for the orchestrator approach, so it is the clear winner. Something simple like setting up a timeout for a request is a pain to do in the choreography architecture and don’t try to find out where your system is failing on a choreography architecture, it is a nightmare if you haven’t spent a lot of time on properly monitoring the system.

However, there are other benefits and tradeoffs that are almost never mentioned…
On an orchestrator architecture, the orchestrator itself contains the business logic. It is there. You can read the actual code. On the other hand, on the choreography architecture, you have to read a document or see a graphic, because only a few people really know the whole end-to-end architecture. This becomes an issue when for example, someone leaves the company, understanding end-to-end complex choreographies can be really a problem.

The choreography approach is often a popular choice done by “cloud-native architects”. This is because most cloud providers have some really good event/messaging systems and there are a lot of tools with buzzwords names that support it. However, for developers, (yes, those that will actually write the code) thinking about a system asynchronously is quite difficult and it means an important mind shift.

The hybrid approach

With all these benefits and tradeoffs of both architectures, there is no clear standard path to choose. Both have their own pros and cons, and the tradeoffs of each approach are different. But if you look closer, they kind of complement each other. If we could have a mix of the benefits from the orchestrator approach and the choreography approach, it would be ideal!

Enters the hybrid approach, and there are many ways you can implement a mix of both architectures, but let’s focus on the next example:

Hybrid architecture

On the hybrid architecture, we combine the two things, an orchestrator and an event tracker. The requests are received by both and the orchestrator starts to execute the business logic but keeping track of the request in sync with the event tracker. This allows the microservice C, B, and D to do decoupled things, based on the event subscription/status, while the orchestrator still orchestrates what happens with the whole flow and directly controls microservices E and F.

With this mix, we can have tightly coupled services when sequentiality is needed, and loosely coupled services when we need to do parallel things “in the background”

When to use each one?

The orchestrator approach is good when you have a business-critical sequential flow. There, the orchestrator can handle the “what if” cases. For example, if one microservice needs to return successfully before calling a second microservice.

The choreography approach is good when you need to do a lot of things in parallel, it allows you to have faster processing times and it scales easily. For example, job processing in the background

The hybrid approach is more flexible, here you can have parallel asynchronous processing while still keeping track of the business logic when it is needed. For example, job processing in the background, but only after a logic sequence of services has been addressed.

Conclusion

Whatever approach you decide to follow, take it as a guide, but not be limited to it.
There are perfectly fit use-cases for each approach, no one is better than the other, I prefer the hybrid approach because of its flexibility.

Service Mesh comparison – Istio vs LinkerD

What is a service mesh?

A service mesh is in charge of managing the network traffic between the services. It does it in a more automated and scalable way compared to otherwise what it would take a lot of manual work. (and we know that manual work is not good 🙂 )

In other words, the service mesh lays in top of Kubernetes, takes over your services networking and makes the communications safe and reliable. It allows to connect, secure and monitor your services.

You can focus on writing your microservices and leverage the observability, network, and security policies to the service mesh.

Key capabilities of a service mesh

We can split the key capabilities in three:

  • Traffic Management
    • mTLS
    • Fault Injection
    • Circuit Breaking
    • Deployment Strategies
  • Metrics/Observability
    • Out-of-the-box monitoring and tracing tools
  • Security/Policies
    • Enforce policies and isolation

A service mesh enables “intelligent route control” capabilities, along with end-to-end encryption in the communications between the services. In addition, it also enables fault injection and advanced deployment strategies like blue/green, canaries, rolling upgrades, and more.

A service mesh becomes the “dashboard of your microservices“, a place where you can view and control what’s going on inside your cluster.

Service mesh comparison

When it comes to service mesh for Kubernetes, there are two big players, Istio and LinkerD. Both have amazing features and work in a very similar way so it is often a complicated choice.

Istio

  • Istio is a Kubernetes native service mesh, but it supports other orchestration tools like Consul and even VMs.
  • It was created by Lyft, but Google and IBM are now supporting it. Today Anthos comes with “Istio” service mesh capabilities.
  • It relies on a control plane and sidecars for working
  • We can say it is the most popular Kubernetes Service Mesh
  • It is very customizable
  • It is complex

LinkerD

  • Mirrors Istio architecture closely. (sidecars, control plane)
  • It is part of the Cloud Native Foundation (CNCF)
  • LinkerD focuses on simplicity instead of Flexibility
  • It is a Kubernetes-only solution
  • We can say it is the second most used Kubernetes service mesh

Comparison Table

Here you can find a summary of a comparison between the features:

ISTIOLINKERD
Mesh Features
Encrypted trafficYes. Auto mTLSYes, but not for TCP. (mongo connection)
Inter microservices traffic managementYes. PoliciesLimited
Microservices metrics for automatic scalabilityYes. PrometheusYes. Prometheus
Real-time mesh status observabilityYes. Kiali. (more graphical)Yes. (own dashboard, simpler)
Microservices Rate LimitingYesNo
Microservices Testing Features (fail injection, delays, % balancer)YesLimited
Microservices Circuit breakingYesNo
Microservices Release mechanisms (canary, green/blue, etc.)YesYes
Monitoring integration and metrics standards (opentelemetry)Yes. Grafana, Jaeger (tracing)Yes
Access Logs GenerationYesNo out-of-the-box (possible with plugins)
Does it work for Inter-cluster communications? (multi-cluster)YesNo
Mesh Generic stuff
ComplexityHighLow
Easy of UseComplexMedium
Companies behind itGoogle, Lyft, IBMBuoyant
Market acceptance/maturity. Community sizeLargeMedium
Support ModelCommunity and Google on AnthosCommunity. Buoyant
License/CostApache 2.0Apache 2.0
ProsMany Features. Can be extended.Easy to Use
ConsComplexDeeply Integrated with kubernetes. Cannot be expanded.

Comparison Conclusion

In conclusion, Istio is very flexible, but it is also very complex, therefore the learning curve is really big, yet it allows you to do more. LinkerD on the other hand focuses on simplicity, so it is easy to use but less customizable.

Microservices contract and versioning

What is it? Why is it important?

This is a series of posts I’m doing about designing microservices. Keep posted I will link them up when they are all ready!

When you are designing a microservices architecture, whether you are using a REST or messaging approach for communication between microservices, you have to design the APIs/messages and how a microservice will interact with each other.

One of the most important aspects of the microservices architectures is the ability to work on and deploy a microservice, completely independent of each other. To achieve this, each microservice must provide a well-defined, versioned contract.

The microservices contract

A microservice contract is between the service and its clients. The main goal is that you can make changes to the service without affecting the clients, nevertheless if the clients are aware of the service changes or not.

Even if you spent a lot of time the first time, designing the initial contract, for certain, the API will need to change over time.

When the time comes to update an API, it is important to understand the difference between breaking and non-breaking changes, when a major release is required and when to dispose of an old version.

When the changes are small, for example, adding a new parameter to the API, if that parameter isn’t business-critical, the clients should be able to consume the API in the old way, without sending or expecting to receive that parameter, and the server should fill the blanks with default values.

However, if you are doing a major, backward-incompatible, change to the API, you will need to maintain the old version for some time because you as a service cannot force your clients to update immediately.
If you are using a REST approach, one way is to add a versioning number in the path, for example /app/v1/service, app/v2/service. This way you can have two or more versions of your microservice available.

That is key to understand, if you are not doing a breaking change, there is no need for a new version of your contract.

Resource management strategy for docker containers on Kubernetes (nodejs + express)

It seems simple, but it is not.

How to properly assign resources requirements and limits to a HTTP dockerized microservice running on kubernetes?

Good question, right?

Well, as you can imagine there is not a single answer to it. But there is a strategy you can follow. This is not a “book definition” what I mean with this is that there might be another way of sizing resource requirements, but, so far, I’ve been using this method successfully.

It is a delicate balance between the hardware size on which your containers will run (ie: k8s nodes) and how many requests the container itself can handle.

The idea is to find that delicate balance so our pods can be scheduled on the nodes, without wasting precious resources that can be used for other workloads.

So, how do we do that?

A simple method:

Let’s take a simple microservice as an example: a nodejs express REST API which talks to a MongoDB.
We will be doing: GET /example

Run the container on your local machine or a k8s cluster, measure the idle resource usage:

  • CPU: 1%
  • Memory: 80Mb


Using a load testing app like Apache JMeter, we fire up 100 concurrent requests in 1 second… measure the resource spike using docker stats or similar:

  • CPU: 140%
  • Memory: 145Mb

We exceeded one CPU core, and that is not good, so we reduce the number of parallel requests to 70

  • CPU: 93%
  • Memory: 135Mb

Now, we have something to work with. We can assume that a single instance of our app, can handle up to 70 requests per second.

Setting Kubernetes resources:

According to how resource limits and requests are set in Kubernetes, we can say that it is safe to consider that our resource requests can be:

  • CPU: 0.1
  • Memory: 100Mb

And we can limit the resources to:

  • CPU: 1
  • Memory: 150Mb

That will ensure that we can handle at least, 70 requests per second, per replica.

Disclaimer: as I said before, this is a simple approach, we are not considering several things, for example, node resources, MongoDB capacity…