{ "version": "https://jsonfeed.org/version/1", "title": "Authress Security and Engineering Blog", "home_page_url": "https://authress.io/knowledge-base/articles", "description": "Authress Engineering articles written and maintained by the Authress engineering team.", "items": [ { "id": "https://authress.io/knowledge-base/articles/2026/03/14/how-aws-can-fix-s3", "content_html": "
info

For help understanding this article or how you can implement auth and similar security architectures in your services, feel free to reach out to me via the community server.

AWS just released a supposed fix for S3 bucket squatting by utilizing what they are calling Account Regional Namespaces. I don't understand the hype, and now I'm going to explain why.

Broken: S3 Bucket Names are Global​

S3 bucket names are global. Not global to your account. Not global to your region. Global to the entire AWS partition — every account, every region, every customer who has ever existed on AWS.

This was not a deliberate design philosophy. It was a default from 2006 that nobody corrected. S3 launched when AWS was essentially a startup with Amazon as its main customer. Global uniqueness was the path of least resistance. Nobody asked whether it would cause problems at scale, because at the time \"scale\" meant a hundreds or thousand developers, not millions of accounts and decades of production workloads.

But, that default is still in place today.

\"A

AWS's relationship with the S3 naming model, circa every year since 2008.

The sad truth is, nobody needs global bucket names. There is no use case that requires your bucket name to be universally unique across every AWS customer on the planet. The value of global uniqueness flows entirely in one direction: it must have simplified the original implementation. The cost of global uniqueness flows in the other direction: two decades of pain for every customer who has ever tried to name a bucket something sensible.

The abomination lives on because someone probably said \"Wouldn't be cool if you could expose your S3 bucket publicly?\" And for that the bucket name would have to be in the URL, and therefore globally unique (and also require that the bucket name be lowercase and RFC 7553 compliant). This is true but also irrelevant. S3 doesn't even support TLS for custom domains. So there is no way to actually serve an asset such as https://assets.mycompany.com directly from your S3 Bucet. None, full stop. Let's break that down, there are three parts to that URL — HTTPS, your domain, and something that maps to the S3 bucket. It has always been, and still is only PICK 2.

Anyone who needs a public URL with a real domain and HTTPS already is using CloudFront as a reverse proxy. As a matter of fact, every SPA out there, must be using CloudFront in order to achieve HTTPS or they must not be using a custom domain. The only suitible URL is the CloudFront distribution's alias, not the S3 bucket name. The bucket name is internal plumbing that nothing outside your AWS account should ever reference directly. I'm here to tell you that not only are global bucket names a mistake, there is actually an easy way to fix it. One has to wonder why AWS hasn't.

The people who think they need global bucket names are the people using S3 Virtual Hosting — mybucketname.s3.amazonaws.com — which does have TLS, but on AWS's domain, not theirs. And of course, there the sad case for supporting this pattern indefinitely because AWS is much nicer than some other cloud providers that constantly deprecate actually required features, such as DNS Zone hosting. Although in recent times that hasn't held up as much, and gives credence to AWS dropping the concept as it would have direct Security and Reliability wins. Not to mention straight out improvement by reducing complexity. There is no case for making it the architectural foundation of an object storage service used by billions of production workloads. And as we will see shortly, exposing that endpoint directly comes with its own expensive problem that CloudFront eliminates entirely.

The reality is none of the following are tradeoffs you agreed to. They are the consequences of a default, set in 2006, that nobody changed. The cost has landed on you ever since. And boils down to basically one core concept.

Name squatting​

The boring version: the bucket name you want — mycompany-prod-logs, myapp-assets, opentofu-state — was registered years ago by someone who no longer works at the company that registered it. AWS has no mechanism for name reclamation. That name is gone until the current owner deletes the bucket, which may never happen. So what you might think, just choose a new name, like you would choose a new username, or website domain. This isn't a new problem after all.

But the reality is: bucket names are predictable, and predictable names are claimable before you need them, and it turns out some bucket names you actually very much need.

The researchers at Aqua Security demonstrated this at Black Hat USA 2024, calling it Bucket Monopoly. AWS services, themselves, create S3 buckets automatically use naming patterns derived from your account ID. Account IDs are not secret — they appear in IAM role ARNs, error messages, S3 URLs, and CloudTrail logs. And while good hygiene means keeping your AWS account ID obscured, the bucket names themselves must be completely public. S3 Virtual Hosting resolves every bucket as a DNS subdomain (mybucket.s3.amazonaws.com), Certificate transparency, and passive DNS collectors observe and index those queries continuously. And while they might not have caught everything, any bucket that has ever received traffic via Virtual Hosting has a name that likely exists in a DNS database outside your control.

Many naming patterns were vulnerable:

The complete impact ranged from data exfiltration to remote code execution to full-service takeover. AWS has patched many of these services after disclosure.

The CDK case may be the worst case. AWS's own infrastructure-as-code tool hack wrapper (because actually the CDK isn't the IaC tool) bootstraps a staging bucket with a name that was never random:

cdk-hnb659fds-assets-{account-id}-{region}

The qualifier hnb659fds is a hardcoded constant in CDK's bootstrap template. It has never changed. Anyone who knows your account ID knows your CDK staging bucket name. If that bucket does not exist — because you deleted it, or becouse you have not bootstrapped yet, or because someone cleaned up an old environment — an attacker can claim it. CDK will then use that bucket to store and retrieve CloudFormation templates. The attacker injects a malicious template. CDK deploys it using an IAM role with broad permissions. Full account takeover.

Aqua Security found over 38,000 accounts susceptible. The vulnerability was present for years before being fixed in CDK v2.149.0 in July 2024.

To be clear, an attacker who learns your AWS Account ID, can register those bucket names before you deploy the service. AWS will see that the bucket exists, trust it, and the route your data into the attacker's bucket. This is happening even without your knowledge. Have you actually checked that every bucket AWS is secretly sending data to is self-owned by your account? Probably not, you probably don't even know which buckets AWS is using.

Security Through Obscurity​

I thought it would go without saying, but I'm sure someone will bring it up: \"Keep your bucket name obscure\" is not a defense, since you can figure out these buckets by just using AWS services. And worse, the bucket name shows up in website hosting CNAMEs, presigned urls, and other places. It is publicly available.

And of course the inverse is also a problem. S3 bucket names carry implicit trust. When your infrastructure reads configuration from my-config-bucket, it assumes the content is authoritative because the name is correct. The global namespace means that assumption is structurally unsound — the name and the owner are not bound to each other in any durable way. An attacker who controls a bucket your infrastructure reads from doesn't need to exfiltrate anything. They inject. Your service pulls the configuration, trusts it, and acts on it.

This is not abstract. Consider the pattern of storing IAM permission mappings in S3 and distributing them via OU StackSets across an AWS organization. Something I actually just wrote about doing. An attacker who controls that bucket — whether by squatting the name, claiming it after a deletion, or exploiting a misconfigured access policy — can inject a permissions map that adds their own identity as a trusted principal. The StackSet propagates the poisoned configuration to every account in the org. Their CICD pipeline assumes the role via OIDC federation. Full organization-wide access, delivered through the normal configuration path, with no credentials created and no anomalous API calls.

This is the same pattern that made Clownstrike's botched configuration update in 2024 so severe. A trusted delivery mechanism pushed configuration that every endpoint pulled and acted on without independent verification. The delivery channel was correct. The content was not. Millions of machines followed instructions from a source they had no reason to distrust.

The difference is that Clownstrike's delivery infrastructure was their own, and the configuration was negligent, not malicious. Whereas the S3 version of this attack does not require compromising the infrastructure owner at all, it only requires claiming a bucket name.

The global namespace is what makes this entire attack class possible. In a correctly scoped namespace, your bucket names are yours, and an attacker in a different account cannot claim them. AWS built a shared global pool and then built their own services on top of it using predictable names, inheriting the vulnerability they created.

Security misconfiguration​

The public access model exists because bucket names are global. Since any AWS account can reference your bucket by name, making a bucket readable without credentials makes it readable by everyone — which is occasionally intentional and routinely catastrophic.

The deeper problem: S3's access control system has never cleanly separated \"accessible by my AWS account\" from \"accessible by the public internet.\" That distinction is not a first-class concept in S3. It has to be constructed from a combination of overlapping controls, each added at a different point in S3's history, each with its own interaction rules:

Each layer was added to contain the blast radius of the previous one. None of them establish \"private to my account\" as the starting point. They establish \"open to everything\" as the starting point and ask you to correctly configure the restrictions. Miss one flag, misread one grantee, inherit one policy from a module you didn't write — and the bucket is likely public.

I like this article from 6 years ago talking a bit about that

\"AWS

IAM access summarized

But then you realize, this is just how IAM works, it isn't how S3 works at all. Sure whether or not IAM grants access is part of the picture, but where's the rest of it? I was trying to find a document in the AWS Docs that does a good job of explaining. There isn't one. There are over One Hundred Pages on access control in S3 alone. Don't believe me, count them. To be fair we have more than one page on similar Authorization concepts in the Authress KB. However, arguably what we designed has to be significantly more complex, since it has to handle literally every possible authorization scenario.

This is not a configuration problem. It is an architecture problem. It is a security problem. The controls are layered on top of a model that was never designed to be private.

And while the likelihood of getting it wrong has gone down significantly, the trade-off has been increased burden on configuration and setup.

Historical Hacks​

Each problem identified by the community attracted a from AWS patch. But no one said they were the right patch.

Forced random suffixes

For buckets operated by AWS Services, you have no recourse, but for buckets you manage for your own platform, you have a small, but not very satisfying alternative. Because the global pool is full of names claimed by other accounts, you cannot have the names you want. my-app-assets is taken. opentofu-state is taken. prod-logs is taken. The community's answer to the problem, years before AWS even started to take any approach, is to use the only reliable strategy available — append a random suffix and stop trying to name things sensibly: my-app-assets-8f2a3c, opentofu-state-a1b2c3, prod-logs-9e4d71.

A list of your S3 buckets is now a list of opaque identifiers. Understanding which bucket belongs to which service requires either tagging discipline — which degrades over time — or reading OpenTofu state, which is stored in an S3 bucket with a random suffix. Not to mention this only gets around the creation problem, and doesn't remotely address the security angle.

This is not a novel problem. Discord ran the same experiment with usernames. Their original system appended a four-digit discriminator to every display name: warren#0088. Globally unique, unambiguous, machine-friendly. I don't remember anyone that could actually remember their discriminator. I can't imagine how many friend requests failed because users entered the wrong tag. With only 10,000 discriminators available per name, popular names of course ran out.

Discord's fix was not to make the discriminator longer. They separated the unique identifier — the username, used for backend lookups — from the display name, which is human-readable and non-unique. The part that needed global uniqueness was the lookup mechanism. The part humans see and share does not need to be globally unique at all.

S3 never made this distinction. The bucket name is simultaneously the unique global identifier, the human-readable label, and the public URL component. When all three concerns are collapsed into one string that must be globally unique across every AWS customer, you get my-app-assets-8f2a3c. That is your discriminator.

Forced predictable suffixes

For us we've taken a slightly different approach. And that's because random suffixes cannot be dynamically used at read time, are not idempotent, and that means usually hard-coding this string in multiple places. Or worse, I've seen many implementations attempt to export the generated S3 name from the infrastructure process to somewhere else, effectively coupling disparate systems that had no business being coupled together.

Our approach is to add the AWS Account ID, the Region, and an internal consistent identifier to ever bucket we create. Now everyone will understand what that means. For example, you can imagine you choose something like -${accountId}-${region}-un1que1d. Is that clever? Not really, but it is far better than having every bucket have a random ID.

The ExpectedBucketOwner property

One hack AWS added was integrating a new parameter into the S3 bucket APIs, which could validate ownership on bucket related actions such as Creation, PutObject, and GetObject. Released in Oct 2020, every S3 API call could now include the expected AWS account ID of the bucket owner. If the bucket exists but belongs to a different account, the call fails. You add this header to your SDK calls, your bucket policies, your presigned URL logic. The problem of AWS created buckets was so bad, that AWS needed an internal security fix for the problem. And this helped a little bit for us users as well. It isn't a real solution though, just something hacked on top.

The problem with this hack though, is that it is security you have to opt into, and if you are using some library or reusable module, good luck assuming that made it in.

CDK v2.149.0

In the July 2024 fix for the CDK boostrap, AWS merged a change that adds a condition to the CDK bootstrap role, preventing the attacker-controlled-bucket scenario. However, the fix still required teams to re-run cdk bootstrap. Any environment bootstrapped with CDK v2.148.1 or earlier and not yet re-bootstrapped remains vulnerable. The hack qualifier still remains hnb659fds, but you can change it, if you want to.

Block Public Access

By 2018, the pattern was clear: teams were misconfiguring bucket policies and ACLs what seemed like on-purpose, as if they were on a mission to win an award. Objects were going public, breaches were making headlines, and the individual controls were too granular and too easy to get wrong. AWS's response was to add a meta-level override: Block Public Access — four boolean flags that sit above all bucket policies and ACLs and veto any access grant that would expose objects to the public internet. To be clear, these flags don't affect the bucket at all, the affect the ability for you to change those other insecure properties on the bucket.

BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy, RestrictPublicBuckets. Each flag a different angle on the same problem.

It is a kill switch. It works, for the most part. It was necessary because the model it was bolted onto had no safe default — the access system started too easy to open and required teams to correctly configure the restrictions, which teams reliably failed to do at scale. Block Public Access does not change that model. It adds a blunt override and calls it a fix. AWS enabled it by default for new accounts in 2022.

Paying for unauthorized access

Did you know until 2024, if someone attempted to access your AWS S3 bucket, even if it was never public, would still incur a charge for you? This massive oversight was fixed under the radar, and you can read more about it in the release Amazon S3 will no longer charge for several HTTP error codes\n. How that ever got off the ground in the first place is honestly shocking.

OU: Block Public Access

Finally, only last year, did AWS release the ability for AWS Organizations to turn off the incredibly insecure configuration by utilizing one of the S3 Org level policies. Now you can actually be sure you don't accidentally get it wrong, or I guess also find out if you did much sooner than you would have.

\"A

The entire history of S3 naming advice, summarized.

Are these hacks? Yes, yes they are. That is because the default considerations for using S3 require more configuration then lesser used strategies. If you want your bucket to be public, you configure less than you do if you want it to stay private. If you want to make sure you are secure and writing to your own bucket, you need to add properties, rather than remove.

What AWS Just Shipped​

The biggest challenges with all of these hacks is — that with each new one being introduced, it required every service, product, application, and library to directly integrate that change. That's because every API, architecture decision, and code path had to account for this change. These weren't just hacks AWS made to solve the problem, these were bad hacks that pushed the burden on customers.

And so, AWS has watched the community embed account IDs, regions, and random identifiers into bucket names for years. That must have meant we loved it, because then they shipped that exact pattern as a first-class feature: Account Regional Namespaces.

\"Success

The feature applies works in that when you create a bucket named myapp-logs and request it in your account-regional namespace: myapp-logs-123456789012-us-east-1-an. The -an suffix signals to the S3 service that this name is scoped to your account and region. Nobody else can register anything-123456789012-us-east-1-an — the 123456789012-us-east-1 segment is reserved for your account. How AWS managed to promise that buckets with an -an suffix don't already exist, and none of those bucket where in a cross-account scenario, is beyond me. Maybe they didn't. The likelihood is very small, that someone already had a bucket with a suffix of -{accountId}-{region}-an, but if they did, and they had a cross account scenario, then that is now broken. Or maybe it isn't, maybe that special bucket according to the new rules was created in the correct account, but in reality someone else owns it.

And so, we can see the same problematic pattern with this one as all the other hacks.

It is opt-in. You must set a special header or use a special property on CreateBucket. Existing buckets are not migrated. Existing tooling does not generate these names. Every piece of infrastructure code that creates S3 buckets needs to be updated to use the new naming convention. And that means every service, SDK, API, library, product, etc... that you are using must also make this change.

It wastes 26+ characters to your bucket name. S3 bucket names have a 63-character limit. You now have at most 37 characters to work with before you hit the wall. If you have a naming convention like {environment}-{team}-{service}-{purpose}, you are already in trouble. Hopefully each team in your organization has their own AWS account, but I know some of us aren't that lucky. You might be asking yourself, why 63? Well this limitation also almost certainly exists because S3 has the expectation you will make your bucket public. And it also made the mistake of expecting that the bucket name has to be part of the url as a subdomain. And DNS parts max out at 63 according to RFC 1123.

It does not address the actual architectural problem. Your bucket is still globally addressable via s3.amazonaws.com. The access model is unchanged. The public bucket problem is unchanged.

And then there is the SDK story.

Clever engineers will immediately ask:

If my bucket name no longer explicitly includes my account ID and region, I cannot just pass around the bucket name. How do I write portable infrastructure?

My answer: You don't.

The obvious AWS's answer: pass the account ID and region as a special token that the SDK resolves at runtime from the current execution environment. Instead of hardcoding 123456789012, you reference a variable that CloudFormation or the SDK resolves from the execution context.

So it's a second hack layered on top of the first one. The question is philosophical but practical, and AWS' answer is technical. That's a weird take.

You now have infrastructure code that creates bucket names by concatenating a prefix with a runtime-resolved account ID and region. Your IaC state needs to capture the resolved name, not the template. Your references to the bucket in other services need to either embed the same resolution logic or accept the full resolved name as an input. Your cross-account pipelines — CI/CD systems deploying into multiple accounts — need to be aware of this resolution mechanism.

AWS did not fix the problem. They added an opt-in feature that partially addresses one symptom, then added tooling to work around the limitations of that feature. You'll notice in the same release post, they also include the changes they had to make to CloudFormation S3 Resource. The people celebrating are celebrating a band-aid on a fracture.

How S3 Is Actually Used​

But the real goal of tis article is actually talk about a solution. And to do that we need to review the fundamental use cases of S3. In practice it exists for four distinct use cases. Which of course have almost nothing in common:

1. Private object storage — build artifacts, backups, data lakes, Lambda packages, database snapshots, OpenTofu, Terraform, IaC state files, and SPA access by CloudFront. No direct external access. Internal AWS service-to-service or IAM-authenticated only. I'm go out on a limb and say this is 99% percent of the S3 usage by volume and by bucket count.

2. Event-driven processing — S3 event notifications triggering Lambda functions. An object is created or deleted; an event fires; a Lambda processes it. (One caveat here is that you MUST Never do this because S3 event notifications are not durable, ensure that all S3 events are sent directly to SQS, and then to Lambda.) The bucket name and ARN arrive in the event payload:

{
      \"Records\": [
            {
                  \"eventSource\": \"aws:s3\",
                  \"awsRegion\": \"us-east-1\",
                  \"eventTime\": \"2024-03-01T12:00:00.000Z\",
                  \"eventName\": \"ObjectCreated:Put\",
                  \"userIdentity\": {
                        \"principalId\": \"AWS:AROAEXAMPLEID:session\"
                  },
                  \"responseElements\": {
                        \"x-amz-request-id\": \"EXAMPLE123456789\",
                  },
                  \"s3\": {
                        \"s3SchemaVersion\": \"1.0\",
                        \"configurationId\": \"upload-processor-trigger\",
                        \"bucket\": {
                              \"name\": \"my-app-uploads\",
                              \"ownerIdentity\": {
                                    \"principalId\": \"AEXAMPLEOWNERID\"
                              },
                              \"arn\": \"arn:aws:s3:::my-app-uploads\"
                        },
                        \"object\": {
                              \"key\": \"uploads/photo.jpg\",
                              \"size\": 1024,
                              \"eTag\": \"d41d8cd98f00b204e9800998ecf8427e\",
                              \"sequencer\": \"0A1B2C3D4E5F678901\"
                        }
                  }
            }
      ]
}

Notice what is not in this payload: a public-facing URL. The bucket.name and bucket.arn reference the internal bucket name. S3 ARNs have never included an account ID or region — arn:aws:s3:::my-app-uploads, not arn:aws:s3:us-east-1:123456789012:my-app-uploads. The identifier in the event is already the private bucket identifier, not a public one. And it would be easy to add the region and account ID to this ARN and likely not break a single thing.

And that's the tell. The event-driven use case has always operated on private identifiers. The Lambda function receiving this event doesn't care what the bucket is called publicly, or whether it has a public URL at all. It cares about the object key and the internal bucket reference — both of which are already account-scoped and private by nature. S3's internal event system was already operating on the right model. The global namespace was never part of this path.

3. Presigned URLs — assets that could be served over CloudFront because they are cacheable, but because you don't want them to be public, such as user owned data, you create a strategy to serve user data directly from S3. And same goes in reverse, you allow users to upload data, but rather than needing to deal with it in your service API, you directly have the client integrate with S3.

4. Direct public access — open buckets, bucket website hosting, ACL-public objects, resolvable by a public DNS. This is the pattern that causes all the breaches, all the confusion, and almost all of the architectural complexity AWS has accumulated in S3 over the years.

Category 4 is a tiny fraction of actual S3 usage by any metric you choose. It is responsible for a disproportionate fraction of the design surface area, the security incidents, and the policy complexity. And all the fixes so far make the usages of (1), (2), and (3) more challenging, while increasing the safety of (4). This is not how you solve architectural problems. You want to play a strategy where the most frequent uses are optimized for security, where the threat model identifies the biggest risk, to subvert that, not protect a screendoor or a fence in the middle of the desert.

The data breaches you read about were almost always S3 misconfiguration involving category 4. A few illustrative examples from a single year — 2017 alone:

The fix in every case should have been \"make S3 harder to misconfigure.\" But the advice and resolution we've seen was instead: \"fix your IAM policies\", \"enable Block Public Access\", \"audit your bucket ACLs.\" Patches. Tooling. Guardrails, Security Hub findings around a footgun that should not exist in the first place.

The reason category 4 exists at all is historical. In 2006, if you wanted to serve a file publicly from the internet, you needed a publicly accessible server. S3 was that server. CloudFront did not launch until 2008. IAM did not launch until 2011. The access model AWS ships with S3 today is the access model from an era when the alternatives did not exist yet. (I'm of course speculating here, because I didn't use AWS until 2008, and couldn't find a great source for this.)

Yet, some of the hacks to fix this problem have happened much later than 2011, and realistically, none of them even required IAM to make this happen.

The Real Root Cause​

All of that complexity — ACLs, Object Ownership, Block Public Access, website hosting, and the hacks added attempt to fix secord-order mistakes. They were pilled ontop of the one thing nobody touched: the naming model. And it's the real feature everyone wants:

Feature 1: The same logical bucket name across multiple AWS accounts.

Take OpenTofu (or any IaC for that matter) for instance. You need remote state storage. The canonical setup: one S3 bucket per account, typically named something like {org}-opentofu-state or {account-name}-tfstate. Simple, readable, deterministic.

In practice, you have a dev account, a staging account, a production account, a security account, a shared-services account. You want 123456798012-opentofu-state in all of them. Under the current global namespace, you cannot have that. You have to name them 123456798012-opentofu-state-dev, 123456798012-opentofu-state-prod, and so on — encoding the account into the name because the namespace doesn't do it for you.

With the new account-regional namespaces, you can now have opentofu-state scoped to each account. In theory. But in practice, all the changed was the interface for creating buckets, the usage of the buckets and their names are still the same as without this latest feature, and worse, without changing anything regarding how the service actually works, now everyone needs to make change. It is the worst of all fates:

  1. OpenTofu's and other IaC's S3 backend configuration needs to be updated to use the new naming scheme
  2. Any modules that reference this bucket by name need to be updated
  3. Any existing state files pointing to the old bucket names need to be migrated
  4. Your bootstrap process — the code that creates the state bucket before OpenTofu can run — needs to support the new CreateBucket header

None of this is easily managed. And while you can opt out of things like (2) and (3), you all know that there is some \"security theater\" going on at large enterprises that will claim a migration here \"increases security\". I'm sure there the associated security hub finding that is going to come out soon with a Critical level. All of it is work that should not have been necessary if the architecture had been correct from the start.

Feature 2: The same logical bucket name across multiple regions.

Multi-region active-active deployments are increasingly common. You want my-app-assets in us-east-1 and eu-west-1. Under the account-regional namespace, these would be my-app-assets-123456789012-us-east-1-an and my-app-assets-123456789012-eu-west-1-an — different names for logically identical resources. Your infrastructure code must now either parameterize the region or generate the full resolved name in every place that references the bucket.

This is the same problem that existed before the fix. The namespace is account-regional — it scopes names to an account and a region. That is correct for preventing name collisions, but it means your logical bucket name is still not portable across regions. The same bucket in a different region is a different name. Your replication configuration, your CDN origin setup, your cross-region failover logic — all of it must carry the full resolved name around. You can have the same DynamoDB Table Name used in every region, but not S3.

The underlying issue is that S3 conflated four separate concerns:

  1. Identity — what is this bucket called?
  2. Location — which account owns it, and which region holds the data?
  3. Addressability — how do external clients find it?
  4. Accessibility — Who should have access to it?

AWS's new feature embeds all four into the name string itself: myapp-123456789012-us-east-1-an. The account ID is in the name. The region is in the name. The identity is whatever is left over after you subtract those 26+ characters. The an limits access. This is not a namespace — it is a naming convention that happens to be enforced by the S3 service on creation only. The four concerns are still coupled; they are just coupled inside the string rather than explicitly as configuration.

Intelligent Design​

I want to be clear, AWS S3 is a fantastic service. It is so great in fact that there are no small number of huge businesses built around duplicating the S3 API. There are 20 years of successes after all. And I don't want to gloss over that:

What S3 gets right​

Object storage is the correct primitive. An opaque key — a bucket name and an object path — maps to a sequence of bytes. Durable, versioned, regionally placed, with a consistent API surface across every SDK AWS ships. Lifecycle rules, replication, object tagging, multipart uploads, and locking (but only recently unfortunately). These are the right tools for managing data at scale, and they work.

Additionally, Presigned URLs are the correct mechanism for temporary access delegation. Credential-scoped, time-limited, no IAM policy change required. The object stays private; the URL grants access for a window. That's also the right design.

Do I need to mention the high durability of 99.999999999%, and the reliability of 99.99% as well?

None of this needs to change. The problem isn't storage. It's two things piled on top of storage: the naming model and the access model.

Secure by default​

Every AWS primitive designed with security in mind starts from the same position: the unconfigured state is safe.

IAM: default deny on everything. No permission exists until you create one explicitly. The account with no IAM policies grants access to nothing.

VPC Security Groups: inbound traffic blocked by default. Every allow rule is explicit. The security group you just created, without touching it? It denies everything. (excluding the default VPC, which I'm not going to get into here)

KMS customer-managed keys: a key with no resource policy grants decryption to nobody — except the account root, which is a recovery mechanism, not an access path. Grants are explicit.

S3 is the exception.

Secure by default doesn't mean \"safe unless you misconfigure it.\" It means safe by construction. The state you reach without doing anything must be the safe state. And for me that also excludes the presence of pits of failure. If it is easy to do the wrong thing, then this a dangerous state. Public access for instance, must require deliberate, explicit, named work. Not the absence of a flag. Not the absence of a policy. Not a default you forgot to change.

S3 had it backwards. And the fix isn't more flags. The fix is a model where a public bucket cannot exist — because public access isn't a property a bucket can have, it's a property of a feature called \"promotion\".

My Prospal: Private by Default, Public by Promotion​

Here is the core insight that AWS released but no one wanted to commit to:

The right model: A Private Bucket Service. If you tilt your head sideways and squint, you might see that such a thing has been here all along, and I'm sure there is even an already existing AWS primative that encapsulates this concept internally.

info

By Private, I mean that the bucket is private to your account, not private in the fact that it just isn't publicly accessible.

I don't think this a novel concept. DynamoDB works exactly this way.

And under this model, my-app-assets in us-east-1 and my-app-assets in eu-west-1 are two separate buckets each globally identifiable via the ARN, and accessible via the region based parameter in the SDK/CLI/API (which by the way is already necessary.) Your infrastructure code references the bucket name as it always has done.

What's missing you might ask?

No 26-character suffix. No runtime SDK token substitution. No encoding of internal topology into names that humans have to read and type. No weird public configuration, no ACLs, no URLs associated with the buckets, no pits of failure.

The Cornerstone Example​

When you create a Bucket today s3PublicClient.createPublicBucket(), let's call it my-app-assets. It has a ridiculous number of limitations for creation, which I will get to later as well as the underlying assumption that you will make some part of it public. It comes with:

That bucket is created with the ARN arn:aws:s3:::my-app-assets.

This doesn't go away, you can still call that API, if you really wanted to. But the truth is that no one would call that API, because very few people need that API. Instead you would call the s3PrivateClient.createPrivateBucket(), and you will get a bucket with an ARN arn:aws:s3:REGION:AWS_ACCOUNT_ID:my-app-assets. That bucket operates with everything you would want in a private bucket:

But it doesn't have any of those things for the public bucket. If you want those things above, you would need to call s3PrivateClient.promoteBucket(). The parameters for that should be something like:

s3PrivateClient.promoteBucket({
      bucket: 'my-app-assets',
      publicBucketName: 'my-app-assets-public',
      ...
})

Doing so at that moment would validate if that public bucket name exists. Everything continues to work the same from a public bucket standpoint, but we are also afforded all the benefits of the private bucket without any of the risks.

This also prevents there being any backwards compatibility isuses as far as infrastructure management and creation goes, because the S3 Public API still exists, the only difference is now there is also the S3 Private API which can be used to create the local buckets, and when desired will be promoted to also be a public bucket. Additionally, you'll see later that migration on the AWS side is necessary to support this.

If I were an S3 Architect, I might ensure that all public bucket names start with public- or exist in the namespace public/ or public:, so that someone could not accidentally write arn:aws:s3:::my-app-assets and get a malicious attacker's promoted S3 private bucket.

That is, if an attacker created arn:aws:s3:us-east-1:666666666666:my-app-assets, and promoted it to be arn:aws:s3:::my-app-assets. Then you could create arn:aws:s3:us-east-1:000000000000:my-app-assets and accidentally reference it as arn:aws:s3:::my-app-assets. In doing so, you would be again using that attackers bucket. Holistically, this is the same problem that has always existed up until this point, so this strategy isn't worse. It is just not perfect. But that's a mistake AWS might need to live it.

It would be better if would have to explicitly add in the public prefix and write arn:aws:s3:::aws-public-buckets/my-app-assets for all public buckets. But that's a breaking change, so likely off the table. However as I mention below, there are great ways to protect against this that AWS can help with.

Public Buckets: How promotion works​

A bucket, once created, is private. The bucket's access state never changes. What changes is what you attach to it.

There are two core public scenarios that I'll call promotion paths that still must have solutions for:

Presigned URLs: Temporary Promotion

You issue a time-limited, credential-signed URL for a specific object. The URL encodes the object path, an expiration, and a signature derived from your IAM credentials. Anyone with that URL can read that object — for the duration you specified. When it expires, access ends. The bucket policy didn't change. The object's access model didn't change. The credential signed the request; the routing table resolved the bucket; S3 validated the signature and served the object.

A presigned URL today looks like https://mybucket.s3.amazonaws.com/file.png?X-Amz-Credential=AKID123%2F20240101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=.... The X-Amz-Credential field already contains the account identifier — derived from the access key ID, which maps to an account. S3 extracts that account, consults the routing table for mybucket in that account, and routes to the right physical bucket. The global uniqueness constraint was never doing the routing work here. The credential was.

I want to say that again, presigned urls will still absolutely work out of the box without any changes.

This is because Presigned URLs are not an S3 concept. They're an IAM concept that S3 validates. To explain, we need to dive into how AWS IAM actually works. AWS IAM uses their custom SigV4 signature strategy for every request to AWS. And every request to AWS goes over the wire on a AWS owned DNS url for the service with all the necessary parameters.

For instance, your SDK computes a SigV4 signature using your IAM credentials — the access key ID and its corresponding secret. No AWS API call is made. The URL is computed entirely locally. This is how it works for every AWS service API. When you call DynamoDB this happens, and the same thing happens when you call S3.

Presigned S3 is a trick. After constructing the full HTTP payload to send to the service, instead of actually sending it, you give it to someone else. Then that person executes the payload. Normally it wouldn't matter who executes it, but what if some part of the payload was allowed to change between the generation of the HTTP payload and the exector executing, let's say for instance: the Binary Body. In this way, you could generate a request that encodes the bucket, the object path, the expiration, and the signature, and hand it to some other user. They present it to S3 with a custom binary.

When S3 receives the request, it extracts the access key ID from X-Amz-Credential, looks up the corresponding IAM entity via STS, re-derives the expected signature, and checks that it matches. Then it checks the expiration. Then it checks that the IAM entity had s3:GetObject permission at signing time. If all three pass, S3 serves the object (or persists it in the case of s3:PutObject).

That's all. S3 is just doing IAM validation, the same thing every other service is doing. It is not checking whether the bucket is public. It is not consulting the access model at all. A fully private bucket — no ACLs, no public access configuration, nothing — can serve objects via presigned URL because the authorization is credential-based, IAM-based, AWS-API based, it is not a unique access-model built into public S3-based buckets.

Public Buckets: Permanent Promotion

Since, public access is not a PrivateBucket property, there has to be some way to still expose public access to the PrivateBucket data. And so the proposal would allow making a PrivateBucket public by requesting a bucket name from the global authoritative S3 Bucket Name list. The same process you already have today for S3 buckets, when you create a new one.

In the new model, the public properties, the ACLs, the website configuration, aren't properties of the bucket. They're a separate resource: a public access configuration. Which today is what is called S3. So you might be able to see why I'm suggesting a name change. When you create one, attach it to your private bucket, and the S3 URL is created. You remove it, and the S3 URL stops existing. The bucket itself never changes state. The URL is a consequence of the configuration, not a property of the storage. And that URL, that's the thing that must be globally unique, and most importantly that doesn't even need to match the original bucket, and it won't.

Website configuration lives there too. Index documents, error documents, redirect rules — these move from bucket settings into the public access configuration. The s3-website endpoint exists because the configuration says it should, not because the bucket was created with a flag set.

And because the user-defined string — the bucket name — is preserved through the Public Bucket configuration. What is no longer true is that this string must be globally unique for the private bucket. That constraint was never load-bearing. It was just there because of the expectation on public usage.

The Custom HTTP domains using S3 website hosting — with CNAMEs pointing to mybucket.s3-website-us-east-1.amazonaws.com or not — continue to work. The website configuration moves into the public access configuration resource; the s3-website endpoint continues to exist as long as that configuration exists. No customer change is required.

Because this functionality is separate, AWS can disable (and hopefully dismantle) in one huge swath all of the public features of S3 that are insecure by default, and lead new AWS accounts down the path of CloudFront for public access. If you need custom domains, TLS termination on your own domain, caching, WAF, HTTP/2, geographic restrictions, or edge functions — that's not an S3 question. That's a CDN question. And the answer is CloudFront as the reverse proxy with a private S3 bucket origin granted access via the Origin Access Control configuration.

The bucket stays private. CloudFront has authorized access to it. Your users get a production-grade delivery layer with every security consideration you need. S3's job is to hold the bytes and serve them to one authenticated caller — the distribution. CloudFront's job is to serve those bytes to the world under your domain, your TLS certificate, your cache rules.

This is already how every serious production setup works. The new model doesn't change that. It just makes it the only coherent option, instead of one option among several confusing ones.

A New CloudFront Opportunity​

Presigned URLs have a structural limitation today that nobody talks about: the SigV4 signature is computed over the canonical request, which includes the Host header. And so the URL is signed against mybucket.s3.amazonaws.com. Change the hostname and the signature fails. Which actually is a huge problem for CloudFront Functions when rerouting requests to a different origin (sometimes it works). This means custom domains for presigned URLs are impossible today. Every download link, every document export, every profile photo URL your product generates contains s3.amazonaws.com. Your customers see your infrastructure provider in every URL. There is no way around it with the current model.

The right fix is for CloudFront to gain first-class presigned URL support: the ability to validate SigV4 signatures on behalf of S3. If CloudFront can validate the signature, the URL can be generated against your CloudFront custom domain — with your ACM certificate, on your domain — and CloudFront handles the validation and the downstream request to S3. The signing mechanism doesn't change. The client code doesn't change. The SDK GeneratePresignedURL call works identically, just against a different hostname. Ironically, CloudFront offers some partial functionality for Signed Request URLs and Signed Cookies, but these actually have a security hole because they don't include the same level of control that IAM policies provide. CloudFront + IAM would be a real game changer for Presigned URLs.

The S3 team's outstanding task​

Now on to easy but annoying part. AWS cannot simply remove public bucket support creation path. It isn't the millions of buckets in production, but rather all the code paths that create buckets and then make assumptions about them. Some of those code paths were written by teams that no longer exist.

Any migration strategy that requires customers to take action will fail for the long run. The path forward has to be one where the default behavior improves without requiring every customer to update their infrastructure. Something that the current history of hacks haven't gotten correct at all. (Although their folly resulted only in decreased security rather than broken configuration.)

AWS can either trudge along with this currently broken S3 architecture riddled with pits of failures. Or they can admit they made a mistake and default all new accounts' buckets to not contain a public access strategy. This is actually the right thing to do, and they can do this safely as they have deprecated even whole AWS services before.

Phase 0 - Private bucket backpopulation​

The core ask is that from all the existing public buckets, AWS creates the local private bucket version. The backpopulation would be a list of S3 private buckets whose names will be the exact same same as the current PublicBucket name. The goal being that all AWS S3 buckets should be referencable by their account-region localized arn, and the relevant console UI exists to display that. That's a script even Kiro could write in an afternoon.

That means for every bucket called arn:aws:s3:::my-app-assets, there will also now exist an ARN arn:aws:s3:us-east-1:123456789012:my-app-assets. Depending on how local name ARN resolution works, that might be a no-ops, but I'm not an AWS S3 architect, a fact that I didn't know I had to repeat so many times 😉.

Phase 1 — New accounts, new defaults​

This doesn't break existing buckets. And new infrastructure gets the right defaults. The blast radius is almost zero. There are some AWS organizations out there that are dynamically creating S3 buckets in automatically provisioned new AWS accounts with assumptions based on how buckets work. When creating a new account and then a bucket in that account, they will see a problem. This just needs to be communicated.

You might be thinking, couldn't there just be a magic flag on bucket creation that specifies that the bucket is account/region bound, call that flag: private: true. The problem is removing the restriction to private buckets MUST BE OPT-OUT. private: true makes the default the legacy insecure current state, and keeps public access is opt-out. And therefore it still allows all the bucket negligence awards that Corey is so keen on giving out. A flag is not sufficient, and instead there needs to be a mature approach to the migration. Which is why the recommendation here is:

  1. Rename S3 everywhere to \"S3 Public Bucket Configuration\"
  2. Reintroduce S3 as a Private Bucket concept

Phase 2 — AWS internal service updates​

AWS has some internal work to do. Luckily most of the mess that was caused is squarely cornered into the S3 Public Bucket Configuration and none of it actually affects our new private bucket creation or usage. That means, after the rename, AWS can go back through all of their services and retarget all interactions with S3 to use the new Private S3 SDKs/API. This is squarely in their control.

S3 Bucket Events + Lambda Event Source Mapping

One area where there is a bit of a cross over are events like S3 Events over SQS => Lambda. But as discussed earlier that's actually a no-op. Similarly, Lambda Event Source Mapping (Lambda ESM), used for automatically polling SQS is a non-issue. But the reason why is worth understanding. An ESM configuration is account-scoped in the first place. When you set up a Lambda trigger, you're making an authenticated API call inside your account: \"Lambda function X should fire on events bucket Y.\" The ESM record lives in your account. The bucket lives in your account. AWS resolves the bucket reference using the account context of that API call — not the public namespace.

The current ESM ARN looks like arn:aws:s3:::mybucket — no account ID, no region, because those were implicit in the global uniqueness guarantee. In the new model, mybucket is a private identifier scoped to your account. The ARN format doesn't change. The resolution just shifts from \"global name lookup\" to \"private identifier lookup within account context\" — which AWS handles internally. No customer touches their ESM configuration. No ARN format changes. No trigger reconfiguration. Future ARN formats for the ESM should take the account ID and the bucket region, but AWS needs to maintain the global mapping table they already have that allows the account-less, region-less ESM bucket ARN to resolve the bucket in the specific region, in the correct specific account. In other words, ESM resource should accept either the global bucket naming strategy or the region-account local one.

The message here \"Update your Event Source Mappings for Buckets so that you have the account ID or region specificed\". This might be the first ever [Action Required] email, that actually has a required action. Or maybe they'll just update Security Hub to include a finding to fix this, and an AWS Config rule that validates it with an automatic remediation.

CloudFront S3 origin compatibility

CloudFront is not part of S3's access model — it's a CDN that sits in front of a private S3 bucket, authorized via OAC. That already works today and obviously must continue to work in the new model. The only S3-specific change AWS needs to make is ensuring that CloudFront's S3 origin configuration resolves bucket references using the private identifier rather than the global name. Again, that is an internal AWS concern. No customer CloudFront configuration changes. I'm sure there is someone out there that is going to request cloudfront have access to S3 buckets in another account. AWS can easily support a similar solution to the ESM as above, CloudFront accepts either the global S3 ARN or the account-region localized one.

Phase 3 — Configuration Split​

Every existing S3 bucket is already the private half of the new model. Customers haven't been creating \"public buckets\" — they've been creating private buckets and then attaching public configuration to them in the form of ACLs, Block Public Access exemptions, Bucket Policies, and website hosting settings. The private bucket has always existed. What hasn't existed is the explicit separation exposed to AWS Account users. That starts now.

Since the buckets themselves and the public access configuration don't actually change here, the only thing AWS has to do is the backpopulation.

Presigned URL configuration handling

As argued above, the Presigned URL configuration already will work out of the box since the exact same problem has already been solved for literally every other resource in AWS. The one caveat here is that there will likely need to be a new method GeneratePresignedBucketUrlForPrivateBucket to make sure it includes the account Id and the region explicitly so that the public bucket configuration isn't necessary to continue to use that option. That's because the current method doesn't take in the account ID or the region, but just the bucket name.

The one exception is cross-account presigned URLs — an IAM identity in Account B generating URLs for a bucket that lives in Account A. I personally don't even know if this is possible, but technically I don't see why not. In this case, if we use the X-Amz-Credential to determine the account, AWS would incorrectly assume the account is B (where the identity is) and not Account A (where the bucket actually lives). But AWS S3 have very competent architects, so I'll leave that challenge for them to solve (I can imagine using this same new GeneratePresigned menthod I just suggested above).

It's also worth noting that potentially the presigned URL configuration could be an explicit resource you create when you need it similar to the public access. And by default just create it for all existing buckets.

Phase 3 — Deprecation

The best part of this design is that regarding deprecations there are none! Since all we are actually doing is changing the same of some SDKs to improve readibily and really just the text in the UI. The only real change that is necessary here is going through all the docs and updating the content with more appropriate and clear naming.

Most importantly, over time, the \"public bucket\" moniker will disappear entirely from the documentation as a concept, from customer usages, and most importantly from the news. And what replaces it? A private bucket with an explicit access configuration attached when needed. Two resources, two concerns, neither coupled to the other by default. The access model that caused two decades of breaches stops being something new engineers get to learn about.

The Objections​

Proposing a fundamental redesign of S3's control plane will attract objections.

Here's another thing I thought would go without saying!

danger

There could definitely be problems somewhere, I'm not saying that there won't be any problems, and no one will run into anything. For sure this strategy doesn't fix everything. That's also not possible. But it fixes most things, and it doesn't come with any obvious and non-obvious downsides even after scrutiny. Sure, there are some very complicated setups that some people could run into, and in that case, maybe AWS offers those organizations/accounts a way to prevent the use of private buckets. But if you wrote some code that contains a time-bomb, AWS changing this isn't likely to be the only thing that would cause your solution to break, please do not blame them, or I guess me either for suggesting it!

So here are the ones I felt like addressing:

What About SPA Websites?​

The most common objection: \"But I host my react/vue/solidjs app on S3 with website hosting enabled, and it works fine.\"

It works, but it isn't correct architecture. Let's be precise about what is actually happening.

Your S3 bucket is serving HTTP at http://my-app.s3-website-us-east-1.amazonaws.com. Your domain is resolved by one two ways:

Option A — CNAME directly to the S3 website endpoint. — You have no TLS. S3 website hosting is HTTP only — it has no mechanism to serve HTTPS for a custom domain. Your users therefore must be on HTTP, so this is not a viable production setup. It actually doesn't work at all.

Option B — CloudFront in front. — CloudFront handles TLS (via ACM), your custom domain, HTTP→HTTPS redirects, the 404 → /index.html behavior for client-side routing, cache headers, compression, and geographic distribution. S3 is behind CloudFront, serving bytes when requested.

Option C — The website domain is the S3 url — You are freely passing out your S3 bucket URL to clients and asking them to remember that custom url. Something for sure is going to break some day, but nothing stopped you from doing it.

Sites that only use S3 website hosting without CloudFront, serving plain HTTP is not a counterexample. It is a site that is broken and getting more broken by the day. Chrome announced in 2023 that it is moving towards HTTPS by default, automatically upgrading HTTP navigations to HTTPS. An S3 website serving HTTP gets upgraded to HTTPS by the browser, and since S3 cannot serve HTTPS on a custom domain, the request fails. Firefox has had an HTTPS-Only Mode available since 2020 that blocks HTTP sites entirely. These are not future concerns. They are not esoteric. They are not nuanced. They are the current state of the web. A site that only works over HTTP is not a production website in 2026. It is a broken website that has not been maintained.

Which means in every functional production scenario, S3 website hosting is doing nothing useful. CloudFront is handling everything. S3 is holding bytes.

Therefore, Option B is every production SPA, S3 website hosting is contributing nothing. CloudFront is doing all the work that makes the setup viable. The bucket does not need to be public. Website hosting does not need to be enabled. The only reason engineers enable website hosting is that they are following a tutorial that predates CloudFront's ability to serve private S3 buckets, and nobody told them the tutorial was outdated. Or more likely, someone did, but they didn't listen.

CloudFront likely has been able to serve our new Private S3 bucket concept since Origin Access Control (OAC) replaced the older Origin Access Identity (OAI) approach. OAC supports server-side encrypted buckets, covers all S3 regions, and signs requests to private S3 using SigV4. Even before OAC, your bucket never needed to be public.

There could be a concern that CloudFront doesn't know how to talk to anything other than a public S3 bucket or a public URL. But interestingly enough, CloudFront now also supports private origins via ALB with VPC origins, which closes the last remaining scenario where direct public exposure might have been argued as necessary. You can run your origin entirely inside a VPC, with no public exposure, and serve it through CloudFront. The gap is gone.

And the \"CloudFront costs more\" objection doesn't land either. CloudFront has a free tier: 1 TB of data transfer per month, 10 million HTTP requests, and 2 million CloudFront function invocations. A landing page or documentation site that fits in an S3 bucket almost certainly fits within that free tier, and even if it doesn't, at scale you are still getting the benefit of the cost reduction.

A complexity argument would be more interesting. Setting up a CloudFront distribution requires more steps than enabling S3 website hosting. That is true. But the complexity exists either way, it is just hidden. And you still need TLS. You still need the index.html routing behavior for client-side routing (or a more expensive CloudFront function). You still end up at CloudFront. The engineers who skip it are the ones serving HTTP from a subdomain with no TLS, which is screams for a denial-of-wallet attack.

And for users who genuinely have not set up CloudFront, a la Option C: the AWS S3 migration plan already answers this. The configuration split means existing public buckets keep their public access configuration intact, those sites keep working. The owner does nothing. When they are ready to do it correctly, the options are available.

But Private Websites?​

Let's get this out of the way really quick. There are no such things as private websites. That's term just doesn't mean anything. You could suggest you are hosting a public website on an S3 Bucket, but using some sort of Bucket Policy to block requests. Or using an internal ALB in a private subnet to route requests to that website while users are connected to the VPC through an VPN.

Walled gardens died a long time ago. But even after death, the ruins remain, and some companies are vehemently trying to resist implementing modern technology such as Zero-Trust often by buying products called \"Zero-Trust Security\" that implement the exact opposite.

Just like with the Public SPAs, nothing stops you from continuing your charade. You can still host your bucket as you always did with the legacy public bucket configuration. However, I would suggest taking a hard look at what you are doing, and reviewing what exactly the reason is. Remember, to use the website, your need users to find it. That means you probably have a DNS set up. And of course you used a DNS hosted in a public AWS Route53 Hosted Zone, and not a private one. And you did that because you are security conscious and know that a private hosted zone will almost immediately create a huge security vulnerability in your architecture. Don't believe me? Check out this Last Week in AWS episode on Split-Horizon DNS.

After that, there is the routing and bucket contents. Since I'm of course advocating for the bucket contents being private, and routing is a trivial aspect handled by AWS, the question that needs to be asked is \"How do I secure content stored in an S3 bucket so that only some users are able to access it\".

The best solution is create a public website, let users login, and then create an API on top of the S3 bucket. You can expose the data just the relevant users by putting authentication and authorization on that API. This works great for internal, low volume, or small file sites.

I know it doesn't work with large or high volume sites. So if you want to go the hard mode route, there exists an alternative solution. Instead, you implement CloudFront with Signed Cookies. This isnt't really an article about that, but the short answer is, CloudFront already works as an identity aware proxy (IAP). There is just one route that is public, and it will host the login page, all the other routes will check for a header that was signed by CloudFront or even better your Corporate Identity Provider (IdP). There's a good example + library available on Using CloudFront as an Identity Aware Proxy.

Bucket Origin Responses​

There is one thing I left out, and I didn't want to bring this up because it's annoying, but I'm sure someone will call me out on it.

When you set up S3 as an origin for your CloudFront, you might have the need to control the response headers. Historically, you were not able to configure anything in CloudFront, let alone do it dynamically. And so using S3 to set the CORS policies or other security policies was required. However now, CloudFront offers response headers, and while it isn't everything, even S3 isn't sufficient for specifying all the relevant headers. While I don't love it, for Authress, we have a CloudFront Function attached to every response. There is a performance hit and a cost hit to do this on literally every S3 related request. But argubly it is a small price to pay to have CloudFront do the thing that it should be doing all along, and not to save this configuration in S3 where it doesn't. Maybe AWS could be nice and still offer this configuration in S3, or be nice and add this as an option to CloudFront, or be nice and make CloudFront functions even cheaper, because why not, API Gateway velocity templates are free after all!

You're asking AWS to blow up a working control plane​

Yes. That is what a migration looks like. The alternative is two more decades of incremental patches, each one adding more surface area and more documentation burden without touching the underlying design, and worst of all, still enables a massive pit of failure.

The control plane does not need to be blown up for customers. The translation layer proposal in the previous section means existing workloads continue working. What needs to change is the model exposed to new infrastructure — the primitives developers learn, the defaults they encounter, and the architecture that tutorials recommend.

AWS has done this before. The IAM role model replaced key-based authentication for most AWS-to-AWS access patterns. And AWS IIC replaces IAM roles for organizations and SSO. CloudFront Origin Access Control replaced Origin Access Identity. Neither replacement was instantaneous, and neither broke existing workloads. The old model continued working through a maintained compatibility layer while the new model became the default for anything new.

The objection treats \"existing behavior must never change\" and \"defaults must never improve\" as the same thing. They are not.

You are still breaking S3 ARNs​

ARNs of S3 bucket type are defined not to have accounts or regions. In some sense there's no technical reason, but it would be fundamentally violating an invariant of ARNs, which is that they either do or don't have accounts and/or regions based on the resource type, and that resources of different types have different ARNs. Knowing AWS, I think that's very unlikely. And further, everybody expects that S3 bucket ARNs look a particular way.

So a couple of things. I can't find any reference that says Account IDs must always be present or never be present. Second, it isn't a breaking change. AWS could easily come out with a completely new service and call it S4, which only offered local buckets. That seems completely unnecessary. Today, people almost certainly split the arn by :, and are just ignoring the account ID and region. However, since they actually need the region to communicate with S3 privately, they have to pull it from somewhere else. This would be a huge improvement. Other than the objection being arbitrary, I can't think of any problem that would come form it. Maybe the one scenario could be breaking someone' database, where they set the column to be a fixed width and contains the S3 ARN VARCHAR(100) and now it is 21+ longer than it was before because the ARN has the account ID and region in it. Before you could be sure that bucket ARNs were at max arn:aws:s3:::63 characters = 76 total. And keep in mind that is only 76 total for the aws partition, we then have gov cloud, the ESC, and China, which all violate that. This adds another 21+ (12 for the account, and 9+ for the region), making buckets ARNs now at least 97. Additionally, since we no longer need the lowercase, 63 character limit. Bucket names could indeed now be even longer. We could pray to AWS to still restrict us to prevent our* collective oversight when it came to storing bucket names in our DBs, but I think that is completely unnecessary.

So highest impact area would be events from S3 that aren't stored in the DB, but end up at SQS or CloudTrail, AWS Config, CloudWatch, and external providers that are surfacing this information. Most of these would be truncating everything except the bucket name anyways, so the likelihood of something breaking would be incredibly negligible. The problem with hyperscalers though is that it is still likely that could break someone, but in those scenarios, it was likely just a matter of time. I can assure you your sacrifice will not go unnoticed, I will personally thank you for it!

\"You

xkcd 1172: Workflow

Automatic public bucket promition​

Anyone who creates a local private bucket who might want a public bucket for it in the future will want the public bucket name to match, and choose their local private bucket names accordingly. To make sure they can get it, they'll promote it to public to make sure it's reserved for them. Any scheme to mitigate this, will end up looking like the namespace solution that S3 has already got.

Some people might do that, but everyone else will not. We should never design technology that solves a made up problem and especially not problems that only affect a very small subset that they intentionally take on themselves. They can do that if they want, nothing stops them. I do agree the solution to that is very similar, if not exactly the same as the current public S3 bucket API, but for everyone more intelligent it would not be a problem. It is almost exactly the same thing as saying, everyone who gets an IPv4 address would want to buy a domain name that matches their ip address. Sure some people can, nothing is stopping them. They can go to town! (except of course you can directly put the ip address in a domain name due to the Domain Name RFC, as a reminder.)

The Better Announcement​

The Account Regional Namespaces announcement solves one real problem, the name collisions, using an opt-in mechanism with a 26-character tax on your bucket names, tooling that requires SDK and CloudFormation updates to remain portable. But it has zero impact on the access model that causes actual harm.

The right announcement would have looked like this:

Instead, we got a feature that requires you to append -123456789012-us-east-1-an to your bucket names, a second feature that lets your SDK dynamically resolve that suffix from the execution environment, and a wave of blog posts explaining how to wire these two features together. And of course we still have to wait for your-favorite-tool™ to implement this funcitonality.

This is not a fix. It is a patch on top of a patch, with new documentation for how to apply both patches correctly. AWS has a long history of excellent engineering, but I don't concern this new functionality to be part of it.

The gap between \"what was shipped\" and \"what would fix the problem\" is not subtle. It is not a matter of resources or engineering difficulty. Name collisions, the problem I can only imagine customers have been filing tickets about for years, was partially addressed. But the access model that still will cause actual harm was not.

Until the access model changes, the endless stream of conflicting advice will remain out there on the internet.

info

For help understanding this article or how you can implement auth and similar security architectures in your services, feel free to reach out to me via the community server.

", "url": "https://authress.io/knowledge-base/articles/2026/03/14/how-aws-can-fix-s3", "title": "Actually Fixing AWS S3", "summary": "The S3 namespace is global, the bucket policy is yours, the vulnerability is yours. A thorough architectural proposal to fix the AWS S3 Control Plane and deliver what everyone always wanted.", "date_modified": "2026-03-14T00:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2026/03/03/securing-aws-accounts-access", "content_html": "

I've seen a lot of complex tooling in my experience, but by far the worst case is designing just one more tool to do something. Especially in the age where software is free, we become burdened by just one more tool. We know at Authress that increased complexity => increased failure rate.

The solution is to utilize the tools we already have, just a little bit better. In this case — \"just a little bit better\" — is adding a trivial amount to your existing AWS built-in technologies, and doing it in a way that you won't even need to add extra management overhead.

info

For help understanding this article or how you can implement auth and similar security architectures in your services, feel free to reach out to us via the community server.

❌ The Wrong Way​

There are lots of ways this could have gone wrong. In fact, if you ask any of the \"Reasoning LLMs\", and are unlucky enough not be told IDK, you will find out things like:

At least the lambda function in every account would work, but it isn't clean, you'll get a lambda in every account, and potentially also region, which comes with at least one IAM role, a CloudWatch Logs Group, and who knows what else.

Someone out there is probably saying \"Why aren't you using OpenTofu for that\", I'll leave that as a challenge for the reader to answer.

The Complete Design​

\"Securing

The design is quite straightforward.

  1. Deploy a Lambda Function to the AWS Management Account which contains the list of permissions for each account.
  2. Deploy an OU StackSet which uses a Custom Resource to call the lambda function in the management account, to fetch the list.
  3. The list is persisted in a GitLab assumable IAM Role
  4. GitLab assumes the role at deployment

🔒 AWS Account Permissions Lambda Function​

Let's do the easy part first. Of course we want to define the permissions somewhere. Since we are using GitLab, what we actually want to do is define for each AWS account, which GitLab projects (and their branches can be used to access that AWS account). At the top here, we'll define the permissions. And at the bottom, we'll receive the account ID from the caller and use that pull the correct permissions out of the map.

Permissioning Lambda Function
const accountPermissionsMap = {
  000000000000: ['project_path:authress/automation/*:ref_type:*:ref:*']
  111111111111: ['project_path:side-projects/*:ref_type:*:ref:*']
};

const sendResponse = (event, context, status, data, reason) => {
    const body = JSON.stringify({
          Status: status,
          Reason: reason || '',
          PhysicalResourceId: context.logStreamName,
          StackId: event.StackId,
          RequestId: event.RequestId,
          LogicalResourceId: event.LogicalResourceId,
          Data: data
    });

    return await fetch(event.ResponseURL, { method: \"PUT\", headers: { \"Content-Type\": \"\", 'Content-Length': body.length }, body });
};

exports.handler = async (event, context) => {
    if (event.RequestType === 'Delete') {
        return sendResponse(event, context, 'SUCCESS', {});
    }

    try {
        const accountId = event.ResourceProperties.AccountId;
        const permissions = accountPermissionsMap[accountId] || [];
        return sendResponse(event, context, 'SUCCESS', {
            GitLabProjects: permissions.join(',')
        });
    } catch (err) {
        console.error('Event:', JSON.stringify(event), 'Error:', err);
        return sendResponse(event, context, 'FAILED', {}, err.message);
    }
};

🟢 Deploying the Lambda Function​

Management Account: CloudFormation Template
// First load the lambda function from the lambda function file
const handlerCode = await fs.readFile(path.join(__dirname, './fetchPermissionsLambdaFunction.js'), 'utf8');

return {
    AWSTemplateFormatVersion: '2010-09-09',
    Parameters: {
        OrganizationId: {
            Type: 'String',
            Description: 'The organization'
        }
    },
    Resources: {
        GlobalConfigLookupRole: {
            Type: 'AWS::IAM::Role',
            Properties: {
                RoleName: 'OU-StackSet-GlobalConfigLookup',
                AssumeRolePolicyDocument: {
                    Version: '2012-10-17',
                    Statement: [
                        {
                            Effect: 'Allow',
                            Principal: { Service: 'lambda.amazonaws.com' },
                            Action: 'sts:AssumeRole'
                        }
                    ]
                },
                ManagedPolicyArns: ['arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole']
            }
        },
        GlobalConfigLookupLogGroup: {
            Type: 'AWS::Logs::LogGroup',
            Properties: {
                LogGroupName: '/aws/lambda/OU-StackSet-GlobalConfigLookup',
                RetentionInDays: 30
            }
        },

        GlobalConfigLookupFunction: {
            Type: 'AWS::Lambda::Function',
            Properties: {
                FunctionName: 'OU-StackSet-GlobalConfigLookup',
                Runtime: 'nodejs24.x',
                Handler: 'index.handler',
                Role: { 'Fn::GetAtt': ['GlobalConfigLookupRole', 'Arn'] },
                MemorySize: 1769,
                Timeout: 30,
                Code: {
                    ZipFile: handlerCode
                },
                LoggingConfig: {
                    LogFormat: 'Text',
                    LogGroup: { Ref: 'GlobalConfigLookupLogGroup' }
                }
            }
        },
        GlobalConfigLambdaPermission: {
            Type: 'AWS::Lambda::Permission',
            Properties: {
                FunctionName: { Ref: 'GlobalConfigLookupFunction' },
                Action: 'lambda:InvokeFunction',
                Principal: '*',
                PrincipalOrgID: { Ref: 'OrganizationId' }
            }
        }
    },
    Outputs: {
        GlobalConfigLookupFunction: {
            Value: {
                'Fn::GetAtt': ['GlobalConfigLookupFunction', 'Arn']
            },
            Export: {
                Name: 'GlobalConfigLookupLambdaArn'
            }
        }
    }
}

▶️ Utilize the Lambda Function​

Then we update the member stack to utilize this lambda function, and create the correct IAM Role.

OU StackSet Member Account: CloudFormation Template
{
    // Pull the values in the Lambda Function
    GlobalConfiguration: {
        Type: 'Custom::GlobalConfiguration',
        Properties: {
            ServiceToken: { Ref: 'globalConfigurationLambdaArn' },
            AccountId: { Ref: 'AWS::AccountId' }
        }
    },

    // The IAM Role for GitHub to utilize
    GitLabRunnerRole: {
        Type: 'AWS::IAM::Role',
        Properties: {
            RoleName: { 'Fn::Sub': 'GitLabRunnerRole' },
            MaxSessionDuration: 3600,
            AssumeRolePolicyDocument: {
                Version: '2012-10-17',
                Statement: [{
                    Effect: 'Allow',
                    Principal: {
                        Federated: { 'Fn::Sub': 'arn:aws:iam::${AWS::AccountId}:oidc-provider/gitlab.com' }
                    },
                    Action: 'sts:AssumeRoleWithWebIdentity',
                    Condition: {
                        StringEquals: { 'gitlab.com:aud': 'https://gitlab.com' },
                        StringLike: {
                            'gitlab.com:sub': {
                                'Fn::Split': [',', { 'Fn::GetAtt': ['GlobalConfiguration', 'GitLabProjects'] }]
                            }
                        }
                    }
                }]
            }
        }
    },

    // Then register the GitLab OIDC Provider to allow GitLab to actually assume the role
    GitLabOIDCProvider: {
        Type: 'AWS::IAM::OIDCProvider',
        Properties: {
            ClientIdList: ['https://gitlab.com'],
            Url: 'https://gitlab.com'
        }
    },
    // ...
}

🏁 Run the Deployment​

One hidden piece of information that might not be so obvious is how we are going to actually deploy that Member Account CloudFormation Template to all the AWS accounts we have in our AWS Organization. For that, we use an AWS Organization OU Stack Set. The stack set automatically deploys the template for every AWS account in the OU, for every region.

Deploy OU StackSet
  import { OrganizationsClient, DescribeOrganizationCommand } from '@aws-sdk/client-organizations';
  import AwsArchitect from 'aws-architect';

  const client = new OrganizationsClient({  region:  'us-east-1'  });
  const { Organization } = await client.send(new DescribeOrganizationCommand({}));
  const parameters = { organizationId: Organization.Id };

  const awsArchitect = new AwsArchitect(packageMetadata, {});
  const deploymentResult = await awsArchitect.deployTemplate(globalConfigurationTemplate,
    stackConfiguration,
    parameters);

  const GlobalConfigurationLambdaArn = deploymentResult.Outputs.find(o =>
    o.ExportName === 'GlobalConfigLookupLambdaArn')
    .OutputValue;
  const memberParameters = {  GlobalConfigurationLambdaArn  };
  await awsArchitect.configureStackSetForAwsOrganization(memberAccountTemplate,
    orgStackConfiguration,
    memberParameters);

And the best part of this is that the lambda function is extensible, so you can include a full configuration in S3 or anything else that you might want to persist in the management account's git repository.

", "url": "https://authress.io/knowledge-base/articles/2026/03/03/securing-aws-accounts-access", "title": "Securing CI/CD Access to AWS", "summary": "Inevitably you'll need to secure the access from your CI/CD platform to your AWS or other cloud provider accounts. Here's how we do it at Authress.", "date_modified": "2026-03-03T00:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/12/03/secrets-building-reliable-software", "content_html": "

\"Slow

When thinking about building reliable software, we mostly focus on technology. While our tech stack, architecture, and good practices all matter, they are not what determines the reliability of our software.

Drawing from my experience of delivering software with strict SLAs, I'll talk about often overlooked foundations for creating software that doesn't flake. Spoiler: it's not about technology at all.

I'll go over aspects that can help or hinder the reliability of your software:


", "url": "https://authress.io/knowledge-base/articles/2025/12/03/secrets-building-reliable-software", "title": "Slow and boring? Secrets to building reliable software", "summary": "When thinking about building reliable software, we mostly focus on technology. While our tech stack, architecture, and good practices all matter, they are not what determines the reliability of our software.", "date_modified": "2025-12-03T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/11/01/how-we-prevent-aws-downtime-impacts", "content_html": "
info

For help understanding this article or how you can implement auth and similar security architectures in your services, feel free to reach out to me via the community server.

🚨 AWS us-east-1 is down!​

One of the most massive AWS incidents transpired on October 20th. The long story short is that the DNS for DynamoDB was impacted for us-east-1, which created a health event for the entire region. It's the worst incident we've seen in a decade. Disney+, Lyft, McDonald's, New York Times, Reddit, and the list goes on were lining up to claim their share too of the spotlight. And we've been watching because our product is part of our customers critical infrastructure. This one graph of the event says it all:

\"Route

The AWS post-incident report indicates that at 7:48 PM UTC DynamoDB had \"increased error rates\". But this article isn't about AWS, and instead I want to share how exactly we were still up when when AWS was down.

Now you might be thinking: why are you running infra in us-east-1?

And it's true, almost no one should be using us-east-1, unless, well, of course, you are us. And that's because we end up running our infrastructure where our customers are. In theory, practice and theory are the same, but in practice they differ. And if our (or your) customers chose us-east-1 in AWS, then realistically, that means you are also choosing us-east-1 😅.

During this time, us-east-1 was offline, and while we only run a limited amount of infrastructure in the region, we have to run it there because we have customers who want it there. And even without a direct dependency on us-east-1, there are critical services in AWS — CloudFront, Certificate Manager, Lambda@Edge, and IAM — that all have their control planes in that region. Attempting to create distributions or roles at that time were also met with significant issues.

Since there are plenty of articles in the wild talking about what actually happened, why it happened, and why it will continue to happen, I don't need to go into it here. Instead, I'm going to share a dive about exactly what we've built to avoid these exact issues, and what you can do for your applications and platforms as well. In this article, I'll review how we maintain a high SLI to match our SLA reliability commitment even when the infrastructure and services we use don't.

📖 What is reliability?​

Before I get to the part where I share how we built one of the most reliable auth solutions available. I want to define reliability. And for us, that's an SLA of five nines. I think that's so extraordinary that the question I want you to keep in mind through this article is: is that actually possible? Is it really achievable to have a service with a five nines SLA? When I say five nines, I mean that 99.999% of the time, our service is up and running as expected by our customers. And to put this into perspective, the red, in the sea of blue, represents just how much time we can be down.

\"What

And if you can't see it, it's hiding inside this black dot. It amounts to just five minutes and 15 seconds per year. This pretty much means we have to be up all the time, providing responses and functionality exactly as our customers expect.

\"5

🤔 But why?​

To put it into perspective, it's important to share for a moment, the specific challenges that we face, why we built what we built, and of course why that's relevant. To do that, I need to include some details about what we're building — what Authress actually does. Authress provides login and access control for the software applications that you write — It generates JWTs for your applications. This means:

And there are of course many more components, that help complete full auth-platform, but they aren't totally relevant to this article, so I'm going to skip over them.

With that, you may already start to be able to see why uptime is so critical for us. We're on the critical path for our customers. It's not inherently true for every single platform, but it is for us. So if our solution is down, then our customer applications are down as well.

If we put the reliability part in the back corner for one second and just think about the features, we can theorize about a potential initial architecture. That is, an architecture that just focuses on the features, how might you build this out as simple as possible? I want to do this, so I can help explain all the issues that we would face with the simple solution.

Maybe you've got a single region, and in that region you have some sort of HTTP router that handles requests and they forward to some compute, serverless, container, or virtual machine, or, and I'm very sorry for the scenario — if you have to use bare metal. Lastly, you're interacting with some database, NoSQL, SQL, or something else, file storage, and maybe there's some async components.

\"The

If you take a look at this, it's probably obvious to you (and everyone else) that there is no way it is going to meet our reliability needs. But we have to ask, just exactly how often will there actually be a problem with this architecture? Just building out complexity doesn't directly increase reliability, we need to focus on why this architecture would fail. For us, we use AWS, so I look to the Amazon CTO for guidance, and he's famously quoted as saying, Everything fails all the time.

And AWS's own services are no exception to this. Over the last decade, we've seen numerous incidents:

(Edit: And since this article was published AWS started keeping track in their own list and the AWS Health Events Dashboard)

And any one of these would have caused major problems for us and therefore our customers. And the frequency of incident is actually increasing in time. This shouldn't be a surprise, right? Cloud adoption is increasing over time. The number of services AWS is offering is also increasing. But how impactful are these events? Would single one of them have been a problem for us to actually reach our SLA promise? What would happen if we just trusted AWS and used that to pass through our commitments? Would it be sufficient to achieve 99.999% SLA uptime? Well, let's take a look.

🕰️ AWS SLA Commitments​

The AWS Lambda SLA is below 5 nines​

\"Lambda

The API Gateway SLA is below 5 nines​

\"API

The AWS SQS SLA is below 5 nines​

\"SQS

Okay, so when it comes to trusting AWS SLAs, it isn't sufficient. At. All.

We can't just use the components that are offered by AWS, and go from there. We fundamentally need to do something more than that. So the question becomes, what exactly must a dependency's reliability be in order for us to utilize it? To answer that question, it's time for a math lesson. Or more specifically, everyone's favorite topic, probabilities.

Let's quickly get through this torture exercise. Fundamentally, you have endpoints in your service, and you get in an HTTP request, and it interacts with some third-party component or API, and then you write the result to a database. For us, this could be an integration such as logging in with Google or with Okta for our customers' enterprise customers.

\"Third-party

💻 Calculating the allowed failure rate​

So if we want to meet a 5-nines reliability promise, how unreliable could this third-party component actually be? What happens if this component out of the box is only 90% reliable? We'll design a strategy for getting around that.

Uptime is a product of all of the individual probabilities:

Ptotal(Success)=P0(S)∗P1(S)∗⋯Pi(S)⋯∗PC−1(S)\\mathrm{P}_{total}(\\text{Success}) = \\mathrm{P}_{0}(\\text{S}) * \\mathrm{P}_{1}(\\text{S}) * \\cdots \\mathrm{P}_{i}(\\text{S}) \\cdots *\\mathrm{P}_{C - 1}(\\text{S})Ptotal​(Success)=P0​(S)∗P1​(S)∗⋯Pi​(S)⋯∗PC−1​(S)

For the sake of this example, we'll just assume that every other component in our architecture is 100% reliable — That's every line of code, no bugs ever written in our library dependencies, or transitive library dependencies, or the dependencies' dependencies' dependencies, and everything always works exactly as we expect.

Ptotal(Success)=1∗1∗⋯P3rdParty(Success)⋯∗1∗1\\mathrm{P}_{total}(\\text{Success}) = 1 * 1 * \\cdots \\mathrm{P}_{3rdParty}(\\text{Success}) \\cdots * 1 * 1Ptotal​(Success)=1∗1∗⋯P3rdParty​(Success)⋯∗1∗1

So we can actually rewrite our uptime promise as a result of the failure rate of that third-party component.

Ptotal(Success)=1−P3rdParty(Failure)\\mathrm{P}_{total}(\\text{Success}) = 1 - \\mathrm{P}_{3rdParty}(\\text{Failure})Ptotal​(Success)=1−P3rdParty​(Failure)

And the only way that we can actually increase the success rate of the uptime based off of failures is to retry. And so we can multiply out the third-party failure rate and retry multiple times.

Ptotal(Success)=1−P3rdParty(Failure)RetryCount\\mathrm{P}_{total}(\\text{Success}) = 1 - \\mathrm{P}_{3rdParty}(\\text{Failure})^{Retry Count}Ptotal​(Success)=1−P3rdParty​(Failure)RetryCount

Logically that makes a lot of sense. When a component fails, if you retry again, and again, the likelihood it will be down every single time approaches zero. And we can generate a really nasty equation from this to actually determine how many exact times do we need to retry.

RetryCount≥ln(P3rdParty(Failure))ln(1−Ptotal(Success)){Retry Count} \\ge \\frac{ln(\\mathrm{P}_{3rdParty}(Failure))}{ln(1 - \\mathrm{P}_{total}(Success))}RetryCount≥ln(1−Ptotal​(Success))ln(P3rdParty​(Failure))​

How many exactly can it? Rather than guessing whether or not we should retry four times or five times, or put it in a while(true) loop, we can figure it out exactly. So we take this equation and extend it out a little bit. Plugging in our 90% reliable third-party component:

RetryCount≥ln(10%)ln(1−99.999%){Retry Count} \\ge \\frac{ln(\\text{10\\%})}{ln(1 - \\text{99.999\\%})}RetryCount≥ln(1−99.999%)ln(10%)​
RetryCount≥5{Retry Count} \\ge 5RetryCount≥5

We find that our retry count actually must be greater than or equal to five. We can see that this adds up to our uptime expectation:

10%5=0.00001⟹99.999%\\text{10\\%}^{5} = 0.00001 \\Longrightarrow \\text{99.999\\%}10%5=0.00001⟹99.999%

Is this the end of the story? Just retry a bunch of times and you're good? Well, not exactly. Remember this equation?

Ptotal(Success)=P0(S)∗P1(S)∗⋯Pi(S)⋯∗PC−1(S)\\mathrm{P}_{total}(\\text{Success}) = \\mathrm{P}_{0}(\\text{S}) * \\mathrm{P}_{1}(\\text{S}) * \\cdots \\mathrm{P}_{i}(\\text{S}) \\cdots *\\mathrm{P}_{C - 1}(\\text{S})Ptotal​(Success)=P0​(S)∗P1​(S)∗⋯Pi​(S)⋯∗PC−1​(S)

We do really need to consider every single component that we utilize. And specifically when it comes to the third-party component, we had to execute it by utilizing a retry handler. So we need to consider the addition of the retry handler into our equation. Going back to the initial architecture, instead of what we had before, when there's a failure in that third-party component, now we will automatically execute some sort of asynchronous retries or in-process retries. And every time that third-party component fails, we execute the retry handler and retry again.

This means we need to consider the reliability of that retry handler.

\"Retry

Let's assume we have a really reliable retry handler and that it's even more reliable than our service. I think that's reasonable, and actually required. A retry handler that is less reliable than our stated SLA by default is just as faulty as the third-party component.

Let's consider one with five and a half nines — that's half a nine more reliable than our own SLA.

PretryExecutor(S)=99.9995%{P}_{retryExecutor}(S) = \\text{99.9995\\%}PretryExecutor​(S)=99.9995%

But how reliable does it really need to be? Well, we can pull in our original equation and realize that our total uptime is the unreliability or the reliability of the third-party component multiplied by the reliability of our retry handler.

Ptotal(Success)=P3rdParty(S)∗PretryExecutor(S){P}_{total}(\\text{Success}) = {P}_{3rdParty}(\\text{S}) * {P}_{retryExecutor}(S)Ptotal​(Success)=P3rdParty​(S)∗PretryExecutor​(S)

From here, we add in the retries to figure out what the result should be:

PtotalWithRetries(S)=(PretryExecutor(S))retryCount{P}_{totalWithRetries}(S) = ({P}_{retryExecutor}(S))^{retryCount}PtotalWithRetries​(S)=(PretryExecutor​(S))retryCount
99.999%≤(99.9995%)RetryCount\\text{99.999\\%} \\le (\\text{99.9995\\%})^{RetryCount}99.999%≤(99.9995%)RetryCount
RetryCount≤2{RetryCount} \\le 2RetryCount≤2

We have a reliable retry handler, but it's not perfect. And with a retry handler that has reliability of five and a half nines, we can retry a maximum two times. Because remember, it has to be reliable every single time we utilize it, as it is a component which can also fail. Which means left with this equation:

5>25 \\gt 25>2

I don't think comes as a surprise to anyone that in fact five is greater than two. What is the implication here?

The number of retries required for that unreliable third-party component to be utilized by us exceeds the number of retries actually allowed by our retry handler.

RetriesRequired>RetriesAllowed{Retries Required} \\gt Retries AllowedRetriesRequired>RetriesAllowed

That's a failure, the retry handler can only retry twice before itself violates our SLA, but we need to retry five times in order to raise the third-party component reliably up. We can actually figure out what the minimum reliability of a third-party component is allowed to be, when using our retry handler:

Min(P3rdParty(Success))=f(RetryHandler)=1−P3rdParty(Failure)RetriesAllowed99.999%≤1−P3rdParty(Failure)2Min(P3rdParty(Success))>99.7%\\begin{align} Min({P}_{3rdParty}(Success)) &= f(RetryHandler) \\\\ &= 1 - {P}_{3rdParty}(Failure)^{Retries Allowed} \\\\ \\text{99.999\\%} &\\le 1 - {P}_{3rdParty}(Failure)^{2} \\\\ Min({P}_{3rdParty}(Success)) &> \\text{99.7\\%} \\end{align}Min(P3rdParty​(Success))99.999%Min(P3rdParty​(Success))​=f(RetryHandler)=1−P3rdParty​(Failure)RetriesAllowed≤1−P3rdParty​(Failure)2>99.7%​​

Which in turn validates that it's actually impossible for us to utilize that component. 99.7%. 99.7% is the minimum allowed reliability for any third-party component in order for us to meet our required 5-nines SLA. This third-party component is so unreliable (~90%), that even using a highly reliable retry handler, we still can't make it reliable enough without the retry handler itself compromising our SLA. We fundamentally need to consider this constraint, when we're building out our architecture.

That means we drop this third-party component. Done.

And then, let's assume we get rid of every flaky component, everything that don't have a high enough reliability for us. At this point, it's good to think, is this sufficient to achieve our 5-nines SLA? Well, it isn't just third-party components we have to be concerned about. We also have to be worried about those AWs infrastructure failures.

🌩️ Infrastructure Failures​

So let's flashback to our initial architecture again:

\"The

We can have issues at the database layer, right? There could be any number of problems here. Maybe it's returning 500s, there are some slow queries, maybe things are timing out. Or there could be a problem with our compute. Maybe it's not scaling up fast enough. We're not getting new infrastructure resources. Sometimes, even AWS is out of bare metal machines when you don't reserve them, request them get them on demand, and the list go on.

Additionally, there could also be some sort of network issue, where requests aren't making it through to us or even throw a DNS resolution error on a request from our users.

\"AWS

In many of these cases, I think the answer is obvious. We just have to declare the whole region as down. And you are probably thinking, well, this is where we failover to somewhere else. No surprise, yeah, this is exactly what we do:

\"Region

However, this means we have to have all the data and all the infrastructure components duplicated to another region in order to do this. And since Authress has six primary regions around the world, that also means we need multiple backup regions to be able to support the strategy. But this comes with significant wasted resources and wasted compute that we're not even getting to use. Costly! But I'll get to that later.

Knowing a redundant architecture is required is a great first step, but that leaves us having to solve for: how do we actually make the failover happen in practice?

🚧 The Failover Routing Strategy​

Simply put — our strategy is to utilize DNS dynamic routing. This means requests come into our DNS and it automatically selects between one of two target regions, the primary region that we're utilizing or the failover region in case there's an issue. The critical component of the infrastructure is to switch regions during an incident:

\"Utilizing

In our case, when using AWS, this means using the Route 53 health checks and the Route 53 failover routing policy.

We know how we're gonna do it, but the long pole in the tent is actually knowing that there is even a problem in the first place. A partial answer is to say Have a health check, so of course there is health check here. But the full answer is: have a health check that validates both of the regions, checking if the region is up, or is there an incident? And if it is, reports the results to the DNS router.

We could be utilizing the default provided handler from AWS Route 53 or a third-party component which pings our website, but that's not accurate enough from a standpoint of correctly and knowing for certain that our services are in fact down.

It would be devastating for us to fail over when a secondary region is having worse problems than our primary region. Or what if there's an issue with with network traffic. We wouldn't know if that's an issue of communication between AWS's infrastructure services, or an issue with the default Route 53 health check endpoint, or some entangled problem with how those specifically interact with our code that we're actually utilizing. So it became a requirement to built something ourselves, custom, to actually execute exactly what we need to check.

Here is a representation of what we're doing. It's not exactly what we are doing, but it's close enough to be useful. Health check request come in from the Route 53 Health Check. They call into our APIGW or Load Balancer as a router. The requests are passed to our compute which can interact and validate logic, code, access, and data in the database:

\"The

The health check executes this code on request that allows us to validate if the region is in fact healthy:

Region HealthCheck validation
import Authorizer from './authorizer.js';
import ModelValidator from './modelValidator.js';

async healthCheck(request) {
    await profiler.start();

    const dynamoDbCheck = accountDatabase.getDefaultAccount();
    const indexerCheck = indexer.authorizationCheck('HealthCheck');
    const sqsValidation = sqsClient.queue('LiveCheck');

    const authorizer = Authorizer.validate();
    const modelValidation = ModelValidator.validate();

    try {
        await Promise.all([
          dynamoDbCheck, indexerCheck, sqsValidation,
          authorizer, modelValidation ]);
    } catch (error) {
        logger.log('HealthCheck Failed', error);
        return { statusCode: 503 };
    }

    await profiler.end();
    return { statusCode: 200 };
}
  1. We start a profiler to know how long our requests are taking.
  2. Then we interact with our databases, as well as validate some secondary components, such as SQS. While issues with secondary components may not always be a reason to failover, they can cause impacts to response time, and those indicators can be used to predict incoming incidents.
  3. From there, we check whether or not the most critical business logic is working correctly. In our case, that's interactions with DynamoDB as well as core authorizer logic. Compared to a simple unit test, this accounts for corruption in a deployment package, as well instances where some subtle differences between regions interact with our code base. We can catch those sorts of problems here, and know that the primary region that we're utilizing, one of the six, is having a problem and automatically update the DNS based on this.
  4. When we're done, we return success or failure so the health check can track changes.

🌿 Improving the Failover Strategy​

And we don't stop here with our infrastructure failover however. With the current strategy, it's good, in some cases, even sufficient. But it isn't that great. For starters, we have to completely failover. If there's just one component that's problematic, we can't just swap that one out easily, it's all or nothing with the Route 53 health check. So when possible, we push for an edge-optimized architecture. In AWS, this means utilizing AWS CloudFront with AWS Lambda@Edge for compute. This not only helps reduce latency for our customers and their end users depending where they are around the world, as a secondary benefit, fundamentally, it is an improved failover strategy.

And that looks like this:

\"CloudFront

Using CloudFront gives us a highly reliable CDN, which routes requests to the locally available compute region. From there, we can interact with the local database. When our database in that region experiences a health incident, we automatically failover, and check the database in a second adjacent region. And when there's a problem there as well, we do it again to a third region. We can do that because when utilizing DynamoDB we have Global Tables configured for authorization configuration. In places where we don't need the data duplicated, we just interact with the table in a different region without replication.

After a third region with an issue, we stop.

And maybe you're asking why three and not four or five or six? Aren't you glad we did the probabilities exercise earlier? Now you can actually figure out why it's three here. But, I'll leave that math as an exercise for you.

As a quick recap, this handles the problems with at the infrastructure level and with third-party components. And if we solve those, is that sufficient for us to achieve our goal the 5-nines SLA?

For us the answer is No, and you might have guessed, if you peaked at the scrollbar or table contents that there are still quite some additional components integrated into our solution. One of them is knowing that at some point, there's going to be a bug in our code, unfortunately.

💻 Application level failures​

And that bug will get committed to production, which means we're going to end up with an application failure. It should be obvious that it isn't achievable to write completely bug-free code. Maybe there is someone out there that thinks that, and maybe even that's you, and I believe you that you believe that. However, I know it's not me, and realistically, I don't want to sit around and pray that it's also my fellow team members. The risk is too high, because in the case something does get into production, that means it can impact some of our customers. So instead, let's assume that will happen and design a strategy around it.

So when it does happen, we of course have to trigger our incident response. For us, we send out an email, we post a message on our community and internal communication workspaces, and start an on-call alert. The technology here isn't so relevant, but tools like AWS SES, SQS, SNS, Discord, and emails are involved.

Incidents wake an engineer up, so someone can start to take look at the incident, and most likely the code.

But by the time they even respond to the alert, let alone actually investigate and fix the cause of the incident, we would long violated our SLA. So an alert is not sufficient for us. We need to also implement automation to automatically remediate any of these problems. Now, I'm sure you're thinking, yeah, okay, test automation. You might even be thinking about an LLM agent that can automatically create PRs. (Side note: LLM code generation, doesn't actually work for us, and I'll get to that a little further down) Instead, we have to rely on having sufficient testing in place. And yes, of course we do. We test before deployment. There is no better time to test.

This seems simple and an obvious answer, and I hope that for anyone reading this article it is. Untested code never goes to production. Every line of code is completely tested before it is merged to production, even if it is enabled on some flag. Untested code is never released, it is far too dangerous. Untested code never makes it to production behind some magic flag. Abusing feature flags to make that happen could not be a worse decision for us. And that's because we can need to be as confident as possible before those changes actually get out in front of our customers. The result is — we don't focus on test coverage percentage, but rather test value. That is, which areas provide most value, that are most risky, that we care about being the most reliable for our customers. Those are the ones we focus on testing.

Root Cause Analysis (RCA)​

Every incident could have been prevented if we just had one more test. The trick though is actually having that right test, before the incident.

And in reality, that's not actually possible. Having every right test for a service that is constantly changing, while new features are being added, is just unmaintainable. Every additional test we write increases the maintenance burden of our service. Attempting to achieve 100% complete test coverage would require an infinite amount of time. This is known as the Pareto Principle, more commonly the 80-20 rule. If it takes 20% of the time to deliver 80% of the tests, it takes an infinite amount of time to achieve all the tests, and that assumes that the source code isn't changing.

The result is we'll never be able to catch everything. So we can't just optimize for prevention. We also need to optimize for recovery. This conclusion for us means also implementing tests against our deployed production code. One example of this are validation tests.

📋 Validation Tests​

A validation test is where you have some data in one format and data in another format and you use those two different formats to ensure referential consistency. (Side note: There are many different kinds of tests, and I do a deep dive in the different types of tests and how they're relevant in building secure and reliable systems). One concrete example could be you have a request that comes in, you end up logging the request data and the response, then you can compare that logged data to what's actually saved in your database.

In our scenario, which focuses on the authorization and permissions enforcement checks, we have multiple databases with similar data. In one case, there's the storage of permissions as well as the storage of the expected checks and the audit trail tracking the creation of those permissions. So we actually have multiple opportunities to compare the data between our databases asynchronously outside of customer critical path usage.

Running the Validation​

On a schedule, via an AWS CloudWatch Scheduled Rule, we load the data from our different databases and we compare them against each other to make sure it is consistent. If there is a problem, then if this fires off an incident before any of our customers notice, so that we can actually go in and check what's going on.

\"The

This sounds bad on the surface that it could ever happen. But the reality of the situation is that a discrepancy can show up as a result of any number of mechanisms. For instance, the infrastructure from AWS could have corrupted one of the database shards and what is written to the databases is inconsistent. We know that this can happen as there is no 100% guarantee on database durability, even from AWS. AWS does not guarantee Database Durability, are you assuming they do, because we don't! So actually reading the data back and verifying its internal consistency is something that we must do.

While it might not seem that this could reduce the probability of there being an incident. Consider that a requested user permission check whose result doesn't match our customer's expectation is an incident. It might not always be one that anyone identifies or even becomes aware of, but it nonetheless a problem, just like a publicly exposed S3 is technically an issue, even if no one has exfiltrated the data yet, it doesn't mean the bucket isn'is sufficiently secured.

🎯 Incident Impact​

There are two parts to the actual risk of an incident. The probability and the impact. Everything in this article I've discuss until now talks about reducing the probability of an incident, that is — the likelihood of it happening. But since we know that we can't avoid ever having an incident, we also have to reduce the impact when it happens.

One way we do that is by utilizing an incremental rollout. Hopefully everyone knows what incremental rollout is, so I'll instead jump straight into how we accomplish it utilizing AWS. And for that we focus again on our solution integrating with CloudFront and our edge architecture.

The solution for us is what I call Customer Deployment Buckets. We bucket individual customers into separate buckets and then deploy to each of the buckets sequentially. If the deployment rolls out without a problem, and it's all green, that is everything works correctly, then we go on to the second bucket and then deploy our code to there, and then the third bucket, and so on and so forth until every single customer has the new version.

\"Rolling

If there is an issue, we stop the rollout and we go and investigate what's actually going on. While we can't prevent the issue from happening to the earlier buckets, we are able to stop that issue from propagating to more customers, having an impact on everyone, and thus reduce the impact of the incident.

As I mentioned before the biggest recurring issue isn't executing an operations process during an incident, it's identifying there is a real incident in the first place. So, How do we actually know that there's an issue?

If it was an easy problem to solve, you would have written a unit task or integration test or service level test and thus already discovered it, right? So adding tests can't, by design, help us. Maybe there's an issue with the deployment itself or during infrastructure creation, but likely that's not what's happening.

Now, I know you're thinking, When is he going to get to AI?

Whether or not we'll ever truly have AI is a separate <rant /> that I won't get into here, so this is the only section on it, I promise. What we actually do is better called anomaly detection. Historically anomaly detection, was what AI always meant, true AI, rather than an LLM or agent in any way.

🔎 AI: Anomaly Detection​

This is a graph of our detection analysis:

\"namely

You might notice that it's not tracking 400s or 500s, which are in reality relatively easy to detect. But in fact don't actually tell us meaningfully what's wrong with our service or whether or not there really is a problem. Impact is measured by business value, not technical protocol level analytics, so we need to have a business-focused metric.

And for us, at Authress, the business-focussed metric we use to identify meaningful incidents we call: The Authorization Ratio. That is the ratio of successful logins and authorizations to ones that are blocked, rejected, timeout or are never completed for some reason.

The above CloudWatch metric display contains this exact ratio, and here in this timeframe represents an instance not too long ago where we got really close to firing off our alert.

\"Anomaly

Here, there was a slight elevation of errors soon after a deployment. The expected ratio was outside of our allowance span for a short period of time. However not long enough to trigger an incident. We still investigated, but it wasn't something that required immediate remediation. And it's a good reminder that identifying problems in any production software isn't so straightforward. To achieve high reliability, we've needed an AI or in this case anomaly detection to actually identify additional problems. And realistically, even with this level of sophistication in place, we still can never know with 100% certainty that there is actually an incident at any moment. And that's because \"what is an incident\", is actually a philosophical question...

🌹 Does it smell like an incident?​

Our anomaly detection said – almost an incident, and we determined the result – no incident. But does that mean there wasn't an incident? What makes an incident, how do I define an incident? And is that exact definition ubiquitous, for every system, every engineer, every customer?

Obviously not, and one look at the AWS Health Status Dashboard is all you need to determine that the identification of incidents is based on subjective perspective, rather than objective criteria. What's actually more important is the synthesis of our perspective on the situation and what our customers believe. To see what I mean, let's do a comparison:

\"incident

I'm going to use Authress as an example. So I've got the product services perspective on one side and our customer's perspective on the other.

Incident Alignment​

In the top left corner we have alignment. If we believe that our system is up and working and our customers do, too, then success, all good. Everything's working as expected.

\"incident

Inversely in the opposite corner, maybe there is a problem. We believe that one of our services is having an issue, and successfully, we're able to identify it. Most importantly, our customers say–yes, there is an issue for us.

It's not great that there's an incident, but as I've identified incidents will absolutely happen, and the fact we've correctly aligned with our customers on the problem's existence independently allows us to deploy automation to automatically remediate the issue. That's a success! If it's a new problem that we haven't seen before, we can even design new automation to fix this. Correctly identifying incidents is challenging, so doing that step correctly, leads itself very well to automation for remediation.

Perspective Mismatch​

One interesting corner is when our customers believe that there's nothing wrong, there have been no incidents reported, but all our alerts are saying – RED ALERT — someone has to go look at this!

\"incident

In this case, our alerts have identified a problem that no one cares about. This often happens in scenarios where our customers are in one region, Switzerland for example, with local region users, a health care, manufacturing, or e-commerce app, is a good example, rather than global, who are likely asleep at 2:00 AM. And that means an incident at the moment, could be an issue affecting some customers. But if they aren't around to experience it, is it actually happening?

You are probably wincing at that idea. There's a bug, it must be fixed! And sure that's a problem, it's happening and we should take note of what's going on. But we don't need to respond in real time. That's a waste of our resources where we could be investing in other things. Why wake up our engineers based on functionality that no one is using?

I think one of the most interesting categories is in the top right-hand corner where:

This is known as a gray failure.

Gray Failures​

\"Gray

And it can happen for any number of reasons. Maybe there is something in our knowledge base that tells our customers to do something one way and it's confusing and they've interpreted it in a different way. So there's a different expectation here. That expectation can get codified into customer processes and product services.

Or maybe our customer is running different tests from us, ones that are of course, valuable for their business, but not ones that we consider. Or more likely they are just using a less resilient cloud provider.

Most fundamentally, there could really be an incident, something that we haven't detected yet, but they have. And if we don't respond to that, it could grow, and left unchecked, escalate, and eventually impact all our customers. This means we need to give our customers an easy way to report incidents to us, which we can immediately follow up with.

For us, every single incident, every single customer support ticket that comes into our platform, we immediately and directly send it to our engineering team. Now, I often get pushback on this from other leaders. I'm sure, even you might be thinking something like — I don't want to be on call for customer support incidents. But if you throw additional tiers in your organization between your engineering teams and your customers, that means you're increasing the time to actually start investigating and resolving those problems. If you have two tiers before your engineering team and each tier has its own SLA of 10 minutes to triage the issue, that means you've already gone through 20 minutes before an engineer even knows about it and can go and look at it. That violates our SLA by fourfold before investigation and remediation can even begin.

Instead, in those scenarios, what I actually recommend thinking about is how might you reduce the number of support tickets you receive in aggregate? This is the much more appropriate way to look at the problem. If you are getting support tickets that don't make sense, then you've got to investigate, why did we get this ticket? Do the root cause analysis on the ticket, not just the issue mentioned in it — why the ticket was even created in the first place.

A ticket means: Something is broken. From there, we can figure out, OK, maybe we need to improve our documentation. Or we need to change what we're doing on one of our endpoints. Or we need to change the response error message we're sending. But you can always go deeper.

The customer support advantage​

And going deeper, means customer support is critical for us. We consider customer support to be the lifeline of our service level agreement (SLA). If we didn't have that advantage, then we might not have been able to deliver our commitment at all. So much so that we report some of our own CloudWatch custom metrics to our customers so they can have an aggregate view of both what they know internally and what we believe. We do this through our own internal dashboard in our application management UIs.

\"Authress

Helping our users identify incidents benefits us; because we can't catch everything. It's just not possible.

💀 Negligence and Malice​

To this point, we've done the math on reliability of third-party components. We've implemented an automatic region failover and added incremental rollout. And we have a core customer support focus. Is that sufficient to achieve 5-nines of reliability?

If you think yes, then you'd expect the meme pictures now. And, I wish I could say it was enough, but it's not. That's because we also have to deal with negligence and malice.

We're in a privileged position to have numerous security researchers out there on the internet constantly trying to find vulnerabilities within our service. For transparency, I have some of those reports I want to share:

“Real” Vulnerability Reports​

\"fake

I am a web security researcher enthusiast. Do you give a monetary reward?

Okay, this isn't starting out that great. What else have we received?

\"appeal

I found some vulnerabilities in your website. Do you offer rewards for ethical hackers?

Well, maybe, but I think you would actually need to answer for us, what the problem actually is. And you also might notice this went to our spam. It didn't even get to our inbox. So a lot of help they might be providing. Actually we ignore any ”security” email sent from a non-custom domain.

\"Phishing

This one was really interesting. We had someone attempting to phish our engineering team by creating a support ticket and putting in some configuration trying to get us to provide them our own credentials to one of our third-party dependencies. Interestingly enough, our teams don't even have access to those credentials directly.

And, we know this was malicious because the credentials that they are referencing in the support request are from our honey pot, stuck in our UI to explicitly catch these sorts of things. The only way to get these credentials is if they hacked around our UI application and pulled out of the HTML. They aren't readily available any other way. So it was very easy for us to detect that this “report” was actually a social engineering attack.

And this is one of my favorites, and I can't make this up:

\"Bugbounty

I have found many security loophole. How much will you pay if you want to working with me like project?

That's the exact quote, I don't even know what that means. Unfortunately, LLMs will actually start to make all of these future \"vulnerability reports\" sound more appealing to read in the future, for better or worse. However, at the end of the day, the truth is that these are harmless. And we actually do have a security disclosure program that anyone can go and submit problems for. I hope the message to white-hat hackers is please use that process, and the legitimate reports usually do go through it. Do not send us emails. Those are going to go into the abyss. Alternatively, you can follow our security.txt public page or go to the disclosure form, but with email, the wrong people are going to get that and we can't triage effectively.

Vulnerabilities in our services can result in production incidents for our customers. That means security is part of our SLA. Don't believe me, I'll show you how:

Multitenant considerations​

It's relevant for us, that Authress is a multitenant solution. So some of the resources within our service are in fact shared between customers.

Additionally, customers could have multiple services in a microservice architecture or multiple components. And one of these services could theoretically consume all of the resources that we've allocated for that customer. In that scenario, that would cause an incident for that customer. So we need to protect against resource exhaustion Intra-Tenant. Likewise, we have multiple customers. One of those customers could be consuming more resources than we've allocated to the entire tenant. And that could cause an incident across Inter-Tenant and cause an incident across our platform and impact other customers.

Lastly, we have to be worried about our customers, our customers' customers, and our customers' customers' customers, because any one of those could be malicious and consume their resources and so on and so forth, thus causing a cascading failure. A failure due to lack of resources is an incident. The only solution that makes sense for this is, surprise, rate limiting.

Helpful Rate Limiting​

So we need to rate-limit these requests at different levels for different kinds of clients, different kinds of users, and we do that within our architecture, at different fundamental levels within our infrastructure.

\"CloudFront

Primarily there are protections at our compute level, as well at the region level, and also place protections at a global level. In AWS, this of course means using a web application firewall or WAF. I think our WAF configuration is interesting and in some ways novel.

Fundamentally, one of the things that we love to use is the AWS managed IP reputation list.

The reputation list is list of IP addresses that have been associated with malicious activity outside of our service throughout other customers at AWS and other providers out there in the world where a problem has been detected. That means before those attacks even get to our service or to our customers' instances of Authress, we can already know to block them, and the WAF does that. This is great, and most importantly, has a very low false positive rate.

However, the false positive rate is an important metric for consideration of counter measures against malicious attacks or negligent accidental abuse of resources, and something that prevents us from using any other managed rules from AWS or external providers. There's two problems with managed rules, fundamentally:

  1. Number one is the false positive rate. If that is even a little bit more, it couldn't be sustainable, and would result in us blocking legitimate requests coming for a customer. This means it is a problem, and it's an incident for them if some of their users can't utilize their software because of something we did. False positives are customer incidents.
  2. The second one is that managed rules are gratuitously expensive. Lots of companies are building these just to charge you lots of money, and the ROI just doesn't seem to be there. We don't see useful blocks from them.

But the truth is, we need to do something more than just the reputation list rule.

Handling Requests at Scale​

And the thing that we've decided to do is — add blocking for sufficiently high requests. By default, any Authress account's service client that goes above 2,000 requests per second (RPS), we just immediately terminate. Now, this isn't every customer, as there are some out there for us that do require such a high load or even higher (as 2k isn't that high). But for the majority of them, if you get to this number and they haven't talked to us about their volume, then it is probably malicious in some way. You don't magically go from zero to 2,000 one day, unless it is an import job.

Likewise, we can actually learn about a problem long before it gets to that scale. We have milestones, and we start reporting loads from clients at 100, 200, 500, 1,000, et cetera. If we see clients hitting these load milestones, we can already start to respond and create an incident for us to investigate before they reach a point where they're consuming all of the resources in our services for that customer. And we do this by adding alerts on the COUNT of requests for WAF metrics.

However, we also get attacks at a smaller scale. Just because we aren't being DDoS-ed doesn't mean there isn't attack. And those requests will still get through because they don't meet our blocking limits. They could be malicious in nature, but only identifiable in aggregate. So while single request might seem fine, if you see the same request 10 times a second, 100 times a second, something is probably wrong. Or if you have request urls that end in .php?admin, when no one has run WordPress in decades, you also know that there's a problem. We catch these by logging all of the blocked requests.

We have automation in place to query those results and update our rules, but a picture is worth a thousand words:

\"WAF

Here you can see a query based off of the IP addresses from the client that are being utilized and sorted by frequency. When we get these requests that look non-malicious individually, we execute a query such as this one and we check to see if the results match a pattern. You can use ip address matching or more intelligently, something called the JA3 or JA4 fingerprints of those requests There are actually lots of options available, I'm not going to get into exactly what they are, there are some great articles on the topic. And there are more mechanisms to actually track these used throughout the security industry, and utilizing them let's you instantly identify: Hey, you know what? This request violates one of our patterns, maybe we should block all the requests from that client.

And so, rather than waiting for them to get to the point where an attacker is consuming 2,000 requests per second worth of resources, you can stop there right away. In the cases where we can't make a conclusive decision, this technology gives us another tool that we can utilize to improve our patterns for the future. Maybe it goes without saying, but of course because we've running our technology to many regions around the world, we have to work on deploying this infrastructure in all these places and push it out to the edge where possible.

\"Authress

🎁 The Conclusion​

I said a lot of things, so I to quickly want to quickly summarize our architecture that we have in place:

  1. Third-party component reliability reviews. I can't stress this enough. Don't just assume that you can utilize something. And sometimes in order to achieve 5-nines, you actually have to remove components from your infrastructure. Some things are just not able to be utilized no matter what. Now maybe you can put it in some sort of async background, but it can't be on the critical path for your endpoints.
  2. DNS failover and health checks. For places where you have an individual region or availability zone or cluster, having a full backup with a way to conclusively determine what's up and automatically failover is critical.
  3. Edge compute where possible. There's a whole network out there of services that are running on top of the cloud providers, which help guarantee your capability to run as close to as possible to where your users are and reduce latency.
  4. Incremental rollout for when you want to reduce the impact as much as possible.
  5. The Web Application Firewall for handling those malicious requests.
  6. Having a Customer Support Focus to enable escalating issues that outside your area of detection.

And through seven years or so that we've been doing this and building up this architecture, there's a couple of things that we've learned:

\"Unsolvable

Murphy's Law​

Everything fails all the time. There absolutely will be failures everywhere. Every line of code, every component you pull in, every library, there's guaranteed to be a problem in each and everyone of those. And you will for sure have to deal with it, at some point. So being prepared to handle that situation, is something you have to be thinking through in your design.

DNS​

DNS, yeah, AWS will say it, everyone out there will say, and now we get to say it. The global DNS architecture is pretty good and reliable for a lot of scenarios, but I worry that it's still a single point of failure in a lot of ways.

Infrastructure as Code (IAC)​

The last thing is infrastructure as code challenges. We deploy primary regions, but then there's also the backup regions, which are slightly different from the primary regions, and then there are edge compute, which are, again, even more slightly different. And then sometimes, we do this ridiculous thing, where we deploy infrastructure dedicated to one customers. And in doing so, we're running some sort of IaC to deploy those resources.

It is almost exactly the same architecture. Almost! Because it isn't exactly the same there are quite the opportunities for challenges to sneak it. That's problematic with even Open Tofu or CloudFormation, and often these tools make it more difficult, not less. And good luck to you, if you're still using some else that hasn't been modernized. With those, it's even easier to run into problems and not get it exactly correct.

The last thing I want to leave you with is, well, With all of these, is that actually sufficient to achieve five nines?

No. Our commitment is 5-nines, what we do is in defense of that, just because you do all these things doesn't automatically mean your promise of 5-nines in guaranteed. And you know what, you too can promise a 5-nines SLA without doing anything. You'll likely break your promise, but for us our promise is important, and so this is our defense.

info

For help understanding this article or how you can implement auth and similar security architectures in your services, feel free to reach out to me via the community server.

", "url": "https://authress.io/knowledge-base/articles/2025/11/01/how-we-prevent-aws-downtime-impacts", "title": "How when AWS was down, we were not", "summary": "Nothing can have an unlimited uptime. But even when AWS and other hyperscalers are having incidents, we've created an architecture that is resilient to those outages. Here's how we do it for Authress.", "date_modified": "2025-11-01T00:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/05/26/what-the-heck-is-auth", "content_html": "

\"What

Authentication remains a complicated yet critical aspect of application security. In this talk, I'll demystify the core concepts, diving into access tokens, refresh tokens, and browser security mechanisms like WebAuthn for hardware-based authentication.\nAdditionally, I'll explore techniques such as session handling, revocation strategies, silent authentication for improved security UX, and the usage scopes for controlling access granularity, and common pitfalls associated with each.

Finally, I'll delve into JSON Web Tokens (JWTs), the use of EdDSA signatures for enhanced security and performance, as well as the common pitfalls that seasoned pro and newcomer alike struggle with when it comes to auth. Here I hope to equip everyone with some additional knowledge to navigate its complexities and build secure, user-friendly systems.

info

For help understanding this article or how you can implement auth and similar security architectures in your services, feel free to reach out to me via the community server.

", "url": "https://authress.io/knowledge-base/articles/2025/05/26/what-the-heck-is-auth", "title": "What the @#!? is Auth", "summary": "Authentication remains a complicated yet critical aspect of application security. In this talk, I'll demystify the core concepts, diving into access tokens, refresh tokens, and browser security mechanisms like WebAuthn for hardware-based authentication.", "date_modified": "2025-05-26T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/05/25/api-gateway-authorizers-vulnerable-by-design", "content_html": "

I had the benefit of joining the AWS Community Day in Zürich this week, most went as expected but, then an interesting question came up...

Does caching in API Gateway create vulnerabilities for products using Authorizer Caching?

Authorization​

When your users call your API, you have an obvious need to verify these requests should actually be allowed. I've talked extensively about this in my academy article on what the @#!? is Auth.

Even if you haven't read that article, if you are well versed in the need for users to authenticate and authorize to your specific service API and endpoints, then you get the gist.

So you have a need to verify the access tokens sent by users on ever request. When using AWS this means using API Gateway, and when using API Gateway that likely means you'll be using an API Gateway Authorizer.

Authorizers in API exist so that you can verify more easily verify the user access tokens. As a reminder an authorization token looks like this:

Decoded JWT authorization token.
{
    \"identityProviderId\": \"https://authress.io\",
    \"userId\": \"TechInternals|test-user-001\",
    \"expires\": 1761483600,
    \"signatureKeyId\": \"example-key\",
    \"signature\": \"SflKxwRJSMeKKF2Qt4fwpMe\"
}

And the process to verify the token looks like this:

Verifying user tokens
const authressClient = new AuthressClient({ authressApiUrl });
const userIdentity =
    await authressClient.verifyToken(userToken);

Of course swapping in your favorite open source JWT verifier. More extensive details on this depending on your identity provider are available

Now I know what you are thinking:

I'm going to get a lot of requests from the same user to my same API, for different resources. That means they are all going to have the same JWT. Wouldn't it be great to cache those results so that I don't need to verify the same JWT over and over again every time this same user makes a similar request for similar data with same JWT.

And you would be right!

Caching​

However, if you wrote the above code and you cache it, you might start to see a problem with it...

Caching by default in API gateway is keyed from the authorization token only and nothing else. This means that the result from one request will interfere with the next one.

Let's take for example the policy result from an AWS API Gateway Authorizer. It might see something like this:

API Gateway Authorizer policy result containing Resource.
const policy = {
      principalId: userIdentity.sub,
      policyDocument: {
        Version: '2012-10-17',
        Statement: [{
            Effect: 'Allow',
            Action: 'execute-api:Invoke',
            Resource: event.methodArn
        }]
      },
      context: {
        principalId: userIdentity.sub
      }
    };

There is actually a problem with this however. The cache key by default is only the JWT, but the result of this policy says that the user is only allowed to one particular event.methodArn. A method ARN as a reminder is like GET /orders/order_id_123.

That means on a followup request with the same JWT to a different endpoint GET /orders/order_id_456, even if the user should have access to that resource and their JWT is still valid, API Gateway will deny that request.

Why?

Well that is simple, because the result is cached based only on the JWT. The cached result specifies only that one route GET /orders/order_id_123 has been authorized.

Worst case scenario, you have a short cache time, and the only thing that happens is a short but confusing user experience, that quickly results in the correct behavior.

But you are smart, you realize there is a fix, instead of passing the event.methodArn as the result policy you specify ['arn:aws:execute-api:*:*:*'] as the resource result.

Now subsequent requests as long as the JWT is still valid, irrespective of the endpoint, will allow the user through!

🎉🎉🎉

And this works great.

But you are thinking why stop there. Can we go further?

And the answer is also yes.

Authorization of Granular Resources Based Access Control​

You might be using solutions such as AWS Verified Permissions hoping to connect it together with Cognito and API Gateway.

Now I know what you are thinking, why is Warren investigating verified permissions when Authress already solves all these problems? Well sometimes even I have to write an article about how the integration of default resources in AWS can cause security misconfigurations.

Your decision is—Not just cache the validity of the JWT, but you also want to cache whether or not the user actually has access to call the endpoint in question. That is, you decide to take the additional step of verifying the user's authorization and you also cache it, then you will have just created a majority security vulnerability in your application.

Do you already see what the problem might be?

In your authorizer you are likely to write:

The authorization check
const hasAccess = await authress.userPermissions.authorizeUser(
   userId,
   'resource',
   'resource:read');

If you are checking the user's access inside the authorizer and it is cached, then subsequent requests to the same API will utilize the cached result.

If the user has:

And then calls

  1. GET orders_123
  2. GET orders_456

They will incorrectly be allowed to access that second order.

That's because the authorizer will have access ALLOW for:

Authorization check for orders_123
const hasAccess = await authress.userPermissions.authorizeUser(
   userId,
   'orders_123',
   'orders:read');

That ALLOW is set as the cache result for the user's JWT:

Stored Cached values by default
JWT_001 => ALLOW

The cache doesn't contain the orderId in it. Or said differently the cache is NOT:

Stored Cached values with additional cache keys
[JWT_001, GET, orders_123] => ALLOW

That means when the second request comes in, we got to the cache table, see the cache already exists for JWT_001, return ALLOW, and never actually check the authorization for orders_456.

Removing the security vulnerability​

It would be nice of API Gateway to be secure by default and require the identity source cache key to include the resource path and method. But it isn't, so it doesn't. And this risk is similar to ones experienced by engineers all day long with caching in CloudFront. And if we think about the frequency of issues with caching in CloudFront which has no security vulnerability, we can realize that—since AWS created the Verified Permissions service and related functionality, this opened a huge security vulnerability potential configuration in API Gateway.

This isn't an explicit vulnerability in the service though, since the vulnerability only exists based on improper configuration, but here the improper configuration is the default. Show me a company using API Gateway and AWS Verified Permissions, and I bet I can show you a Security Bounty waiting to be collected.

The resolution here is to force the API Gateway Authorizer to cache also on the httpMethod (Context) and path (Context).

\"API

Once that is done, now API Gateway will close this security hole because the cache key will match the authorization check performed by your authorization provider.

Recommendations​

On your side there is little you can do to remove the pit of failure. Review documentation, invest in deep understanding of the tools you used especially when security is involved. I guess also keep reading my posts as I often try to focus on security related topics.

On the AWS side, there is absolutely a strategy that would have fixed this by design. The authorizer should not have access to the Path and Method properties of the HTTP request unless the identity source cache key includes them. This would require breaking existing configurations, but it would be in the name of security by default.

Going further​

There are actually lots of different ways to cache permissions results in AWS when not even using Verified Permissions and for an extensive list of the options and my personal recommendations check out this Auth Academy article on the topic.

info

Want to chat more about this topic? Join our community!

", "url": "https://authress.io/knowledge-base/articles/2025/05/25/api-gateway-authorizers-vulnerable-by-design", "title": "API Gateway Authorizers: Vulnerable By Design (Be Careful!)", "summary": "API Gateway authorizers provide a unique opportunity to exposure yourself to malicious requests when defaults are used, and even when the defaults aren't used.", "date_modified": "2025-05-25T00:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/03/18/meeting-impossible-slas", "content_html": "

\"Meeting

Running critical components requires a completely different mindset when the required uptime is five nines. Can a service even have a 99.999% uptime guarantee? It's easy to promise, but delivering on that is something else. What works at two or three nines can't work when components become critical. The math actually tells us this.

You'll get a full review of how we managed to promise such a high up time, if it is even possible to make it happen, and how we think about it. From this talk attendees will learn key trade-offs in elevating the reliability of their solutions.

Key Takeaways — The core components of a highly reliable solution, Lessons learned in the process, Architecture strategies to increase the reliability.

info

For help understanding this article or how you can implement improved reliability or a similar architecture in your services, feel free to reach out to the Authress development team or follow along in the Authress documentation.

", "url": "https://authress.io/knowledge-base/articles/2025/03/18/meeting-impossible-slas", "title": "Meeting Impossible SLAs: How we made our uptime 99.999%", "summary": "Running critical components requires a completely different mindset when the required uptime is five nines. Can a service even have a 99.999% uptime guarantee? It's easy to promise, but delivering on that is something else. What works at two or three nines can't work when components become critical. The math actually tells us this. Presented at L8 Conf Warsaw 2025.", "date_modified": "2025-03-18T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/03/13/using-systems-thinking-to-understand-unintended-consequences", "content_html": "

As engineers, we’re used to thinking in algorithms where causality is clear and the same set of actions always produces the same results. When we apply algorithmic thinking to complex systems, however, our well-meaning actions often result in unintended consequences. Whether driving organizational change or fixing a small bug in your monitoring system, it’s not enough to consider the most immediate and direct result.

In this talk, I’ll go over examples of simple changes causing large scale unintended consequences, explain how to recognize when your actions could impact more than you wish for, share techniques for anticipating ripple effects so you can use them to your advantage, and help you become more adept at reasoning about complex systems.

info

Want to chat more about this topic? Join our community!

", "url": "https://authress.io/knowledge-base/articles/2025/03/13/using-systems-thinking-to-understand-unintended-consequences", "title": "Unintended consequences of well-meaning changes", "summary": "As engineers, we’re used to thinking in algorithms where causality is clear and the same set of actions always produces the same results. When we apply algorithmic thinking to complex systems, however, our well-meaning actions often result in unintended consequences. Whether driving organizational change or fixing a small bug in your monitoring system, it’s not enough to consider the most immediate and direct result.", "date_modified": "2025-03-13T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/02/03/your-source-code-is-not-that-special", "content_html": "

Whenever the topic of protecting developer machines comes up, I find myself dismantling the same fallacies over and over again. One of those fallacies goes like this: we must protect our source code! So far so good. Protecting your source code makes sense. But then it turns into: our source code is precious! It is our Intellectual Property! And then: our precious source code is on developers’ machines, we must do everything in our power to protect these machines! And that’s a fallacy.

It turns out, your source code itself, that treasured IP, is really not that special. Let’s think about it for a moment.

How much of your code is truly unique and innovative? How much of it is copy/pasted from Stack Overflow? How much is done by third party libraries you depend on? If you were given a chance to start from scratch and completely rewrite all your company’s software, how much of the end result would look exactly like your today’s code? (Ignoring, of course, those few who would jump at the opportunity because they haven’t yet had the privilege of spending countless years doing a migration project…) Unless you’re working on developing something new at the already cutting edge of software technology, your code is largely a glue holding various common pieces together. There aren’t many companies out there that have some secret sauce hidden in their code, and even if they do, that secret sauce is a tiny fraction of the whole code base.

Here’s a thought experiment—Imagine that your source code suddenly goes public. What is the worst that would happen? Would some indie hacker just copy it and spin up a new business just like yours to become your biggest competitor over night? Or what if it is your biggest competitor getting access to your code? Would they go “ah-ha!” and replace their code with yours? Spoiler alert: no engineer I’ve ever worked with has had the thought while looking at someone else’s code “Wow this is great, we would never be able to write this better.”

If exposing your source code was such a catastrophe, how do we explain all these software companies who make their code available to the public? Multiple very successful businesses nowadays operate under that principle and chances are the company you work for is paying for some of them. It’s worth noting that most of the open-core or source-available software is not simple nor does it solve some trivial problem. Yet all these companies are doing well, commercially.

These days, the code is rarely the source of competitive advantage. And if it is, it’s not down to how well or clever it’s written. What matters is how you position yourself in the market, how you acquire your customers, how you retain them, and how you convey your value to the customers. What also matters is your pricing model, your operational efficiency, your margins, and how well you control the cost over time. Notice how these have very little to do with the code you’ve written? As a matter of fact, the code a company writes is a result of those exact strategies. Even if we look at the software itself, there are multiple other factors about it that are more important than the source code. Are there bugs in production? How often and how quickly do they get fixed? How easy is it for a customer to get help when something isn’t working? How easy is it to get started and onboard new users? How reliable is your software? How often do you leak your customers' data and do they care?

The true value of your code is not the source, but rather how well it supports the business model, how it’s operated and how it runs in production. And of these three, the most crucial bit is the latter - your source code running in production. That is the only thing truly worth protecting. So that code on your developer machine? Who cares.

", "url": "https://authress.io/knowledge-base/articles/2025/02/03/your-source-code-is-not-that-special", "title": "Your source code is not that special", "summary": "Whenever the topic of protecting developer machines comes up, I inevitably hear that we must protect the machines in order to protect the source code. It's a fallacy. Your source code is really not that special.", "date_modified": "2025-02-03T10:00:00.000Z", "author": { "name": "Dorota Parad", "url": "https://dorotaparad.ch" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/01/20/your-security-teams-job-is-not-security", "content_html": "

There is a very common misunderstanding in tech that I keep running into. Many people believe that a person hired by their company who has “security” in their title is there to ensure that your software and operations are secure. It’s especially heartbreaking to see new graduates who studied the topic and are looking for their first job in cybersecurity, full of hopes that they’ll get to fight the baddies, make things better, or at least make sure that software engineers are doing things properly.\nThe truth is, when a company hires a CISO and invests in a security team, those people’s jobs have nothing to do with security. They are there to ensure compliance. It’s a very important job, but a rather different one from what the name implies. In fact, compliance and security are often at odds with each other.

So what is the job of a CISO and the security team they may be in charge of?

The job of a CISO​

No matter what the PR release may say, companies hire a C-level security person for one reason only - to ensure compliance with whatever security-ish requirements may be imposed on the business. Compliance in security boils down to convincing some third party authority that your company is doing a good enough job minimizing security risks. It’s important to highlight two things here: it’s not about the job being perfect, but merely good enough, and it’s not about eliminating the risks, but merely minimizing them. And, to be cynical, it’s not even about doing these things but rather convincing the stakeholders that you are. Hopefully it’s clear why we see so many cases of security theater out there.

Putting the cynical view aside, the job of a CISO is really to craft a convincing narrative on how your company minimizes security risks. People to be convinced are usually associated with an external third party, which can fall in one of three categories:

As you can imagine, the stakeholders I’ve listed above have no idea about the specifics of your business. They also tend to have limited knowledge of software and technology in general. And your poor CISO needs to convince them that your company sufficiently minimizes security risks. It’s a tough job!

Security team’s dilemma​

Since security teams get created to ensure compliance, which is all about creating a convincing security narrative for non-technical, external stakeholders, you’d imagine they would be composed of people good at communication, or at least great at writing. But it is rarely the case.

A lot of people working in security, and especially those bright-eyed new graduates joining the profession, do so because they love the idea of good security. They may be motivated by thinking about all the threats and ways to thwart them, or by laying down a foundation to do things “correctly”. Regardless of which flavor of cybersecurity these people are drawn to, almost no one gets into this discipline by thinking “wow, I really want my job to be maintaining paperwork and ensuring adherence to corporate procedures!”\nPeople who are motivated by the latter end up as auditors rather than security professionals. As a result, we have a large number of professionals who want to, and often believe they should, do one thing but are instead paid to do something completely different. It’s no surprise they would face pushback.

There may be pushback from senior business leaders, who may not understand why should the company spend money or effort to implement this particular protection strategy over this other one which seems much simpler to grasp (and usually involves simply paying six figures for a specific tool).\nThere may be pushback from other employees, who don’t want to do any extra steps or change any of their well-established procedures, and who are under pressure to deliver whatever their department is paid to do.\nThere may be pushback from the tenured people in security organizations, who are comfortable with the status quo and may have strong attachment to “how we do things around here”.

It can all be disheartening.

Now imagine you’re a CISO, by nature a very specialized role, yet your organization is entirely separate from the structures where the actual work happens: the IT, software development, operations. You may have a passion for security, but all your company cares about is the optics. You have to make your company look good in front of people who care nothing about your business model, have no understanding of modern technology, and don’t even know what security is. It doesn’t exactly look like a dream job, does it?

Compliance at the cost of security​

Crafting a convincing narrative about a deeply nuanced, technical topic for people who don’t care about nuances and have limited technical knowledge requires finesse. If you also want that narrative to come with actual security practices that are fully integrated into your company’s daily operations, then that’s a whole other level of difficulty. It’s one that is far beyond the scope of compliance. Why bother then? It is tempting to optimize for the ease of audits. It is even more tempting to just blindly copy whatever someone else did that got them to pass their audit, even if what you’re copying doesn’t make sense in your situation.

This is why so many businesses leak their customers’ data, get infected by ransomware, suffer random downtimes for unexplained reasons (if you don’t look, you don’t see; if you don’t see, you don’t have to report it) all the while having a fully staffed security team and investing over 13% of their IT budget in security.

The truth is, lazy compliance is often at odds with actual, proper security. The sad part is, in the industry today, the vast majority of compliance is of the lazy kind.

The certification auditor doesn’t understand modern cloud technology? Instead of relying on cloud provider’s built-in security, which you get almost for free but which may be difficult to explain, your security team decides to go old-school and locks down access to corporate resources by only allowing specific IP addresses to connect. This means you need VPN. Some time later, hackers are happily using your “secure” VPN client solution to gain access.

Struggling to explain how your zero trust architecture makes endpoint protection obsolete? Instead of spending effort to document your reasoning, your security team decides to install spyware on all your employee machines. Some time later, spyware has a bug, now none of your machines boot up. Or perhaps your “secure” spyware provider was hacked again and all the data from your devices got wiped. Or that tool that was supposed to prevent credential theft gets used by the attackers to do precisely that.

You may argue that no software is bug free and any software can get hacked and that is no reason not to use security tools. I agree. But the risk that these tools mitigate better be much greater than the inherent risks coming with the tools themselves. In a lot of cases, that equation doesn’t check out. In a lot of cases, no one even does that sort of analysis, because the job is not to actually minimize the security risk but to convince the third party stakeholders that the company is doing so. And if the company is paying six or seven figures for the cause, that surely must be sufficient, isn’t it?

But it’s not just about using paid tools as shortcuts. Lazy compliance can lead to other contradictions. Here’s my favorite quote from last year (the person asked to remain anonymous for obvious reasons, but it’s not some obscure tiny company):

“So a ‘senior IT security engineer’ in my company wants me to email credentials for an internal system to some external penetration testers.
\nI confirmed with him via chat and even got on a phone call with him and as far as I can tell it's a genuine request but I can't tell if I'm being trolled right now or not.”

So if your security team’s job is not security but simply compliance, then whose job is it to make sure your company’s software systems are secure? The cynical view would be - it’s no one’s job. Businesses routinely ignore security in favor of compliance, meaning security isn’t valuable enough to make it anyone’s job. But considering that the global number of successful attacks increases year over year, and the insurance companies are getting more assertive on what sort of losses they cover and when, ignoring actual security is not sustainable in the long run. So whose job is it? I’ll let you answer this one for yourself.

", "url": "https://authress.io/knowledge-base/articles/2025/01/20/your-security-teams-job-is-not-security", "title": "Your security team’s job is not security", "summary": "Sometimes your CISO and security team impose annoying rules on the development team. Sometimes these rules may seem at odds with security. There is a reason why that happens.", "date_modified": "2025-01-20T10:00:00.000Z", "author": { "name": "Dorota Parad", "url": "https://dorotaparad.ch" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/magic-links-passwordless-login", "content_html": "

I’ve seen a lot more apps ask for just an email for sign up and sign in recently. In some ways this is an awesome alternative to using username + passwords. After spending some time chatting with the development teams, we’ve learned they see it as a low barrier to getting users in, especially since passwords are a bane for users.

Their auth systems usually consist of authN and authZ (for more information check out the article on the difference between these). Their authN is just enter your email. Sweet and quick. Then they’ll use the email or dynamically generated userId as their identifier in their authZ access control.

It’s the quick and easy solution​

Seems relatively easy to set up and implement, perhaps a quick database table with userId associations as well as something to generate those magic links to send out.

I like it because I also hate passwords. When you want to invite other users, which is notoriously difficult, you have to use an email or at least share a link. It’s also pretty nice because they can reuse the flow for sign in. Also it frees up the team to not have to worry about a messy password reset flow, one that probably involves copying and entering codes.

Additionally, since the bar is always moving on how to best store passwords, it avoids that nasty business as well. You may see some suggestions on using bcrypt with crypto random salts, that’s actually not even the best solution out there. You may have thought since the brand names use bcrypt, they must be the best. But it turns out that isn’t the case. So unless you follow the latest trends or even using a SaaS solution, you could be in trouble.

So there seems to be a lot going for a passwordless type of solution. Probably the most notable benefit being built-in email verification. That is, no one can fake your email/username when using a site, because it is coupled to another login process.

So why not use it then?​

That’s about where the benefits of using email links as login stops being so great though. There are a fair bit of edge cases that come up quite frequently which cause solutions like this to fall apart, when they are the only way to log in.

  1. How will you handle multiple logins? Perhaps you are hoping that the email provider let’s users add the email label (adding a + to any email), but how many users know about that sort of functionality?
  2. How will you handle changing email addresses? This is one of those cases that will happen, but you hope you’ll never have to deal with it. When a user wants to change their email address, then they should be able to change it. So now you need another complex flow to make that happen. When a user is coupled to a userId generated separately by the user, that’s less of a problem.
  3. You also open yourself up to common attack vectors that hackers use, and that is account impersonation. By having only a single string to represent each user, an attacker can continually guess email addresses until they have a hit. From there they can choose who they want to target. When you need both an email/username and a password, you can block requests where either is incorrect.
  4. Another common problem is insecure email addresses. What happens when your email provider has a security breach. Account logins now are a huge issue, because each one of those is the login itself. Or what if a user is connected to a vulnerable hotspot and accesses their email in an unsecured way, a link as a login offers its own insecurities. This is the main reason why Oauth2.1 specifically forbids the use of tokens in query strings. Tokens in URLs are not safe.
  5. Replay is a huge problem, what happens for a user with multiple tabs or closing and opening, or miss-clicking. The links will have to be one-time use only, but that means, if the user “logs out” and then returns they’ll need to go through the email flow again.
  6. You can’t predict every user’s email provider, that means that the user will have to leave your app to log in. They’ll have to access their email provider and click there. If they have tons of emails or in the middle of doing something else it’s easy to have them get lost.
  7. Attackers can use your service to send login emails to many different users. You start paying for the resources. If you are using SMS to send those links, then an attacker will cause you to fraudulently send messages to every number they put in.

Fundamentally, you will have to deal with these issues:

  1. Your links will expire and users will still click them, the resultant experience will be bad for them.
  2. Attackers will attempt to abuse your magic links to do email validation.
  3. Attackers will attempt to abuse your magic links for beg-bounties.
  4. Attackers will attempt to abuse your magic links for fun.
  5. OTP via email is phishable. Email links are also phishable if those links enable cross device log in flows. If the link must be resolved on the same browser/tab that the login was started in, these are less-phishable.
  6. Email scanners that check for spam and malicious emails automatically click links in your emails. This will include your one time login links. Thy can and will definitely interfere here.
  7. The entire flow is significantly worse UX than every other log in strategy (sans passwords), the fact is, users need to separately go to their email and there is no way to deep link to that email. Now you can optimize a little bit by guessing their email provider and hot linking to it to open that email, but this only works for quite the pittance of email providers. Using this Email-Linker significantly improves the UX, but still is limited.
  8. Network connections are not perfectly reliable. Which means your users will attempt to use a valid code via the link, the connection will be dropped mid exchange causing the code to be marked as used, but the user won't have received a valid access token/auth cookie/etc from you.
  9. Email access flows are susceptible to MITM attacks unless the originating application uses some sort of PKCE to ensure closed loop auth to the originated trusted application. However this directly breaks multi-device flows, and users with multi-device expectations will click on the these links and have a bad time.
  10. These emails will eventually end up in spam for someone, so you'll still need to support a first class auth strategy.

The next step​

Actually logging in via the link side-steps the whole benefit that federated login providers offer, think Log in with Google, Office 365, Twitter, etc…

These providers offer something that other identity providers don’t, and that is MFA through that email login. Taking another step forward from just email, these offer 2FA after password, depending on email links or hardware token validation. And some go even further with security precautions like deviceId checking, IP remembrance, and session tracking, which email can’t use.

But there are many of these, and aggregating the logins across federated login providers is challenging. Forgetting which login strategy you picked and making duplicate accounts is a problem. Or merging two accounts both using the same Google login. To solve this identity aggregation is necessary.

Going further​

It’s easy to start with magic links but the challenges start to break down pretty quickly. Additional issues with user session tracking and multiple devices just add to the complexity. And these aren’t something that you want to deal with. That’s even before you have to start worrying about the attack surfaces for your product application.

Start with an Identity Aggregator and solve the complexities with login from the beginning. They make it simple to integrate and provide your application and users the protections they need.

", "url": "https://authress.io/knowledge-base/articles/magic-links-passwordless-login", "title": "Magic links and Passwordless login", "summary": "Making user signup and login easy is critical to having a successful app. Magic links and passwordless login helps, however it can create many problems longer term, and not ones that have easy solutions.", "date_modified": "2025-01-15T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/01/15/google-oauth-workspace-vulnerability", "content_html": "
info

This article is a rebuttal to Truffle Security's post on Millions of Accounts Vulnerable due to Google's OAuth Flaw. (Alt link) Even more ridiculous might be that their post got picked up by no small number of news outlets that all should be ashamed of themselves, far too many to actually link in this post.

Are millions of accounts vulnerable due to Google's OAuth Flaw?

In a true Betteridge's law of headlines fashion, the answer is a resounding No. Which explains why Google ignored this vulnerability in the first place:

\"Google

The TL;DR of the source article claims that due to the nature of how Google OAuth works, \"Millions of Americans' data and accounts remain vulnerable\". It relies on the nature of Domain Ownership.

The Claim​

Google’s OAuth login doesn’t protect against someone purchasing a failed startup’s domain and using it to re-create email accounts for former employees.

Domains are the root of trust* for many businesses. At Authress we rely on authress.io to establish trust with our customers, just as at your business you rely on your domains for your customers. This is \"Root of Trust\" with an asterisk because in reality the root of trust lies with the domain authority, the domain registrar, and the issuer of your TLS certificates for HTTPS encryption. But that is outside of the scope of this article.

The claim in the original article is that it is OAuth and specifically Google's OAuth that is at fault and nothing else. And that somehow domain ownership is linked to the exposure of customer data.

Background​

Gaining access to your trusted domain is one way in which attackers attempt to circumvent your security strategy and compromise your users. If malicious attackers can utilize your domain to trick your users, then they can impersonate your business and steal their personal information, bank accounts, and credit card numbers. This is the basis for why phishing is popular today. As a matter of fact phishing is so popular because compromising a domain is incredibly hard, and is usually executed through a DNS Poising attack. The strategy behind phishing is to purchase alternative domains that look and feel like the valid domain as the next best thing (Typosquatting). These facsimiles exist for exactly that reason.

Besides using separate domains attackers will often also attempt Subdomain takeovers which is a mesh between domain compromise and using an alternative domain.

However, in this case, attackers cleverly will attempt to use your existing corporate domain after you believe you are done with it. The expected flow involving Google Workspace's OAuth looks something like this:

  1. You buy a domain for your company, let's call it yourcompany.com.
  2. Sign up for an Employee Identity Solution (IdP) that provides OAuth, there are actually many solutions here, Google Workspace, Okta, Microsoft Entra ID, Ping Identity
  3. Then your employees use that identity solution to sign into to a third party product such as Stripe, AWS, PostHog, etc...
  4. Lastly you give critical data to that product, business sensitive information, like your pets' birthdays.
  5. That third party application saves that data because they like data very much.

\"Corporate

Identity​

When you log into your favorite third party application, there needs to be an identifier sent from the Employee Identity Solution to that third party. The Third Party trusts your chosen identity solution as well as that identifier. Here is an example token generated by Google Workspace:

Google Workspace Identity Token
{
      \"iss\": \"https://accounts.google.com\",
      \"sub\": \"210169484474386\",
      \"iat\": \"1736946817\",
      \"exp\": \"1736996817\",

      \"email\": \"warren@yourcompany.com\",
      \"hd\": \"yourcompany.com\",
      \"name\" : \"Warren Parad\",
      \"picture\": \"https://lh3.googleusercontent.com/a/ACg8ocImaIv0P4B8ZNyM4bgIkeUK4TSIXdPhfcLHugB4Arzq=s96-c\",
      \"given_name\": \"Warren\",
      \"family_name\": \"Parad\",
      \"locale\": \"en\"
}

The identifier in the token is the sub claim with the value 210169484474386. This is my User ID (Note: this is not actually my user ID, feel free to do with it as you wish, but I made it up for the purposes of this post.)

Your third party application uses this sub property to uniquely identify you, and then authorize you to your company's sensitive cat photos.

The Vulnerability​

Now, imagine that you close your Google Workspace account, because your company goes bankrupt (This frequently happens because as much as we want to believe companies are successful through hard work, the truth is that it is actually luck). Along with your Google Workspace Account will likely be your expired domain yourcompany.com, unless you have some secret prayers that one day you will be able to sell it instead of expiring worthless. Let's assume that yourcompany.com domain is now available for anyone to purchase. By purchasing that domain, an attacker can create a new Google Workspace account, in hopes to gain access to those exact same third parties you had used for your business.

This actually isn't even the first time something like this has been attempted, and frequently it works due to hard-coded solutions in many applications. In a cruel twist of fate, here is a great example of being able to compromise the attackers themselves because they had a used a application which relied on expired trusted malicious domains.

This actually doesn't happen with Google OAuth. When you close the google workspace account, the User ID with the value 210169484474386, ceases to exist. This is what Google is confirming by closing the original bug report. An attacker recreating the Google Workspace account is unable to generate the same sub again. So that even if an attacker attempted to create a new Google Workspace from the expired and unclaimed domain yourcompany.com, the sub would be different and your third party application would reject access.

What's the problem?​

The issue is some third party applications decided not to use the sub claim. The author of the Truffle Security post suggests that this is due to some bug in the Google OAuth implementation, but the reality is OAuth has nothing to do with this problem. The failure to use the sub claim stems from this shiny property in the identity token called email. In the original token above you can see the users email there warren@yourcompany.com.

A third party that utilizes this email address to uniquely identify users means that they are allowing malicious attackers who compromise employee identity providers through expired domains to take over your account. There are lots of reasons they do this, but primarily it is because they like the way the @ looks in their database.

That means this is actually a vulnerability on the third party application side. Any third party application that allows users to log in with just an email are inherently creating a vulnerability in their own platform and setting themselves up to expose their (ex-)users data.

Vulnerability review​

So, actually this has nothing to do with Google Workspace at all. And an attacker can actually use any email provider to perpetrate this attack:

  1. Buy an expired domain and register your domain in a new email provider
  2. ...
  3. Profit

Although in this case the ... is simply: Attempt a password reset or magic-link authentication for that third party application. In a similar attack a vulnerability was utilized by attackers through an email support system.

1. The real vulnerability​

This shows us that OAuth and Google Workspace aren't actually the source of the issue here, it's the third party application. I've frequently condemned Magic-Link based Authentication, and while there are some areas where it unfortunately still provides value, it isn't worth it if you care about security. The fact that the email is provided by Google is just unfortunate. Emails are helpful for identify where to send messages to users who want emails, but it should never be used anywhere related to security.

2. Dismantling the solution​

The original article suggests that adding yet two more additional claims/properties to the User Identity Token, will solve the problem. One claim isn't good enough, let's have three!

Given that the problem is that third party applications are ignoring the already existing sub claim. I find it to be quite the naïve suggestion. No amount of additional claims will prevent third parties for incorrectly substituting in their beliefs where actual security is necessary. This is just an unfortunate truth. We see this every day and it is one of the reasons we built Authress in the first place. The defaults that exist in SDKs, frameworks, protocols, and standards, are just not enough for people to do the right thing, explicit investment had to be made in prevention of doing the wrong thing.

My favorite line in the whole article is:

\"The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week\".

It's an attributed quote, the author didn't say where it came from, and the good people of Hacker News, were very kind about dismantling this particular ridiculous statement. Google Identity is very clear about expectations regarding the sub claim. And a huge thank you to @loginatnine who shared some of their experience with this:

If the sub changes, it's because it's not necessarily the same person so have a flow ready for that. It could be an employee left and came back, a domain change, an IT error that lead to a reprovisioning of the user, etc.

I've been working with an app that uses Google to login for the past 10 years, and I've had problems with sub changing when these situations happened : - Domain change - Company being bought by another one and being integrated in their Google Workspace - Employee leaving and coming back

To us, it's very very far from the quoted 0.04% which is to me very high. I had to deal with it 5-6 times in the past 10 years but of course that number will vary depending on the usage of your app and I'm not gonna venture and put a percentage on it.

3. Third Party Application responsibility​

The last part of the problem is that the author in the original article claims

What can Downstream Providers do to mitigate this? At the time of writing, there is no fix.

Which just isn't true. Third party applications that allow email based authentication, must delete user data after account deactivation. Once you stop paying for a third party application, that data must be deleted and never exposed again unless you resume access and the third party verifies identity. I prefer taking guidance from the NIST 800-63A.

As a user you too, can do something to. If you have sensitive data, you could decide not to use any third party applications, unless of course you actually pay for it and ensure that you delete your account before your company stops using that application. If you give someone your data, they have it, assume the worst. We can and should put more responsibility onto these third party application services who are utilizing unsafe email addresses and often SMS numbers of authentication. As long as you treat email auth as a valid solution, everyone will forever be just as culpable as third parties who rely on it. Use OAuth and SAML for your business authentication and make sure to provide sufficient secure options to the users of the products and services you build.

Consumer exposure​

The original article also seems to conflate risks to consumers directly. There is nothing about this vulnerability that directly affects consumers. Sure there are impacts to consumers regarding data privacy, but the vulnerability discussed in this article doesn't include them.

That's because as a consumer when you use an application, that application stores data in their primary databases. When the company that manages that application fails, both their databases and their bank accounts are empty. You don't have to worry about that data. But you do have to worry about who they gave your data to. You have to worry about that irrespective of the company, or its state. Many companies out there have started to be investigating because of just that. This is the whole premise of the Facebook's Cambridge Analytica scandal. Facebook gave user personal data to Cambridge Analytica when they should not have access to it. Facebook didn't even need to be bankrupt for there to be a problem.

The core of the issue isn't the data you have given to the company, the problem is data the they have shared to others. But no amount of praying or technological solutions is going to fix that. The problems proposed in this article regarding the domain vulnerability in question are related to the data given to the third party applications secured with by the company's corporate domain. The data that is most vulnerable in these circumstances is the business-to-business relationships. Billing information, strategic partnerships, invoices, business strategies, these are at risk.

For example, at Authress we use Stripe, sometimes. In stripe we have customer account information, including customer emails for sending invoices. If you are using Stripe or another payment provider, then chances are you too are storing some sort of customer data in Stripe. If your company goes bankrupt, and attacker uses the domain vulnerability to do a password reset on your Stripe account, they will now have access to your old company's customer invoice and email data. You probably don't care, but you should.

Conclusion​

So I think we can say definitely, no there aren't millions of people at risk with this vulnerability. Sure your data is at risk, it always had been at risk, it always will be at risk, but Google's OAuth implementation, while problematic, honestly doesn't change anything at all. You can continue to file your data deletion requests with your third party application providers when you don't think they are doing too well. But if they aren't doing that well, I sincerely doubt they are deleting your data, let alone deleting your data from their third party providers. I don't know what will become of the original published articles or Google's response, but I had felt strongly to first educate regarding the problem rather than lambast Google Workspace over their responses. The claim by the original author that millions of accounts vulnerable due to Google's OAuth Flaw is just irresponsible.

If this article provides interesting insight for you, and something worth discussing more, please join our community.

", "url": "https://authress.io/knowledge-base/articles/2025/01/15/google-oauth-workspace-vulnerability", "title": "Are millions of accounts vulnerable due to Google's OAuth Flaw?", "summary": "Are millions of accounts vulnerable due to Google's OAuth Flaw? It is certainly possible, but not the one that Truffle Security would have you believe.", "date_modified": "2025-01-15T00:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2025/01/03/bliss-security-framework", "content_html": "

When it comes to cybersecurity approaches, there are two main camps. One is rooted in the traditional defense-focused mindset, very common among security professionals. We have to make our internal assets impenetrable to the attackers! We have to close every gap they can sneak through, eliminate all vulnerabilities! This mindset leads to a heavy handed process that is poorly understood by engineering teams. Security becomes something that’s a concern for the security team only. It’s us vs them. It’s not helped by the fact that the software engineers themselves are often seen as a potential threat, with corresponding, very obvious protections to counter it.

The second camp recognizes that this traditional approach to security is costly. It slows everything down and adds extra steps no one really understands. And for what? Most businesses can get away without thinking about security until their first serious incident, which may or may not happen and if it does happen, maybe it won’t happen this year. It’s a risk that a lot of companies are willing to take. But then those pesky regulators demand that businesses protect their users’ data and insurance companies offer enticing discounts for those who implement certain traditional cybersecurity practices. What ends up happening is some flavor of security theater where we do things that sure look good on paper, but that don’t actually make us more secure. Security becomes a set of checkboxes to file and forget.

Neither camp is right. With the sheer amount of attacks carried out on everything that’s connected to the internet and with the amount of resources poured into cyber crime by certain nation states, every software system will get compromised eventually. Ignoring this may pay off in the short term, and keeps paying off when the alternative is to require a team of security specialists and practices that slow every engineer down. But what if there was a third way, a pragmatic way, that actually improves your security while being easy to grasp by every engineer? And what if that way was so cheap to implement that it changes your traditional risk equation? What if you could make your security BLISS?

This may be disappointing, but it’s not the latest, coolest product nor a service you can pay for and keep ignoring the topic. BLISS is a framework that helps you think about security in a healthy, pragmatic way so that you don’t have to spend a whooping 13% of your IT budget on security with mixed results. It stands for Bulkheads, Levels, Impact, Simplicity, and pit of Success. Let’s dive into details of what this means.

Bulkheads - limiting the blast radius​

Bulkheads are a feature on all submarines (well, maybe not Titan) — where in case a section of a ship has some catastrophic failure, it gets sealed so that the problem doesn’t spread to other sections. We can use the same principle in software security — if a part of our system gets compromised, we should be able to seal that part off, or remove it so that the problem is contained. Think of it as limiting the blast radius.

The ability to isolate, turn off, and swap or redeploy a component of a system in the event of an attack could make a difference between a total hard down of production and only a partial loss of functionality. It is also something that costs almost nothing to introduce, as long as it’s been considered when designing the system. Good security starts with architecture and this is a great example. The extent to which this principle can be applied and its effectiveness are going to be constrained by your architecture choices.

In the case of microservice architectures, where each service is deployed independently and services are communicating solely through REST APIs, the service boundaries are our natural bulkheads. The risk of the whole system going down when one microservice gets compromised is very small and the ability to redeploy the service quickly further minimizes the risk. In the case of a monolith, we don’t have this capability available, so we need to get more creative with placing our bulkheads. Things like deploying separate instances for different customers or using containers would be good examples.

Regardless of architecture choices, it’s worth adding isolation to the infrastructure to limit the impact when some part of it inevitably gets compromised. Consider having separate git repositories for different projects, and separate cloud accounts for deploying different environments and modules, each with separate access controls only authorizing people involved in respective work. That sort of separation can be added without great effort even for legacy software.

It’s important to keep in mind that adding bulkheads is just one tool out of many for improving security. As such, it makes no sense to overdo it. If your software is already serverless, meaning each function runs in isolation by definition, then deploying a separate instance for each customer would be an overkill, at least for the purpose of limiting the blast radius of security incidents. If you have a big-ball-of-mud-style monolith where everything is stored in a single git repository and has to be deployed all together, trying to add bulkheads may not be the most prudent first step.

Levels - protection proportionate to the risk​

Protecting data and operations necessarily means adding little obstacles — extra login factors or additional steps to perform to either prove who the user is or reduce the exposure. However, not all data is equally sensitive and not all operations equally critical. It makes no sense to have the same level of protection applied to all of them because that ends up in a compromise that is either too strict for regular, daily operations or too lax for the sensitive ones. Instead, it’s more practical to introduce different levels of protection proportionate to the security risk involved.

Having multiple levels in protection strategy enables fine tuning the approach to each situation. That way, people doing the work and operating the systems don’t get overburdened with security where it’s not needed, and are more likely to pay attention in areas where targeted security measures indicate heightened risk. Last thing we want is people getting desensitized to security and becoming lax or complacent. Most successful attacks happen due to human error after all.

When defining security measures, it makes sense to look at your whole scope holistically as well as from the user or operator perspective. Consider groups of actions and the data involved - are there sets that get performed frequently together? What is the sensitivity of the underlying data? Are there distinct steps that can be treated separately? What is the security risk associated with each? Once you have that granular view, consider:

It may be tempting to put a single, reinforced gate in front of everything and call it a day, resist that temptation. If an operation consists of multiple steps, it’s rare that the first step already requires the strictest protection, so consider layering different means throughout the process.

Let’s use a developer machine as an example, treating the whole machine as the most precious thing deserving ultimate protection would be a mistake. Instead, it makes sense to protect individual data types and operations. After all, accessing cat memes on the internet is not the same as accessing your company’s compensation data even though they both may use the browser and https as protocol. One of them likely happens every day and should be possible to perform freely. The other one happens rarely and should require strict identity checks and access control behind the scenes. Likewise, viewing source code is not the same as committing code — which is not the same as merging the commits — which is not the same as deploying those commits to production. Each of these steps can have its own protection, with the strictness and corresponding obstacles increasing progressively at each step.

Notice how the step or action that has the highest risk in these examples - accessing salaries and deploying the code - is also one that happens less frequently than the other ones? It’s not a coincidence. In normal business operations, the vast majority of things we do aren’t inherently risky, so trying to protect all of them is a waste.

Impact - minimizing the consequences of an incident​

The traditional mindset among security professionals is to protect, with a heavy focus on preventing the attacks. This was a very good approach in the early days of the internet when we still had a lot of low hanging fruit in securing our software and infrastructure. Nowadays, a lot of the measures intended to reduce the likelihood of a breach are built into the technology we’re using and are so ubiquitous that we’re not even paying attention to them. Cloud technologies, Single Sign-On (SSO), and deployment automation are just some of the examples. Attempting to further strengthen prevention is providing diminishing returns. It makes more sense to focus on reducing the impact of a successful attack when it inevitably happens.

Think of what happens when something gets compromised. Are there bulkheads in place that could contain the incident to a relatively small scope? What can the attacker do with their access - are there more levels of protection further down the line before they get to the juicy bits? What’s the worst that can happen? How much will it cost the company or your team, including the cleanup? What can be done to further reduce this cost? There are a lot of techniques that help limit the impact of a security incident. Other than the above mentioned bulkheads and levels, there is encryption — in transit and at rest — granular permissions, and restricting who and when can grant other people access. Let’s not ignore measures typically used to ensure high reliability, such as backups, redundancies, and rollback strategies - they also serve as means to minimize the damage done in an attack.

Let’s take an example of a common threat: a malicious actor getting their hands on one of our engineer’s cloud account credentials. What’s the worst that can happen here? If we applied the principle of bulkheads, then those credentials are limited in scope to a single cloud account, which may or may not host our production environment. Even if it does, if we utilize modern software development practices, the access would be limited to read-only and innocuous configuration changes, since resource creation and deletion would be automated and only permitted as part of CI/CD pipeline. Let’s say that account includes access to a database containing user data if we applied proper encryption, the data is effectively useless to the attacker.

Another common threat that could have business-ending consequences is a ransomware attack. If we make full use of bulkheads combined with reliability techniques, then our production environment is distributed, meaning only part of our system would be affected. What’s more, we would be able to roll back the deployment that included the malware (if that was the attack vector), restore a database backup in a matter of minutes, or quickly switch to an unaffected alternative region/server.

Trying to simply prevent the incidents in the above examples would be a prohibitively costly endeavor, involving multiple remediations to account for all the different attack vectors. Limiting the impact, on the other hand, is a holistic solution that doesn’t take as much effort or cost, and often comes with additional benefits of increased robustness. This isn’t to say it’s ok to completely ignore preventing the attacks. There may still be places where investing a little in prevention will return outsized benefits. In most cases though, looking at impact first is more pragmatic.

Simplicity - simpler means more secure​

The more complex something is, it is expected that the more errors or accidents will happen. If a process is difficult to follow, people find workarounds. More tools added to the tool stack mean more vulnerabilities to patch. A complicated security strategy filled with technical jargon is going to be difficult to execute by anyone outside the security team. If a security practice causes friction for the users, it’s less likely to be correctly applied. As a rule of thumb — the simpler something is, the easier to secure.

Biometric login is going to be more secure than forcing the users to type in their passwords, because there’s nothing to write down on a sticky note, nothing to reset, and stealing biometric credentials is difficult to execute.\nHaving to log in once a day is going to be more secure than forcing the user to log in every hour, because it creates fewer points in which the credentials can be sniffed or hijacked.\nHaving a basic CI/CD pipeline is going to be more secure than adding every possible vulnerability scanner, end to end test, or multiple manual gates, because of fewer opportunities for human errors that could be exploited and fewer temptations to just log onto the server and make “just this little fix” directly.

Notice there is some tension between the principle of simplicity and the other principles of the BLISS framework. Adding bulkheads means adding potential complexity, it means we have to deal with multiple parts instead of a single one. On the other hand, which system is more complex - one big thing that does everything, or multiple small ones that do one thing each? The answer depends on the situation. Similarly, multiple levels of protection make things more complicated, because we have multiple strategies to apply in different situations. On the other hand, having those levels helps keeping things simple for, hopefully, the majority of day to day operations. Simple doesn’t mean simplistic. The best way to think about it is in terms of tradeoffs - when there’s a choice, the simpler option is better.

Pit of Success - security by default​

A pit of success is when it’s so easy to do the right, the secure thing, that it happens almost by accident, that people fall into a pit of success. It should be obvious what the correct thing is, the default option should be the right one. Or to put it differently, doing the wrong thing should be so difficult that people have to go out of their way to make it happen. This means that the obstacles should be reserved for the attackers or operations that are very risky and should happen only rarely.

You can’t have security without considering the user experience. This includes developer experience, because software engineers have the power to bring the whole system down and introduce new security vulnerabilities. If maintaining security requires people to take extra steps or makes their daily jobs even a little harder, no one is going to do it. Good security should be invisible, built into the way a company operates.

A very good yet underrated way to create a pit of success in security is automation. It is rare that companies recognize all the things we do to make our code deployments smoother and more reliable as improving security. But continuous integration, infrastructure as code, utilizing serverless technologies - just to name a few, all make it difficult to make mistakes that would result in severe security incidents. Continuous integration means a robust, repeatable process to integrate new code into the codebase without breaking anything along the way, which means a malicious piece of code has to go through multiple testing and review steps to remain undetected - it can’t be included by accident. Infrastructure as code means that getting new code into production is fully automated and standardized, which means that tampering and potentially planting backdoors with production servers is something that requires a conscious effort. Going serverless means there is no server to install malware on, nothing for an engineer to log in and accidentally infecting the production machine. While it’s not enough to simply eliminate the ways in which engineers’ negligence could cause a problem, it goes a long way. Social engineering techniques are one of the most common attack vectors and people are likely to think twice when asked to do something that requires them to go out of their way.

If you want people to follow your security policies, make them easy to read and understand. If you want good password hygiene, don’t force arbitrary complexity requirements or password rotations. If you don’t want your test environments to be an open invitation to a breach, make them short-lived, created and torn down automatically, and scoped to a single pull request.

Wherever there is a potential for a data leak, credential theft or other security incident, consider: can this process be simplified? What is the impact in the worst case, what can be done to reduce it? Is this a reasonable level to introduce protection? Are there bulkheads in place that would limit the scope of this problem? This is all in contrast to a more typical, adversarial approach, where security teams focus on making it harder for people to do the wrong thing. Instead of putting all the effort towards creating obstacles for engineers and other employees preventing them from compromising the security, investing that effort in creating a pit of success results in better outcomes.

Summary​

To make security BLISS instead of a checkbox filled theater, use Bulkheads to introduce isolation that limits the blast radius of incidents, apply protection on multiple Levels proportionate to the level of risk, focus on reducing the Impact of security incidents when they inevitably happen, keep your tools and processes Simple, and create a pit of Success for your team members so that they do the right thing by default.

Will all of this help you pass your next certification audit? Not necessarily. But it’s not going to make it harder and, at the same time, it will make everyone’s lives easier. This is how you create a foundation for security by default and how you can stay secure without spending a fortune on fancy tools, extra headcount, or engineering productivity loss.

", "url": "https://authress.io/knowledge-base/articles/2025/01/03/bliss-security-framework", "title": "Forget the checkboxes, make your security BLISS", "summary": "BLISS is a framework that embodies a pragmatic approach to security. Traditionally, security is either all about defense and creating obstacles, or about ticking compliance checkboxes. It doesn't have to be this way.", "date_modified": "2025-01-03T10:00:00.000Z", "author": { "name": "Dorota Parad", "url": "https://dorotaparad.ch" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/11/19/saas-billing-infrastracture-and-usage-tracking", "content_html": "

It shouldn't be any surprise that we rely on DynamoDB over at Authress for a variety of things. For context, we have a any number of databases of different types, and DynamoDB is just one of the ones we use. As we operate at significant scale, that means we are privileged with the opportunity to test the limits of our tools, of which DynamoDB is one.

To get to the scale we are at, with the reliability we need for our customers, idempotency is table stakes.

Failures will happen, and they will happen in every component. That means DynamoDB will fail and that means retries. We need some way to have hundreds of thousands RPS be idempotent and be retried without requiring serial processing. However, the pure RPS that needs to be idempotent does not need to be as high. That's because, while our service RPS is much higher, individual item write RPS is much lower. This might be in the range of only 100 RPS. The probability of the same item being updated more than 100 times per second is unlikely and out of scope for our service, and more importantly this article.

A good example of this is bank transactions. While some banks might have millions+ RPS, the number of finance transactions per account is usually low, and can often even be serialized. How many payments or withdrawals do you have per day? 1? 10 max? Actually for the purposes of bank accounts, what is even more important is serialization to ensure that the account does not overdraft. That is, a bank needs to prevent an account from withdrawing funds that it does not have.

I honestly hadn't expected this to be surprising, but apparently it isn't that obvious how to actually achieve this sort of idempotency in DynamoDB, so the rest of this article reviews our use case, and exactly what we do to achieve idempotent requests into DynamoDB without using DynamoDB transactions.

The setup​

Authress provides login and access control for the apps you write. Therefore it is infrastructure, and rather than charging by some ridiculous metric like Monthly Active Users, we charge by usage. You get what you pay for. However, this means we have the unique problem of actually trying to calculate exactly how many logins and authorization requests there were, so that we can bill for it.

And more importantly we need to display this data in an aesthetically pleasing way to our customers:

\"Authress

The DynamoDB use case​

Multiple writes are going to come into your table at the same time. Before we can even get to the actual use case with handling idempotency, I want to review all the different ways concurrent writes could be an issue for you.

\"Concurrent

Normally, each user would likely be writing to the same table, but different items. (For simplicity, I'm going to model this the DynamoDB requests using pseudo code and javascript, of course the examples could be in any language).

await writeToTable(Table1, { pk: 'user_001' }, dataOne);
await writeToTable(Table1, { pk: 'user_002' }, dataTwo);
await writeToTable(Table1, { pk: 'user_003' }, dataThree);

This is the simplest and works out of the box. Even in more complex scenarios DynamoDB works out of the box.

Concurrent writes to same item.

await writeToTable(Table1, { pk: 'user_001' }, dataOne);
await writeToTable(Table1, { pk: 'user_001' }, dataTwo);

If we did this, then the last write would win. The problem here is that we usually care about ordering these writes. Luckily DDB supports ordered writes. All we need to do is first fetch the data before writing, and use a condition key.

const currentData = await readFromTable({ pk: 'id' });
await writeToTable({ pk: 'id' }, newData, currentData.expectedLastUpdatedDateTime);

// Or in explicit DynamoDB Syntax:
await dynamoDB.put({
  Item: {
    data: newData,
    lastUpdated: now.toISO()
  },
  ConditionExpression: '#lastUpdated = :expectedLastUpdated',
  ExpressionAttributeNames: {
    '#lastUpdated': 'lastUpdated'
  },
  ExpressionAttributeValues: {
    ':expectedLastUpdated': currentData.lastUpdated
  }
});

With more than one process writing to the same table, we could end up with either:

In the case of the first one, the updates will be in serial so everything is fine. (And if this were something where it wouldn't be fine, then you would use a user supplied value of the expectedLastUpdated time instead of the one from the previous read.)

In the second case, the latter write will get a ConditionalCheckFailedException exception from DDB, and you can decide what to do. In many cases, the right thing to do is retry:

In other cases, you will want to throw the error back to the user.

Caveats​

We still haven't reached the goal of this article, but even at this point there are some concerns. For instance, you can't just read the DB and then perform another write. You absolutely need to consider, what the correct outcome should be. Sometimes the correct outcome is \"409 Conflict\" or \"412 Precondition Failed\" when the user needs to take action. Other times, these updates are orthogonal. For instance, let's say there are two properties A and B.

Request One updates A and Request Two updates B. In some scenarios, these might be independent properties, and so both can be persisted. In other scenarios, B should be calculated based on the current value of A, so a recalculation will work. However, if B's value should be determined by a process outside of the database, you want to throw the 4XX status code for the user to decide what to do.

The real scenario​

And that is good and all, but now it is time to discuss the real scenario. What happens when DDB encounters an issue? Or the connection to DDB fails? Or the electricity powering the virtual machine where your process is running fails and terminates the compute container. Was the data in the database actually updated?

With direct persistence, or PUT semantics, you can just rerun some code (because the end result is fixed) However, when you are updating the item properties you could run into a problem because the end result is deterministic based on the current values.

So, let's go back to the Authress example. One user completed 10 api calls to Authress and now we want to record this information in the database.

Our code looks like this:

const currentValue = await readFromTable({ pk: 'id' });
await writeToTable({ pk: 'id' },
  currentValue + 10,
  currentData.lastUpdated);

Notice the issue?

At a small scale, this isn't an issue because with concurrent execution we can throw and retry later. However, there are actually two problems here:

  1. With scale of say 10 concurrent requests, 1 will succeed, but the other 9 will fail the first time, then 8 will fail the second time (after the retry), and etc... We might be able to be clever with \"when\" we execute the retry, but when we are unlucky for N updates, it would take N2 requests to DDB, with `N2 - N` failing. That's pretty bad.
  2. We have the critical problem of tracking, in the case when there is a connection drop we have no idea if the + 10 was applied or not.

Idempotency​

The only solution is to tell DDB to only process the + 10 if it wasn't processed before. So we need code that looks like this:

await writeToTable({ pk: 'id' }, 10, updateId);

We can't rely on passing in the currentValue because that violates problem (1) work at scale and only apply once. And we need to have an updateId to handle problem (2).

But DynamoDB doesn't support idempotency out of the box, but it does support operations to make this happen.

We will store the updateId and validate it on writes:

await dynamoDB.update({
  UpdateExpression: 'SET #history =
    list_append(#history, :keyList)',
  ConditionExpression: 'NOT contains(#history, :key)',
  ExpressionAttributeNames: {
    '#history': 'history'
  },
  ExpressionAttributeValues: {
    ':key': idempotencyKey,
    ':keyList': [idempotencyKey] // DDB requires array
  },
});

Getting to scale​

This will work for any small scale. It handles idempotency for anything where the size of the item remains under 400KB. 400KB is the max size of a DDB item. In our case we get an update from our trigger, CloudFront, fairly often, but not too often. The trigger is an id with a hash value. ECloudFrontId/authress-account-id/2024-11-22/15.dfc31777. That's 56 characters, or ~56 bytes. Which means we could stick 400KB / 56 or 7142 updates before the item fills up. (Remember we are pushing every idempotencyKey into the #history List)

However, since we index our customer billing by date, we only need the hash in the table (or for us the hour and the hash, as CloudFront doesn't provide higher granularity than an hour). That means 15.dfc31777 or 11 bytes, which amounts to 36000 updates per day or 25 billing updates per minute. CloudFront doesn't have that level of granularity, so one item per day is more than sufficient for us.

If we needed more granularity, then we could split the billing item rows in DynamoDB by hour, and then only save the hash dfc31777 (8 characters) or 50k updates per hour (14 updates per second). Even CloudFront is smarter than to do that level of reporting, but that's a problem we would love to have to deal with.

Going even further​

Let's say however, that wasn't good enough for us or an item you have requires having the 100 RPS I talked about that the beginning of the article, and those updates all need to be idempotent and cannot use the lastUpdateDateTime to drive concurrency handling, then what can you do?

DynamoDB can still support this, but we have to be clever. Let's define a gauge.

A Gauge is an arbitrary limitation that helps us restrict the solution space and make the problem easier to solve. Then once we solve the problem, we will see that this is actually the general solution as well.

So our Gauge is going to be Idempotency keys only prevent duplicates in the last 5 minutes. (Curious why I picked that? It's because 5 minutes is the same ridiculously short time span that Message Deduplication IDs in SQS permits. If it is good enough for AWS, maybe it is good enough for us).

If we only need to save idempotency keys for the last 5 minutes, and we have 100 RPS, then that means we have to save 5 min * 60 seconds * 100 RPS = 30k keys.

Luckily that is below our limit, we calculated in the previous section (which came to be 50k keys), it makes you seriously wonder if DynamoDB is used with SQS deduplication...

In any case, how the heck are we going to store only the last 30k keys in the item. The answer is the magic REMOVE DynamoDB UpdateExpression Operation. DynamoDB supports REMOVING List items in a DynamoDB item List property. More details on the REMOVE operation if you are curious.

The AWS example shows this CLI command:

aws dynamodb update-item \\
    --table-name ProductCatalog \\
    --key '{\"Id\":{\"N\":\"789\"}}' \\
    --update-expression \"REMOVE RelatedItems[1], RelatedItems[2]\" \\
    --return-values ALL_NEW

Can we use that to help us here?

Yes!, Yes we can. Let's update our DDB update expression to add new idempotency keys to the end of the list, and remove the oldest one from the front:

await dynamoDB.update({
  UpdateExpression: 'SET #history =
    list_append(#history, :keyList)
    DELETE #history[0]',
  ConditionExpression: 'NOT contains(#history, :key)',
  ExpressionAttributeNames: {
    '#history': 'history'
  },
  ExpressionAttributeValues: {
    ':key': idempotencyKey,
    ':keyList': [idempotencyKey] // DDB requires array
  },
});

Now when ever we add a new idempotency key to the list we also remove one from the front of the list.

However this isn't the whole solution. It is also important to remember that if there are zero elements in the list, then this will cause a problem, also we need to get to this steady state. We need 30k values in our list already. So we can do two things:

  1. Update the ConditionExpression to include attribute_exists(#id), that will force a ConditionalCheckFailedException when it doesn't exist.
  2. Fallback on the item not existing to creating it with a initial empty #history of 30k blank values.
await dynamoDB.put({
  Item: {
    history: [...Array(30000)].map(() => 0)
      .concat(idempotencyKey)
  },
  ConditionExpression: 'attribute_not_exists(#id)',
  ExpressionAttributeNames: {
    '#id': 'id',
    '#history': 'history'
  }
});

And then from that moment on the list will always have every key from the last 5 minutes.

The Math​

One last thing I wanted to throw in here is a formula. That is the relationship between RPS, Idempotency Key Size (Ks), and Idempotency Key Lifetime (Kl).

Given we have a limit of 400KB for a DDB item, we will have:

KeyLifetime * KeySize * RPS <= 400KB

That is if you have 10 request every second, and you want to save the keys for 10 minutes, then your key sizes are limited to

KeySize <= 400KB / (6000) = 70-byte keys

However there is one more limitation here, the size of the keys must account for significantly greater than then the number of keys that will be used. For instance, if we were allowed only 3-byte keys (which happens if we have key lifetime of 1 hour and ~40 RPS), we will have a problem. While 128*3 (ascii 7-bit) is 2097152 and much larger than the number of total keys that would be saved in your KeyList (144000), performing an optimized hashing to get from the idempotency value the user provides to one of only 3 characters, is quite a challenge. Having less than 128 byte key is usually an issue. However, you might be able to optimize by using a SHA256 hash is 32 bytes long. Which means that with a SHA256 hash for idempotency keys, your `KeyLifetime RPSwould support12500` keys. (Or only 42 RPS for 5 minute key LifeTime)

K-Byte Idempotency Keys​

This graph shows the formula plotted for each of different K-Byte sizes, 32, 64, and 128, as well the maximum RPS * Lifetime you can have for each one:

K-byte key restrictions

Limitations​

This of course has some limitations:

(1) The idempotencyKey size is limited. The SQS messageDeduplicationId is limited to 128 bytes. Ours is going to be a bit smaller

(2) At an RPS of 100, that would require a DynamoDB item to allow a size of at least 4MB to support 128-byte keys. So clearly each SQS FIFO queue is not using a single DDB item to store this. However, for a smaller Deduplication ID or a lower RPS we can easily fit everything in. For instance, even with a 128-byte keys, a customer RPS of 10 or less can fit everything in there. DDB can actually return 1MB per query, so actually AWS had more room to design this feature (if AWS wanted to use DDB to solve this problem).

(3) Our actual RPS must be <= 100, it can't be 101, even in burst, or else there will be a problem. Or more specifically, there will be a problem if a request gets dropped and requests an idempotent retry.

(4) You need to do the analysis to know what your RPS is per item and what your users and customers expect. If you are directly exposing this mechanism to a process you don't control it might actually overwhelm your database table (for example your REST API). But if you are in full control of what is happening (eg background processing), and monitor the usage, this is the perfect solution.

(5) Don't try to get too smart with DDB operations unless you really know what you are doing. Coming up with strategies like this is not simple, obvious, or quick. At Authress we spent years discovering opportunities to improve the usage and performance of our databases.

", "url": "https://authress.io/knowledge-base/articles/2024/11/19/saas-billing-infrastracture-and-usage-tracking", "title": "SaaS billing infrastructure and usage tracking", "date_modified": "2024-11-19T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/11/19/security-or-convenience-why-not-both", "content_html": "

\"Security

Traditionally, security is all about creating obstacles and making it difficult to access data. This is at odds with our drive for a more smooth and faster development process. How can we keep the software we’re building secure without adding friction for engineers? Can security ever be something other than a costly nuisance?

In this talk, Dorota explains how to create a foundation for security by design, go over quick wins at the cross section of security and productivity that most companies overlook, and share my tips for building secure software without sacrificing productivity.

", "url": "https://authress.io/knowledge-base/articles/2024/11/19/security-or-convenience-why-not-both", "title": "Security or Convenience - Why Not Both?", "summary": "Traditionally, security is all about creating obstacles and making it difficult to access data. This is at odds with our drive for a more smooth and faster development process. How can we keep the software we’re building secure without adding friction for engineers? Can security ever be something other than a costly nuisance?", "date_modified": "2024-11-19T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/oauth-identity-login-trust-relationship", "content_html": "

If you've landed here, on this page, it's probably because you too have asked, why does Auth and Login have to be so complicated for everyone. Fundamentally, the problem comes down to trust.

Background​

I create a Stack Overflow account using my GitHub Identity via \"Log in with GitHub\"

Stack Overflow has the possibility to remove the GitHub login now.

If they do that I will no longer have access to my Stack Overflow account.

The problem comes down to \"how does a client application such Stack Overflow know who the logged in user is?\".

Some definitions:

Where does the trust come in?​

The short answer is that Stack Overflow is in control of this because they decide whom to trust. Stack Overflow decides which Identity Providers to trust, and how to identify users. Thus they decide which login mechanisms can be used.

If they delegate this to another provider, then that provider is on the hook for whom to trust. For instance Stack Overflow can delegate all of login to Google Login or directly to Authress.

Trust matters because without it a malicious identity provider could issue fake tokens that allow one user to impersonate another. The identity providers that Stack Overflow picks can decide to self issue tokens that represent their users and steal your data from Stack Overflow by pretending to be you. You might not be concerned if GitHub decided one day to delete some of your Stack Overflow posts or even remove your account. But what if you used GitHub to log into your bank.

You might say \"I will never use GitHub to log into my bank\", and that answer works for GitHub, but since Authress is an Identity Provider and our customers are Banks, there is a concern of trust here. However, unlike GitHub, when you implement Authress it comes with a terms of service and a contract.

This is also an issue with supporting Third party identity providers. One common problem is scaling up to enable SSO for your B2B customers. This is where you let them bring their own Identity Provider. By nature this Identity Provider is untrusted. You can't make the same trust claims about it as you might be for GitHub, Google, or Authress. However...You still need to trust it to verify identities that come from that provider. Some identity providers let you specify Third party identity providers, but unless they explicitly state that it's a solution for managing SSO in B2B contexts, take a step back before continuing. Just because you can, doesn't mean you should. And when it comes to adding identity providers, most cases you shouldn't. For in-depth details of which providers to select under different circumstances, check out the current Auth situation report.

In essence, this means only some identity providers can be trusted. As an application owner, you can't decide to arbitrarily trust any identity provider. Each identity provider you trust creates a new attack vector for your users' data. These are your First party identity providers. That subset of trusted identity providers is a problem that Stack Overflow cares about, and therefore has solved by selecting the Identity Providers they see as trustworthy. If they remove one from the list, it is because it was deemed untrustworthy.

There is no way around this.

So, how do we standardize?​

Because Stack Overflow is on the hook for handling whom they Trust, it follows that every application has this same problem. So, standardization requires standardization of trust. Is that even possible?

NO!

Fundamentally, this isn't, and we'll see why below. To potentially circumvent the problem of standardized trust, let's review the existing three solutions:

1. Identity Aggregators​

Authress and other identity aggregators allow for your application to select whom to trust. Upon trusting an identity provider, subsequently when the user logs in, your application will receive an opaque, but trusted identity token. This token is secured by Authress, we've spent a lot of work deciding how to trust every OAuth provider we support out of the box, and how to support all the rest that follow the OAuth2.1 specification.

One alternative solution is first-class browser support for identity management. Instead of your users logging into your application directly, they would log into their browser. Then the browser would allow your application to fetch these credentials from the browser assuming the user has approved this interaction. This is how Chrome supports Google identities. No other providers exist for this directly. Because, really this is just implementation detail. It doesn't matter how your application gets the user's identity, there is a user identity, and you trust some non-user relying party to sharing that identity with your application. In OAuth, these Identity Providers are actually known as the Authorization Server (AS).

Your application trusts the Identity Provider, which generates the user identity. However these identity providers don't just give out tokens to whomever asks for one. It would be nice if they did, because then your application could figure out the user dynamically without the user having to take extra steps. User goes to your application, and automatically you know who they are. This is exactly what you do for existing users using the session tokens generated by Authress, but this doesn't work for users that never logged in. And, in any case, there are privacy concerns here, so this is actually more problematic. In reality, this is a problem with ALL OAuth providers. And that is because they require client applications to register. Google Login refuses to generate tokens to your application without you going to Google, configuring your OAuth credentials, and then setting up your Identity Aggregator of choice, all before the user logs in the first time. They have to approve your application. (The reason for this is a bit complicated, but it boils down to the fact that some malicious actors ruined it for everyone by pretending to be Google Drive.)

It's worth mentioning that Dynamic registration is a partial solution. Dynamic Registration is a strategy that allows your application to ask Google for the user's token at runtime, without preregistration. That means you would not need to follow a complicated guide. This is really cool and could help reduce the burden here, but actually Google does not support dynamic registration. The fact is, many OAuth providers, which clients and users actually use, don't provide dynamic registration!

2. Web3 (Crypto)​

So there's no trust-based solution for OAuth without dynamic registration as long as we have an Authorization Server or need an Identity Provider. The second solution attempts to do away with the identity providers altogether. For this we need to allow users to become their own Identity Provider. And there is actually an already available standard for this, known as Decentralized Identifiers DIDs.

However, this doesn't necessarily solve the problem either. And that's because your application still needs to decide which implementation to trust. Accepting DID tokens from anywhere is fine, as long as you implement the right logic, and the user's provider actually follows a DID implementation standard. The problem is that many Web3 implementations don't actually support DID, but you could of course limit yourself to ones that did. The standard exists, we can't force Stack Overflow to use it and neither can anyone force your app to trust DID based tokens. That still means at some point you could decide to stop supporting some subset of tokens generated. Even Web3 clients don't trust every provider. Often they only enable this mechanism and limit it to one trusted crypto, like BTC or ETH, or whichever one they are trying to sell to you. Most Web2.0 applications don't offer logging in with Web3, because they want your email address. Logging in with a DID solution means you don't know who the user is other than the fact that they hold some private key. And most applications still want to email their users as if that would solve the global climate crisis.

Since you are asking for an email, and an unwilling to use an alternative, that means that arguably email address is the solution to this problem, but let's quickly see solution #3.

3. WebAuthn​

The last exciting existing solution is WebAuthn. WebAuthn provides a similar implementation to the Web3 consistent Standard for login. However, again Stack Overflow and your application have to decide to trust the implementation.

WebAuthn and really FIDO2 has tried to make it as easy as possible for client applications to accept the one Standard, however as we've seen that there are still problems such as the improper usage of the ResidentKey implementation for hardware tokens. Even without the problems, most application portals today do not support WebAuthn--even though some identity aggregators do support it, such as Authress and identity providers such as GitHub and Google. Again it's up to your application to decide if you want to support it.

Solution: Email?​

Arguably, email is the solution and has always been the solution. The problem is E-Mail is E-Mail. The standard for email itself is old and bad, and the providers for E-Mail are also old and bad.

With an email address you can:

There's still a problem associating the email across login mechanisms, logging in with GitHub via my email is not the same as logging in via Google. Those are different identities unless your application trusts both of them to have verified your email address. In other words, as an application do you trust GitHub when it gives you a user's email? How do you know that user's email is what GitHub thinks it is, or even did GitHub verify that email address before giving it to you. Thankfully, we know GitHub did, but others definitely didn't. That means you can't just send emails to any email address any identity provider gives you, unless you trust that they did their job.

And in any case, that means we are back to the trust problem.

One way around this would be browsers to implement identities that are tied to email addresses and then provide those to the client application to let the user pick which one to use.

\"\"Credit xkcd

Conclusion​

As we've seen up to this point, as a user, we really have to trust the application with our data. Not just with the data we give them, but that they delegate to the right identity provider and in the right way. Because fundamentally, we can't take our identity with us. With some of the provider implementations listed above we can certainly try, and some users do that with extensions, a special browser, a special OS, or a hardware device to allow them to move from one browser to the next one.

Obviously, one way around this would be legal mandates on client applications, we could require applications implement some standards, but on the web, voting happens my majority, and that's if we are lucky. Thankfully to counteract this, GDPR and the like require you as a user to have the ability to take your data with you. You can take your data with you at any point, so it doesn't matter if you can't log in, you can still have those precious binary ones and zeros.

However, none of these are real solutions. Fundamentally, the application decides whom to trust, and we can't make an application provider trust other things, just because as a user we want them to. Likewise, if Stack Overflow delete our data, and there is very little we can do about it, irrelevant of what many \"data protection / privacy advocates\" believe. Stack Overflow owns it, maybe not legally, but factually. And they could at any moment, just like InfluxDB did.

✶ ✶ ✶

Figuring out which identity providers to trust is hard, but we've already done the work with Authress Authentication, which optimizes for SSO security for your customers.

", "url": "https://authress.io/knowledge-base/articles/oauth-identity-login-trust-relationship", "title": "OAuth Login should be standardized and this is why it cannot be", "summary": "The problem of trust relationships in OAuth, how clients hold all the power, and why login/user identity should be standardized.", "date_modified": "2024-10-27T00:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/09/04/aws-ensuring-reliability-of-authress", "content_html": "

\"How

Authress has an expected 99.999% SLA. There can be many requirements for uptimes--regulatory needs, core dependencies, or your service offers life-saving responsibilities. In these circumstances, it's critical to never let your service go down. Here, we'll review how we’ve architected, built, and maintained a solution with such a high SLA, the frameworks and strategies we’ve established, and what it takes to keep up that commitment. Presented at AWS Summit Zurich 2024.

Transcript​

Okay. Hello and welcome. I'm Warren Parad, and I'm the CTO at Authress. And today, I'm going be talking about how we built one of the most reliable Auth solutions available. When I say reliability, what I mean is that we offer a five-nines SLA.

That's 99.999% of the time, what our service offers is up and running, and it's working as expected by our customers and their end users. To put this into perspective, in this giant blue rectangle I have up on the screen, the red in here represents the amount of time in which we can be down. And if you can't amounts to just five minutes and fifteen seconds per year that we need to maintain uptime to match our SLA promise. It very much means that we just can't be down. For the rest of the presentation, I want to walk through how some of the challenges that we've had to overcome in order to make this a reality as well as how we actually achieve this, utilizing AWS as our infrastructure.

And to do that, I want to add a little bit of context of what Authress actually does. So Authress is a login and access control solution designed specifically to help add the functionality to and protect the applications that you write. It offers, user management, user identity, authentication, access control, permissions, role and resource based authorization, as well as API key management for both—your services and your end users to interact with your APIs. It also offers audit trails and a number of other features to help complete the picture. So hopefully with this said you can already start to imagine why we have to offer a five-nines SLA and some of the challenges are that we may have.

If wee forget for a moment about the need to actually maintain a high uptime, and we thought about what the initial architecture is that we could we could have to actually build our product, you may end up with something that looks a little bit like this. We could have had a single region with, some HTTP interface, either a gateway or an ALB, to handle the request and pass them to, Lambda or EC2 or some other compute, as well as having a database, RDS or DynamoDB, and some ancillary services. Now, I'm sure you're thinking a single region with these components. It's already pretty obvious that there's a problem here that we're going have to overcome. And I think the Amazon CTO is frequently attributed to having said that \"everything fails all the time\".

And AWS's serve services are certainly no exception. Over the last decade, there have been more and more incidents in AWS, for which any one of those would have had a serious impact on our services, and on the services that we offer to our customers. And if we actually tried to rely on the AWS infrastructure directly and just take AWS's SLAs as our own, we would very quickly run into a problem. AWS's Lambda's SLA is below five-nines, the API Gateway SLA is below five-nines, The AWS SQS SLA is below five-nines. We're not really, getting any success here.

And so, fundamentally, the first kind of problem that we have to overcome is infrastructure failures. These are problems with the infrastructure that we're utilizing in AWS, by how it's delivered to us by default. So just for an example, going back to the initial architecture, if we had gone with this and there's an issue, with our database and we can't connect to it or connections are intermittent, timing out, returning five hundreds, or if there is an issue with our compute resource, either the control plane, spinning up new containers is a problem, or getting a request, or maybe there's an issue with, actually handling the incoming requests and return responses to our customers—networking could be an issue. In any of these cases, we pretty much just declare bankruptcy if we're only running in a single region and just declare that the region is down.

So this is obviously not sufficient for us. We have to handle this very explicitly. And the way we do that is we duplicate the infrastructure one for one, from primary region to a secondary region. And when there's an issue in that primary region, we fail over automatically to the secondary region. So our customers hopefully have never aware that there's actually an incident.

This requires having all the infrastructure and the data to be duplicated in both places. Okay. That's great that we know how to actually what we want to achieve. How how do we actually do this in AWS? To do this we use dynamic routing and Route 53.

We designate one of the regions as primary and one of the regions as the failover region. And when there's an issue, Route 53 will automatically failover to the appropriate region. We have to have the same infrastructure in every single region that we operate in. And since we operate in five different regions total, we actually have 10 sets of our infrastructure out there in case of a failover event. The failover region in the set is geographically close to the primary region so that if there's an incident, it's very nearly undetectable by our customers during a failover.

The way we actually know that there's an incident and the way we communicate through Route 53 that there's a problem is using Route 53 health checks. Route 53 health checks allow you to actually communicate, that there's a problem to the dynamic failover records and for Route 53 three to route those new requests to the appropriate location. Our health check looks something like this. It comes in and communicates with the gateway and verifies that it's up. That executes a request, on in our compute, which may interact with the database.

Now you could be utilizing the standard, health checks that are come out of the box, but we actually significantly configure the health checks we're using and use something custom that allows us to have greater control over when to failover from one region to another one. This is really critical for us, because it means that, you the last thing you want to do is failover from one almost working region to another region that isn't working, or failover and it not actually fix the problem. That can make it worse! So instead of just relying on just the health checks that AWS gives us, we have in-depth execution which verifies whether or not we're down in a particular region. I have a code snippet up here, for example.

It's not exactly what we're doing, but I think it's a pretty good proxy. About 22 times a minute, the Route 53 health check will fire off from different places around the world and verify that this code worked as expected. What we're actually doing is we profile our request to make sure that we're handling all our customer requests in an optimal way. If it's too slow to actually respond, then the profiler will will be able to validate that and, also mark the region as down.

Fundamentally, we check a few of different things:

The first one is we go to our database and we make sure that we can actually fetch most of the resources that we need correctly in order to perform authorization checks. We make sure that the core logic in our authorizer is working correctly. This is the primary functionality that most of our customers use. We may verify secondary services in AWS. If enough of them are down, it could also cause an impact that is visible to our customers, and that could be a reason to failover as well.

And then we have some logic validations to make sure that the code that we actually deploy is working correctly. Maybe there was an issue with the deployment package, maybe what we're actually running there isn't isn't correct or there's an issue with it, this would catch it. Then of course we evaluate the result of these these requests and return success or failure back to the health check. We actually don't stop here. Realistically with this configuration and this infrastructure, it requires having a complete duplicated set of all the components running in multiple regions even if you're never utilizing them.

And if only one component is down, it's it's a shame to just fail over and run everything in a second region. Sometimes that could create unnecessary load and extra complexity both for your writing your service as well as some of the routing that could take place. So in some cases, where we're afforded an opportunity to change our architecture, we move to a decentralized edge compute solution. And we do that using Lambda@Edge through CloudFront Distributions. So instead of having a failover, when a request comes in instead of coming to a region, it gets executed on the edge, and the closest region to that edge node, goes to the compute, which has to be a Lambda@Edge function, and it directs from the database.

Well, components in that region could be down, an example would be DynamoDB, and in that case we automatically fall back to executing a database request to an adjacent region, and contact the database there to read and write as necessary. And if there's a problem there, then we fall back again to a third region. At at this point, you may be like, well, why three, why not four or five? At at three, we've identified that it's significantly rare for such an event to ever occur that, would be far above what our SLA promise actually is, and would potentially create, negative impact on the request that are coming into our service from our customers. For instance, if you have to fill over to four regions, that's four potential checks in your service. That really increases the amount of execution time—the latency before returning the successful request to our customers. So it's much better in those cases to give up after three and just return a failure and have have the client retry as necessary.

Okay. So at this point, we've handled I think we've handled infrastructure in AWS, but that's not the only situation in which we have to deal with, as a service provider. It would be it would be really nice if this just solves all of our problems. We also have to deal with application failures.

It's obviously something you would want to achieve to have bug free code, but the reality of the situation is that at some point there's going to be an issue that is going end up in production and could affect some of our customers. If we waited until we got a Discord message, email, or started an on-call incident to trigger one of the engineers on our team to get online and investigate. By the time they're online, we've already violated our SLA. Realistically, getting online in less than five minutes and fifteen seconds is just not going happen, let alone actually fixing the problem.

So instead, we need to optimize for having automation in place over just alerting. So wherever there's an alert, we really need to think about how can we automatically resolve this problem without involving someone. Naturally, the the primary way to handle this is to have a lot of testing in place, automated testing. After all, every single incident you could ever have, we've ever had, could be easily unsolved by having just One.More.Test.

We test efficiently before deployment so that by the time changes get to production, we're confident as possible.

We don't focus on test coverage though, but rather test value. And the tests we create are the ones that we believe have the most value. But I'm not going into that further. However, having all the right tests all the time is almost impossible for a service that's constantly improving and adding more and more features every day.

For every additional test we want to write, the time to actually create and maintain that grows exponentially. So the challenge to get to a 100% complete test coverage would actually take an infinite amount of time. This is known as the Pareto principle, or the 80/20 rule. We will never catch everything in the amount of effort to get even a little bit closer, very quickly gets away from us. So instead, we have to optimize both for prevention by running the right tests, but more importantly recovery when an issue gets to prod.

The only conclusion we can come to is that we need to have additional tests running against our production environment. An example of what we do is our validation tests. An example of a validation test that you could run is use the data that you've got within your database, and the logs that you have to verify—what you're seeing—to verify what you've got already in the database. So for our service, in some cases we have the same data stored in multiple different places. It's not exactly the same, so it gets used for performance searching, quick lookups, and pre-populated caching,.

Other times we mutate the data for optimized throughput. We can act on a schedule we can fire off some compute, which will go and verify the data between these databases. And then, if there's an issue we can send an alert which would allow someone to actually go investigate, and see what's going on here. If you don't not everyone has a situation where they have multiple databases, with similar data. But if you don't, you can utilize potentially past logs or the current logs that are coming in and reprocess them and verify the data that's in your database is exactly as you expected to be.

You can rerun it through the same code and see if the same result happens. You can sample your logs so that, you're not just reprocessing all the data. Another thing we do is, of course, incremental rollout. So when we create a change that's going go to production, we need to make sure that if there's an issue with it, that it impacts the lowest number of our customers and their users as possible. To do that, we bucket our customers into small groups and deploy changes to an individual customer group one at a time.

If that change is successfully deployed, then we'll move on to the next group, and the next group, until it's deployed everywhere. This is another incremental rollout. If there's a problem, then we'll stop the deployment, alert our team that there's an issue with it, and then dive in and see what the possible problem could be.

This is a good strategy, but we actually need to figure out what makes sense to alert on. And I don't think it would be a presentation at AWS conference if I didn't at least mention AI one time. And so I'll say that, we are using anomaly detection in CloudWatch and a couple other tools to identify automatically if there's an issue with the code that we've rolled out to production for one of those deployment groups. The anomaly detection evaluates a specific metric that's relevant for us. The metric we are using is what we call the authorization ratio. That's the ratio of successful authorizations or login attempts versus ones that are blocked, cancelled, or never completed. If there's a problem there, then anomaly detection will alert automatically, and someone will jump in and attempt to resolve it. You can on this graph of alerting data, that we got pretty close to breaching the threshold and thus identifying an anomaly.

The detection was one cycle away from realizing a potential problem and alerting to us. I just happened to find this when I went and looked at the logs. In this case, there was no issue so we just moved on as as planned. Deployment went on to the next deployment group. This ratio is specific to our business at Authress, but there are of course relevant business metrics that make sense to alert on in your situation.

It doesn't make sense to just use an arbitrary thing like APM or number of 200s or 500s or even 400s that you're getting. It should actually be relevant to your use case. And I think this is a really important point, because it can be actually quite a challenge to correctly identify whether or not our service is up and running and that it also matches what our customers also believe. It's very difficult to have an accurate picture of what our customers are seeing.

Let's break that down. The Authress service health versus our customer's perspective.

A. If we correctly identify that our service is up and running, and our and our customers don't see a problem, then we're all good. Customers think everything is great. We think everything is great. Fantastic.

B. Inversely, maybe we identified that there is an incident currently happening and so do our customers. And in which case, we probably have deployed some automation here, which automatically fixes the problem or at the very least alerts the team to go and investigate. While it's not great that there's an alert, we correctly identify that there is an incident. That's a huge step forward.

G. One interesting category is this bottom left-hand corner, where our customers think our service—everything is great. There is no alerts from them. But for us internally, we've identified that there may be a problem. And you may be thinking, well, how how could this happen? Well, our service alerting could be a little bit too sensitive. Or we have an issue in production, but the users who care about it or who are utilizing it maybe aren't online. Take our customers in Australia at this moment. They're all asleep.

So we could potentially reduce our alerting, and not fire off an alarm in these in these particular. Reducing these is important, because they amount to extra work without necessarily extra value. I think the most interesting quarter is up here on the top right-hand corner, which is called a Gray Failure. Our customer says—\"Hey, there's a problem with your service.\" And we look at our, dashboards, our metrics, our alerts, and there's nothing happening.

Everything looks fine to us. You may be thinking, how is that possible? How could that happen? Well, realistically it's possible that we missed on alert.

We don't catch everything. There's something there that we're missing, and there's an incident, and we are not alerting on it. It's very rare though. More likely what happens is the expectations from our customers don't match our expectations.

Maybe there's something in our knowledge base which is confusingly worded, and this reported incident gives us an opportunity to review that documentation and improve it. Maybe it's a poorly named variable. Maybe our customer has some sort of alerting that's relevant for their service, and they're alerting on that even though Authress hasn't changed at all. A good example is one of our customers has a different network configuration or just using the not as good cloud provider. In these circumstances, we really need to care about what's going on from their perspective. They may have additional tests that we're not running, which we can get the benefit from them. And so it's critical that we consider our customer support pipeline to be invaluable for us. If there's a problem and our customers identify, we want to know about that as soon as possible.

That means triaging and exposing customer support requests as alerts in our system to our team. If you get too many alerts, then it isn't realistic to alert on customer requests. You may be tempted to add in additional support levels into your organization to combat the problem. However, for every additional level of support, you reduce the turnaround time for solving support requests. And if one of those requests identify a potential incident and offers insight into a problem with your service, then that means you're actually reducing the possible SLA that you can offer.

So instead, we spend a lot of time thinking about how to eliminate support requests that aren't related to incidents or triage them differently. That means we evaluate every single support request that comes in. We try to answer the question of why did we get this request? Is there a problem on documentation? Is there something confusing? Is there a way that we can avoid this problem in the future, make something simpler, change something around?

We consider our support to be one of the lifelines of our service and allowing us to maintain, the five-nines SLA that we offer. We track these metrics internally that we utilize to know about the status of our service. And in some of these cases, because our customers have a different view, it helps to expose the same information to them.

So actually in the dashboard of our product, we expose some of these metrics specifically so that they can compare what's going on. The most used resources in Authress from their perspective, the users that are utilizing their service at that time, as well as some specific messages and warnings related to Authress functionality usage, that they may want to take a look at. And they could utilize that additional information to identify if in their particular situation they have some feedback for us, or for them this additional data helps point to an internal incident on their side, and it would be great for us to know what's going on and why.

I wish I could say that after implementing everything that I've said so far in this talk, automated region failover, incremental rollout, having a huge, customer support focus, that that would be enough. But realistically, we also need to deal with negligence and malice.

We're lucky enough to be such a popular product that we have free security researchers out there on the Internet constantly reminding us by spamming our email about the identification of a \"possible problem\" with our service that we should go and and solve. And it's important that we take some of these seriously because if we have a multi tenant solution, I'm sure I'm sure most of you do, and that means we share some infrastructure resources between customers. So if one customer is malicious or negligent and you're over utilizing resources or that customer has a user who is malicious and is over utilizing their resources, that have an impact on our infrastructure resources. Then that could potentially have a negative impact for our other customers. The way we defend against this problem is by adding rate limiting of a few different kinds. We add rate limiting into our application level at the region where the code is executing, and we also add it into the CloudFront distribution. And we do this using a web application firewall.

Web Application Firewall (WAF), if you're not familiar, is a component from AWS, which, tracks requests and you can match on a lot of different things. There's a lot of different configurations out there, but I want to share exactly what we do. So fundamentally, our WAF configuration includes some AWS managed rules, specifically the IP reputation list. This is a great rule to have because AWS automatically populates it with things things that they've identified as malicious.

And so that means that you don't without even thinking about it, you can automatically reduce your attack surface by known problematic entities. If one of these entities is performing a DDoS attack on AWS' resources and affects many different customers, we'll have the benefit of being protected. We don't stop there, but we don't use any other managed rules though. There's a lot in the marketplace and also by AWS, bot control rules, etc. From our perspective, overpriced. They don't really offer a lot, and sometimes they have too many false positives or are challenging to configure. So we don't actually use them.

We do however add BLOCK rules for high rate limits.

So anything that we consider that no customer would ever likely hit, realistically, and if they were to hit then there's probably an issue going on, we just immediately stop those requests. We match on IP address as well as some other properties, and we just immediately cut that out. Not every customer is below this limit. We have some that are above, and so we do make special exceptions for those. But we know when we expect that's going to be happening.

Additionally, we add a whole bunch of COUNT, which is just a way of logging that a particular user is utilizing resources in our service. So add a couple different denominations, 1000rps, 2000rps, 5000rps, etc. We automatically track requests, and this can help us early identify that there's an attack on the horizon, something is coming, and utilize this information to prepare ourselves and our service, or to get AWS on the line, to ask—\"Hey. What's going on? Is there something else that we should be doing here?\"

And so of course we have alerts on these WAF configurations. Remember, this is not sufficient to actually prevent attacks. We we need automation in place. And so the way we do that is we log on web application firewall requests, and then we filter that data to try to understand what's happening on the fly. So automatically when we get one of these alerts, we're running some automation, which is evaluating what's actually happening.

And here's an example of some of our query automation where we're pulling out highest used IP addresses over a small period of time or some specific parameters. This is goes into CloudWatch and we query that data using their API. And depending on this and some of our own internal rules and logic, we'll dynamically update the WAF rules so that they will block attacks as they happen automatically.

And because we're running in multiple regions, we have the same technology to deploy to all of our regions, and it's also extended out to the edge compute as well.

Review​

Okay. I said a lot of things. Let's quickly review what we're doing.

  1. So we have Route 53 automatic failover and health checks to handle multi region deployments and dealing with infrastructure issues.
  2. We have edge compute to help push off some functionality and decentralize it, so not everything is in the same place. It also gives us improved throughput, and reduces latency to our customers' end users.
  3. We implement incremental rollout, to reduce the impact of problematic changes to production.
  4. We have a customer support focus.
  5. Lastly, we have a Web Application Firewall (WAF) with AWS managed and custom roles.

Insights​

I want to jump into a few quick insights.

There'll be failures everywhere. So not only will things fail, everything will fail all the time. I think it really speaks to the fact that wherever you've got components or code, or functionality or an endpoint exposed. Something will happen to it. For us, DNS is still a single point of failure. I don't love this about our architecture and I wish there was something that we could do, but I'm not confident there is more we could be doing at this moment.

The last one, deploying almost identical infrastructure is actually really hard. So it'd be one thing if we just deployed our technology to 5 regions or 10 regions and we're done. But some of those regions are backup regions. So they're not primarily used. So the configuration is slightly different.

The failover is slightly different. Some records are primary, some are secondary. So there's a lot of there's actually a lot of minor differences that we have between different regions and this adds significant complexity to our deployment story.

If anyone has any great suggestions on any of this, I'm happy to discuss them.

info

For help understanding this article or how you can implement improved reliability or a similar architecture in your services, feel free to reach out to the Authress development team or follow along in the Authress documentation.

", "url": "https://authress.io/knowledge-base/articles/2024/09/04/aws-ensuring-reliability-of-authress", "title": "Ensuring the reliability of Authress", "summary": "Authress has an expected 99.999% SLA. There can be many requirements for uptimes--regulatory needs, core dependencies, or your service offers life-saving responsibilities. In these circumstances, it's critical to never let your service go down. Here, we'll review how we’ve architected, built, and maintained a solution with such a high SLA, the frameworks and strategies we’ve established, and what it takes to keep up that commitment. Presented at AWS Summit Zurich 2024.", "date_modified": "2024-09-04T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/07/21/user-resource-sharing-challenges", "content_html": "

The goal is to have a simple UI share link for users that does the right thing:

\"Create

Sharing resources between users seems like it should be simple, then why are there no obvious simple solutions? Maybe it isn't so simple after all. So let's at least review the bad options.

Background​

In most software applications and platforms the ability for users to share resources is essential for collaboration and communication. With no shortage of data breaches and privacy concerns, developing a secure but user-friendly solution is crucial.

The most common pattern is that one user user A creates a resource. Then later that user shares access to that resource to another user user B. The permissions granted to user B, maybe the same or different than user A has, and the ability for user B to share access with another user user C may or may not be restricted.

Another pattern with sharing resources is that users might be part of a shared customer account tenant. Multiple users might own and access multiple resources in the same account. There might be groups of users, and multiple resources or groups of resources might need to be shared at the same time.

The last scenario relevant for sharing is that users might be anonymous. That is, our user B might not exist in our platform, yet. When user A attempts to share a resource with them. That means we won't have a user identifier representing the user before staring the sharing process.

An additional complexity is that resources may or may not be owned by a single service. In other words, shared resources might be owned and operated through a third party plugin or extension. If you have a Platform Marketplace, then sharing might have further complex scenarios. However in most cases those scenarios just extend from the core ones listed above, and by using Authress Platform Extensions these are supported out of the box. See the platform extensions documentation for more information.

\"The

To demonstrate these usage patterns we'll be using the document repository implementation example which entails:

There are many documents. Each document can be access by any number of users and each of those users may have different permissions to that document. Users will share documents and change other users’ permissions as well as create, modify, and delete documents.

info

This article references the document repository implementation example to help explain the following concepts. It helps to take a quick review to better understand the core product concept.

Potential Solutions​

There are actually multiple solutions to support resource sharing, and depending on your core software needs, any number of them might be right for your situation. For higher frequency of user usage, scale, or more complex user stories, more robust solutions will be necessary. More complex doesn't necessarily mean scale. You could have scalability concerns but only simple needs, or complex needs, but small scale. As we'll learn this doesn't change the feature set, however. This is important to keep in mind. And next, we'll iterate through some different options available.

Resource property or user list​

The most common solution for storing resource access permissions is to push them onto the resource or list the resources a user has access to in the user record. We know from the academy article on Access Control Solutions that these are the Inline access policy and the Access control list strategies. And neither of them sufficiently support our needs. Additionally neither of these scale. Eventually users will have access to many resources (scale), and resources will to support multiple users that need access (many-to-many).

With a hierarchy of documents in many directories, permissions on the user or the resource would not be sufficient to allow users to fetch all the documents they have access to, to get all the users that have access to a resource.

The reason is in either of these cases, to support our scenarios our system would need to add additional users to the resource list, or add the resources to the user. At this point if we aren't including the permissions for the resource then there is a huge problem, because there will be no way to distinguish what actions a user could take for that resource. Many-to-many user to resource mapping doesn't include the permissions we have to have.

That means for each authorization check we need to verify three things:

Only with those three things can we actually verify that the user has the correct access to the correct resource.

Going forward we'll assume that there is a dedicated first-class system that supports the storage of authorization and permissions for each user. This system doesn't have to be full featured for each of these solutions, but without something in place, most of these are impossible to support.

The user attribute​

Now that we have figured out that we'll need to store additional information regarding the permission that a user has and that different users will have different access, a followup might be Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC). However both of these systems lack the support we need as well.

With RBAC, a set of strings is assigned directly to the user, such as Admin or Viewer. But it doesn't include the granular permissions nor the resources that a user should have access to. For instance, in the case of user A sharing resources with user B, they might want to only grant read-only permissions. With RBAC they can do that might assigning a ReadOnly role to user B. However, the problem here is that user B will then have ReadOnly access to all of the resources of user A or the Account Tenant that user A is a part of, which is not what we want.

Likewise with ABAC, a similar problem exists. There is no one attribute we could put on user B to signify which resources they should have access to. Further, then we would be stuck will all the pitfalls of ABAC, further listed and discussed in the Access Control Solutions article, the problems with ABAC is so long that this is not the place to discuss them, but the most fundamental of which is the ability to list all the users that have access to resource R. Queries like that aren't possible with ABAC, and being able to query for resources or users is a critical component of an access control system.

The iterative approach​

It's probably not a surprise that in today's software engineering of systems, many software applications, services, and platforms take an agile iterative approach to building up access control. They might start with one of the above solutions. That means our starting point is that access is granted directly to the user on the document:

\"User

Our access check is very simple, does the user have access to the specific document:

VIEW /documents/doc_001

Then later, we might realize that users are actually in accounts and that documents are actually owned by one account in particular, not by a specific user most of the time. We achieve this by storing which users are in which accounts, and including the account in the document resource uri. If the user has access to this new hierarchy then they have access to the specific document.

\"Resources

The access control check would be altered to to support the tenant.

VIEW tenants:tenant_001/document/doc_001

This is often an easy change for your system because at this point you likely haven't gotten that far. But if this sort of migration is a challenge, good luck for what comes next.

Eventually, inside of those accounts, naturally different groups of users will exist for each account and have access only to a subset of all the document in that account.

\"Access

But here's when the iterative approach starts to break down. Eventually we'll want to identify that group of resources, so that we can define users or groups of users that should have access to it. This grouping is what causes a problem, and that's because we actually need a new element in our document hierarchy that wasn't there before. And we can't just add in a group segment to that hierarchy because the document might not exist in a document group from a user perspective. Is Document Group 001, a thing the user knows about? It is possible, but it is likely it is an arbitrary construct.

\"Access

From a directory perspective a document only exists in one hierarchy, but from an access standpoint, a user might need access to same group of documents as another user, directory irrelevant. For instance two documents might be /root/path/1/doc_001 and root/path/2/doc_002, and user B needs access to only these two documents but isn't allowed to have access to the rest of /root/path/1. That would require explicitly giving the user access to these two documents, which works. Now imagine that there are 1000 additional users that need the exact same access to these documents and all other documents that consider similar information. For instance /root/path/*/financial_report_year_end/*. There also might be too many documents or users to explicitly give the users access to all the documents in that group.

That means that we can't just extend our document uris to be of this format:

VIEW tenants:tenant_001/document-groups/group_001/documents/doc_001

The documents already exist in a hierarchy which is a business logic requirement, we don't want to alter that to satisfy our access control needs.

Actually trying to do this and make the migration here is often an impossible burden. But assuming we attempt it, there are actually three problems with this approach:

  1. Documents can be in multiple groups, so if users have access through different groups, we actually need to check which group they are trying to access. We don't know up front which group will grant access. That would require first iterating all the groups the user has access to and then checking to see if the user has access to the document through that group. The one caveat here is that this is actually possible to do in Authress via the wildcard tenants:tenant_001/*/documents/doc_001, however since most solutions don't support this, this might not always be an available solution to you.
  2. This doesn't scale across multiple services. In the case you have multiple resources and multiple services in a service mesh, microservice architecture, or a monolith, then users will need access to multiple resources of different types at the same time. Or at the very least, every service you have will end up needing to reimplement the same solution. If users care about sharing resources in one part of your system, they'll almost for sure need to share resources in both related in unrelated parts as well.
  3. I'm sure you are already thinking, what happens when I need sub-groups. And for sure that day is inevitable.

One mistake that is made far too often is attempting to solve this with any sort of attribute-based access control. While it seems like it could help to support this case, it breaks down cross service, or when any sort of auditing is necessary, and it fails to support all our other needs as use cases. So while it works here, it doesn't support the user experiences we need, see above for all the ABAC failure modes.

Resource groupings pitfalls​

Above we mentioned creating document groups as part of the resource uri. This would give us a lot of control if it worked. If every document was part of only exactly one hierarchy, then adding the resource group to the our resource uri would work. For us this is actually the directory path, and we can see that this is implemented in almost every document/file system: /root/path/directory/file.ext. However, this obviously won't work for us, when it comes to sharing subsets of documents, as these sets of documents exist outside of any defined hierarchy. In other words, Users must not be restricted how they share, based solely upon how they have structured their directories. And while you could add that restriction to your service, both your product managers and users won't be happy because for sure some user made a mistake somewhere in their resource management and rather than trying to \"fix it\", they'll just go to a different platform.

We know we need to support multiple groups, we could be tempted to make a document-group a first class notion in our document repository service API. There would be CREATE /group and UPDATE /group to manipulate the groups of documents. Users would be given full control over this, and when we want to check for access, we would check to see which groups a user has access to and which resources are in that group.

The code to perform that check looks like this in our service:

Check if user has access to resource via resource group membership

async function checkAccess(userId, resourceId) {
        const groupsThatContainResource = await database.getResourceGroups(resourceId);
        foreach (const groupId in groupsThatContainResource) {
                const hasAccess = await authService.checkAccess(userId, groupId);
                if (hasAccess) {
                        return true;
                }
        }
        return false;
}

Not only is this code poorly written, there isn't really a great way to optimize it. So it creates a pit of failure for our services. It's a pit of failure, because It's easy for our services ho have it implemented wrong, and there aren't even any good solutions. And there are still more problems:

  1. Every service we have in our platform or monolith would need to create the exact same functionality to support resource groups. We literally have duplicate the same code in every service and resource we have. And we probably want all those things to be the same as well, since permissions management that is consistent provides emergent benefits for our users.
  2. We actually don't care about resource groups in our service. Resource groups are almost never a business concept. We only implemented resource groups because we needed a way to grant access. So this isn't even something we want in our service in the first place. Of course there are some exceptions where a resource group may make sense in your context, and then you want to go and implement it, but for our document repository it just adds confusion (and technically we already have groups in the form of a directory structure).

Pseudo-copy resource​

Due to the nature of the problems created with the Resource groupings, many implementations opt for sharing only at the Account tenant level, and thus do not have groups. That means a user would have to be assigned access at the top level and therefor not have granular access to specific resources. However, as identified this isn't a full solution and often is insufficient for the user experience we as well the audit logs we need to keep. It also feels wrong.

The next alternative is to actually copy resources. At this maturity of software, you likely have a solution in place that could support groups, account tenants, and direct access for users at the resource level, but it misses one or more features to actually get all the way there. One trick is that when a document is shared with a user instead of giving that user ReadOnly access to the document, we could create a copy of that document and mark that document as ReadOnly. This avoids all the problems identified with groupings so it can seem like a great alternative to having an authorization and permissions management solution that supports first-class resource sharing.

\"Copying

This of course isn't without its own problems though:

  1. The primary problem is Which Users - If we don't know who the user is that we will be sharing with, then we won't know at sharing time where to duplicate the document. If that user is part of an Account Tenant, we also won't know which tenant to share with. Most systems will completely ignore this problem and tell you outright that: you can't share with an unknown user, which can tell you a lot about the state of their security implementation.
  2. All other services need to implement this as well - As with the resource grouping pitfalls, every service that wants to support resource sharing will fall into the problem of needing to implement the resource sharing code. They will all need to support an extra flag on every resource. Forcing every service and resource to add an extra flag tells us there is something wrong with this solution.
  3. Potential mess everywhere - There are ongoing questions about how to manage multiple read only resources for different users. For instance if the same resource is shared multiple times, how many readonly copies should we make. If we only make one ReadOnly in total, then we need to add some explicit code to check if we have already made it, for.every.single.resource. And if we make more than one, then...we'll get that to that.

There are actually some critical issues as well:

Users who are not part of the main resource owner group for the original tenant can only have ReadOnly access to the whole resource. It won't include answers to questions of the form:

And if that wasn't bad enough, by far the worst problem of all is attempting to keep these copies in sync with the original resource. When the resource changes, that means now our service has to sync at least one version to another copy of the resource. Getting objects in a database in sync is an incredibly hard problem, as it is often unreliable and no perfect solution to guarantee consistency. And if there are multiple copies then at best we've just unnecessarily doubled our DB size, and it could be even worse.

All of that, and we still don't get much further than a very small edge case, because all of the more sophisticated access patterns still don't have a solution. So lots of questions, no answers.

The complete solution​

So far it seems like every partial solution comes with major drawbacks that make iterating on it not possible without completely changing the strategy. So instead, let's just focus on the best way to achieve the Click here to generate a sharing link that works in every situation.

To do this we'll take the best parts of all the previous solutions and combine them to eliminate the problems and achieve our end goal. The full list of the things we want to achieve has become:

This is very nearly a full list of everything that is necessary for resource sharing and we can't do without. To support it we'll introduce a solution called Resource Bundles.

Resource bundles​

Resource bundling is a first-class strategy we'll need our authorization solution to support. Resource bundling is simply the ability to define a list of resources and a list of users we want to have access to that list of resources.

It would allow us the ability to group resources in our access control permissions system. This means that our standard access control solution needs to support it in order to get the benefit. However, given that everything discussed in this article is access control related, this is the right place to go. It eliminates most of the items from our list just by moving out the logic from the individual services.

To achieve resource bundling we'll need a place where we can define those lists. For the purposes of this article to give them a label that can be reused, let's call that an Access Record. For bundling to work, it must at least support our required fields and properties: users, resources, and roles (or permissions):

\"Access

If our authorization solution supports listing out all of the resources in a single place, then this is our bundle. For every bundle of resources, we can assign who should have access to that bundle. A strategy like this can also be extended to grant groups of users, organizations, or whole tenants access to the bundle, rather than just individual users. Some access control solutions support this. Most importantly though, we don't have to put the group in the resource hierarchy as the bundling of resources is authorization related not resource definition related.

info

It is worth calling out that, there are some cases that make sense for our solution and our users which require grouped resources, and in these cases the group should remain in the resource uri. Otherwise we now can take it out.

The resource uri without the group in the hierarchy
tenants:tenant_001/documents/doc_001

Our resources can remain permission group agnostic, unless we actually want them to have a grouping as a concept. And because the bundling can include arbitrary resources and resource uris from any service in our software, application, or platform, this also ensures there is a single location for access. Every service gets support by design for free.

Putting it altogether​

Now that we have a strategy in place with Resource Bundling, we can actually go through the flow of how that works in practice.

Flipping back to that copy link button from earlier:

\"Create

When the user is on this screen they have two options:

Since sharing is really just assigning permissions to a user in disguise, in the first case, we would create (or update) an access record for the invitee:

Creating an access record for the new permission
await authressClient.accessRecords.createRecord({
    // Select a predictable recordId so we can update this later
    recordId: `rec_Document:${documentId}:User${inviteUserId}`,

    // A helpful name for the record
    name: `Document ${documentId} access`,

    // Invitee UserId
    users: [{ userId: inviteUserId }],
    statements: [{
        // Grant the Editor permissions
        roles: ['Editor'],
        // To the document in our tenant
        resources: [{ resourceUri: `tenants:tenant_001/documents/${documentId}` }]
    }]
});

In the case that we want to invite someone to edit all documents in a hierarchy:

Creating an access record for the wildcard sub directories
await authressClient.accessRecords.createRecord({
    // Select a predictable recordId so we can update this later
    recordId: `rec_Document:all-documents:User${inviteUserId}`,

    // A helpful name for the record
    name: `All document access`,

    // Invitee UserId
    users: [{ userId: inviteUserId }],
    statements: [{
        // Grant the Editor permissions
        roles: ['Editor'],
        // To the document in our tenant
        resources: [{ resourceUri: `tenants:tenant_001/documents/*` }]
    }]
});

To update a resource bundle with all the new users it would look similar to this:

Update access record resource bundle for multiple users
await authressClient.accessRecords.createRecord({
    // Select a predictable recordId so we can update this later
    recordId: `rec_Document-Bundle-001`,

    // A helpful name for the record
    name: `Access for document bundle 001`,

    // Invitee UserId
    users: [{ userId: 'user A' }, { userId: 'user B'}],
    statements: [{
        // Grant the Editor permissions
        roles: ['Editor'],
        // The relevant document bundle
        resources: [
                { resourceUri: `tenants:tenant_001/documents/doc_001/sub-resources` },
                { resourceUri: `tenants:tenant_001/documents/doc_002/*` },
                { resourceUri: `tenants:tenant_002/documents/*` },
                { resourceUri: `tenants:*/documents/*/finance-docs/*` }
        ]
    }]
});

And for the second case, when we don't know who the user is yet, we would instead want to generate an invite instead. There are many ways to do that. For completeness an example from the Authress Knowledge Base is included here:

Inviting a user to a shared document
import { AuthressClient } from '@authress/sdk';

// The Authress API Url: https://authress.io/app/#/api?route=overview
const authressClient = new AuthressClient({ authressApiUrl: 'https://auth.yourdomain.com' });

// First, generate the invite
const inviteResponse = await authressClient.invites.createInvite({
    statements: [{
        roles: ['Editor'],
        resources: [{ resourceUri: `tenants:tenant_001/documents/${documentId}` }]
    }]
});
const inviteId = inviteResponse.data.inviteId;

Once we have the inviteId we just need to get it to the user. Realistically, you probably would want to host your own Invite Acceptance UI to help guide the new user. For instance they might want to register a new account, or use an anonymous user to authorize to the document they have just been given access to.

And we are able to do this only because:

Additional advantages​

User agnostic invites​

There are some additional edges cases from the above list which you might have to support. In order to generate a generic link for a user to click on, the invite might need to be user agnostic. That is because we don't know who the target user is yet. This means that our access control system must have a concept of access without a user that can be claimed by a user at invite acceptance time. This is important for two reasons:

Invites are a complete topic in themselves, and can be sophisticated depending on needs of your software. Check out the full guide on implementing invites yourself.

One time sharing​

Invites also serve a secondary benefit as well. If we've implemented resource sharing using invites that means that one time resource share links can also be provided. A one time link grants a user access to those permissions only one time via single session. Instead of logging the user in and granting them permanent access, we can utilize the features in the authorization for access records to refine exactly how that access works. If that access should expire after 1 day or 1 month, then we can easily do that. With the other strategies, every individual resource and service would need to be extended with that functionality. Every single additional feature we need, would require every service in our platform to support it before we could offer a consistent solution.

Revoking access​

Access records also extend to control over revoking of access as well. While the topic of revoking access is so complex that there is an entire academy article on invaliding user access, if an access control solution is utilized to grant cross tenant, cross user, or cross group access via resource bundles, then revoking access works the same as if the resource was only shared within an account tenant. Having a consistency in how to revoke access creates a powerful pit of success. If to revoke access to an in-account document, we need to update the permissions, but to revoke access to someone who we shared the resource with externally we would need to do something different such changing the explicit resource as well, then we've created a pit of failure--easy to get wrong and likely to write code which doesn't execute the right thing. By having the same flow irrespective of who the user is, where the resources are located, or how they received access, we can be sure that the right code and right flow will be executed every time.

Followup​

There are many ideas expressed in this article which have corresponding additional articles, from the Security blog, the Auth Academy, or the Authress Knowledge Base, here are some of the listed for ease of access:

", "url": "https://authress.io/knowledge-base/articles/2024/07/21/user-resource-sharing-challenges", "title": "Challenges building solutions for user sharable resources", "summary": "Sharing resources between users seems like it should be simple, then why are there no obvious simple solutions? Maybe it isn't so simple after all.", "date_modified": "2024-07-21T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/07/02/devweek-why-you-should-check-your-secrets-into-git", "content_html": "


In any software, platform, or application that involves more than one user, you will have to deal with authentication. And when you have more than one service or microservices, you will have to deal with credentials. If getting credentials right can be a headache, then keeping them secure during events such as — production deployments, engineer off-boarding, and credentials rotation — is a nightmare. Here, we’ll explore the different ways to manage your secrets by discussing the advantages and best practices for keeping your sensitive information, private keys, and service clients secure.

", "url": "https://authress.io/knowledge-base/articles/2024/07/02/devweek-why-you-should-check-your-secrets-into-git", "title": "Why you should check your secrets into git", "summary": "In any software, platform, or application that involves more than one user, you will have to deal with authentication. And when you have more than one service or microservices, you will have to deal with credentials. If getting credentials right can be a headache, then keeping them secure during events such as — production deployments, engineer off-boarding, and credentials rotation — is a nightmare. Here, we’ll explore the different ways to manage your secrets by discussing the advantages and best practices for keeping your sensitive information, private keys, and service clients secure.", "date_modified": "2024-07-02T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/06/18/estimating-roi-on-security", "content_html": "

How to put a number on the cost of something that may not even happen? How to assign value to abstract and subjective constructs like “brand reputation” or “customer trust”? How do we know if we’re spending enough on security, and how to tell if we’re spending too much? Assuming we have the budget for software security, where should we invest it? And in the absence of a budget, what can we do to obtain it?

", "url": "https://authress.io/knowledge-base/articles/2024/06/18/estimating-roi-on-security", "title": "How to estimate ROI on Security", "summary": "How to put a number on the cost of something that may not even happen? How to assign value to abstract and subjective constructs like “brand reputation” or “customer trust”? How do we know if we’re spending enough on security, and how to tell if we’re spending too much? Assuming we have the budget for software security, where should we invest it? And in the absence of a budget, what can we do to obtain it?", "date_modified": "2024-06-18T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/06/14/build-versus-buy-versus-open-source", "content_html": "

\"Build

All of us have an approach for deciding whether to build that next thing in house, get something off the shelf, or use an open source solution. I’d like to share a method I’ve used and refined over the years to simplify such decisions, and I’ve distilled it into a flowchart. Say goodbye to overthinking, decision paralysis, and projects that are doomed to fail!

Flowchart​

\"Device

Simplified​

\"Device

", "url": "https://authress.io/knowledge-base/articles/2024/06/14/build-versus-buy-versus-open-source", "title": "Build, buy, or use open source", "summary": "All of us have an approach for deciding whether to build that next thing in house, get something off the shelf, or use an open source solution. I’d like to share a method I’ve used and refined over the years to simplify such decisions, and I’ve distilled it into a flowchart. Say goodbye to overthinking, decision paralysis, and projects that are doomed to fail!", "date_modified": "2024-06-14T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/05/10/techspot-building-security-first-apis", "content_html": "


Embrace a security-first mindset in API development to proactively prevent malicious attacks. Learn how to integrate fundamental security building blocks, authenticate requests, validate access control, implement secure communication channels, identify potentially dangerous actors, and dynamically prevent attacks as they happen.

", "url": "https://authress.io/knowledge-base/articles/2024/05/10/techspot-building-security-first-apis", "title": "Building a Security-First API", "summary": "Embrace a security-first mindset in API development to proactively prevent malicious attacks. Learn how to integrate fundamental security building blocks, authenticate requests, validate access control, implement secure communication channels, identify potentially dangerous actors, and dynamically prevent attacks as they happen.", "date_modified": "2024-05-10T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/03/08/decompiled-adding-security-to-your-architecture", "content_html": "

It's easy to make security an all or nothing approach. Many architectures thrive in delivering the perfect product vision while others attempt to deliver something quickly. Often what can get lost is attention to security in a race to deliver the best architecture for the best product. Here I'll discuss when's the right time to think about security and how to do it in a agile way without losing focus on your core.

", "url": "https://authress.io/knowledge-base/articles/2024/03/08/decompiled-adding-security-to-your-architecture", "title": "Adding Security to your Architecture", "summary": "It's easy to make security an all or nothing approach. Many architectures thrive in delivering the perfect product vision while others attempt to deliver something quickly. Often what can get lost is attention to security in a race to deliver the best architecture for the best product. Here I'll discuss when's the right time to think about security and how to do it in a agile way without losing focus on your core.", "date_modified": "2024-03-08T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2024/02/04/fosdem-building-security-first-apis", "content_html": "


Embrace a security-first mindset in API development to proactively prevent malicious attacks. Learn how to integrate fundamental security building blocks, authenticate requests, validate access control, implement secure communication channels, identify potentially dangerous actors, and dynamically prevent attacks as they happen.

", "url": "https://authress.io/knowledge-base/articles/2024/02/04/fosdem-building-security-first-apis", "title": "Building a Security-First API", "summary": "Embrace a security-first mindset in API development to proactively prevent malicious attacks. Learn how to integrate fundamental security building blocks, authenticate requests, validate access control, implement secure communication channels, identify potentially dangerous actors, and dynamically prevent attacks as they happen.", "date_modified": "2024-02-04T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2023/07/11/codemotion-why-you-should-check-your-secrets-into-git", "content_html": "


In any software, platform, or application that involves more than one user, you will have to deal with authentication. And when you have more than one service or microservices, you will have to deal with credentials. If getting credentials right can be a headache, then keeping them secure during events such as — production deployments, engineer offboarding, and credentials rotation — is a nightmare. Here, we’ll explore the different ways to manage your secrets by discussing the advantages and best practices for keeping your sensitive information, private keys, and service clients secure.

", "url": "https://authress.io/knowledge-base/articles/2023/07/11/codemotion-why-you-should-check-your-secrets-into-git", "title": "Why you should check your secrets into git", "summary": "In any software, platform, or application that involves more than one user, you will have to deal with authentication. And when you have more than one service or microservices, you will have to deal with credentials. If getting credentials right can be a headache, then keeping them secure during events such as — production deployments, engineer offboarding, and credentials rotation — is a nightmare. Here, we’ll explore the different ways to manage your secrets by discussing the advantages and best practices for keeping your sensitive information, private keys, and service clients secure.", "date_modified": "2023-07-11T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/auth-situation-report", "content_html": "

If you’re someone who builds software, no matter if you’re on the backend or frontend or even on the product side, sooner or later you have to concern yourself with securing the thing. Or you realize that data privacy laws are very real and you must have a strategy for user data sharing. So you want to implement some sort of authentication. More likely, you’re looking for a solution, open-source or otherwise, that will solve this problem for you. And here comes confusion.

The auth space is currently very active, with new players joining and old players repositioning themselves (with or without changing what they offer). They each use a different language on their landing pages. Or worse, they use the same words to mean different things. With this article, I hope to bring some clarity on what parts of auth and data security are relevant if you’re building software, and how to think about the space.

I’ll clarify the terminology around it, list the most important considerations when picking a solution, and mention any caveats or pitfalls you may hit along the way.

If you’re looking for concrete recommendations, feel free to jump straight to the end.

In many places, I list example solutions. These are not exhaustive lists and there are many other products out there, each with their pros and cons. I picked the examples based on what popped up most often in my user research, and I only list those where people were saying good things. Last but not least, as a founder, I’m always going to be rooting for our own solution, Authress, and it is a good reminder to do your own research based on your exact needs.

Login​

When most people hear “auth”, they think login. Login is authentication, or auth-N for short. It’s the first step of securing your software.

Even if you’re making a small side project just for yourself, as soon as you host it somewhere outside of your machine, you want to implement some sort of authentication. The days when obscurity that saved your little inconsequential personal website or crawler from DDOS attacks are long gone. It’s really simple - if it runs on the internet, it has to require login. Unless it’s just a static website, but let’s not be pedantic.

Good news is, it’s really easy to add login to your software and it will cost nothing if you play it right. Bad news is, it can get really complicated and expensive unless you spend a moment to think about what you need exactly.

Username and password​

Username and password may seem like the easiest, most basic thing to do - just plop a form, maybe use a library to support it, and you have your login. Wrong. This is a trap.

First of all, it puts you on the hook for storing and processing personal identifiable information (PII). That’s because for this to work you need to collect and store a user’s email address. Why? Because sooner or later someone will forget their password, or will want to change their existing one. To facilitate it, you need an email - and just like that, you now have to comply with local regulations surrounding PII: GDPR, CPRA, CPPA, LGPD, PDPA, the list goes on... You really don’t want to do that unless you absolutely have to and have the resources to do so.

If that’s not enough to convince you, there’s still the matter of user experience. Remembering passwords is not fun. If your users are smart, they will use a password manager. If they’re lazy, they’ll reuse their passwords. Either way, you’re adding friction right at the start of your process.

Social login a.k.a. federated login​

When you want to add login functionality to your project quickly, use a social login. This is also known as federated login provider, and it is the way to go. It’s free, fairly painless to configure (although there are caveats) and you won’t have to worry about handling personal identifiable information - the provider will do that for you.

Login with Facebook, Google, Microsoft, Github, …the list goes on and on. Theoretically, all those implement the same standard, OpenID. And theoretically, integrating them into your software will be consistent across all of them. In practice, there are two types of engineers - those who follow the spec and those who get creative. It seems that the majority of federated login providers employ the latter kind. This means that for most of the providers, you’ll have to jump through a few hoops to get it working.

If you stick to a single provider, integration ranges from super easy to manageable, but as soon as you want to support multiple options, it may be easier to use an Identity aggregator or a login box solution. Those usually aren’t free.

SSO (single sign-on)​

If your software is used by other companies, and your users are their employees, you want to allow those users to login via their standard corporate login. There are multiple standards here and each company will have their own authentication format - SAML, OpenID, OAuth, or something custom. And just like the federated login providers, they too are usually creative with their implementations.

It gets complicated real quick, that’s why you probably want to go with one of the identity aggregators or a full featured auth SaaS. If you’re after enterprise customers, you likely have other requirements that mean you really can’t just cobble something together on top of your existing solution. It’s worth paying for the convenience and peace of mind in this case. On the other hand, some solutions will make you pay the enterprise tax for no good reason, so make sure to compare different providers and read what’s really included.

Some key providers in this space are Auth0, FusionAuth, Ory, Authress, each with their pros and cons.

It’s worth noting that this is different than using SSO internally at your own company. You can’t extend your own SSO to your customers. Solutions dedicated to providing access to enterprise employees, such as Google Workspace, Okta or Azure AD (Entra), aren’t the ones that will help you add such functionality into the software you’re building, it’s an entirely different category.

Login box and identity aggregation​

If you know your users will need multiple options for login or you simply don’t want to worry about the whole authentication part, there are quite a few providers that offer a login box you can simply drop into your website. These also act as identity aggregators - they will accept whatever login method you have configured, or your customers want to configure, and return a consistent token that your software can consume directly. Convenient.

Providers include Authress, Auth0, or FusionAuth. There’s also Keycloak if you’d rather self-host and are willing to become an expert in the topic.

When picking a solution, make sure that all the login options your users will care about are supported, that may include things like passkey, WebAuthn, or passwordless login, on top of the most common social logins. Also check if there’s an enterprise tax and whether you have to pay extra for SSO integration. Some providers charge for the number of users and also for the number of customers and each extra SSO integration you add.

There are a few potential gotchas here. Sometimes an aggregator promises support of all the different login providers but when you go ahead and try to integrate, you end up having to do all the hard work yourself. That’s something you can only evaluate through a proof of concept. Most of the open source frameworks fall into this category. There’s usually multiple extra packages, and whole sets of additional configuration to set up.

Another thing to look for is reliability - you need an SLA that’s at least as good as the one you plan on offering yourself. If login is down, your software is down.

What also happens is that some aggregators will make it easier for themselves to integrate with all the different identity providers and use a security standard that’s a lowest common denominator. This means that sometimes an identity provider offers a more secure standard to login, but it’s not utilized by your aggregator. For instance, some providers offer EdDSA token signatures, but most aggregators only provide RS256. It’s not something that’s easy to tell by just looking at their website, as obviously no one will advertise this. If you care about security, you’ll have to poke around documentation and code samples to find out the real story.

Access control​

Now that your users can log in, let’s talk about the hard part - the actual authorization, auth-Z for short. Historically, this was lumped together with login - the scope of what a user could do was determined solely by whether or how they are authenticated. It turns out this is not good enough, especially if your software involves any sort of data sharing. That’s where authorization, or access control comes into play.

Solutions range from classical sticking the claims into a user token, to fully fledged authorization decision engines which can support even the most sophisticated cases.

Sadly, which one is the “easy way” is a strong case of “it depends”.

Simple authorization​

If all your users can do exactly the same things in your software, or if you’re building it just for yourself, the quickest way to secure access is by piggybacking on the access tokens. This means sticking the resources that a given user should have access to directly into the token using claims.

This works when you have a simple access model - users can see and do everything in your software, nasty bots can’t. It may also work if your model is slightly more nuanced, as long as all your users follow the same pattern. It breaks as soon as you have a lot of users, your users can have multiple roles, or your resource hierarchy is a little more complex than a very short list. That’s when simple becomes simplistic. In some cases you’re not even able to rely on this method, as you easily exceed the maximum field or token size.

This is where many login box providers, such as Auth0 and FusionAuth, fall short - they only offer simple authorization as of now. If that’s sufficient for your use case and you’re happy with their pricing for the other features you need, no need to look further.

DIY authorization​

Even 5 years ago, if you wanted something more robust, you had to build your own authorization engine. Everyone who’s ever built software in the B2B space knows this very well. It may seem like a simple thing, just a single database with some simple checks - 5 months later you’re still tweaking and fixing and cursing. It’s a death by a thousand cuts. In the past, there was no choice. Nowadays there are multiple providers you can choose from with much better developer experience, tighter SLAs, more features, and at a fraction of your own cost.

These days, you really don’t want to build your own, unless you’re a masochist and have extra time and money to throw away.

Authorization as a service​

There are many SaaS solutions that offer the authorization decision engine as a service. They will handle all the complexity, reliability, and security, and you only have to map your permissions model to theirs.

With growing awareness on security and data privacy, this is a quickly growing space, and new startups are popping up to solve the problem seemingly every other month. This is good for you as the user, but also confusing, as everyone thinks about it slightly differently and uses different words to talk about it.

There are a few things to consider when picking your solution. The main one is obviously the features - other than providing a structured way to model your permissions, you may also want intelligent integrations with things like machine to machine authentication, permissions for service clients, and login that complements what you were planning on using. Also think whether you want to be running the solution yourself - you probably don’t, unless it doesn’t have to be very reliable, but you may have no choice if you’re working in a highly regulated environment. If you have specific requirements for data residency, also make sure your selected solution will accommodate it.

Another consideration is the ease of getting started and overall developer experience. Is there documentation? How quick can you get support if you have a question? Is support provided by technical people or an offshore customer support center on the other side of the world? Do you even need support, or is everything clear enough for you to have something running within hours?

Then there’s the ease of use once you’ve set everything up. Software rarely stands still and your permission model will evolve. How easy is it to change it once you have hundreds of thousands users in production relying on a given authorization system? Is the model inherently flexible and extensible?

There’s also a matter of pricing. There are a few strategies that providers use. Most common one is based on the number of active monthly users, which also means you can easily estimate how much it will cost you. This is a good approach if your unit of value directly corresponds to an active user. It gets expensive if you offer a free tier, your users use your software only a couple of times a week, or you often run marketing campaigns that result in spikes of new logins that don’t fully convert. A few providers offer metered pricing, similar to how your cloud provider charges you. This is a bit more tricky to estimate your actual cost, but in the grand scheme of things, it usually ends up cheaper than per-user pricing. Some providers will want you to pay a monthly minimum. Most offer some sort of volume discount. If you consider authorization engine a part of your infrastructure, metered pricing is a good fit.

There’s a bunch of options to choose from, some of the more established ones include Authress, Ory, Oso, or Permit.

Internal security​

All you’ve read so far was about application-level security - all the things that make your software secure for your users. When people hear “security” though, they usually think “infrastructure security” - all the things that make your software secure to build and run.

This is a complex topic of its own and only tangential to what this article is about, but auth is an important part of it, so let’s include it for the sake of completeness.

Cloud security​

If you’re hosting your software in the cloud, which is most likely the case when you’re building something new, then you most certainly want to secure that infrastructure, if only to prevent someone else’s bitcoin miners running at your expense.

Luckily, all cloud providers offer plenty of options for you to do so. First thing you want to look at is to use your cloud provider’s IAM to configure access policies for your account. There are many other things you could do for cloud security, so if you work in the cloud a lot, invest in learning about it.

Most cloud providers offer some sort of tool that may look like it will let your users login into your app - AWS Cognito, Entra Azure AD, or Firebase. It may be tempting to try and use these. Thing is, these are good choices if you’re building something for yourself or to be used by your own developers, or family members, as they will be the ones with access to your cloud account. When you need something to handle your end users’ identity, cloud native options just don’t cut it.

API security​

Securing your APIs consists of two main parts. It is analogous to securing your application by forcing your users to log in in order to authorize them.

First part is the machine-to-machine authentication - you don’t want to let just any service call your API after all. Your options are plaintext API keys, mTLS, public/private key pairs, or a combination of them - there are libraries in each major language that should help with these. Or you could use one of the SaaS solutions that handles machine to machine authentication as a first-class concept, such as Authress or Ory.

Authenticating your clients is better than nothing, but if you leave it at that, you’re still open to Broken Object Level Authorization (BOLA) vulnerabilities. This means that if you want your API to be public at any reasonable scale, you need to consider the second part - an authorization layer on top of the authentication. If you’re using a SaaS for access control, see if they offer the same functionality for service clients.

At the time of writing, Authress and Ory are the only providers that do this in a straightforward manner, but the space is evolving rather quickly, so new solutions may have come out by the time you’re reading this.

Secret sharing​

Sometimes you want to securely share credentials (if sharing credentials can ever fall in the category “secure”). You may need to share a login to a particular piece of software, e.g. if you’re paying per seat or for whatever other reason. If so, you’re looking for a password manager for business. That’s what solutions like 1Password or Zoho Vault are for. LastPass is another solution worth mentioning if only for their inability to keep your passwords secure.

Then there’s the separate matter of using secrets by your services in production, to access a third party integration. This is what AWS Secrets Manager and Hashicorp Vault enable. If you’re running your software in the cloud however, there is a more secure way of handling it though, by encrypting your credentials and using your cloud provider’s Key Management Service.

Employee access control​

If you’re a big company, you may want to manage access to all your third party applications that you have licenses for - Figma, Salesforce, Zendesk, etc. You probably also already have an employee directory such as Entra, Google Workspace, Microsoft AD, Rippling or the likes. In that case, you can hook up your directory to something like Okta to let your employees use their single sign on to login into the apps. This is less about security and more about convenience of managing your software licenses, especially if you pay per seat.

Summary and recommendations​

If you’ve read this far, you hopefully have a good understanding of what types of problems exist in securing your software and your user’s data, and what are some of the software products that address them. Here is a visual summary of what all the different providers I mentioned offer:

\"Main

If all the above is too much to read, or you have no time to think about all the nuances, let me make things really simple by providing recommendations for most common scenarios. These are opinionated and don’t include all the intricacies of your individual situation, so use these as a starting point and do your own research.

Side project, personal use​

When you’re building a small side project, or something just for yourself, friends and family, you don’t want to overcomplicate things and you want things cheap, if not free. For login, use one of the social logins, like Facebook or Google. This is a standard OAuth implementation, which you should get familiar with anyway, and most languages have libraries that help.

You want to run the thing in the cloud rather than your own machine - it’s super easy, comes with a lot of security options out of the box, and it costs peanuts.

If you need to restrict access to specific resources (e.g., your mom can delete her own data, but she can’t delete yours), use claims and the simple authorization access control in the access token to save this information.

MVP for public use​

When you want to build a product quickly to validate it, you may need something slightly more robust, especially if you’re going to share it with the wider public. If you can get away with a single social login, great. If not, use any provider that offers a login box cheaply. Authress' or Auth0’s free tier fit the bill.

If you have a monolithic app and don’t need SSO, Auth0 should be good enough. If you are building microservices and you need machine to machine authentication, Auth0 gets expensive real quick. You should be able to work around this using your cloud provider’s Key Management Service and storing your encrypted keys in git.

B2C software​

If you build something intended for consumers (individuals rather than businesses) at any reasonable scale, you shouldn’t cut corners regarding the login experience. Use a well-established login box provider, such as Auth0 or AWS Cognito.

If your software involves any sort of data sharing, you can use Auth0 if you only have one or two simple permissions. For fine-grained access control, use Authress or Ory. The latter two also include login box support. Which one to use depends on how complex your use case is - you don’t want to spend weeks configuring and tweaking if things are simple, but you also don’t want to rewrite your stuff after finding out your initial approach doesn’t scale.

If your software integrates with third party tools, you definitely want to look at Authress or Ory for secure machine to machine authentication and access control.

B2B software​

If your software is meant to be used by companies, regardless if we’re talking about small businesses or enterprises, you absolutely have to support SSO. Some login box providers offer this: Auth0, FusionAuth, Ory, or Authress, although most of them charge an absurd amount of money for the privilege. Still, if you’re selling to enterprises, this may pale in comparison to your other costs, but it could land you in a spot on SSO tax.

Proper access control is essential in B2B, because most companies will have a notion of user roles with different levels of access and different data visibility depending on who’s logged in. If you would like a solution that only handles access control, use Oso or Permit. If you want one solution that also handles sign-on, machine to machine auth, monitoring, and auditing, use Authress or Ory. If, for whatever reason, you really want to host the solution yourself, both Oso and Ory offer that as an option.

Public API​

When your product is an API, you don’t have all that much choice. You want to go with a solution that supports API security as a first-class notion, that is, Authress or Ory. This will give you machine to machine authentication, as well as resource-based access control, to protect yourself from BOLA.

And that's it - all you need to know before adding auth to your project.

", "url": "https://authress.io/knowledge-base/articles/auth-situation-report", "title": "Auth situation report", "summary": "Before adding auth to your project, there is a lot to consider. The auth space is currently very active, with new vendors appearing and old ones repositioning themselves. How to pick the right one? What to search for? What are the common pitfalls? Read to find out.", "date_modified": "2023-05-28T10:00:00.000Z", "author": { "name": "Dorota Parad", "url": "https://dorotaparad.ch" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/2022/09/11/agile-architecture-adding-security-to-your-architecture", "content_html": "


It's easy to make security an all or nothing approach. Many architectures thrive in delivering the perfect product vision while others attempt to deliver something quickly. Often what can get lost is attention to security in a race to deliver the best architecture for the best product. Here I'll discuss when's the right time to think about security and how to do it in a agile way without losing focus on your core.

", "url": "https://authress.io/knowledge-base/articles/2022/09/11/agile-architecture-adding-security-to-your-architecture", "title": "Adding Security to your Architecture", "summary": "It's easy to make security an all or nothing approach. Many architectures thrive in delivering the perfect product vision while others attempt to deliver something quickly. Often what can get lost is attention to security in a race to deliver the best architecture for the best product. Here I'll discuss when's the right time to think about security and how to do it in a agile way without losing focus on your core.", "date_modified": "2022-09-11T00:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/authorization-token-access-token-jwt-myths", "content_html": "

Identity providers solve the issue of identity verification, but never include solutions for CIAM access management. These CIAM providers handle authorization separate from the authentication provider. This separation creates opportunities for confusion, so we'll discuss some of the common misconceptions surrounding JWT authorization below.

JWT access tokens containing roles solves authorization​

These are common user stories, and often can be solved by adding a single string identifier to the JWT access token containing the relevant Tenant ID and the Role (User or Admin in this situation). This can definitely work, although it doesn't scale. If an application has one api, and the number of users that share a tenant is small (for example less than 5), then each user may have a different role and everything works. However as soon as there are more users, more services, or more resources, it starts to break down. Very soon every user will need to have a different role to be able to correctly identify the right permissions.

Further, a role's meaning will be obfuscated as one service understands the roles Admin and User but another also has Editor. Which permissions should the User get and to which service. Users and services trying to understand access get frustrated or worse incorrect grant access to secured resources.

It's easy to add resource access to a JWT​

One way to solve role confusion is to include resources also in the JWT, User X has Y role for Z resource. But this doesn't scale. JWTs are limited in size, and more than a couple of resources will break the JWT usage.

Another critical problem are permission changes. Often, when a user should receive access to a resource, they need it immediately, it can't wait. JWTs expire on the order of hours. This means new permissions, if stored in the token, and removed permissions won't take effect until after the user gets a new token. No identity provider has a way to expire tokens, this isn't how JWTs work. Having a CIAM authorization service separates the roles the user has from the token, allowing both to freely change as necessary.

Making calls to an authorization service is too expensive​

Having the roles in the token is easy, it's quick, and it's cheap from an external service standpoint. But only short term. It actually isn't expensive to create a service to handle all these requests, but it expensive to manage it. Using a CIAM authorization scales as your usage does. Additionally it separates the concerns about the user identify from the access control. CIAM services are fast because they perform the access checks directly knowing about resources, whereas identity providers have no knowledge of how resources are organized.

While running an AuthZ CIAM service is subject to the difficulty of successfully building an authorization service (Think Build), it might be considered expensive to reach out to an external service at runtime (Think Buy). Most present day architectures already separate services, external authorization services fit perfectly into existing tech stacks and support migrations to other architectures such as microservices. With out of the box caching and opportunities for client side optimizations authorization calls are fast and reliable.

Migrating authorization to an external service is costly​

The critical point for migration to an authorization service is at the time of rearchitecture. Legacy services still need to be supported but new service paradigms are being created. Existing services and new services need to be able to successfully depend on always available apis and easy to integrate with access control interfaces. The migration is the easy part however, since your application has already done the difficult part of deciding what features are necessary and modeling the services and their permissions and roles, this directly translates to to Authress' resources, roles, permissions, and access records without additional work. Authress also provides migration wrappers and concierge migration support to ease transitions to or from any other system, because it focuses on abstracting the complexity away. For example, with using AWS, Authress provides an EventBridge integration to enable crossing the cloud gap.

Identity Providers handle authorization because they create authorization tokens​

Probably the most confusing concept is what identity providers actually do. Identity providers often suggest they create authorization tokens. These tokens are JWTs, which should be used as Bearer Tokens in the Authorization header of an HTTP request. This is true, what isn't true is that there is any information about what access this token provides. A token represents an identity, the data in the token about the identity is very slow changing, and that's it. These tokens are verifiable via public key certificates.

So, where's the authorization part? The tokens are used as the identity for services to authorize users. That is, the Authorization token is an identity, used to authorize the user. They say This is the token to be used for authorization.

Aren't scopes usable for access control?​

No. Scopes represent the permissions an identity gives to the holder of the JWT access token, and Not the permissions the user actually has. A scope can say: profile--meaning that the holder of the token can have access to the user's profile. But a scope can't say access to resource X, because scopes are directly controlled by the user, the user could elevate access by telling the identity provider that I approve access to resource X without every having it. If you use scopes for permissions then you are let your users control what they have access to themselves. An example of this attack against vulnerable services, can be seen using AWS Cognito.

The token generator has no idea if the user has access to resource X to be able to delegate it to the holder of the token. If it did that, then you would be leaking access to resources, since scope properties are directly controlled by the user. Scopes must always represent resources that a user has access to a priori and not ones that are configurable for access via an authorization rules such as ACL, RBAC, ABAC, or Resource-Based.

Authorization checks are always necessary​

Not every resource in your api needs to check for permissions. Only shared data, data shared between users, but is not public needs to have checks. Even in those circumstances, users with access to only one tenant can depend on less complex checks. In cases such as social media where everything is public, or a game and streaming service where everything is private, there is no reason to have authorization checks.

The total cost of ownership (TCO) of permissions management is low​

The initial set of features for a CIAM service is small. It is only used by a couple of services, with shared context about the expectations. But this doesn't last long. The average time for a CIAM system to become legacy tech debt is 6 months. At the sixth month mark CIAM systems become a source of pain for users, time sinks for development teams, and fountain of vulnerabilities as they lack the attention and dedicated resources needed to maintain them. This is a large topic, so there is a dedicated article on building your own authorization service.

CIAM is a feature of an identity provider​

Most identity providers support configuring some sort of roles, unless you are using a dedicated provider or a social login provider. But these barely solve even 1% of access control needs. That means using an identity provider and a CIAM provider. These don't have to be the same, and frequently they aren't. Depending on the needs of the product or application, it makes sense to fully evaluate the existing identity providers to pick the right one for your requirements.

The roles users have will never change​

When roles are defined in the JWT access token, they can't be allowed to change. That's because tokens are immutable and frequently there is no way to get another one until the current one expires even if the user logs out and back in. And in technology everything changes. So if these roles are not static, if they are user editable then putting roles in the token will cause issues for your application or platform. For more information on issues with RBAC in general, check out the security article on Choosing an access control strategy.

", "url": "https://authress.io/knowledge-base/articles/authorization-token-access-token-jwt-myths", "title": "JWT access token misconceptions", "summary": "Tackling the myths of JWT roles at scale, what scopes are, and are CIAM providers expensive?", "date_modified": "2021-08-10T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/security-of-long-running-transactions", "content_html": "

There are many articles dedicated to the OAuth access token exchange for user login, and then using that token with an API. While there are complexities around how to get the token from an authorization provider, there are many articles and examples which can help. This is the simple case.

When there is one user, one UI, and one service, things are simple. Access control is still necessary, roles are still necessary, permissions configuration is still necessary. But the user is logged in, access tokens are valid, and responses are quick and synchronous. This happens rarely, more frequently, some part of the requests your application has will be long running or executed when a user is offline. Some of these are as simple as a email distribution for new features. In those cases, service client api keys can be used to allow one service to authorize another service. These actions are performed in the context of a calling service.

Long running transactions​

However, when service access needs to be done on behalf of a user, then the service needs to have a valid, non-expired access token. And this token may need to be valid, days, weeks, or months later. To make sure a request only accesses the appropriate resources, these should be done as the service impersonating the user, and as a matter of fact, the calling service may not even have direct access to the resources owned by another service client. This happens more frequently when the data is owned by a third party, or a secure area of your application platform, and it isn't enough to have your authorization provider (which may or may not be Authress) since that third party won't know about your authorization capabilities.

The authorization code flow​

Using the OAuth flow a service can receive a long lived refresh token a secured service can use to reauthenticate with a third party api and allow it to make subsequent authorized calls.

\"long

Extending the model to microservices​

When there is only one service that needs access to these refresh tokens, a simple database with user <=> refresh token can work. Since there is only one refresh token, and every new refresh token requires a new user interaction, only one service can manage these tokens. In many cases there is more than one service, and more than one of these needs access to these tokens. In those cases, having a central Credentials service can reduce the friction. However, this introduces the need to secure that service. In those cases, it is necessary to create a Credentials resource scoped to the user, which would allow other services to request access to these refresh tokens.

Create the necessary role​

Create the necessary credentials related roles.

\"creating

Assign the role to the relevant clients​

Then, assign the the new credentials related roles to the appropriate internal clients.

\"assign

Credentials access check​

Finally, when a request comes in to access these refresh tokens or other credentials make the appropriate access check:

Get a user's third party access token
import { AuthressClient } from '@authress/sdk';
const authressClient = new AuthressClient({ authressApiUrl: 'https://auth.yourdomain.com' });

async function getUserCredentials(serviceClientCallerId, userId) {
  // Check that caller has access to the relevant credential for a user
  await authressClient.userPermissions.authorizeUser(serviceClientCallerId,
    `/user-credentials/refresh-tokens/users/${userId}`,
    'credentials:read');

  // If the caller has access then lookup the credentials
  const storedRefreshToken = await GetRefreshTokenForUser(userId);

  // Get an access token for the refresh token
  const accessToken = await GetAccessTokenFromRefreshToken(storedRefreshToken, {...});
  return {
      statusCode: 200,
      body: { accessToken }
  };
}

Going further​

Building the credentials service is easier said than done though, while the permissions is a huge part, security of the data the is even more important. Security of the endpoints as well as encryption of the data (at rest, in transit, and in memory if not actively being used). For this reason to make it simple, Authress provides automatic credential handling of refresh tokens through any configured Authress OAuth provider connection.

", "url": "https://authress.io/knowledge-base/articles/security-of-long-running-transactions", "title": "Handling security of long running transactions", "summary": "A deep dive into security of long running offline transactions using refresh tokens and service client tokens.", "date_modified": "2021-07-23T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/aws-gitlab-cicd-login-authentication", "content_html": "

For eons there have only been three ways to \"secure\" your build platform or servers. All of them have been historically bad, for different reasons:

tip

Now you can stop using access key and secret today!​

The solution​

The perfect solution is if GitLab could directly authenticate with AWS and you give gitlab access directly to the resources it needs, in the context of your job. And now it exists.

Gitlab generates signed JWTs that you can use with AWS to get temporary access tokens. (You don't need any complexity brought by Hashicorp's Vault, you'll notice this works even without it.)

Gitlab's tokens look like this:

gitlab-jwt.json
{
  \"iss\": \"gitlab.com\",
  \"iat\": 1585710286,
  \"nbf\": 1585798372,
  \"exp\": 1585713886,
  \"sub\": \"job_1212\",
  \"namespace_id\": \"1\",
  \"namespace_path\": \"mygroup\",
  \"project_id\": \"22\",
  \"project_path\": \"mygroup/myproject\",
  \"user_id\": \"42\",
  \"pipeline_id\": \"1212\",
  \"job_id\": \"1212\",
  \"ref\": \"auto-deploy-2020-04-01\",
  \"ref_type\": \"branch\",
  \"ref_protected\": \"true\",
  \"environment\": \"production\"
}

Now it's just a matter of setting up AWS to accept that token and allow it to generate an STS token.

We'll be using AWS IAM's AssumeRoleWithWebIdentity to convert the token into an identity

1. Create the Authress Gitlab connection​

Navigate to the quick set up in Authress, add a custom OAuth connection, and then swap the the incoming JWT with one that AWS will accept.

Create the connection​

\"Set

2. Add an AWS IAM identity provider​

Create Identity Provider, selecting OpenID Connect as the type. Specify the details in AWS that matches the Authress connection. The provider url will be your account domain from the connection configuration and the AWS Audience will be the connectionId from Authress. If you are using a custom domain with Authress, this will be the custom domain instead:

\"AWS

3. Create a role for that identity provider​

Assign Role:\n\"Create

Important: Update the Trust Policy to restrict the valid token sources (make sure to replace AUTHRESS_ACCOUNT_ID.login.authress.io with your account's identity. If you are using a custom domain, this should be the custom domain value same as from step #2):

gitlabRunnerIamTrustPolicy.json
{
  \"Version\": \"2012-10-17\",
  \"Statement\": [
    {
      \"Effect\": \"Allow\",
      \"Principal\": {
        \"Federated\": \"arn:aws:iam::AWS_ACCOUNT:oidc-provider/AUTHRESS_ACCOUNT_ID.login.authress.io\"
      },
      \"Action\": \"sts:AssumeRoleWithWebIdentity\",
      \"Condition\": {
        \"StringEquals\": {
          \"AUTHRESS_ACCOUNT_ID.login.authress.io:aud\": \"AUTHRESS_CONNECTION_ID\"
        },
        \"StringLike\": {
          \"AUTHRESS_ACCOUNT_ID.login.authress.io:sub\": \"gitlab-runner|GitlabGroup/*\"
        }
      }
    }
  ]
}

4. Call AWS STS at build time to get your credentials​

Exchange the gitlab CI_JOB_JWT token using Authress and then call AWS STS. Here's a javascript example which sets the AWS SDK credentials appropriately:

Exchange incoming token for an Authress signed JWT
import axios from 'axios';
import { WebIdentityCredentials, config } from 'aws-sdk';

// Update these values to match your environment
const authressHostUrl = `https://${AUTHRESS_ACCOUNT_ID}.login.authress.io`;
const authressConnectionId = 'CONNECTION_ID';
const iamRoleName = 'AWS_IAM_Gitlab_Runner_RoleName';

const tokenResponse = await axios.post(
  `${authressHostUrl}/api/authentication/gitlab/tokens`,
  { client_id: authressConnectionId, grant_type: 'client_credentials'},
  { headers: { Authorization: `Bearer ${process.env.CI_JOB_JWT}`} }
);

config.credentials = new WebIdentityCredentials({
  WebIdentityToken: tokenResponse.data.access_token,
  RoleArn: iamRoleName,
  RoleSessionName: `GitLabRunner-${process.env.CI_PIPELINE_ID}`,
  DurationSeconds: 3600
});

For more information on the AWS credentials configuration, see the AWS AssumeRole CLI Docs.

FAQs about GitLab configuration​

Are the connectionId and api url sensitive data?​

While it's a good idea to obscure the connectionId, nothing here is sensitive other than the relevant JWTs. That's the whole point of this authentication mechanism, it removes the sensitive material. It's still good to avoid sharing the AWS account ID as well, as it would allow attackers to call your accounts API and potentially cause DDoS on your infra. But it's still 100% secure, if you have the need to keep these values in the clear.

ErrorCode: Unauthorized​

You'll want to revalidate that the token being sent to the Authress Login endpoint is correctly specified. If you aren't sure your code is correct, test out using Request Inspector or ngrok by replace the Authress endpoint call. You'll be able to directly evaluate the data being sent there.

", "url": "https://authress.io/knowledge-base/articles/aws-gitlab-cicd-login-authentication", "title": "AWS + Gitlab - Leveling up security of your CICD platform.", "summary": "Stop using aws access keys and secrets today!", "date_modified": "2021-06-14T10:00:00.000Z", "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/breach-enabling-emergency-data-protection-case-study", "content_html": "

The importance of data security has not been left off anyone’s radar. And, in the wake of unauthorized access to the US Capitol building the approach for some is to wipe everything. Potentially malicious attackers on premise, able to access user data and user sessions left unlocked. The historical lack of sufficient technical experts in leading government areas have left reasonable controls out of the picture. The lack of attention to data security in some of the most critical areas results from a number of antiquated mindsets and overall deficit in talent.

The situation​

However, many companies also find themselves, while not in possession of highly classified and confidential materials, equally sensitive and vulnerable to unauthorized access. Security experts will point to increased availability of cloud solutions with a bifurcation of security responsibility \"Of the Cloud\" and \"In the Cloud\". Cloud usage helps, that’s no longer a question, but it's only the beginning of the story. As an organization you are in control of your data, your security, and possible remediation strategies when there is a critical incident. But how will you help your users tackle these issues when something happens to them outside of your control?

A quick browsing through social media will expose just how many possible vulnerabilities where in place:

Having the appropriate security controls in place is important for a variety of reasons. While not everything protects your users' data inside your cloud solution, some of it helps protect your users' digital identity. We’ll iterate through the available options below to see what additional controls can be implemented.

Physical Management​

The first layer is everything physical and connected to a network. This is known as endpoint management. Users log in to physical devices in a dedicated location or one that is treated as a zero-trust network. These endpoints need to offer basic protection:

User Authentication​

Once your users are past their device login, network and internet access have further protections. Many apps in 2021 still lack some of these basic protections, so when choosing what to use ensure that they follow these guidelines.

Many authentication SaaS providers have at least some of these features if not all of them. Building a custom login solution will always be less secure unless it contains at least these mentioned criteria.

User Authorization​

Authorization or identity access management (CIAM) is critical for securing user connected applications. As we’ve seen above there are important areas missing from physical and authentication security that need to be closed.

Conclusion​

Using these different pieces together is the only way to ensure security amid data emergencies and not have to resort to drastic measures to clean up. Combining physical management with federated user login and authorization apis gives a full working security approach to almost any digital infrastructure. The solutions for teams and organizations that are responsible for these resources are available, all it takes is simply to implement them.

Interested in figuring out which auth and physical data security tools, checkout how to pick the best auth solution.

", "url": "https://authress.io/knowledge-base/articles/breach-enabling-emergency-data-protection-case-study", "title": "Breach - Enabling emergency data protection", "summary": "In the wake of unauthorized access to the US capitol building presents a unique cybersecurity opportunity to reexamine best practices in this data security case study.", "date_modified": "2021-01-08T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/how-to-pick-best-auth-solution", "content_html": "

You’re considering using an auth product like Auth0, Okta, Firebase, or Cognito and want to pick something that fits your situation best. You start searching for alternatives and are quickly swamped by conflicting sources of information. I know this pain. Let me help you a little.

When looking for solutions in the auth space, your main obstacle will be the conflation of terminology. Products with entirely different functionality are marketed using the same words. More words means more search keywords, which may be great for SEO, but doesn’t help us poor users.

Let me give you an example. A solution that helps reduce login friction for your users looks completely different than a solution to restrict your app functionality based on user roles. You may care about one and not the other. Yet, products that are strong in the latter are described almost the same as ones that mainly deal with the former.

It doesn’t help that most product websites suck. They suck because they’re made by marketing people for not-so-technical decision makers. Try to explain the difference between authentication and authorization to a marketing person… They sound similar, so must be the same, right?

Then there are all those comparison sites listing SaaS products and their competitors. They may give you some ideas on who the players are in the space but fundamentally suffer from the same problem - they lump different things together. This problem becomes immediately obvious when you see LastPass in the same list as Auth0.

Let me tease all these apart and show you the most important categories in the space, so that you’re better equipped to do your own comparisons. My goal is to give you vocabulary to go out and search for solutions that meet your criteria and a decent starting point in terms of existing products.

You want to make logins more secure​

Main keyword: authentication.

You’re not an expert in security and don’t want to deal with the complexities of sensitive data storage, multi factor authentication, or preventing data leaks.

What to look for​

There are two crucial things here: security and reliability. Duh, but what does that mean? Forget about all those shiny certification badges you see on websites - they don’t mean anything. You want to look at reviews, perhaps talk to the sellers to see how they approach it. Look at their SLAs and if they have a website showing their current service status.

Example solutions​

You want to combine identities across apps or platforms​

Main keyword: single sign on (SSO).

Your users may login using different mechanisms: their company’s active directory, common federated login providers, their social media account. You want to allow them access regardless of what login provider they use, and don’t want to write separate code that handles all these providers.

What to look for​

This is an area that can get very expensive quickly. Scrutinize the pricing and see what’s actually offered. See how easy it is to set up a new connection - some solutions out there boast they integrate with X in their marketing materials, but then you find out you have to pretty much write all the code yourself to handle the connection. Also pay attention to the authentication standard used in each integration. Sometimes, identity providers support higher security standards than the aggregator uses to connect. You want PKCE and not implicit flow authentication here.

Example solutions​

This is an area where you can find the most products. Many solutions are good, but the devil is in the details. Are you trying to create a single entry point for your internal corporate users, or are you writing an app that should integrate with different login sources? Some products can do only one, others do both.

SSO for enterprise employees:

SSO for the users of your app or B2B solution:

You want to manage user access and roles​

Main keyword: authorization.

You have shared resources in your application or different groups of your users can perform different actions. You want your user’s data to remain secure without adding friction for shared resources.

What to look for​

You want granular permissions for access control, otherwise you’ll find yourself hacking workarounds into the system. Consider how many hoops you’ll need to jump through to change the assigned permissions or roles. Ideally, you want a nice administration dashboard that gives you an overview of what’s going on and lets you configure business-relevant parts without calling your development team. Also needs to support fetching permissions outside of user tokens.

Example solutions​

This is an area where you won’t find many ready to use solutions. Some of the authentication providers do a bit of authorization, but it’s usually clunky.

You need to share credentials between your software modules and services​

Main keyword: machine to machine authentication.

Your various services, clients, and other parts of your software need to act on behalf of your users or simply communicate with each other on the public internet. You don’t want to expose yourself to security breaches. Additionally, your services may need to also connect with third party services using OAuth tokens.

What to look for​

Pay attention for Credential Vaults and for Machine to Machine authentication, these are the two important pieces.

And for some reason, this is an area where pricing differs dramatically between different providers. You may be tempted to think that more expensive solutions offer better functionality, but that’s not the case here. Why? Who knows. Pay attention how many machine to machine tokens are offered for what price, and how easy it will be to scale that number. Publicly verifiable tokens are a must.

Example solutions​

One stop shop solution​

You wish it existed, don’t you? You want one provider to solve all your auth needs - current and future. One vendor to deal with, so you don’t need to make dozens of decisions and deal with your software purchasing process more than necessary. One solution to rule them all.

I have bad news for you. It doesn’t really exist in the auth space, at least not yet. There really isn’t any single product that fulfills all your auth-related needs. Some vendors, such as Auth0 or Authress are trying to get there, but neither are truly comprehensive at this point.

Rather than looking for a one stop shop, I’m afraid you’ll have to consider what’s most important for you and pick a product that checks the most boxes, possibly augmenting it with another one. There really isn’t any other way.

Overall considerations​

Besides functionality, there are other factors to consider when picking an auth solution provider. Take these into account once you’ve picked a few providers that offer all the features you want.

Ease of onboarding and migration​

How easy is it to start using the solution? Do you have to rewrite all your existing logic to do that? How strong is the vendor lock in once you start using the product - will you be able to change solutions without a massive headache in the future? It may not be easy to answer these questions without some serious digging. Good documentation, or better yet - API endpoints to play with, can give you a sense of how much effort could be involved in the migration.

Documentation and support​

How good is the documentation? Is it easy to access? Will your engineers understand it or will you have to contact support with every little question? How accessible is the documentation? Are there product usage related questions + answers on Stack Overflow or similar platforms? How good are the answers? If you contact support, will you talk to an engineer or a salesperson? These topics aren’t always the first priority when you’re looking for a product, but they’re well worth considering. Your engineers will thank you for that.

SLAs​

For SaaS products running on the web, are their SLAs published anywhere? How good are they? Do they fit your use case (it makes no sense to pay for 99.9% uptime if your users only log in once a week)? If you’ve decided for a self-hosted or self-managed solution, it means your team is on the hook for reliability and uptime of the service. How good of an SLA can they provide? Will you need them to debug and fix server problems in the middle of night or on the weekend?

Overall cost transparency​

Some products list pricing on their websites. Others require you to go through a salesperson to even begin estimating your overall cost. When talking about self-hosted solutions, you have to add the cost of maintenance by your own staff on top of any licensing fees. All of it makes it harder to compare different offerings.\nFrom my experience, if the price is not listed on the website, you’ll most likely end up paying more than you need to, unless you have access to a skilled negotiator used to playing this game with vendors.\nAlso from my experience, self-hosted solutions seem cheaper at first, but end up more expensive in the long run. People tend to discount the amount of effort that goes into running a server smoothly, but it’s a real cost that adds up over time.

", "url": "https://authress.io/knowledge-base/articles/how-to-pick-best-auth-solution", "title": "How to pick the best auth solution", "summary": "Finding an auth provider that suits your needs is tricky. Product pages are loaded with SEO keywords that mean little in practice. Comparison websites can't distinguish between authentication and authorization. Let me help.", "date_modified": "2020-11-16T10:00:00.000Z", "author": { "name": "Dorota Parad", "url": "https://dorotaparad.ch" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/how-to-verify-jwt-in-web-app", "content_html": "

Securing a web application or api requires actually validating the access token that is being used. When using JSON web tokens (JWTs), there are two mechanisms for doing this. But the core of the solution requires inspecting that JWT, understanding who the authority is, and using that authority for verification.

The properties or fields in a JWT are called claims. JWTs contain an ISS claim. This is the Issuer. The issuer is the authorization server (AS) which is marked by the issuer. As such the AS provides a full document about how JWTs are constructed and how to verify them. This document must always be found at https://${Issuer}/.well-known/openid-configuration (according to RFC 8414). Here’s a real life example.

What’s more is that the openid configuration may inform you of an introspection endpoint. By passing the token there the AS will tell you if the token is a valid one. However, not only is this optional, it is expensive since the results can not be cached. If you are verifying 1000s of tokens per second, it is far too prohibitive to verify them like that.

JWTs are signed, that means they have a signature which allows them to be verified. With the signature, they also contain a kid which specifies which public key was used to sign the token and create the signature.

A better alternative is to use the issuer, kid, and signature to verify the token. To do this get the relevant JSON Web Keys (JWK). Use the issuer to get the keys, find the right key using the kid, and then verify the signature using the key. Therefore keys allow you to self verify the token much faster and the keys themselves are cacheable, that means that you can avoid frequent API calls by using the JWKs. This still has a similar problem as the introspect endpoint; that is you are implicitly trusting the issuer. So step one becomes:

After that, verify to token, just break open the token grab associate JWK and verify the signature

info

Note: This is a generic authorizer. For Authress specific verifiers, see the verifying JWTs

Validate request JWT
import axios from 'axios';
import { jwtVerify } from 'jose';
import jwkConverter from 'jwk-to-pem';

const ISSUER = 'https://login.authress.io';
const PUBLIC_KEY_URL = `${ISSUER}/.well-known/openid-configuration/jwks`;

class Authorizer {
  async getUser(request) {
    const authorization = Object.keys(request.headers).find(key => {
      return key.match(/^Authorization$/i);
    });

    const token = request.headers[authorization] ? request.headers[authorization].split(' ')[1] : null;
    if (!token) {
      throw Error('Unauthorized');
    }

    const unverifiedToken = jwtManager.decode(token, { complete: true });
    const kid = unverifiedToken && unverifiedToken.header && unverifiedToken.header.kid;
    if (!kid) {
      throw Error('Unauthorized');
    }

    const issuer = unverifiedToken && unverifiedToken.payload && unverifiedToken.payload.iss;
    if (!issuer) {
      throw Error('Unauthorized');
    }
    const key = await this.getPublicKey(PUBLIC_KEY_URL, kid);

    try {
      const pemKey = await importJWK(key);
      const options = { algorithms: ['EdDSA'], issuer };
      const verifiedToken = await jwtVerify(token, pemKey, options);
      return identity.sub;
    } catch (exception) {
      throw Error('Unauthorized');
    }
  }

  async getPublicKey(jwkKeyListUrl, kid) {
    if (!this.publicKeysPromises[jwkKeyListUrl]) {
      this.publicKeysPromises[jwkKeyListUrl] = axios.get(jwkKeyListUrl);
    }

    try {
      const result = await this.publicKeysPromises[jwkKeyListUrl];
      const jwk = result.data.keys.find(key => key.kid === kid);
      if (jwk) {
        return jwkConverter(jwk);
      }

      // If the public key isn't found it could be because this token was signed with a new public key, so try fetching a new version
      const retryResult = await axios.get(jwkKeyListUrl);
      const newJwk = retryResult.data.keys.find(key => key.kid === kid);
      if (newJwk) {
        this.publicKeysPromises[jwkKeyListUrl] = retryResult;
        return jwkConverter(newJwk);
      }

      // Otherwise this is an old jwk, so fall through and throw Unauthorized Error

    } catch (error) {
      // If there is a problem looking up the keys, we have no choice but to return a 401 to the caller.
      // * It's possible that there was a problem connecting to the jwks endpoint. In those cases, adding automatic retries to the HTTP calls here, is recommended
      // * If the retries don't work, and there is still a problem, return a 401 to the caller.
    }

    throw Error('Unauthorized');
  }
}

module.exports = Authorizer;
", "url": "https://authress.io/knowledge-base/articles/how-to-verify-jwt-in-web-app", "title": "Validating JWTs in Web APIs", "summary": "Securing a web application or api requires actually validating the access token that is being used. When using JWTs, there are two mechanisms for doing this.", "date_modified": "2020-11-08T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/creating-a-multitenant-application", "content_html": "

A Multitenant application​

Multitenancy is the concept that your application serves distinct non-overlapping accounts, with resources assigned to and belonging to each account. A simple example of this is an off-line console video game. Each game copy is bought and paid for by a single owner (“account”), and in that copy there may be some amount of gameplay saved data, development data, and user configuration which is unique and sequestered to that copy. In some cases that data may need to be synced with other copies that the user owns. For example a PC version or potentially a status update on their mobile device.

On the opposite side of the spectrum there are open communities where no data is restricted to the users, a simplified version of Twitter is a good example. Tweets are public, anyone can read them. While a user may be able to post new tweets and delete ones they posted, the environment that user sees is a combination of all users on that platform.

In these extremes, access control policies aren’t that meaningful, since users either have access to nothing (except their resources) or everything. It becomes required and thus more challenging, when resources are partially shared. Take a photo sharing application. While photos may be owned by a single account, a user may want to share their photos with their friends. Additionally they could create albums which their whole family can see, or public ones they intend to demonstrate their photography prowess.

Users create separate accounts which have their own data, own configuration, but may need to share that data with users in that account, or users in other accounts. This is at the core of what is considered a multitenant application.

Application resources​

If we use a document repository example we can start to build out what the recommendations might be to secure that data. To start off we’ll look at the types of actions users may want to perform, these are often called user stories.

This is just some of the functionality necessary to be built into a document repository. While it is trivial to save documents (depending on their size), getting the permissions right so that they are both easy to maintain but can also provide the necessary flexibility for your users to control access they want to is difficult. We’ll walk through one possible implementation below.

Modeled Actions​

  1. Create Account - Let’s assume no restricted access is necessary here, anyone can create an account.
  2. Invite User to Account - Perhaps something like
    1. Resource: accounts/{accountId}/users
    2. Permissions: AddUser/InviteUser, RemoveUser, ReadUsers
  3. List accounts - I’ll assume accounts are private. A user can only list accounts they are part of, potentially if you share a document a user might need to be able to see some aspects about that account that owns the document:
    1. Resource: accounts/{accountId} and accounts/{accountId}/info
    2. Permissions: updateName/Description, ReadAccount
  4. List documents - Need to be able to list the documents in the account
    1. Resource: accounts/{accountId}/documents/(documentPath)
    2. Permissions: AddDocument, DeleteDocument, ReadDocument, EditDocument
  5. List users - Need to be able to list users that have access to a document
    1. Resource: accounts/{accountId}/documents/(documentPath)/members
    2. Permissions: ShareDocument, RemoveAccess, UpdateAccess, AssignDocumentOwner

These resource uris are a good match for our user stories about a document repository. For these we would create the relevant roles. So far we listed out the permissions, since the permissions are checked by services we’ll want role abstractions that contain these permissions for the resources.

Relevant Roles​

Relevant Resources​

Standard Multitenant Resource Recommendations​

It can be difficult to get the resource paths just right for your application. What’s important is matching up the user stories to necessary access control checks. Since Authress provides scoped permissions and resources, the best recommendation are resource uris that looks similar to the following:

NS:tenants/{tenantId}/parentResources/\n{parentResourceId}/resources/{resourceId}/\nsubResources/{subResourceId}

(Another example exists in the Zoom Case study)

In some cases resources are very fluid and too much scoping can be a problem, but in Authress these can be changed by updating access records and a simple migration can be used to propagate them if the access control model needs to be changed.

", "url": "https://authress.io/knowledge-base/articles/creating-a-multitenant-application", "title": "How to secure a multitenant application architecture", "summary": "How to create and secure an application where multiple users share one account, are part of multiple organizations, and interact with other accounts.", "date_modified": "2020-07-12T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/choosing-the-right-http-error-code-401-403-404", "content_html": "

Selecting the right response to API requests helps secure your application. While it may not seem so on the outside, every unnecessary piece of information makes it easier for an attacker to understand how to gain access. And on the flip side every piece of missing information makes it harder for a consumer of your API to understand the response to an HTTP request.

Here we’ll break down the most common HTTP error responses used for the purposes of API security. When a request is successful, that means that:

Then a successful response status code can be used => 2XX

Fundamentally there are three relevant error codes: 401, 403, 404.

401​

401 - indicates that the request has not been applied because it lacks valid authentication credentials for the target resource. The user is not authenticated. (See here for more information on the difference between authentication and authorization). The API requires a valid user, this is determined by the Authorization header in the request. A 401 is the right error code when:

403​

403 - indicates that the server understood the request but refuses to allow it. The action the user wants to take is forbidden. The user attempted to perform an action, but the token that identified the user does not have sufficient permissions to do that. It’s really helpful to return the permissions that the user is missing so that they can go request them from an admin. Google presents to you a page like this:

\"Google

404​

404 - indicates that the server can't find the requested resource. Links which lead to a 404 page are often called broken or dead links, and can be subject to link rot. If the url path otherwise known as the resource doesn’t exist, then a 404 is appropriate.

Picking the right error code​

That seems easy to handle, and most of the time, they go in order:

  1. Validate Token
  2. Verify user permissions
  3. Check resource existence

Sometimes 2 and 3 happen in the reverse order depending on what’s easier for the application server. However, when you do this you could be opening your resources up to exposing too much information. What happens if a user doesn’t have access to a resource and that resource exists? 403 seems reasonable. What happens if that resource doesn’t exist? 404?

In that case, without access to a resource, you return different information based on whether that resource exists. A user without permission can start scanning all your endpoints and potential resources searching for existing ones. If those resources could be publicly shareable like Zoom sessions, you could end up with more Zoombombing on your hands.

Instead we should break down 403’s into more nuanced categories. If a user knows about a resource and doesn’t have permission then return a 403. You may still want to share the missing permissions or two to request access from. However, if the user shouldn’t know about the resource, then neither return who to contact, the missing permissions, nor a 403 suggesting that the resource exists. In this case return the 404.

A concrete example is what Authress provides when authorizing users. Your application has resources. Access to those resources is stored in Authress access records. Access records are account specific. If an Authress user from another access asks to read one of your accounts access records, they get a 404. If they ask to see the account info, 404. If they attempt to get a list of users with access to a resource, 404. Under no circumstance do they get any other than a 404, except if they:

And that’s about it.

Returning a 403 is great for your actual users, but can start to expose important details of your application. Don’t let potential attackers know which resources exist.

Going further​

Want to see how Authress uses error codes to ensure security? Checkout the Authress Management Portal - API section.

", "url": "https://authress.io/knowledge-base/articles/choosing-the-right-http-error-code-401-403-404", "title": "Choosing the right error code 401, 403, or 404", "summary": "Here we’ll break down the most common HTTP error responses used for the purposes of API security.", "date_modified": "2020-07-01T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/authn-vs-authz", "content_html": "

You frequently hear authentication and authorization listed together in one breath. Often, people conflate these terms and simply say “auth”. It’s no wonder that some of us confuse the two. If you care about security of your application and privacy of your users, it is important to know that there’s a difference between these two terms though. So what is the difference and why does it matter?

When speaking about authentication (also called AuthN), think about identity. Authentication tries to answer “is this person who they say they are?” It’s a software equivalent of a passport or national ID check. Or to put it in more realistic terms, authentication is a similar process to that moment when you look at another person’s face to recognize that this is your friend from college and not your annoying second floor neighbor.

On the other hand, authorization (also called AuthZ) is all about permissions. Authorization answers a question “what is this person allowed to do in this space?” You can think of it as your house key or office badge. Can you open your front door? Can your annoying neighbor enter your apartment at will? And more, once in your apartment, who can use the toilet? Who can eat from your secret stash of cookies tucked away in your kitchen cupboard?

While related, authentication and authorization are entirely different problems. It may be tempting to lump them together, but that usually results in one of them being neglected. Sadly, the part that tends to be neglected the most is the authorization. It’s easy to see why.

What does it mean in a software context?​

In software, you can’t really perform authorization without prior authentication. Unless every user can do everything and everything is public, you have to be able to distinguish between the users. That’s why it starts with authentication. Authentication is a very visible process - you get a login form, or a button to use a federated identity provider (login with Google, Twitter, Office 365, etc.) And since authentication is the first step that’s very visible, it gets a lot of attention from developers. When you’re used to thinking about AuthN and AuthZ simply as “Auth”, after you have spent a considerable amount of time ensuring that users can log in securely, it’s tempting to think you’re mostly done with your “auth”.

In reality, that’s where the actual work starts. Once you know that the user is who they say they are, you need to consider what they are able to access and what operations they can perform. That’s authorization. Where authentication drops off (user is logged in), authorization picks up and continues as long as the user interacts with your software. There are a lot of intricacies involved in this process. How many different resources are there? Are they treated the same or different? Can users share resources? If so, who decides what’s shared and what isn’t? What happens to a shared resource when it gets deleted by the owner? The list goes on. If you don’t consider authorization as its own, separate class of problems, your users will encounter a lot of edge cases that may or may not result in a PR disaster for your company.

If you don’t tease authentication and authorization apart, you run into a risk of building a lot of inconveniences if not bugs into your software. Depending on what your software does, either AuthN or AuthZ may be more important. It makes sense to design your solution with full awareness which one is which.

Authorization in practice​

Let’s illustrate this with an example. You’re in charge of security of an apartment complex. Your task is to ensure that people can access their apartments and no unwanted visitors can do the same.

If your first thought is authentication, you will likely post a guard at the entrance, asking everyone to show their ID in order to get in. Now that you have a guard, you will give the guard a list of all tenants, so that only the right people get in. Problem solved. Oh, but wait! You came back from work only to see your annoying neighbor is there on your couch, eating from your secret stash of cookies. The guard at the entrance obviously didn’t prevent her, she’s a tenant in the building after all. In software terms, this would be an equivalent of Zoombombing or private pictures leak. Not so unheard of.

If you approach the same problem as authorization-first, perhaps instead of posting a guard, you would give each of your tenants a set of keys instead. Everyone would get the same kind of key to open the main door, and additional key unique to each apartment. Now your annoying neighbor can only visit you when invited, and she surely won’t get the cookies! This is definitely a much better solution. Still, it’s not complete. What if you lose your keys? What if someone finds your lost keys and decides to make themselves at home in your apartment? You wouldn’t know, because in that scenario you’re not getting into the building ever again. In software, this would be the same as your ex hacking into your email account and changing the password. Not very fun.

The truth is, you need to think very carefully both about authorization (what can each user do) and authentication (is this user who they say they are). Despite them having similar names, they are two different classes of problems, each with their own unique edge cases to consider. If you lump them together, you end up with a problem one way or another.

", "url": "https://authress.io/knowledge-base/articles/authn-vs-authz", "title": "To authenticate or to authorize - what is the difference?", "summary": "Authentication vs authorization - which one is which? Even experienced software developers confuse the two. Let’s make it crystal clear once and for all - what is the difference and why it matters.", "date_modified": "2020-06-09T10:00:00.000Z", "author": { "name": "Dorota Parad", "url": "https://dorotaparad.ch" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/zoom-case-study", "content_html": "

It’s called success when a company reaches the billion dollar mark, and while there isn’t anything inherently special about that particular base10 number, critical issues affecting the business become public knowledge. In this case, Zoom Video Communications is that relevant company, and the critical issues are those linked to their flagship product Zoom.

In a rush to scale a product to millions of users, products are rapidly spun up and delivered. Startups have become more and more agile and focus more and more on whether it could scale rather than making sure it does. In this race, all too often it comes down to cutting corners and skipping important milestones in hopes the product idea will be your goldmine. “We can do it later”, “It would be wasted effort if we did it now”, and other musings fly around product-engineering meetings at the best entrepreneurial businesses. But that’s a tangent of how or why those companies gamble on security. What’s important is--to make sure you are successful, you need to forgo implementing non-user-visible features, even if they are critical.

Many products are fraught with this peril but as their companies squander their opportunity to gain traction, the issues never come to light. Others that have over-engineered and over-invested in technology and solutions that no one is interested in, also disappear. In essence, it is a never ending struggle to release a successful product while still solving every technical problem that appears along the way.

And the worst of these critical, but non-user-visible, issues are security related. Security is the most hidden aspect to your product, one that you don’t think about as a user, but are fully expectant that it is there in full capacity. Additionally, while the concept is simple, the implementation and knowledge required for understanding it are extremely esoteric. It isn’t common to find engineers who think about the details of security, and even harder to find those who can implement a solution.

Over time, Zoom encountered a number of issues related to its product, I’ll reiterate through some of the key security related ones here:

(This list actually goes on and on, you can find more of examples and their details if you search the internet)

Why did these happen? Were they avoidable?

To answer questions like these, we need to look at the core features that are required to exist, and then we can talk about what solutions to these security challenges would look like.

The features​

We’ll focus on having great user experience, which means being able to deliver core functionality. I’ll enumerate those features that are related to the above security flaws. The goal is to design and deliver the best video sharing experience without sacrificing security. Can we do it?

That’s all very specific to Zoom, so I’ll abstract these a bit:

That list starts to sound really familiar, actually too familiar. Almost every single application ever has to develop solutions to these exact problems. Now we know what we need to implement, how do we go about it? This isn’t a short list, and every one of these requires attention to make sure it has the right user experience, but also the best security practices.

The implementation for these security based features has to look something like:

Getting it right​

While we can easily suggest that having a stronger attention to security related aspects on day one might have allowed Zoom to avoid these disasters, that suggests too much hindsight for me to like. So what’s the alternative? Assume that these problems exist everywhere and the users just have to suffer the consequences? I don’t like that answer either.

The only thing I can think of, is to delegate the responsibility to security-first systems. Zoom is attempting this with the acquisition of Keybase, a firm known for their public GPG repository and security knowledge, but that doesn’t solve everything. Knowing you need to make changes is one thing, but it still takes time. Not to mention that not every company can just go out and purchase a security team on the spot, so there has to be a better solution.

The list is long of necessary features that also focus on security, and none of them are free and just fundamentally take time. They require a complete understanding of the space and focus on their delivery to get there. It isn’t surprising that apps like Zoom didn’t implement them to start. And while they are trying to get through them as fast as possible, security isn’t Zoom’s core competency, and chances are it isn’t yours either. Just like email isn’t my core competency, and I rely on email providers to handle it for me (mostly because I hate SMTP protocol because it isn’t RESTful), figuring out how to handle security isn’t yours.

There are many solutions sitting out there that can help implement these security features out of the box, so finding the right one becomes the main challenge. When searching, look for how easy it is to implement or integrate with the solution. The easier it is, the quicker your team can deliver the security your users expect. As an example, let me detail how Zoom could configure Authress to work for their users:

  1. Scope all resources in the following way: ZoomAccounts/{AccountId}/Meetings/{MeetingId}/Resources The main meeting will be the top level MeetingId and anything created as part of that meeting would be a sub resource.
  2. Create an access record per meeting. Those access records can contain as many people as make sense along with their permissions to the individual resources. The limits on resources and users in access records is much higher than most products need, ~500 for Zoom.
  3. Create the following permissions--EnterMeeting, EditConfiguration, Host, MuteOthers, UnMuteOthers, Record, etc… Then on each action the explicit permission could be checked to ensure that the user can enter. On the Meeting resource you can set it as either Public or Private, meaning that users without access cannot join or can join, but only after approval.
  4. Use an app directory for integrations. Rather than trying to keep track of the users in an account, use the GSuite, Microsoft, etc… app directory for the app. In the case you want it to be open and not use SSO or a default user directory, create one per account using the Organizations Manager. Using the domain name by default is a great start, but allowing users to automatically get permissions by the domain is not.
  5. Create Authress Service Clients and grant them access to only the part of the user or meeting they should have access to. On the client side, clients can be generated per user, but also per meeting depending on the integration. When configuring an integration, pass credentials to the app that it can use at a later point for integration. It’s important to pick a provider with OIDC compliant JWTs as the standard otherwise every app would need to write a custom integration.

It’s easy to see how many companies get to the place where product features exist, but critical security features do not. Spending precious time working on security means diverting attention from your core competency. Instead of doing that, find the existing security solutions that meet all the requirements a high volume app needs.

Even further​

And in 2025 Zoom is still unsafe by design, with one-click access for any attacker you connect with on Zoom: Deep dive on Zoom RCE

", "url": "https://authress.io/knowledge-base/articles/zoom-case-study", "title": "Zoombombing - a case study of data protection", "summary": "Zoombombing is a relatively recent phenomenon, although underlying causes aren't new. In this case study, I take a look at what went wrong and how a company can protect itself from similar issues.", "date_modified": "2020-06-08T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/so-you-want-your-own-authorization", "content_html": "

Perhaps it happened to you. You're working on a project, developing some fancy software where users can do all sorts of cool things.\nAs part of this work, your software needs to be able to recognize your users. Perhaps there's something your users store for later, some content they create, or you simply need to know who paid and who didn't.

That's simple, right? You authenticate the users using a federated login provider and the only extra work you need to do is to store the link between the user ID and whatever resource in your software that you want tied to that user. It's a simple data model - each user has access to a single data blob. This is the user and this is the resource they own. You store it in a database, problem solved.

\"Authorization

Sometimes the story ends here. More often than not though, your product gets new features and the number of things you need to keep track of for each user keeps growing.\nPerhaps there are different types of content each user can create? Or different features they should have access to based on what they pay? It may even be something as simple as listing all documents or pictures the user owns in your software. Your database needs indexing now. You're still tracking the link between a user ID and resource, except the 'resource' is now a list. It's still simple, but you have to start considering performance and optimize data fetching. Oh well, nothing too complicated.

\"Things

Now imagine that you have some sort of interaction between different users. Maybe there's a whole company using your software, with administrators and regular users. Or perhaps users want to share their content with others. Or there's some way for more than one person to collaborate on the same resource. Your old data structure, where you had a user ID and a linked list of resources, is no longer sufficient. Now you not only need to keep track of the users and their respective resources, you also need to know who can do what actions on a given resource. Your old tables may still work, but you need a different strategy for indexing. You also definitely need more tables and more ways to access the data. We're starting to talk about many to many relationships. Things quickly get messy. It's still all manageable, probably.

\"You&#39;ll

If your project is complete, congratulations. However, likely, you'll keep adding features to your product, and you'll start to discover the more things your users can do, the more roles they can play and more resource types they can engage with. This means your simple data model gets more and more complicated. Your indexing needs to be a first class notion, and you need to think more often about the speed of data fetching. What started as a single database table is now multiple tables with relationships that may need to change as you work on new features. What a headache.

\"As

Speaking of change, it's unlikely that your original vision of resource hierarchy and user permissions will hold true forever. As you add more and more features, as requirements change, you're making tweaks to your roles and types of data you're storing. Perhaps your notion of resources changed and now you have sub-resources? Or you need to deal with user or role groupings? One day, your old queries no longer work. Your old tables can't simply be extended to hold the additional complexity, they have to be redesigned. You have a data migration on your hands.

\"You&#39;ll

What's more, your indexes have to be rebuilt. Things start to get slower and slower. You make some changes and it turns out that you accidentally let one user see another user's private resources. Luckily, you discover that bug right away and spend the evening fixing it - phew, hopefully no actual users found out!

Over time, your product grows. You get more and more users. Whenever you fail to quickly and correctly resolve what resources should be available to which user, many people get annoyed. Whenever your authorization code fails for some reason, your whole application is severely affected. What started as a simple bit of code, is now a critical, ever growing component that you have to maintain in top shape. Perhaps it's time to have a dedicated team working on it?

I've been through this journey myself, more than once. Every time, I was looking back and cursing myself for ever thinking that authorization is a simple problem. It may start simple, but rarely remains so. It usually ends up in a headache and lots of wasted hours.

It may seem like a good idea to build it all yourself. While your application is unique, the authorization problems it faces rarely are. It's just one of the areas where it's better not to reinvent the wheel and use something off the shelf.

So what's the solution​

The short answer is: Authress. It solves all these issues so you don't have to worry about it. It provides a scalable, multitenant, fully compatible RESTful authorization API with user identity management, access control, permissions, groups, and more. However, if you've got a list of possible providers already, and aren't sure which one to pick, check out the guide on picking the best auth solution.

", "url": "https://authress.io/knowledge-base/articles/so-you-want-your-own-authorization", "title": "So you want to build your own authorization?", "summary": "When writing new software, it's hard to notice complexity creeping in. Authorization is one of the aspects where things start deceptively simple and before you notice, you end up in a zoombombing scandal.", "date_modified": "2020-05-29T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/companies-gamble-on-privacy", "content_html": "

I vividly remember how shocked I was after reading a research paper on how freelance developers handle data security of their solutions (hint: they really don't). Turns out, unless you explicitly state you want your application data (such as username and password) to be stored securely, you won't get it, even if your main requirement is to create a user registration page.

For me, as a user of many web apps, it was horrifying. Of course I want my personal data secure! This should go without saying!

What gave me an even bigger pause was thinking back to my years as a Product Manager. I remember how it was working with deadlines. Data privacy, if mentioned at all, would be one of the topics you work on last. That's just how it is when you're developing a new product. You focus a lot on core features, the ones that can set you apart from your competition. And if you're like me, you've hired smart experienced developers whom you trust to do the right thing. Of course the data has to be secured, it goes without saying! Turns out, if you don't say it, it won't get done unless you're extremely lucky.

In a way, it makes sense. No one picks one application over the other because this one protects user data better. As a user, you look at the core features first, and only when two products are roughly the same you may look at security as a deciding factor. Businesses know that. Keeping user data private is a cost, it's not what brings money.

So we end up in situations like Google accidentally sending your private photos to a random stranger after you requested a download (this was luckily fixed in the meantime). Zoombombing is still a thing.

It's not like these are small or underfunded companies. It's also not like they don't care about security or user data privacy. So why do we see such scandals? Well, data privacy means you need to manage who should have access to what at what time. It's hard to get it right. What's more, security isn't usually a core competence of companies making software products. User data privacy is rarely a fundamental feature. Therefore, securing your users' data always ends up as an afterthought. Even if you create a dedicated team with sole focus on the topic, they will have to compete for resources with teams who deliver actual business value. Guess who wins?

When your data privacy is only a cost center, you won't do it right. It's hard to get it right. It's even harder to get it right without affecting your software's performance. When your core competency lies elsewhere, it will always be a better investment for you to work on new features to wow your users and set your product apart from competition.

That's why, if you have an opportunity to let someone else take care of securing your app, you should. And I don't mean you should leave this task to contractors (based on the research I mentioned earlier, that should be the last thing anyone does). I mean search for an actual off the shelf solution, where some other company made data privacy and cyber-security their core competencies.

It may feel scary at first - if user privacy is your concern and you know it's difficult to get it right, why should you let another company do it for you? Wouldn't they make a mess of it? Possibly. But there is a high chance that a company providing app security as a service optimized the heck out of their solution. They likely use the most efficient technology. They most certainly are squashing bugs as soon as they're discovered. They can offer you support and SLAs that your internal team could never match. That is because for those companies, data privacy is precisely what brings them money. Can you say the same?

I bet you're not producing your own electricity, even though your company couldn't exist without it. You also probably aren't producing your own CPUs to run the software you're building. Security of your application isn't all that different.

", "url": "https://authress.io/knowledge-base/articles/companies-gamble-on-privacy", "title": "Why companies gamble on user data privacy", "summary": "Despite GDPR, we still hear about embarrassing data leaks, often at big tech companies. What is so difficult about protecting your users data? Turns out, it's just business...", "date_modified": "2020-05-27T10:00:00.000Z", "author": { "name": "Dorota Parad", "url": "https://dorotaparad.ch" }, "tags": [] }, { "id": "https://authress.io/knowledge-base/articles/securely-store-client-key-secret", "content_html": "
info

A comprehensive review of Credential Management available in the Authress Academy.

Securing APIs requires the generation of API keys. These api keys will be used in the CLI and SDK versions of your service passed to the REST or gRPC api via the Authorization header. The generation of API keys examples can be found in the article for Creating API Keys. If the generation of API keys is the service side action, then the storage of these api keys is the client side responsibility. Here we'll discuss the best solution for storage of these keys.

API secrets go by many names, CLIENT_SECRET, PASSWORD, API_KEY, ACCESS_KEY, and the list goes on. These may be used to access an API, log in to an existing service, or verify symmetric encrypted data blobs. In almost all cases these should be asymmetric encrypted secrets never stored by the credentials server. (If a service allows you to reveal or display a secret more than once, it is a red flag that it isn't well secured in the service.)

API keys generated for a service client hold all the authority in an application, they can represent a user, a group, full access to API service resources, or be used to impersonate an admin user. And as such the secure storage of these keys is critical. One prevalent pattern is storing the credentials on the client in unencrypted form on the production container. This is not safe however. While the location is of the better ones available, the deployment of unencrypted api keys is rife with security issues. How do you get unencrypted credentials on a target container without exposing those credentials to a system that shouldn't have access? And further who has seen these unencrypted credentials? Limiting exposure of credentials is hugely important to security.

A better solution than this is resolving these credentials at runtime, so the unencrypted form is only available in memory in the target service. Even as such there are two ways to resolve this information:

Using a secret name​

Credentials can be stored in a vault or other secure storage and then loaded from that storage using the name of the secret. The usage of roles and other access control via cloud providers can be used to store the credentials and then securely resolve them at runtime.

This is a great solution for the storage of the secret, but requires generating a secret name, and potentially exposes this concept other places. It also encourages the reuse of secrets by other services, which violate microservices if that is a pattern being used. In some cases this a good pattern, but picking one that doesn't encourage the reuse of the secret is best.

Another possible problem with this pattern, is that it lacks transparency of secret changes. In the case you will want rotate the secret key, the tracking of the rotation happens outside the purview of the service, and therefore is difficult to track.

Using encrypted credentials​

Instead of storing the credentials in another service, a more powerful and secure pattern is to encrypt the credential and store it in the source code. This prevents exposure of the encrypted credentials directly, by limiting visibility to the service's source. While anything can copy the credentials, it's clear that the service owns them and won't be directly used without something copying that full encrypted text. To generate and resolve the credentials at runtime:

Additionally this solves the problem of audit, since the version control system now tracks exactly when the credential has been updated. Access control is still enforced by similar mechanisms when using the secret name, but the scope of the credential is limited in space to the service where the encryptedCredentials are stored. Rotating the key is as simple as updating the encrypted credentials blob, and the change is transparent for everyone. It also allows handling multiple changes without needing the use of complicated mechanisms like secret key versions. The version is the one in the source!

Comprehensive review of credential security options​

A comprehensive review of Credential Management available in the Authress Academy.

", "url": "https://authress.io/knowledge-base/articles/securely-store-client-key-secret", "title": "Securely store client IDs and secret access keys", "summary": "API access is provided through client IDs and secret access keys and because of the authority attached to these credentials they must be secured as safely as possible.", "date_modified": "2020-02-10T10:00:00.000Z", "author": { "name": "Warren Parad", "url": "https://warrenparad.net" }, "tags": [] } ] }