This is a story about two search APIs that were quietly ruining user experience, and the one architectural question that fixed everything.

The Problem

We had 15 APIs in total across a platform. 13 of them were performing well, P95 response times in milliseconds. Clean, fast, no issues.

But two search APIs were consistently showing P95 latency of 8.2 seconds and 10.5 seconds during performance testing. Every time. Under every load pattern.

That kind of consistency is actually useful information. When something fails consistently, it is telling you something specific.

What We Tried First

The usual performance tuning cycle started immediately.

Database query analysis. Network bandwidth checks. Connection pool tuning. Parallelism adjustments. Various configuration experiments.

None of it moved the needle.

When the standard fixes do not work, that is the signal to stop tuning and start investigating the architecture.

Finding the Real Cause

Both APIs shared something in common. Each request depended on a single attribute that had to be fetched from an external system. Everything else in the request was handled locally. But that one attribute required a synchronous call to another service every single time.

During load testing, hundreds of requests were hitting these APIs within a short window. Each request independently called the external system. Even though our own system was well optimised, the response time of that external dependency dominated the overall latency on every single request.

We discussed it with the other team. But sometimes dependencies are outside your control. The system ran this way for quite some time.

Then we asked ourselves a simple question.

What if we did not fetch that attribute during the request at all?

The Fix

We introduced a small table in our own database. It stored just two things. The entity identifier and the required attribute value.

A background process kept this table synchronised with the external system. The data stayed current. But the API no longer needed to make a live call to the other system during request processing.

The request flow became entirely local.

The Result

Search API A: 8.2 seconds down to 0.39 seconds. A 95% reduction.

Search API B: 10.5 seconds down to 0.47 seconds. A 96% reduction.

Zero infrastructure changes. No additional servers. No heavy tuning. No code rewrite.

Just removing a synchronous external dependency from the critical request path.

The Takeaway

When you are chasing a latency problem and nothing is working, ask yourself this before you try the next configuration change.

Is the work happening in the right place?

Sometimes the system is not slow. It is just waiting for someone else. And that is a different problem with a different solution.

Performance tuning is not always about optimising what you have. Sometimes it is about rethinking where the work happens.