What Makes AI Infrastructure Worth Using? AI 基础设施，什么时候值得用？

00 Why Would a Team Need This?

My first few weeks learning AI infrastructure were about mechanics: prefill and decode, GPU memory, KV cache, batching, scheduling. I could begin to explain how a serving system works.

But I ran into a much more basic question: Who actually needs this? If I am building a small AI application, I can call a hosted model API. Why would I run my own model? Why would I need an inference engine, let alone a managed infrastructure service?

That question changed the way I studied. Instead of stopping at the technical mechanism, I started asking four things about every product:

Understanding the mechanism tells me what technology can do. Understanding the user tells me whether that capability matters.

01 One AI Application, Different Infrastructure Choices

Consider a hypothetical startup building an AI coding agent. The end user asks the agent to fix a bug. From the outside, it is just a product experience. Behind the scenes, the company must decide how to run the model.

There is no universal answer. A team may rely on a hosted frontier-model API, operate an open-weight model itself, or use a managed service that operates a chosen model on its behalf.

Below the application sits a wider stack: model weights, inference engines, GPU software, hardware, networking, monitoring, and operations. A separate post-training path can improve a model through supervised fine-tuning or reinforcement learning; a training framework may use an inference engine to generate rollouts. The product categories cooperate, but they solve different jobs.

An important lesson for me: not every AI developer is an infrastructure customer. Someone validating a small MVP with a hosted API may have no reason to operate an inference engine at all.

02 Who Is the User, Really?

I originally imagined “AI developers” as one broad group. That hides major differences. A model lab deploying its own architecture has a different job from a startup optimizing an agent's cost per task; a GPU cloud has a different job again.

User ≠ Buyer

Another distinction matters in B2B software. The engineer evaluating a tool may not control the budget. An infra lead worries about migration risk; a CTO may care about total cost and delivery speed. A product message has to speak to the right question without pretending they are the same person.

03 Connect a Feature to a Result

I had learned how prefix caching avoids redundant computation. But knowing the mechanism was not enough. To explain a technical feature as a product, I needed a longer chain of reasoning.

The same caution applies to batching. A higher throughput number can come with worse latency for an individual request. A good product story has to name the workload and the trade-off, not just display the biggest benchmark.

For an AI coding agent, I might care about cost per successfully completed task and the time a user waits for that task, not just raw tokens per second. For a cloud provider, requests served within latency targets per GPU might matter more. The right metric depends on the job.

Open source is free. Operating it is not.

My first answer to “Why pay for a managed service?” was about reliability and focus. A team may be able to download the software, but must still provision machines, tune performance, monitor the system, manage upgrades, and respond to incidents.

The decision is not automatically “managed is better.” A mature infrastructure team may prefer self-hosting. A smaller team might reasonably pay to focus on its product. The product question is whether the service creates enough net value for this particular customer.

04 When a Customer Already Uses Another Engine

Here is the case that most changed how I think about positioning: a developer says, “I already use an inference engine. It works reasonably well. Why should I switch?”

My first instinct is no longer to list features. I want to understand the current system.

This is a product discovery problem before it is a product comparison problem. If the current engine is meeting the customer's needs, not switching may be the right answer. If there is a real bottleneck, we can compare alternatives on the actual workload instead of on an unrelated public benchmark.

That was also a lesson from learning about post-training: a wrong model output does not automatically mean “run RL.” The agent may be missing context, calling a tool incorrectly, or using unclear instructions. Training only makes sense after understanding the failure mode.

05 A Great Feature Is Not the Same as Adoption

Now suppose an engineering team releases a performance optimization. Its benchmark is strong, but developers are not adopting it. My data-science instinct is to ask, “What exactly counts as adoption?”

A GitHub star, a documentation visit, a successful first run, and a sustained production deployment are four different events. Traffic and attention may help, but they are not the outcome.

Where does the developer journey break?

If 400 developers try a feature and only 80 complete a first successful run, buying more traffic may be the wrong move. The activation rate among trial users is only 80 / 400 = 20%. Before changing anything, I would check measurement quality, segment by model and hardware, and look for recurring setup failures.

That is where the Growth Hacking framework started to feel familiar to me. Acquisition, activation, retention, referral, and revenue describe different moments in the journey. As someone trained in statistical analysis, I also recognize the danger of mistaking correlation for causation.

A small experiment, not a big campaign

Suppose qualitative research suggests that the quickstart guide is unclear. A possible intervention is a tested, model-specific guide with clearer compatibility requirements and error messages. If measurable and practical, the team could compare it with the existing guide; otherwise it could start with usability testing and a staged documentation improvement.

A practical caveat: open-source projects often cannot observe private self-hosted deployments. So these metrics may be unavailable or biased. Honest product research can use opt-in surveys, community feedback, support issues, and targeted usability sessions without pretending to have perfect telemetry.

06 Where a Product-Facing Person Can Contribute

The most interesting part of this exercise is that I did not need to become an inference-kernel engineer to find meaningful problems. I did need enough technical literacy to understand the limits of a feature and enough product judgment to find where users struggle.

In a small infrastructure team, product management, product marketing, developer relations, technical writing, growth, and engineering may overlap. I would begin by making the developer journey more observable and easier to complete.

These are proposed contribution paths, not claims about a particular company's internal priorities. The first step would always be to ask the team what its most important user problem actually is.

One feature, two conversations

The same performance improvement needs different explanations depending on who is listening.

07 What I Now Ask Before Recommending a Tool

This is the small checklist I want to carry into any technical product conversation:

1. Who has the problem? Be specific about the user, their workload, and who makes the decision.
2. What are they doing now? Understand the current solution, including what already works.
3. What fails, and why? Diagnose before selecting a technology.
4. What would improve? Translate a feature into a measurable user or business outcome.
5. What would adoption require? Consider documentation, migration, reliability, and the cost of change.
6. How would we know? Choose an honest metric and acknowledge its blind spots.

A faster engine is a technical achievement. Helping the right developer succeed with it is a product achievement.

I still want to understand the underlying systems. But now, when I learn a concept like caching, scheduling, or post-training, I also ask who needs it, what trade-off they face, and what evidence would convince them. I want to use that technical knowledge in user research and product work.

08 Public References & Boundaries

The technical examples above describe general product patterns. I have not used private interviews, internal product metrics, confidential company details, or claims about proprietary roadmaps. The funnel and customer scenarios are explicitly illustrative. My own analysis is not an endorsement or official representation of any project.

1. SGLang documentation: open-source model serving, performance, APIs, and deployment basics.
2. SGLang public Q4 2026 roadmap: usability, cookbooks, deployment simplicity, and compatibility as publicly stated development goals.
3. SGLang public 1.0 tracker: production stability and configuration clarity as an example of developer-experience work.
4. vLLM documentation: another open-source inference-serving framework and an example of the existing alternatives a developer might evaluate.
5. AI Infra From Zero: Note 01: my earlier technical learning note on prefill, decode, KV cache, and prefix reuse.