Calling a hosted model from your main application is a few lines of code, and for a lot of AI features that is exactly the right design. At some point, though, the feature outgrows it. You need an embedding model that only has a Python library, a document parser, a reranker, or long-running jobs that don't belong in a web request. That is when a separate AI service starts to make sense.
This is the shape I use: the main application stays in Laravel, and a small Python service built with FastAPI does the AI work. The language pair matters less than the boundary between them, so most of what follows applies to any stack.
When to split, and when not to
Split when at least one of these is true:
- The work depends on the Python ecosystem: embeddings, tokenizers, document parsing, rerankers, evaluation tools.
- The workload scales differently from your web app: bursty batch jobs, heavy CPU, or a GPU.
- Requests are long-running or streaming and would tie up web workers.
Don't split just because "AI is in Python". A feature that sends a prompt to a hosted model and returns the answer can live in the main app with a well-built HTTP client. A second service brings deployments, monitoring, auth, and failure modes. Pay that cost only when you get something for it.
Who owns what
The boundary is the design. Get it right and both sides stay simple.
The main application owns users, tenants, permissions, billing, credits, and the business data. It decides who may ask what, and about which records.
The AI service owns prompts, model selection, retrieval, and inference. It doesn't know about users or plans. It receives exactly the context it is allowed to use, does the work, and reports what it cost.
Because the main app enforces permissions before anything reaches the AI service, a retrieval request carries the tenant and the set of allowed sources with it. The AI service filters its search by those values. It never searches everything and trusts the main app to hide the results afterwards.
Three ways to talk
Synchronous HTTP for short tasks, such as classifying a ticket or extracting fields from one document. Set a tight timeout and have a fallback ready.
A queue for long tasks, such as indexing a thousand documents or generating a report. The main app creates a job record, publishes a message, and the AI service updates the job when it finishes. The user sees progress instead of a spinner that times out.
Streaming for chat-style answers. FastAPI streams tokens with server-sent events, the main app proxies the stream to the browser, and the user sees the answer form within a second instead of waiting ten.
Contracts both sides can trust
Define the request and response once, version them, and validate on both sides. On the Python side, Pydantic models make the contract explicit:
class AnswerRequest(BaseModel):
request_id: str
tenant_id: int
question: str
allowed_source_ids: list[int]
max_output_tokens: int = 800
class AnswerResponse(BaseModel):
request_id: str
answer: str
citations: list[int]
input_tokens: int
output_tokens: int
model: str
On the Laravel side, map the same shape into a typed object and reject anything that doesn't match. When the contract changes, add a version to the path (/v2/answer) instead of changing the meaning of an existing field under a running client.
The request_id is created by the main app and travels everywhere: logs, metrics, the job record, and the usage entry. When a customer reports a bad answer, one ID connects every hop.
Service-to-service auth
The AI service should not be reachable from the internet. Put it on a private network and authenticate every call from the main app, with a signed service token or mutual TLS. Treat its own outbound calls to model providers like any other secret-bearing dependency: keys in a secret store, rotated, and never logged.
Metering and credits
AI features cost money per request, so usage has to be counted as carefully as payments. The AI service reports token counts and the model used on every response. The main app turns that into a cost and records it in a usage ledger: one row per request, tied to the tenant and the request ID.
If customers have credits or quotas, debit them from the ledger inside a transaction, and check the remaining balance before sending an expensive request, not after. A credit system that only finds out at the end of the month that a tenant used ten times their plan isn't a credit system.
Streaming without leaks
Streaming adds two problems that plain requests don't have.
- Disconnects. When the user closes the tab, the main app must cancel the upstream stream. Otherwise you keep paying for tokens nobody reads.
- Timeouts. Proxies and load balancers often close idle connections after 30 to 60 seconds. Send periodic keep-alive events and set every timeout in the chain on purpose, not by default.
Observability across two services
Log the request ID, tenant, model, token counts, latency, and outcome for every call on both sides. Store prompts and outputs with personal data redacted, because they are your only way to debug a bad answer. Track cost per feature, not just in total, so you know which feature to optimise first.
Failure is a feature
The AI service will be slow or down sometimes, and so will the model provider behind it. Decide in advance what users see when that happens: a cached answer, a simpler non-AI path, or a clear "try again" message with the request saved. What they should never see is a hung page or a charge for an answer they didn't get.
The checklist
- Split only for the Python ecosystem, different scaling, or long-running work.
- The main app owns users, permissions, and billing; the AI service owns models and retrieval.
- Permissions are applied before retrieval, and the allowed sources travel with the request.
- HTTP for short tasks, a queue for long ones, server-sent events for streaming.
- Versioned, validated contracts with a request ID that travels everywhere.
- A private network and service auth between the two.
- A usage ledger, with credit checks before expensive calls.
- Cancelled upstream streams on disconnect, and deliberate timeouts.
- A defined fallback when AI is unavailable.
Done this way, the AI service stays small and replaceable, and the main application stays in charge of the things it must never get wrong.
