Want a private AI server for coding assistance and infrastructure automation? A single NVIDIA RTX 3060 12GB system running Rocky Linux 9, Podman, LocalAI, and Open WebUI can provide a practical self-hosted AI foundation for home labs, small IT teams, and internal automation workflows.
Build goal: deploy a private local AI server that can run useful coding, documentation, and infrastructure-assistance models while keeping prompts, internal notes, and experimentation inside your own environment.
Executive Summary
This runbook builds a self-hosted AI server using Rocky Linux 9, Podman, LocalAI, Open WebUI, and an NVIDIA RTX 3060 12GB GPU. The goal is not to compete with hosted frontier models. The goal is to create a controlled platform for private prompts, coding assistance, runbook drafting, infrastructure automation, and future retrieval augmented generation experiments.
The build is intentionally modest. It works well for homelabs, small IT teams, and administrators who want a practical entry point into local AI without immediately designing a multi-node platform.
Reality check: you do not need enterprise GPU hardware to begin self-hosting AI. An RTX 3060 12GB remains useful for 4B, 8B, and many quantized models that fit inside 12GB of VRAM, especially for private experimentation and operational learning.
Architecture Overview
This is a single-server AI platform. Rocky Linux provides the operating system, Podman runs the containers, LocalAI exposes an OpenAI-compatible inference endpoint, and Open WebUI provides the browser interface.
Browser
|
+-- Open WebUI on TCP 3000
|
+-- LocalAI API on TCP 8080
|
+-- NVIDIA RTX 3060 12GB
|
+-- Local models and configuration under /opt/containers/localai
| Component | Purpose |
|---|---|
| Rocky Linux 9.x | Stable enterprise-style Linux base for the AI host. |
| Podman | Container runtime for LocalAI and Open WebUI. |
| LocalAI | OpenAI-compatible local inference endpoint. |
| Open WebUI | Chat-style web interface for users and administrators. |
| RTX 3060 12GB | GPU acceleration for practical local model inference. |
Build Checklist
- Rocky Linux 9.x installed and updated.
- NVIDIA RTX 3060 12GB installed and visible to the operating system.
- Administrative shell access available.
- Network access available for package installation and container pulls.
- Static IP or DHCP reservation planned for the AI server.
- Trusted LAN clients identified before opening firewall access.
Recommended Hardware Baseline
The NVIDIA RTX 3060 12GB remains one of the better value cards for local AI inference because the 12GB of VRAM gives you enough room to run small and mid-sized models that are useful for chat, shell help, documentation, and infrastructure troubleshooting.
This does not replace large hosted frontier models, but it gives administrators a controlled environment for experimenting with private prompts, internal runbooks, code assistance, and automation workflows without sending every request to a public cloud service.
- GPU: NVIDIA RTX 3060 12GB
- CPU: 8+ core Ryzen, Intel Xeon, or Intel i7
- Memory: 32GB minimum, 64GB recommended
- Storage: NVMe SSD preferred
- Operating system: Rocky Linux 9.x
- Container runtime: Podman
- Network: 1GbE minimum, 10GbE preferred for larger environments
Model Selection
Use models that are actually visible in the LocalAI model gallery on this build. The screenshots reviewed during testing showed the Qwen3 family and several DeepSeek-related variants, but not the older Qwen2.5-Coder, DeepSeek-Coder 6.7B, or Mistral 7B examples. Start with a model that proves the stack works before moving into larger or more specialized models.
- qwen3-4b: recommended first validation model. It is small enough for a clean first test on an RTX 3060 12GB and is useful for basic chat, shell, and documentation prompts.
- qwen3-8b: practical second test after
qwen3-4bis working. It provides a better real-world feel while still fitting comfortably in GPU memory according to the LocalAI model gallery. - qwen3-14b: larger fit test after the smaller Qwen3 models are confirmed. Use it only after the baseline path is stable because it consumes more VRAM and can make troubleshooting harder.
- qwen3-1.7b or qwen3-0.6b: fallback smoke-test models if you only need to confirm that the API, download path, and UI connection work.
- DeepSeek and DeepSeek-derived variants: advanced next-step models when they appear in the gallery. Use them after the baseline model works because they may require extra model variables, backend settings, or tuning to load cleanly.
Practical tip: install qwen3-4b first. Once it loads, answers a prompt, and shows GPU activity, move to qwen3-8b, qwen3-14b, or DeepSeek-derived variants. Do not begin troubleshooting with a large or specialized model.
Deployment
Step 1: Prepare the Rocky Linux Host and Detect the GPU
After installing Rocky Linux 9, update the system and install the core tools required for this build. On a minimal Rocky Linux 9.x server install, pciutils may be missing, and that package provides lspci for hardware detection.
sudo dnf update -y sudo dnf install -y podman git wget curl pciutils net-tools bind-utils *** Optional packages I recommend *** sudo dnf install -y cockpit-podman cockpit-storaged ************************************* sudo dnf install -y epel-release sudo dnf config-manager --set-enabled crb sudo dnf clean all sudo dnf makecache lspci | grep -i nvidia
Rocky Linux note: the tested Rocky Linux 9.8 Server path did not require manually installing epel-release, kernel-devel, kernel-headers, gcc, make, dkms, or elfutils-libelf-devel before installing the NVIDIA driver. If the driver module install reports missing build dependencies, install the specific packages it requests and retry.
Step 2: Install the NVIDIA Driver
Install the host NVIDIA driver before installing the NVIDIA Container Toolkit. The container toolkit does not provide nvidia-smi; it only helps containers use a GPU that already works on the host.
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo sudo dnf clean all sudo dnf install -y kernel-devel kernel-headers gcc make dkms elfutils-libelf-devel sudo dkms autoinstall sudo dnf module install -y nvidia-driver:latest-dkms sudo reboot
Step 3: Validate the Host NVIDIA Driver
After the reboot, confirm that the host can see the NVIDIA driver and kernel module. Do not move on to container GPU work until nvidia-smi works on the host.
nvidia-smi lsmod | grep nvidia
Step 4: Install the NVIDIA Container Toolkit
Install the NVIDIA Container Toolkit after the host driver is working. This toolkit is needed for GPU access inside Podman containers, but it is not required for the host-level nvidia-smi command.
sudo curl -sSL -o /etc/yum.repos.d/nvidia-container-toolkit.repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo sudo dnf install -y nvidia-container-toolkit
Toolkit note: nvidia-container-toolkit is not normally available from the stock Rocky Linux or EPEL repositories. Add NVIDIA’s libnvidia-container repository first, then install the toolkit.
Step 5: Generate the NVIDIA CDI Spec and Validate Podman GPU Access
For Podman, use NVIDIA CDI instead of the older OCI hook path. CDI exposes the NVIDIA device nodes and driver libraries to containers through a generated device specification.
sudo mkdir -p /etc/cdi sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml ls -lah /etc/cdi/nvidia.yaml grep " name:" /etc/cdi/nvidia.yaml podman run --rm --security-opt=label=disable --device=nvidia.com/gpu=all docker.io/nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi podman info
CDI validation: do not deploy LocalAI until the CUDA test container shows the RTX 3060 with nvidia-smi. If the test works, Podman can inject the GPU into containers with --device=nvidia.com/gpu=all.
Step 6: Create Persistent Storage Paths
Keep all container-backed application data under /opt/containers. This keeps Podman workloads centralized and makes backups, rebuilds, and troubleshooting much easier.
sudo mkdir -p /opt/containers/localai/models
sudo mkdir -p /opt/containers/localai/data
sudo mkdir -p /opt/containers/localai/config
sudo mkdir -p /opt/containers/openwebui/data
sudo mkdir -p /opt/containers/localai/backends
sudo mkdir -p /opt/containers/localai/cache
Now recreate the directories with the SELinux volume flag:
sudo chown -R 0:0 /opt/containers/openwebui/data
sudo chmod -R u+rwX,g+rwX /opt/containers/openwebui/data
sudo restorecon -RFv /opt/containers/openwebui 2>/dev/null
sudo chown -R 0:0 /opt/containers/localai
sudo chmod -R u+rwX,g+rwX /opt/containers/localai
sudo restorecon -RFv /opt/containers/localai 2>/dev/null
Step 7: Deploy the LocalAI Pod
With the NVIDIA driver, container toolkit, and CDI configuration validated, the next step is to start LocalAI as a standalone Podman container. This container exposes the LocalAI API on port 8080, mounts persistent model and data directories, and grants the container access to the RTX 3060 through the NVIDIA CDI device mapping.
This approach keeps the deployment simpler than a shared Podman pod. LocalAI runs as its own container, which makes the later systemd service easier to generate, start, stop, inspect, and troubleshoot.
sudo podman run -d --name localai --device=nvidia.com/gpu=all -p 8080:8080 --security-opt=label=disable -v /opt/containers/localai/models:/models:Z -v /opt/containers/localai/data:/data:Z -v /opt/containers/localai/backends:/tmp/localai/backend_data:Z -v /opt/containers/localai/cache:/root/.cache:Z -v /opt/containers/localai/data:/data docker.io/localai/localai:latest-gpu-nvidia-cuda-12
Deployment note: this build uses NVIDIA CDI for Podman GPU access. If LocalAI reports libcuda.so.1 missing or the container has no /dev/nvidia* devices, regenerate /etc/cdi/nvidia.yaml and confirm the CUDA test container works before troubleshooting LocalAI.
Step 8: Add the Optional Open WebUI Interface
Open WebUI provides a familiar ChatGPT-style browser interface for interacting with the LocalAI endpoint. This is useful if you want a web UI instead of relying only on API calls or command-line tools.
sudo podman run -d --name openwebui -p 3000:8080 -v /opt/containers/openwebui/data:/app/backend/data:Z -e OPENAI_API_BASE_URL=http://host.containers.internal:8080/v1 -e OPENAI_API_KEY=localai ghcr.io/open-webui/open-webui:main
Step 9: Generate a Systemd Service for the LocalAI Pod
Because this build creates a Podman pod named localai-pod and then runs the LocalAI container inside that pod, generate the systemd unit from the pod. This lets systemd recreate and manage the pod and the LocalAI container together after reboot.
cd /tmp sudo podman generate systemd --name localai --files --new sudo podman generate systemd --name openwebui --files --new
Install the unit files.
sudo cp /tmp/container-localai.service /etc/systemd/system/
sudo cp /tmp/container-openwebui.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/container-localai.service
sudo chown root:root /etc/systemd/system/container-openwebui.service
sudo chmod 644 /etc/systemd/system/container-localai.service
sudo chmod 644 /etc/systemd/system/container-openwebui.service
sudo restorecon -v /etc/systemd/system/container-localai.service
sudo restorecon -v /etc/systemd/system/container-openwebui.service
sudo systemctl daemon-reload
Verify that systemd owns the pod and that LocalAI is responding.
sudo systemctl status container-localai
sudo systemctl status container-openwebui
sudo podman pod ps
sudo podman ps --pod
curl http://127.0.0.1:8080/v1/models
Step 10: Open the Firewall
Allow access to LocalAI and Open WebUI from trusted systems on your network.
sudo firewall-cmd --add-port=8080/tcp --permanent sudo firewall-cmd --add-port=3000/tcp --permanent sudo firewall-cmd --reload
Security reminder: do not expose LocalAI or Open WebUI directly to the public internet without authentication, TLS, access controls, and a reverse proxy strategy.
Accessing the Stack and Loading the First Model
After the containers are running, validate the pod, reach the services from a browser, and load qwen3-4b as the first baseline model before testing larger Qwen3 or DeepSeek-derived models.
Confirm the Containers Are Running
podman pod ps podman ps podman logs localai --tail=50 podman logs openwebui --tail=50
Open the Service URLs
From a workstation on the trusted LAN, replace <server-ip> with the Rocky Linux server address.
| Service | URL | Purpose |
|---|---|---|
| LocalAI | http://<server-ip>:8080 | LocalAI web interface and API service. |
| LocalAI API | http://<server-ip>:8080/v1 | OpenAI-compatible API base URL. |
| Open WebUI | http://<server-ip>:3000 | Browser chat interface. |
Connection note: when configuring Open WebUI, do not point it at localhost:8080 unless LocalAI is running inside the same container. From the Open WebUI container, localhost means Open WebUI itself. Use http://<server-ip>:8080/v1 as the LocalAI API base URL.
Install qwen3-4b as the Initial Test Model
In the LocalAI model gallery, search for qwen3 and install qwen3.5-4b. This is the recommended first model for this runbook because it is visible in the tested gallery, is small, fits comfortably on the RTX 3060 12GB, and is enough to prove that model loading, the LocalAI API, Open WebUI, and GPU inference are working.
- Open the LocalAI interface at
http://<server-ip>:8080. - Search the model gallery for
qwen3. - Install
qwen3.5-4bfirst. - Confirm the model appears through the API before testing Open WebUI.
- Keep manual model files and LocalAI model configuration under
/opt/containers/localai/models.
ls -lah /opt/containers/localai/models curl http://127.0.0.1:8080/v1/models podman logs localai --tail=100
Make sure the backend is also installed:
- Open the LocalAI interface at
http://<server-ip>:8080. - Search the backend gallery for
llama-cpp. - Install
llama-cpp.
12GB GPU Tuning: Context Size and mmproj
On a 12GB GPU, the LocalAI model gallery may default to an 8K context window, and some models may install with an mmproj multimodal projector file. That can push even a small model over the VRAM limit during KV cache allocation.
If qwen3.5-4b fails with cudaMalloc failed: out of memory, failed to allocate buffer for kv cache, or an error mentioning an mmproj file, edit the model YAML directly for the first text-only validation test.
grep -Rni "context_size|mmproj|Qwen" /opt/containers/localai/models sudo nano /opt/containers/localai/models/<model-config>.yaml
For the initial validation run, set the context size to 2048 and remove or comment out the mmproj entry if the model is being used only for text generation.
context_size: 2048
After saving the model configuration, restart LocalAI and test the model again.
podman restart localai curl http://127.0.0.1:8080/v1/models watch -n 1 nvidia-smi
Run Basic Initial Test Prompts
Start with simple prompts that are easy to judge. The goal is to confirm that the model responds, not to test the limits of model quality.
Explain what this server is running in three bullet points. Write a short Bash script that prints CPU, memory, and disk usage on Rocky Linux. Summarize the difference between Podman and Docker for a Linux administrator. Create a five-step troubleshooting checklist for a Linux service that will not start.
While the prompt runs, watch GPU activity from the host. You should see GPU memory increase and GPU utilization move while the model is generating.
watch -n 1 nvidia-smi
Connect Open WebUI to LocalAI
Open WebUI can connect to LocalAI because LocalAI exposes an OpenAI-compatible API. In Open WebUI, open the admin settings, add an OpenAI-compatible connection, and use the LocalAI API base URL.
LocalAI API base URL: http://<server-ip>:8080/v1
After the connection is saved, return to the Open WebUI chat screen, select qwen3.5-4b from the model selector, and send one of the basic test prompts above.
Advanced Next Step: DeepSeek Variants
DeepSeek and DeepSeek-derived models are good advanced tests after the baseline LocalAI path is proven. Do not use them as the first validation model. In the tested gallery, DeepSeek-related entries appeared as variants rather than the older DeepSeek-Coder 6.7B example, and some variants may need additional model variables, backend options, context tuning, or template adjustments before they load and respond cleanly.
Advanced model note: if a DeepSeek variant fails while qwen3.5-4b works, treat that as a model/backend configuration issue, not a failed GPU deployment. Confirm the baseline model first, then review the DeepSeek model card or LocalAI template for required variables.
Optional Free Web Search with SearXNG
Open WebUI can also use web search results when a prompt needs current information. For a free, self-hosted search option, use SearXNG. SearXNG is a local metasearch service that Open WebUI can query and then pass a small number of search results into the model context.
Recommended free option: use SearXNG for Open WebUI web search. Paid providers such as Brave Search API or SerpApi can be easier to operate, but SearXNG keeps the search layer self-hosted and avoids a paid API key.
Web Search Architecture
Browser
|
+-- Open WebUI on TCP 3000
|
+-- LocalAI on TCP 8080
|
+-- SearXNG on TCP 8081
|
+-- Public search engines
Step 1: Create the SearXNG Configuration Directory
Create persistent storage for the SearXNG configuration.
sudo mkdir -p /opt/containers/searxng sudo nano /opt/containers/searxng/settings.yml
Use this starter configuration. The important setting is formats, because Open WebUI needs JSON output from SearXNG.
use_default_settings: true server: bind_address: "0.0.0.0" port: 8080 secret_key: "CHANGE_THIS_TO_A_LONG_RANDOM_VALUE" search: safe_search: 0 autocomplete: "" default_lang: "en" ui: static_use_hash: true outgoing: request_timeout: 10.0 formats: - html - json
Generate a random secret and replace CHANGE_THIS_TO_A_LONG_RANDOM_VALUE.
openssl rand -hex 32
Step 2: Run SearXNG with Podman
Run SearXNG on host port 8081. The container listens on port 8080 internally.
podman run -d --name searxng -p 8081:8080 -v /opt/containers/searxng:/etc/searxng:Z docker.io/searxng/searxng:latest
Verify that SearXNG returns JSON from the host.
podman ps curl "http://127.0.0.1:8081/search?q=rocky+linux&format=json" | head
Step 3: Recreate Open WebUI with Web Search Enabled
Recreate Open WebUI with SearXNG configured as the web-search provider. The SearXNG URL must be reachable from inside the Open WebUI container, not only from the host shell.
podman stop openwebui podman rm openwebui podman run -d --name openwebui -p 3000:8080 -v /opt/containers/openwebui/data:/app/backend/data -e ENABLE_RAG_WEB_SEARCH=true -e RAG_WEB_SEARCH_ENGINE=searxng -e RAG_WEB_SEARCH_RESULT_COUNT=3 -e RAG_WEB_SEARCH_CONCURRENT_REQUESTS=5 -e SEARXNG_QUERY_URL="http://host.containers.internal:8081/search?q=<query>&format=json" ghcr.io/open-webui/open-webui:main
Container networking note: do not use localhost in the Open WebUI SearXNG URL unless SearXNG is running inside the same container. From inside the Open WebUI container, localhost means Open WebUI itself. Use host.containers.internal or the server’s LAN IP.
Step 4: Test Open WebUI to SearXNG Connectivity
Test from inside the Open WebUI container.
podman exec -it openwebui bash -lc 'curl "http://host.containers.internal:8081/search?q=test&format=json" | head'
If that fails, test with the server LAN IP instead and update SEARXNG_QUERY_URL to match.
podman exec -it openwebui bash -lc 'curl "http://<server-ip>:8081/search?q=test&format=json" | head'
Step 5: Enable Web Search in Open WebUI
In Open WebUI, open the admin settings and confirm web search is enabled.
- Open Admin Panel.
- Open Settings.
- Open Web Search.
- Confirm the engine is set to
searxng. - Set the result count low, usually
2or3, when using a local 6K context model. - Start a new chat and enable the web-search toggle only when current information is needed.
Recommended Local Model Settings for Web Search
Web search adds context to the prompt. On a local model with a 4K to 6K context window, keep search injection small and task-specific.
| Setting | Recommended Value |
|---|---|
| Search results | 2 or 3 |
| Concurrent requests | 5 or fewer |
| Knowledge files | Attach only when needed |
| Tools | Use small task-specific tools |
| Web search | Enable only for prompts that need current information |
SearXNG Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| No web search results | Open WebUI cannot reach SearXNG | Test with podman exec openwebui curl .... |
| Search works from the host but not Open WebUI | Wrong container URL | Use host.containers.internal or the server LAN IP. |
| JSON parse errors | SearXNG JSON output is not enabled | Add json under formats in settings.yml. |
| Context exceeded errors | Too many search results plus tools, knowledge, or chat history | Lower search results to 2 or 3 and start a fresh chat. |
| Slow responses | SearXNG is waiting on slow engines | Lower request timeout or reduce concurrent requests. |
Web search operating rule: let Open WebUI perform web search and pass a small amount of search context to LocalAI. Do not try to route web search through the model itself, and do not leave web search enabled for every local-model prompt.
Validation Checklist
Do not call the build complete until the core services survive basic functional checks and a restart test.
nvidia-smishows the RTX 3060 from the host.- The NVIDIA CDI file exists at
/etc/cdi/nvidia.yaml. - A CUDA test container shows the RTX 3060 with
--device=nvidia.com/gpu=all. podman inforeturns healthy runtime information.- The LocalAI container remains running after deployment.
qwen3.5-4binstalls and appears in/v1/models.- Open WebUI loads in a browser on TCP 3000.
- A basic test prompt returns a useful response.
- GPU utilization changes during inference.
- The LocalAI service starts again after reboot.
podman ps systemctl status pod-localai-pod.service nvidia-smi
Day 2 Operations
The first successful chat response is not the end of the build. Treat the server like a small production service: back up the right data, document the model inventory, and verify that updates do not break GPU acceleration.
| Operational Area | Recommended Practice |
|---|---|
| Backups | Protect /opt/containers/localai/models, /opt/containers/localai/data, /opt/containers/localai/config, and /opt/containers/openwebui/data. |
| Updates | Update the host, NVIDIA stack, and containers intentionally. Validate GPU access after each major update. |
| Monitoring | Watch GPU memory, container restarts, disk growth, and response latency. |
| Access Control | Limit access to trusted systems. Add reverse proxy, TLS, and authentication before broader use. |
| Capacity Planning | If users, models, or RAG workloads grow, plan for more VRAM or a split architecture. |
Operations note: document which models are installed, where they came from, and what they are used for. Model sprawl becomes a maintenance problem quickly once experimentation starts.
Troubleshooting Matrix
| Symptom | Likely Cause | First Check |
|---|---|---|
| GPU is not visible | NVIDIA driver or hardware detection issue | Run nvidia-smi on the host. |
| Container starts but inference is slow | CPU fallback or GPU injection issue | Watch nvidia-smi during inference. |
LocalAI logs show libcuda.so.1 missing | Container was started without CDI GPU injection | Regenerate /etc/cdi/nvidia.yaml and recreate LocalAI with --device=nvidia.com/gpu=all. |
cudaMalloc failed or KV cache allocation fails | Context window is too large for available VRAM, or an mmproj projector is consuming extra memory | Edit the model YAML, set context_size: 2048, remove mmproj for text-only validation, restart LocalAI, and retest. |
| LocalAI container fails to start | Container image, volume, SELinux, CDI, or model backend problem | Run podman logs localai. |
qwen3.5-4b works but a DeepSeek variant fails | Advanced model requires extra variables, backend options, or template tuning | Keep the baseline model installed and review the DeepSeek model card or LocalAI template. |
| Open WebUI loads but cannot chat | API endpoint mismatch or LocalAI unavailable | Confirm Open WebUI points to the LocalAI endpoint. |
| Service does not survive reboot | Systemd unit missing, disabled, or generated incorrectly | Check systemctl status pod-localai-pod.service and review journalctl -u pod-localai-pod.service -n 80 --no-pager. |
| Disk fills quickly | Model downloads or logs growing under persistent paths | Review /opt/containers and container logs. |
nftables compatibility note: warnings such as nft_compat will not be maintained in a future major release or nft_compat_module_init are usually informational on Rocky Linux 9.x. They indicate that an iptables compatibility path is being loaded on top of nftables. Do not treat them as a LocalAI or Podman deployment failure unless firewall rules or container networking are actually failing.
Where to Take This Build Next
After the base LocalAI environment is stable, the real value comes from connecting the model to internal knowledge and automation workflows.
- Add Qdrant or pgvector for retrieval augmented generation.
- Connect VS Code using the Continue extension.
- Experiment with Aider for AI-assisted coding workflows.
- Place Nginx in front as a reverse proxy.
- Add internal TLS certificates.
- Ingest Git repositories for code-aware assistance.
- Index Ansible playbooks and internal runbooks.
RavenHawkTech Analysis
This build works because it keeps the first local AI milestone achievable. It does not require Kubernetes, multi-GPU scheduling, distributed storage, or a full enterprise AI platform. It gives administrators a concrete place to learn how models behave, how GPU acceleration fails, how local inference changes workflow design, and what operational controls matter before the stack grows.
The main limitation is the VRAM ceiling. A 12GB card can run useful models, but it is not a replacement for larger hosted models or multi-user inference infrastructure. As usage grows, the natural next step is to separate the inference service from the user interface, search layer, and vector database.
Upgrade path: treat this LocalAI server as the first rung. Once the workload is stable, move toward a split architecture where inference, web UI, search, and vector storage can be rebuilt or scaled independently.
Final Thoughts
A private AI server built on an RTX 3060 12GB, Rocky Linux 9, Podman, and LocalAI is a realistic platform for self-hosted AI experimentation. It gives administrators a controlled environment for coding assistance, infrastructure automation, and future RAG integration without sending every task to an external service.
For homelabs and small teams, this is a strong starting point: affordable hardware, enterprise-style Linux, containerized deployment, and plenty of room to expand into more advanced internal AI workflows.