Every DevOps pipeline eventually hits a wall when it needs to talk to the outside world. Your build agents need to pull dependencies from public registries. Your deployment scripts need to reach artifact repositories. Your monitoring tools need to send data to external dashboards. Without a central point of control, every request becomes a security question mark. That is where a custom proxy server steps in. It gives you a single place to enforce policies, log traffic, and isolate your internal network from the chaos of the public internet. Building one for your internal DevOps pipeline is not as hard as it sounds. Here is exactly how to do it.
A custom proxy server for your DevOps pipeline gives you full control over outbound traffic from build agents, deployment runners, and CI/CD tools. You can enforce allowlists, cache dependencies to speed up builds, and audit every request for compliance. This guide walks you through choosing the right protocol, setting up Squid or Nginx, configuring access controls, and integrating with tools like Jenkins, GitLab CI, and GitHub Actions. You will also learn common pitfalls and how to avoid them.
Why Your Pipeline Needs Its Own Proxy
Off the shelf proxy services work fine for general web browsing. But your DevOps pipeline has different needs. It makes thousands of requests per minute. It needs to authenticate with private package registries. It must bypass the proxy for internal services like your own Git server or artifact store. A generic proxy cannot handle these nuances without a lot of configuration pain.
A custom proxy server gives you:
- Granular access control. You decide which external hosts your build agents can reach.
- Traffic inspection. Log every request for compliance audits without relying on a third party.
- Caching. Store frequently downloaded packages locally to speed up builds and reduce bandwidth costs.
- Resilience. Fail over to backup proxies when your primary server goes down.
If you are managing a team of DevOps engineers or platform architects, a custom proxy is one of the best investments you can make for security and network isolation.
Choosing the Right Proxy Protocol for DevOps Traffic
Not all proxies are created equal. For internal pipeline traffic, you have three main options. Each one serves a different purpose.
| Protocol | Best For | Weakness |
|---|---|---|
| HTTP/HTTPS forward proxy | Package managers (npm, pip, Maven), API calls | Does not handle arbitrary TCP traffic |
| SOCKS5 proxy | SSH tunnels, database connections, any TCP stream | Slower for HTTP, no built in caching |
| Transparent proxy | Legacy tools that cannot be configured with proxy settings | Harder to debug, limited access control |
For most DevOps pipelines, an HTTP/HTTPS forward proxy is the right choice. Tools like curl, wget, and most CI runners support the HTTP_PROXY and HTTPS_PROXY environment variables natively. You can point your entire pipeline at it with a single configuration change.
If you need to route SSH or database traffic through the proxy, add a SOCKS5 proxy alongside your HTTP proxy. Many teams run both on the same machine using different ports.
Step by Step: Building Your Custom Proxy Server
Let us walk through setting up a production ready proxy using Squid, one of the most reliable open source forward proxy servers. These steps assume you are running Ubuntu 24.04 LTS, but the same concepts apply to any Linux distribution.
1. Install and Configure Squid
sudo apt update
sudo apt install squid -y
Once installed, edit the main configuration file at /etc/squid/squid.conf. Start by defining which internal networks are allowed to use the proxy.
acl internal_network src 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
http_access allow internal_network
http_access deny all
This ensures only your internal infrastructure can send requests through the proxy. Block everything else.
2. Set Up an Allowlist for External Hosts
You do not want your build agents talking to random servers on the internet. Create an allowlist file at /etc/squid/allowed_sites.acl.
.registry.npmjs.org
.pypi.org
.docker.io
.github.com
.maven.org
Then add this line to your squid.conf:
acl allowed_sites dstdomain "/etc/squid/allowed_sites.acl"
http_access allow allowed_sites
http_access deny all
Now your pipeline can only reach the registries and services you have approved. Everything else gets blocked.
3. Enable Caching for Faster Builds
Caching is where a custom proxy really shines. Add these lines to your squid.conf:
cache_dir ufs /var/spool/squid 10000 16 256
maximum_object_size 512 MB
cache_replacement_policy heap LFUDA
This allocates 10 GB of disk space for cached objects and keeps frequently accessed packages in memory. Your next build will pull dependencies from the local cache instead of downloading them again.
4. Configure Authentication for Private Registries
Some requests need authentication headers to access private packages. Squid can forward these headers without stripping them. Add this line:
request_header_access Authorization allow all
If you need to authenticate the proxy itself, use basic authentication:
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwords
acl authenticated proxy_auth REQUIRED
http_access allow authenticated
5. Integrate with Your CI/CD Tools
Most CI systems let you set environment variables at the pipeline level. For Jenkins, add these to your global environment or to each job:
HTTP_PROXY=http://proxy.internal:3128
HTTPS_PROXY=http://proxy.internal:3128
NO_PROXY=localhost,127.0.0.1,.internal.company.com
For GitLab CI, add them to your .gitlab-ci.yml:
variables:
HTTP_PROXY: "http://proxy.internal:3128"
HTTPS_PROXY: "http://proxy.internal:3128"
NO_PROXY: "localhost,127.0.0.1,.internal.company.com"
The NO_PROXY variable is critical. It tells your tools to bypass the proxy for internal services like your Git server, artifact repository, and monitoring dashboards.
Common Mistakes and How to Avoid Them
Building a custom proxy server is straightforward, but a few misconfigurations can cause headaches. Here are the most common ones I have seen teams run into.
| Mistake | Symptom | Fix |
|---|---|---|
| Forgetting NO_PROXY | Builds fail trying to reach internal Git server | Always set NO_PROXY for internal domains |
| Allowing all destinations | Security audit flags unexpected outbound traffic | Use a strict allowlist, not a blocklist |
| Disabling caching | Slow builds, high bandwidth usage | Enable cache and monitor hit rates |
| Using HTTP for sensitive data | Credentials sent in plaintext | Force HTTPS for all outbound requests |
| No monitoring or logging | Cannot trace which agent made a request | Enable access logs and ship them to a central log server |
Expert advice: Start with a strict allowlist and loosen it only when you have verified that a new destination is necessary. It is much easier to add a domain to the allowlist than to clean up after a security incident caused by an overly permissive proxy.
Monitoring Your Proxy for Health and Performance
A proxy that goes down in the middle of a deployment can halt your entire pipeline. You need visibility into its health. Set up a simple health check endpoint that your monitoring tools can hit.
Squid exposes basic statistics via its cache manager. Add this to your squid.conf:
acl manager_acl src 10.0.0.0/8
http_access allow manager_acl
cachemgr_passwd secret_password all
Now you can query the proxy from your monitoring system:
curl -u "user:secret_password" http://proxy.internal:3128/squid-internal-mgr/info
If you want a deeper look at performance, check out our guide on how to monitor proxy server health and performance in real time. It covers metrics like cache hit ratio, connection latency, and request throughput.
Security Hardening for Production Use
Your proxy is a gateway to the internet. That makes it a target. Follow these steps to lock it down.
- Run the proxy as a non root user. Squid drops privileges by default, but double check that the process runs as the
proxyuser. - Restrict access by source IP. Only allow your build agent subnets to reach the proxy.
- Use TLS for client to proxy communication. This prevents other services on your network from sniffing requests.
- Rotate logs daily. Large log files can fill a disk and cause the proxy to fail.
- Set rate limits. Prevent a single agent from overwhelming the proxy with requests.
For a complete security checklist, read our guide on 7 essential proxy security checks every IT admin should run in 2026.
Scaling Your Proxy for Growing Pipelines
As your team grows, a single proxy server can become a bottleneck. Here are three ways to scale.
Horizontal scaling with DNS round robin. Deploy multiple proxy servers and point a single DNS name to all of them. Your CI tools will pick a random server on each connection.
Load balancing with HAProxy. Put a load balancer in front of your proxy pool. This gives you more control over traffic distribution and lets you take servers offline for maintenance.
Geographic distribution. If your team spans multiple data centers, deploy a proxy in each region. Configure your CI tools to use the nearest proxy based on DNS resolution.
For more on load balancing strategies, see our guide on 5 best practices for load balancing with proxy servers in 2026.
Your Pipeline Deserves a Proxy That Works for You
Building a custom proxy server for your internal DevOps pipeline is one of those projects that pays off almost immediately. Your builds become faster because dependencies are cached locally. Your security team sleeps better because every outbound request is logged and controlled. Your compliance audits become simpler because you can point to a single configuration file that defines exactly where your pipeline can talk to.
Start small. Deploy a single Squid instance, configure an allowlist with your most common registries, and point one CI job at it. Once you see how much control it gives you, you will wonder why you did not do this sooner. The full guide on building a custom proxy server for your DevOps pipeline is available on our site, along with templates and configuration examples to get you started.