Introduction

ARC2 Proxy and NGINX are versatile software solutions that offer similar capabilities and can be used interchangeably in various scenarios. Both support features like serving cached responses, which are crucial for improving the performance and scalability of web applications. In this round of testing, we aim to evaluate their efficiency in serving a cached "Hello World!" response.

By conducting performance tests under identical conditions, we aim to highlight any differences in response times, resource utilization, and scalability between ARC2 Proxy and NGINX. This comparative analysis will assist developers and system administrators in making informed decisions when selecting the most suitable proxy server for their specific requirements.

Features to be tested in this round

  • Serve a cached "Hello World!" response

Setup

Tests will run on a consumer-grade computer with a Linux-based OS and with the following specifications:

  • Kernel Version: 6.11.3-200.fc40.x86_64 (64-bit)
  • Processors: 12 × Intel® Core™ i5-10400 CPU @ 2.90GHz
  • Memory: 23.3 GiB of RAM
  • Linux Distribution: Fedora Linux 40
  • KDE Plasma Version: 6.2.0
  • ARC2 Proxy Version: oxyredox-v2/2.1.29
  • NGINX Version: nginx/1.26.2

For benchmarking, K6 will be used with the following configuration:


import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
    stages: [
        { duration: '5s', target: X },
        { duration: '30s', target: X },
    ],
    insecureSkipTLSVerify: true,
};

export default function () {
    http.get('http://localhost/');
    sleep(1);
}

At the start, there will be a 5-second ramp-up to X VUs so that we can guarantee resource allocation and smooth expansion.

The configuration of ARC2 Proxy:


listening_address = "0.0.0.0"
listening_port_http = 80
listening_port_https = 443
logging_level = "error"
add_caching = true
add_rate_limiting = false
add_logging = false
disable_default_body_limit = false
add_sql_injection_protection = false

[[proxy_rules]]
domain = "localhost"
max_age_seconds = 3600
paths = ["/"]
forward_ipv4 = "127.0.0.1"
forward_port_http = 81
rule_type = "Whitelist"
ignore_query_string = false
enable_logging = false
enable_sql_injection_protection = false

Note that in total ARC2 Proxy will run with 4 workers active (not present in this configurations file).

The configuration of NGINX:


worker_processes 100;

events {
    worker_connections 4096;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    proxy_cache_path /opt/nginx keys_zone=mycache:60m;
    sendfile        on;
    keepalive_timeout 65;

    server {
        listen       79;
        server_name  localhost;

        proxy_cache mycache;
        proxy_cache_valid any 60m;

        location / {
            proxy_pass http://127.0.0.1:81;
        }
    }
}

A Rust Actix-web-based test server is running on port 81.

The results

For 100 VUS

NGINX

NGINX 100 VUS K6 cached response performance results

ARC2 Proxy

ARC2 Proxy 100 VUS K6 cached response performance results
Reference Table
Program Request Latency Delta Total Requests
ARC2 Proxy 170.54 μs 0.4959% 3297
NGINX 171.39 μs - 3297

For 1000 VUS

NGINX

NGINX 1000 VUS K6 cached response performance results

ARC2 Proxy

ARC2 Proxy 1000 VUS K6 cached response performance results
Reference Table
Program Request Latency Delta Total Requests
ARC2 Proxy 153.96 μs - 32975
NGINX 127.98 μs 16.8745% 32970

For 5000 VUS

NGINX

NGINX 5000 VUS K6 cached response performance results

ARC2 Proxy

ARC2 Proxy 5000 VUS K6 cached response performance results
Reference Table
Program Request Latency Delta Total Requests
ARC2 Proxy 186.04 μs - 164887
NGINX 177.44 μs 4.6227% 164888

For 10,000 VUS

NGINX

NGINX 10000 VUS K6 cached response performance results

ARC2 Proxy

ARC2 Proxy 10000 VUS K6 cached response performance results
Reference Table
Program Request Latency Delta Total Requests
ARC2 Proxy 301.06 μs 3.0652% 329709
NGINX 310.58 μs - 329571

For 15,000 VUS

NGINX

NGINX 15000 VUS K6 cached response performance results

ARC2 Proxy

ARC2 Proxy 15000 VUS K6 cached response performance results
Reference Table
Program Request Latency Delta Total Requests
ARC2 Proxy 406.3 μs 3.3631% 494374
NGINX 420.44 μs - 494334

For 20,000 VUS

NGINX

NGINX 20000 VUS K6 cached response performance results

ARC2 Proxy

ARC2 Proxy 20000 VUS K6 cached response performance results
Reference Table
Program Request Latency Delta Total Requests
ARC2 Proxy 2.81 ms 306.4057% 656343
NGINX 8.61 ms - 646381

Conclusions

Analysis

  • Scalability: ARC2 Proxy demonstrated better scalability, maintaining lower latency as the number of requests increased. The substantial difference in latency during the maximum load test highlights ARC2 Proxy's ability to handle high traffic efficiently.
  • Consistency: ARC2 Proxy provided more consistent performance across different load levels. Except for the moderate load test where NGINX had a slight advantage, ARC2 Proxy consistently showed lower latency.
  • Resource Utilization: The similar number of total requests handled by both proxies suggests that resource allocation was comparable. However, the differences in latency imply that ARC2 Proxy utilizes system resources more effectively to process requests faster. During the tests a bit lower CPU utilization was observed for ARC2 Proxy.

Implications

  • Performance Optimization: For applications where low latency is critical, especially under heavy load, ARC2 Proxy appears to be the superior choice. Its ability to maintain lower response times can enhance user experience and reduce server strain.
  • High-Traffic Environments: In scenarios involving a large number of concurrent users or requests, ARC2 Proxy's scalability offers a significant advantage. It can handle spikes in traffic without a proportional increase in latency, making it suitable for high-availability services.
  • Efficiency in Caching: Since both proxies were tested on serving cached responses, ARC2 Proxy's performance indicates a more efficient caching mechanism or better handling of cached data retrieval. Specifically it uses in-memory caching for repeated requests.

The benchmarking results indicate that ARC2 Proxy generally outperforms NGINX in serving cached responses, particularly under high load conditions. Its lower latency and consistent performance make it a compelling option for applications requiring efficient request handling and scalability. While NGINX remains a robust and widely-used proxy server, ARC2 Proxy's advantages in this specific use case suggest that it may be better suited for environments where performance and scalability are paramount.

By understanding the strengths of each proxy, developers and system administrators can make informed decisions that align with their application's requirements and performance goals.