Advanced Usage of ProxyHub in Python

Take your scraper to the next level with advanced features: concurrent rotations, caching responses, custom headers, and Tor integration.

📖 Table of Contents

• Concurrent Requests & Pooling
• Response Caching
• Injecting Custom Headers
• Tor Network Support
• Logging & Metrics

1. Concurrent Requests & Pooling

Use Python’s concurrent.futures to fire parallel requests and rotate IPs seamlessly:

import requests
from concurrent.futures import ThreadPoolExecutor

API_KEY = "YOUR_API_KEY"
BASE = f"https://proxy.montgomerynx.com/{API_KEY}"

def fetch(url):
    return requests.get(f"{BASE}/{url}", timeout=10).text

urls = ["http://httpbin.org/ip"] * 10

with ThreadPoolExecutor(max_workers=5) as executor:
    results = executor.map(fetch, urls)

for body in results:
    print(body)

2. Response Caching

Avoid redundant network calls by caching successful responses:

from functools import lru_cache

@lru_cache(maxsize=128)
def get_page(path):
    res = requests.get(f"{BASE}/{path}")
    res.raise_for_status()
    return res.text

print(get_page("data.json"))  # cached on subsequent calls

3. Injecting Custom Headers

Add your own headers (e.g., User-Agent, cookies) to mimic browsers:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept-Language": "en-US,en;q=0.9"
}

resp = requests.get(f"{BASE}/httpbin/headers", headers=headers)
print(resp.json())

4. Tor Network Support

Route through Tor by appending a query flag:

resp = requests.get(f"{BASE}/force_rotate_ip?use_tor=true")
print("New Tor exit IP:", resp.json()["ip"])

5. Logging & Metrics

Instrument your scraper for visibility:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("scraper")

def fetch_and_log(url):
    try:
        resp = requests.get(url)
        resp.raise_for_status()
        logger.info(f"200 OK: {url}")
        return resp.text
    except Exception as e:
        logger.error(f"Error fetching {url}: {e}")
        raise

Use Prometheus or your favorite APM to collect these logs for dashboards and alerts.