A year of hosting in Palitana: a retrospective

We’ve been hosting customers from the Palitana data center for a full year now. Here is the retrospective — what worked, what didn’t, and what we’re planning for year 2.

The numbers

• Customers hosted: 47 active subscriptions
• Apps running: 12 (WordPress + ERPNext + 1 development instance per major app)
• P95 latency (India): 18 ms
• P95 latency (US): 142 ms
• P95 latency (EU): 158 ms
• Uptime: 99.92% (a single 6-hour outage during the BSNL power-cut incident in Q2)
• Support tickets: 312 (median first response: 47 minutes; median resolution: 2.4 hours)
• Bandwidth: 18 TB / month egress

The outages

Three notable ones in the first year:

1. BSNL power-cut (Q2): 6-hour outage. The UPS bridge worked for 30 minutes as designed; the generator had a faulty transfer switch. Lesson: test the generator transfer switch quarterly, not annually. We now have a quarterly drill.
2. CDN cache poisoning (Q3): 4-hour partial degradation on the static assets. The cache key derivation was missing a query-string component, causing legitimate cache-busting URLs to share a key with stale ones. Lesson: derive cache keys from the full URL, not just the path.
3. Database connection pool saturation (Q4): 2-hour incident on the S3 Standard tier. The pool size was hard-coded to 10 connections; load-testing had shown we needed 40. Lesson: load-test every tier quarterly, not just at launch.

The wins

• Latency. 18 ms P95 to the closest Indian ISP is competitive with the US-based CDNs (typically 80-150 ms). Most of our customers’ customers are in India, so this matters.
• WhatsApp support. Median first-response time of 47 minutes is the metric we’re most proud of. No ticket portal, no email-only support, just WhatsApp.
• INR pricing. The plan-page INR price is the invoice price. No FX surprises.

The lessons

1. Document the failover matrix. When the power went out, the runbook said “transfer to generator after UPS bridge” but didn’t cover what to do if the transfer switch failed. We now have a 2-page decision tree + a quarterly drill.
2. Invest in operational observability. The CDN cache poisoning took 4 hours to diagnose because we didn’t have the right dashboards. We now have a Grafana + Prometheus stack on the same network with 30-second resolution metrics + PagerDuty alerting.
3. Capacity-planning is ongoing. The database pool saturation taught us that load-testing at launch isn’t enough. We now re-run quarterly.

The roadmap for year 2

• Multi-region replication. Currently single-region (Palitana only). Adding a second region (Bangalore) for redundancy is a Q1 priority.
• Edge compute. Custom routing logic at the CDN edge for traffic shaping. Not a CDN-managed feature; we’d ship a small Node service.
• GPU plans. A handful of customers have asked for GPU-backed containers for ML inference. The L3+ instances can host an A100 or H100; pricing TBD.
• Backup target upgrade. Currently the daily backups land on the S3 Standard tier. We’re moving them to the new Infrequent Access tier to save cost on the 30-day+ retention that’s not in the product yet but is on the roadmap.