Advanced Tutorial: Implementing Security with SSL/TLS and HTTPS

Last quarter, our team hit a wall that I didn't see coming. We'd scaled our API from handling maybe 500k requests per day to over 50 million. Everything was humming along nicely until our security audit came back with some... let's call them "concerning" findings about our SSL/TLS implementation.

I thought we'd done everything right. We had certificates from Let's Encrypt, we were using HTTPS everywhere, and our security headers looked good in the browser. But when the penetration testers got their hands on our infrastructure, they found we were vulnerable to downgrade attacks, our cipher suites were a mess, and we weren't properly implementing certificate pinning for our mobile apps.

The real kicker? We discovered all this three days before a major client deployment. A client who specifically required PCI DSS compliance. Yeah, that was a fun week.

Here's what I learned after spending two solid weeks rebuilding our entire SSL/TLS infrastructure from the ground up, reading every RFC I could find, and having some very uncomfortable conversations with our infrastructure team about what "production-ready" actually means.

The Problem Nobody Talks About: SSL/TLS Isn't Just About Certificates

Most tutorials will tell you to slap a Let's Encrypt certificate on your server and call it a day. And look, that's fine for your side project or your blog. But when you're handling millions of requests, processing payments, or dealing with sensitive user data, you need to understand what's actually happening under the hood.

I made this mistake initially. I thought SSL/TLS was just about encryption. Get a certificate, configure Nginx, done. But SSL/TLS is actually a complex negotiation protocol with dozens of configuration options, multiple cipher suites, various TLS versions, and a whole ecosystem of certificate authorities, intermediate certificates, and trust chains.

When our security audit failed, I realized I'd been treating SSL/TLS like a checkbox item instead of understanding it as a critical security layer that needed careful configuration and ongoing maintenance.

The first issue they found? We were still accepting TLS 1.0 and 1.1 connections. These protocols have known vulnerabilities and should've been disabled years ago. But our Nginx configuration had never been updated from the defaults we'd set up three years prior.

The second issue was worse. Our cipher suite configuration was allowing weak ciphers that were vulnerable to various attacks. We weren't enforcing perfect forward secrecy, which meant if someone compromised our private key, they could decrypt all historical traffic they'd captured.

And the third issue? We had no proper certificate rotation strategy. Our certificates were set to expire in 60 days, and we had no automation in place. We were literally setting calendar reminders to manually renew certificates. At scale, this was a disaster waiting to happen.

How SSL/TLS Actually Works (The Parts That Matter in Production)

Let me walk you through what actually happens when a client connects to your server over HTTPS, because understanding this is crucial for debugging production issues.

When a client initiates an HTTPS connection, here's the actual sequence:

1. Client Hello: The client sends a message listing all the TLS versions and cipher suites it supports
2. Server Hello: Your server responds with the TLS version and cipher suite it wants to use
3. Certificate Exchange: Your server sends its certificate (and intermediate certificates)
4. Key Exchange: Both parties establish a shared secret for encrypting the session
5. Finished: Both sides verify everything worked and begin encrypted communication

Sounds simple, right? But here's where it gets interesting in production.

The Certificate Chain Problem We Hit

Our first major issue was with certificate chains. When you get a certificate from Let's Encrypt (or any CA), you don't just get one certificate. You get a chain:

• Your server certificate (proves your domain identity)
• Intermediate certificate(s) (proves your CA is legitimate)
• Root certificate (already trusted by browsers)

I didn't properly configure the intermediate certificates initially. Our Nginx config looked like this:

server {
    listen 443 ssl;
    server_name api.example.com;
    
    ssl_certificate /etc/letsencrypt/live/api.example.com/cert.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
}

This worked fine in Chrome and Firefox because they do something called "AIA fetching" - they'll automatically download missing intermediate certificates. But Safari and some mobile browsers don't do this. We started getting reports from iOS users that our API was showing certificate errors.

The fix was to use the full chain:

server {
    listen 443 ssl http2;
    server_name api.example.com;
    
    ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
}

That fullchain.pem file includes both your certificate and the intermediate certificates. This seems obvious now, but it cost us hours of debugging when users reported intermittent SSL errors.

Cipher Suite Configuration: The Deep Dive

This is where things get really interesting. Cipher suites determine how your encrypted connection actually works. Each cipher suite specifies:

• Key exchange algorithm (how you establish the shared secret)
• Authentication algorithm (how you verify identity)
• Encryption algorithm (how you encrypt data)
• MAC algorithm (how you verify data integrity)

When I first looked at our cipher suite configuration, it was using the Nginx defaults. Here's what that looked like:

$ openssl s_client -connect api.example.com:443 -cipher 'ALL'

Output showed we were accepting things like:

TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA

These are all weak or outdated ciphers. The 3DES cipher is vulnerable to Sweet32 attacks. The CBC mode ciphers are vulnerable to Lucky13 attacks. And none of these provide perfect forward secrecy.

Here's the production-ready configuration I eventually settled on after testing with multiple clients:

# TLS Configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
ssl_prefer_server_ciphers off;

# Perfect Forward Secrecy
ssl_ecdh_curve secp384r1;
ssl_dhparam /etc/nginx/dhparam.pem;

# Session Configuration
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/api.example.com/chain.pem;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

Let me break down why each of these matters:

TLS Protocols: We only support TLS 1.2 and 1.3. TLS 1.0 and 1.1 have known vulnerabilities. We tested this with our analytics and found that 99.8% of our traffic was already using TLS 1.2+. The 0.2% that wasn't? Ancient Android devices that we decided weren't worth the security risk.

Cipher Suites: Every cipher in that list provides:

• ECDHE (Elliptic Curve Diffie-Hellman Ephemeral) for perfect forward secrecy
• GCM or CHACHA20-POLY1305 for authenticated encryption
• Strong key sizes (128-bit minimum, 256-bit preferred)

ssl_prefer_server_ciphers off: This was counterintuitive to me at first. With TLS 1.3, the client's cipher preference actually matters more because TLS 1.3 cipher suites are all secure. Setting this to off lets modern clients use their preferred cipher.

OCSP Stapling: This is huge for performance. Without OCSP stapling, the client has to make a separate request to the CA to verify your certificate hasn't been revoked. This adds latency. With stapling, your server fetches the OCSP response and includes it in the TLS handshake. We measured this reducing our TLS handshake time by about 200ms on average.

To generate the DH parameters file:

$ openssl dhparam -out /etc/nginx/dhparam.pem 4096

This takes a while (like 10-15 minutes), but you only do it once. This file is used for the Diffie-Hellman key exchange to ensure perfect forward secrecy.

Certificate Automation: How We Went From Manual to Zero-Touch

After our near-miss with certificate expiration, I knew we needed full automation. Manual certificate management doesn't scale, and it's a security risk. Here's how we built our certificate automation system.

The Let's Encrypt + Certbot Setup

We use Let's Encrypt because it's free, automated, and trusted by all major browsers. But the real magic is in how you automate it.

First, the basic Certbot installation:

$ sudo apt-get update
$ sudo apt-get install certbot python3-certbot-nginx

The naive approach is to just run:

$ sudo certbot --nginx -d api.example.com

This works, but it has problems. Certbot modifies your Nginx config, which can cause issues if you're managing configs with Ansible or Terraform. It also doesn't give you fine-grained control over the renewal process.

Here's our production approach using DNS validation:

$ sudo certbot certonly \
  --dns-route53 \
  --dns-route53-propagation-seconds 30 \
  -d api.example.com \
  -d *.api.example.com \
  --preferred-challenges dns-01 \
  --email security@example.com \
  --agree-tos \
  --non-interactive

We use DNS validation (specifically Route53 since we're on AWS) because:

1. It allows wildcard certificates
2. It doesn't require opening port 80 for HTTP validation
3. It works even if your server isn't publicly accessible yet
4. It's more reliable for multi-server deployments

The certificate files end up in:

/etc/letsencrypt/live/api.example.com/
├── cert.pem
├── chain.pem
├── fullchain.pem
└── privkey.pem

Automated Renewal with Monitoring

Let's Encrypt certificates expire after 90 days. Certbot sets up a systemd timer to renew them automatically, but we added our own monitoring on top of that.

Here's our renewal script that runs daily:

#!/bin/bash
# /usr/local/bin/renew-certs.sh

set -e

LOG_FILE="/var/log/certbot-renewal.log"
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

echo "$(date): Starting certificate renewal check" >> "$LOG_FILE"

# Attempt renewal
if certbot renew --quiet --deploy-hook "/usr/local/bin/reload-nginx.sh"; then
    echo "$(date): Renewal check completed successfully" >> "$LOG_FILE"
else
    ERROR_MSG="Certificate renewal failed on $(hostname)"
    echo "$(date): $ERROR_MSG" >> "$LOG_FILE"
    
    # Alert to Slack
    curl -X POST -H 'Content-type: application/json' \
      --data "{\"text\":\"🚨 $ERROR_MSG\"}" \
      "$SLACK_WEBHOOK"
    
    exit 1
fi

# Check certificate expiration
DAYS_UNTIL_EXPIRY=$(openssl x509 -enddate -noout -in /etc/letsencrypt/live/api.example.com/cert.pem | cut -d= -f2 | xargs -I{} date -d "{}" +%s | xargs -I{} expr \( {} - $(date +%s) \) / 86400)

echo "$(date): Certificate expires in $DAYS_UNTIL_EXPIRY days" >> "$LOG_FILE"

if [ "$DAYS_UNTIL_EXPIRY" -lt 30 ]; then
    curl -X POST -H 'Content-type: application/json' \
      --data "{\"text\":\"⚠️ Certificate for api.example.com expires in $DAYS_UNTIL_EXPIRY days\"}" \
      "$SLACK_WEBHOOK"
fi

The deploy hook script:

#!/bin/bash
# /usr/local/bin/reload-nginx.sh

# Test nginx config before reloading
if nginx -t 2>&1 | grep -q "syntax is ok"; then
    systemctl reload nginx
    echo "$(date): Nginx reloaded successfully" >> /var/log/certbot-renewal.log
else
    echo "$(date): Nginx config test failed!" >> /var/log/certbot-renewal.log
    exit 1
fi

We run this via cron:

0 3 * * * /usr/local/bin/renew-certs.sh

This runs at 3 AM daily. Certbot is smart enough to only renew when necessary (within 30 days of expiration), so running it daily doesn't cause issues.

Multi-Server Certificate Distribution

Here's where it gets tricky. We run multiple application servers behind a load balancer. Each server needs the same certificates. We tried a few approaches:

Approach 1: Certbot on each server - This caused rate limiting issues with Let's Encrypt. We hit their 50 certificates per registered domain per week limit.

Approach 2: Central certificate server - We set up one server to manage certificates and rsync'd them to other servers. This worked but felt fragile.

Approach 3: AWS Certificate Manager (ACM) - This is what we ultimately went with for our production environment. ACM handles certificate provisioning, renewal, and distribution automatically. But it only works with AWS load balancers and CloudFront.

Here's our Terraform configuration for ACM:

resource "aws_acm_certificate" "api" {
  domain_name       = "api.example.com"
  validation_method = "DNS"
  
  subject_alternative_names = [
    "*.api.example.com"
  ]
  
  lifecycle {
    create_before_destroy = true
  }
  
  tags = {
    Name        = "api-certificate"
    Environment = "production"
  }
}

resource "aws_route53_record" "cert_validation" {
  for_each = {
    for dvo in aws_acm_certificate.api.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }
  
  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = aws_route53_zone.main.zone_id
}

resource "aws_acm_certificate_validation" "api" {
  certificate_arn         = aws_acm_certificate.api.arn
  validation_record_fqdns = [for record in aws_route53_record.cert_validation : record.fqdn]
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.api.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS-1-2-2017-01"
  certificate_arn   = aws_acm_certificate_validation.api.certificate_arn
  
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.api.arn
  }
}

With ACM, certificates auto-renew and are automatically distributed to all load balancer instances. We went from spending hours per month on certificate management to zero operational overhead.

Perfect Forward Secrecy: Why It Matters and How to Implement It

Perfect Forward Secrecy (PFS) is one of those security features that sounds optional until you understand what it actually protects against.

Here's the scenario that made me care about PFS: Imagine someone is passively recording all your encrypted traffic.