CDN Content Routing — [Notes]

17 min readAug 7, 2023

link for all System design articles

· Upload Flow /Image Upload Process to Cloud Storage:
∘ a. Image Upload Process to Cloud Storage:
∘ b. Upload through the backend:
· Image download/render process via CDN
∘ Example CDN configuration (using AWS CloudFront):
∘ How the CDN determines which content to serve:
∘ Best Practices:
· multi-CDN strategy
∘ DNS Configuration:
∘ Application Configuration:
· Best Practices for Multi-CDN:
· Download/render flow
· Process of resolving CDN URLs

When Instagram forwards content to a CDN (Content Delivery Network) server, it uses a process called content routing to determine which server the data should be forwarded to.

Content routing is a mechanism that uses various criteria, such as the user’s location, the content’s origin, and the CDN’s server locations, to decide which server should handle the request.

When a user in India uploads an image on Instagram, the content routing mechanism will take into consideration the user’s location and the location of the CDN servers. If there is a CDN server located in India, the content will be forwarded to that server to reduce latency and improve the user’s experience. If there is no server located in India, the content routing mechanism will forward the content to the nearest server based on network conditions, such as network latency and bandwidth availability.

Similarly, when the user’s friends in Europe access the image, the content routing mechanism will forward the content to the nearest server in Europe to reduce latency and improve their experience.

In summary, the content routing mechanism in the CDN servers is responsible for determining the best server to handle the request based on various factors, including the user’s location, the content’s origin, and the CDN server’s locations.

When Instagram sends material to CDN servers, it uses a process called content routing to determine which location server the data should be sent to. Content routing is the process of selecting the appropriate server to serve a particular piece of content based on factors such as the user’s location, the availability of the content on different servers, and network conditions.

To accomplish this, Instagram uses a combination of techniques, including DNS-based content routing and IP geolocation.

Upload Flow /Image Upload Process to Cloud Storage:

There are multiple ways to handle uploads:

a. Image Upload Process to Cloud Storage:

a) Pre-signed URL Generation:

The client requests a pre-signed URL from the application server.
The server, authenticated with the cloud storage service (e.g., AWS S3), generates a pre-signed URL. / The backend generates a pre-signed URL for S3 (or similar storage service).
This URL is time-limited and grants temporary permission to upload a specific object to a specific location in the storage bucket.

b) Direct Upload:

The client uses the pre-signed URL to upload the image directly to the cloud storage.
This upload bypasses the application server, reducing server load.

c) Upload Confirmation:

Upon successful upload, the client notifies the application server.
The server then records metadata about the uploaded image in its database.

d) CDN registration:

This step isn’t always necessary if the CDN is configured to pull from the storage service directly.
In some setups, the backend might indeed register the asset with the CDN or invalidate existing cached versions.

Example using AWS S3 and Node.js:

const AWS = require('aws-sdk');
const s3 = new AWS.S3();

async function getPresignedUrl(fileName, fileType) {
  const params = {
    Bucket: 'your-bucket-name',
    Key: `images/${fileName}`,
    ContentType: fileType,
    Expires: 600 // URL expires in 10 minutes
  };

  try {
    const signedUrl = await s3.getSignedUrlPromise('putObject', params);
    return signedUrl;
  } catch (error) {
    console.error("Error generating pre-signed URL", error);
    throw error;
  }
}

Significance of sending the upload URL:

Security: The pre-signed URL provides temporary, scoped access to the storage bucket.
Performance: Direct upload to storage reduces the load on application servers.
Flexibility: The server can control where and how files are stored without exposing these details to the client.

b. Upload through the backend:

- The client uploads the image to the backend server.
- The backend then uploads it to S3 or another storage service.
- This method gives more control but puts more load on your servers.

In both cases, the backend typically stores metadata about the uploaded file (e.g., URL, size, type) in its database.

Image download/render process via CDN

When a client requests an image, the process typically follows these steps:

a) Client Request:

The client requests an image using a URL, e.g., https://cdn.example.com/images/profile-pic-123.jpg

b) CDN Edge Server Handling:

The request hits the nearest CDN edge server.
The CDN checks its cache for the requested image.

c) Cache Hit Scenario:

If the image is in the cache and has not expired, the CDN will serve it directly to the client.
This is the fastest scenario, significantly reducing latency.

d) Cache Miss Scenario:

If the image is not in the cache or has expired, the CDN requests it from the origin.
The origin could be your application server or directly the cloud storage (e.g., S3).
The CDN retrieves the image, caches it, and serves it to the client.
Subsequent requests for this image will be cache hits until the cache expires.

Example CDN configuration (using AWS CloudFront):

{
  "DefaultCacheBehavior": {
    "MinTTL": 0,
    "DefaultTTL": 86400,
    "MaxTTL": 31536000,
    "ForwardedValues": {
      "QueryString": false
    },
    "ViewerProtocolPolicy": "redirect-to-https",
    "TargetOriginId": "S3-your-bucket-name"
  }
}

How the CDN determines which content to serve:

URL Structure: The URL path (e.g., /images/profile-pic-123.jpg) is mapped to the storage location.
Origin Configuration: The CDN is configured with rules to map URL patterns to specific origins.
Cache Key: The CDN uses the URL (and potentially other factors like headers) as a cache key to store and retrieve content.

Best Practices:

Use content versioning in URLs (e.g., /images/v2/profile-pic-123.jpg) for cache busting when content updates.
Set appropriate Cache-Control headers on your origin to guide CDN caching behavior.
Use CDN features like origin shield to reduce load on your origin servers.
Implement proper error handling and fallback mechanisms in case of CDN failures.

This setup allows for efficient, scalable, and fast delivery of images, leveraging the global distribution of CDN edge servers while maintaining control over the content through your cloud storage solution.

multi-CDN strategy

Using multiple CDNs, a multi-CDN strategy, is a common practice for large-scale applications to improve performance, reliability, and global reach.

In a multi-CDN setup, the process typically works as follows:

DNS-based CDN Selection:

The client is given a single hostname (e.g., content.example.com) for all CDN requests.
This hostname is managed by a DNS-based load-balancing system.

2. Intelligent DNS Resolution:

When the client performs a DNS lookup for this hostname, the DNS system (often a Global Server Load Balancing or GSLB solution) determines the optimal CDN based on factors like:
Geographic proximity
CDN health/performance
Cost
Traffic distribution policies

3. CDN-specific URL Resolution:

The DNS system returns the IP address of the chosen CDN’s edge server.
The client then makes the request to this IP address, using the original hostname.

4. Request Handling:

The chosen CDN handles the request as normal, either serving from the cache or fetching from the origin.

5. CDN-specific URLs:

In most cases, you don’t need CDN-specific URLs. The same URL works across all CDNs.
If you do need CDN-specific URLs (rare cases), you can use:
URL rewriting at the CDN level
Dynamic URL generation in your application

DNS Configuration:

Use a DNS provider that supports GSLB, like AWS Route 53, Cloudflare, or NS1.

Example Route 53 configuration:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "MultiCDNRecord": {
      "Type": "AWS::Route53::RecordSet",
      "Properties": {
        "HostedZoneName": "example.com.",
        "Name": "content.example.com.",
        "Type": "A",
        "SetIdentifier": "Primary",
        "Weight": 90,
        "AliasTarget": {
          "DNSName": "d123456abcdef8.cloudfront.net.",
          "HostedZoneId": "Z2FDTNDATAQYW2"
        }
      }
    },
    "SecondaryMultiCDNRecord": {
      "Type": "AWS::Route53::RecordSet",
      "Properties": {
        "HostedZoneName": "example.com.",
        "Name": "content.example.com.",
        "Type": "A",
        "SetIdentifier": "Secondary",
        "Weight": 10,
        "AliasTarget": {
          "DNSName": "example.global.fastly.net.",
          "HostedZoneId": "Z2FDTNDATAQYW2"
        }
      }
    }
  }
}

Application Configuration:

In your application, always use the generic hostname:

const CDN_HOSTNAME = 'content.example.com';
const imageUrl = `https://${CDN_HOSTNAME}/images/profile-pic-123.jpg`;

CDN Configuration: Ensure all CDNs are configured to handle requests for the same hostname and path structure.
Monitoring and Optimization: Implement monitoring to track the performance of each CDN and adjust your DNS weights or policies accordingly.

Best Practices for Multi-CDN:

Use consistent URL structures across all CDNs.
Implement proper cache control headers to ensure consistency across CDNs.
Use a central configuration management system to keep CDN settings in sync.
Implement robust health checks and failover mechanisms.
Consider using a CDN broker service for more advanced multi-CDN management.

This approach allows you to leverage multiple CDNs while maintaining a simple, consistent URL structure for your clients. The complexity of CDN selection is handled at the DNS level, making it transparent to both your application and end-users.

Download/render flow

In a well-optimized setup, the client typically contacts the CDN directly for static content without first reaching out to the backend/origin server. This direct contact is crucial for maximizing the CDN's performance benefits.

Client Contact for Static Content:

In a well-optimized setup, the client typically contacts the CDN directly for static content, without first reaching out to the backend/origin server. This direct contact is crucial for maximizing the performance benefits of a CDN.

2. CDN Selection Process:

When multiple CDNs are in use (a multi-CDN strategy), the selection process typically works as follows:

a. DNS Resolution:

The client is given a single hostname for all CDN requests (e.g., content.example.com).
When resolving this hostname, the request goes to a Global Server Load Balancing (GSLB) system.

b. GSLB Decision:

The GSLB system determines the optimal CDN based on factors like:
Geographic proximity
Current CDN performance and health
Cost considerations
Traffic distribution policies

c. IP Return:

The GSLB returns the IP address of the chosen CDN’s edge server to the client.

d. Content Request:

The client then requests the content from this IP address using the original hostname.

3. CDN Content Serving Process:

The CDN determines which content to serve based on the URL path and any additional parameters. Here’s a typical flow:

a. URL Structure:

The URL typically includes a path that maps to the content location, e.g.: https://content.example.com/images/profile-pic-123.jpg

b. CDN Request Handling:

The CDN edge server receives the request and extracts the path (/images/profile-pic-123.jpg).
It checks its cache for this specific content.

c. Cache Hit:

If the content is in the cache and not expired, it’s served directly to the client.

d. Cache Miss:

If not in the cache, the CDN requests the content from the origin server.
The origin server (or storage service like S3) returns the content.
The CDN caches the content and serves it to the client.

4. Client Request Type:

The client typically makes a standard HTTP GET request for the content. For example:

GET /images/profile-pic-123.jpg HTTP/1.1
Host: content.example.com

5. Detailed CDN Content Delivery Process:

a. Initial Setup:

During application development, you decide on a URL structure for your static content.
You configure your CDN to map these URLs to the correct locations on your origin server or storage service.

b. Content Upload:

When content is uploaded (e.g., to S3), your application records the appropriate CDN URL for that content.

c. Content Delivery:

Your application serves HTML that includes these CDN URLs for static content.
When a browser renders the page, it makes separate requests for each piece of static content to the CDN.

d. CDN Processing:

The CDN receives these requests and processes them as described above (cache hit/miss).

e. Optimization:

CDNs often employ additional optimizations like:
Automatic image format conversion (e.g., WebP for supported browsers)
On-the-fly resizing
Compression

Here’s a code snippet showing how you might generate a CDN URL in your application:

function getCdnUrl(imagePath) {
  const CDN_HOSTNAME = 'content.example.com';
  return `https://${CDN_HOSTNAME}/images/${imagePath}`;
}

// Usage
const profilePicUrl = getCdnUrl('profile-pic-123.jpg');

This setup allows for efficient delivery of static content, leveraging the global distribution of CDN edge servers while maintaining control over the content through your origin server or cloud storage solution. The key is that once the initial HTML is served (which may come from your origin server), all static content references use CDN URLs, allowing direct client-to-CDN communication for these resources.

Download/Rendering Flow: Your understanding is correct. Let’s address your questions within this context:

Client-CDN Direct Contact: In most optimized setups, the client contacts the CDN directly for static assets. However, there are scenarios where the client might first contact the origin server: a. Dynamic CDN URL generation:

If CDN URLs are generated dynamically based on user context or other factors.
Example: Personalized content delivery or A/B testing of CDN providers.

b. Token-based authentication:

When the CDN requires a token for access, the client might need to obtain this from the origin server first.

c. Initial page load:

The HTML containing CDN URLs is typically served by the origin server.

2. Multiple CDN Selection: When multiple CDNs are used (multi-CDN strategy), selection typically happens at the DNS level:

A Global Server Load Balancing (GSLB) system is used.
When the client performs a DNS lookup for the CDN hostname, the GSLB system determines the optimal CDN.
Factors for selection include geographic proximity, CDN health, cost, and traffic distribution policies.
The client receives the IP of the chosen CDN’s edge server.

3. CDN Content Identification and Client Request:

a. Content Identification:

The CDN identifies content based on the URL path and query parameters.
Example URL: https://cdn.example.com/images/profile-pic-123.jpg
The CDN extracts the path /images/profile-pic-123.jpg to locate the asset.

b. Client Request:

The client makes a standard HTTP GET request:

GET /images/profile-pic-123.jpg HTTP/1.1
Host: cdn.example.com

Comprehensive Overview:

Here’s a detailed flow incorporating all components:

Initial Setup:

The origin server is configured with rules for the CDN, including caching policies, TTL, and origin pull settings.
DNS is set up to route CDN hostname(s) to the appropriate CDN edge servers or GSLB service.

2. Asset Upload

async function uploadAsset(file) {
  // Get pre-signed URL from backend
  const { uploadUrl, assetId } = await getPresignedUrl();
  
  // Upload to storage service
  await fetch(uploadUrl, { method: 'PUT', body: file });
  
  // Notify backend of successful upload
  await notifyBackend(assetId);
}

3. Backend processing

async function notifyBackend(assetId) {
  const response = await fetch('/api/assets/uploaded', {
    method: 'POST',
    body: JSON.stringify({ assetId }),
  });
  
  if (response.ok) {
    const { cdnUrl } = await response.json();
    return cdnUrl;
  }
}

4. Asset retrieval

function getCdnUrl(assetId) {
  return `https://cdn.example.com/assets/${assetId}`;
}

async function displayAsset(assetId) {
  const cdnUrl = getCdnUrl(assetId);
  document.getElementById('asset').src = cdnUrl;
}

5. CDN Processing:

When the CDN receives a request, it first checks its cache.
On a cache miss, it requests the asset from the origin, respecting any cache-control headers.
The CDN may perform on-the-fly optimizations like image compression or format conversion.

6. Origin Interaction:

The origin server receives requests from the CDN for cache misses.
It serves the requested asset along with appropriate caching headers.

This setup allows for efficient, scalable content delivery while maintaining control over assets. The CDN acts as a distributed cache and intelligent routing layer, optimizing delivery based on user location and network conditions.

Key points:

Direct client-to-CDN communication for most static assets
Intelligent CDN selection for multi-CDN setups
Flexible origin server interaction for dynamic content or authentication needs
Optimized caching and delivery strategies implemented at the CDN level

This architecture provides a balance of performance, scalability, and control, allowing for efficient global content delivery while maintaining the ability to update and manage assets as needed.

Process of resolving CDN URLs

Now, let’s break down the process step by step:

Initial Request:

The client needs to load static content (e.g., an image) with a URL like https://cdn.example.com/images/photo.jpg.

2. DNS Resolution:

The client performs a DNS lookup for cdn.example.com.
This request is forwarded to a Global Server Load Balancing (GSLB) system.

3. CDN Selection:

The GSLB system selects the optimal CDN based on factors like:
Geographic proximity
Current CDN performance and health
Cost considerations
Traffic distribution policies

4. IP Return:

The GSLB system returns the IP address of the chosen CDN’s edge server to the client.

5. Content Request:

The client sends an HTTP GET request to the CDN edge server:

GET /images/photo.jpg HTTP/1.1
Host: cdn.example.com

6. CDN Processing:

The CDN edge server receives the request and checks its cache.
If the content is in the cache and not expired, it serves it directly to the client (6a in the diagram).
If the content is not in the cache or has expired, the CDN requests it from the origin server (6b in the diagram).

7–11. Origin Server Interaction (if needed):

If the CDN needs to fetch the content, it sends a request to the origin server.
The origin server retrieves the content from its storage (e.g., a database or file system).
The origin server sends the content back to the CDN.
The CDN caches the content according to caching headers set by the origin server.
The CDN serves the content to the client.

Key Points:

Direct CDN Contact:

The client typically contacts the CDN directly for static content, not the origin server.
This is crucial for maximizing CDN performance benefits.

2. CDN Selection with Multiple CDNs:

When multiple CDNs are used, the selection happens at the DNS/GSLB level.
The client is unaware of multiple CDNs; it simply uses the IP address provided by the DNS resolution.

3. Content Identification by CDN:

The CDN identifies which content to serve based on the URL path and any query parameters.
In our example, /images/photo.jpg tells the CDN exactly which asset to serve.

4. Request Type:

The client makes a standard HTTP GET request to the CDN.
The CDN may support other HTTP methods, but GET is most common for static content.

5. Caching:

CDNs cache content at edge servers to reduce latency and origin server load.
Caching behavior is typically controlled by headers set by the origin server.

Here’s a simple code example of how a client might request content from a CDN:

async function loadImage(imageId) {
  const CDN_HOSTNAME = 'cdn.example.com';
  const imageUrl = `https://${CDN_HOSTNAME}/images/${imageId}.jpg`;
  
  try {
    const response = await fetch(imageUrl);
    if (!response.ok) throw new Error('Failed to load image');
    const blob = await response.blob();
    const imgElement = document.getElementById('myImage');
    imgElement.src = URL.createObjectURL(blob);
  } catch (error) {
    console.error('Error loading image:', error);
  }
}

// Usage
loadImage('photo');

This setup allows for efficient, global content delivery while maintaining control over your content. The CDN acts as a distributed cache and intelligent routing layer, optimizing delivery based on user location and network conditions.

Excellent question. You’ve highlighted an important aspect of the CDN content delivery process that I should have addressed more clearly. Let me explain how the client obtains the CDN domain URL in the first place.

The CDN domain URL is typically provided to the client in one of several ways:

Embedded in HTML/CSS/JavaScript:

When the client (usually a web browser) initially loads a web page, the HTML, CSS, or JavaScript contains references to assets using the CDN domain. This initial page load usually comes from the origin server.

For example, in HTML:

<img src="https://cdn.example.com/images/photo.jpg" alt="Example photo">

Or in CSS:

.background {
  background-image: url('https://cdn.example.com/images/background.jpg');
}

2. Dynamically Generated by JavaScript:

The CDN URLs might be constructed dynamically by JavaScript, often based on configuration data provided by the server.

const CDN_BASE_URL = 'https://cdn.example.com';

function getImageUrl(imageName) {
  return `${CDN_BASE_URL}/images/${imageName}`;
}

// Usage
const imgElement = document.createElement('img');
imgElement.src = getImageUrl('photo.jpg');
document.body.appendChild(imgElement);

3. Returned by API Responses:

When the client makes API requests to the backend server, the responses might include CDN URLs for related assets.

async function fetchUserProfile() {
  const response = await fetch('/api/user-profile');
  const data = await response.json();
  
  // data.avatarUrl might be "https://cdn.example.com/avatars/user123.jpg"
  document.getElementById('avatar').src = data.avatarUrl;
}

4. Configuration Provided on Initial Page Load:

The server might include a configuration object when serving the initial HTML, which includes the CDN base URL.

<script>
window.appConfig = {
  cdnBaseUrl: 'https://cdn.example.com'
};
</script>

Then in your application JavaScript:

function getAssetUrl(assetPath) {
  return `${window.appConfig.cdnBaseUrl}/${assetPath}`;
}

5. Service Worker Interception:

In more advanced setups, a service worker might intercept requests and rewrite URLs to use the CDN.

// In a service worker file
self.addEventListener('fetch', (event) => {
  const url = new URL(event.request.url);
  if (url.pathname.startsWith('/images/')) {
    const cdnUrl = `https://cdn.example.com${url.pathname}`;
    event.respondWith(fetch(cdnUrl));
  }
});

In this updated flow:

1–4. The client first loads the initial page from the origin server, which includes CDN URLs. 5–8. When the client needs to load a resource from the CDN, it goes through the DNS resolution process we described earlier. 9–10. Finally, the client requests and receives the content from the CDN.

This process ensures that:

The origin server maintains control over which assets are served via CDN.
The client can efficiently retrieve assets without repeatedly contacting the origin server.
The CDN selection process remains transparent to the client application.

By providing the CDN URLs in the initial page load or through API responses, your application can leverage the performance benefits of a CDN while maintaining flexibility in how and when CDN resources are used.

Excellent question. Handling CDN failures is a crucial aspect of maintaining a robust and resilient web application. Let’s explore the strategies for dealing with CDN failures and implementing fallback mechanisms.

Client-side Handling of CDN Failures:

The client can implement error handling and fallback mechanisms in JavaScript. Here’s a typical approach:

Here’s a JavaScript implementation of this fallback strategy:

async function fetchWithFallback(cdnUrl, originUrl, maxRetries = 2) {
  let retries = 0;
  
  while (retries <= maxRetries) {
    try {
      const response = await fetch(cdnUrl);
      if (!response.ok) throw new Error('CDN request failed');
      return await response.blob();
    } catch (error) {
      console.warn(`CDN request failed (attempt ${retries + 1}):`, error);
      retries++;
      if (retries > maxRetries) break;
      await new Promise(resolve => setTimeout(resolve, 1000)); // Wait before retry
    }
  }
  
  // Fallback to origin
  try {
    const response = await fetch(originUrl);
    if (!response.ok) throw new Error('Origin request failed');
    return await response.blob();
  } catch (error) {
    console.error('Both CDN and origin requests failed:', error);
    throw error;
  }
}

// Usage
const CDN_BASE = 'https://cdn.example.com';
const ORIGIN_BASE = 'https://origin.example.com';

async function loadImage(imagePath) {
  try {
    const blob = await fetchWithFallback(
      `${CDN_BASE}${imagePath}`,
      `${ORIGIN_BASE}${imagePath}`
    );
    const imgElement = document.getElementById('myImage');
    imgElement.src = URL.createObjectURL(blob);
  } catch (error) {
    console.error('Failed to load image:', error);
    // Handle the error (e.g., display a placeholder or error message)
  }
}

loadImage('/images/photo.jpg');

2. Origin Server Handling of CDN Failures:

The origin server can also play a role in handling CDN failures:

a. Health Checks:

Implement regular health checks from the origin server to the CDN.
If failures are detected, the server can update its response to clients.

b. Dynamic URL Generation:

Instead of hardcoding CDN URLs, the server can dynamically generate them based on CDN health.

function getAssetUrl(assetPath) {
  if (isCdnHealthy()) {
    return `https://cdn.example.com${assetPath}`;
  } else {
    return `https://origin.example.com${assetPath}`;
  }
}

app.get('/api/asset-url', (req, res) => {
  const assetPath = req.query.path;
  res.json({ url: getAssetUrl(assetPath) });
});

3. DNS-level Fallback:

In more advanced setups, DNS-level fallback can be implemented:

Configure your DNS provider to automatically failover to the origin server’s IP if the CDN’s health checks fail.
This approach requires careful configuration and monitoring but can provide seamless fallback for all clients.

4. Multi-CDN Strategy:

Implementing a multi-CDN strategy can provide additional resilience:

const CDN_PROVIDERS = [
  'https://cdn1.example.com',
  'https://cdn2.example.com',
  'https://origin.example.com'
];

async function fetchWithMultiCdnFallback(assetPath) {
  for (const cdnBase of CDN_PROVIDERS) {
    try {
      const response = await fetch(`${cdnBase}${assetPath}`);
      if (response.ok) return await response.blob();
    } catch (error) {
      console.warn(`Request to ${cdnBase} failed:`, error);
    }
  }
  throw new Error('All CDN and origin requests failed');
}