Daily Compass: Designing a Serverless Azure Data Pipeline

Daily Compass started with the idea of ​​a personal dashboard that collects TODO, calendar, emotion-based phrases, and real-time news needed for daily planning on one screen. Although it is a small web application when looking at its functionality, it became a serverless data pipeline design problem when organizing where to collect data, what storage to put it in, what API to provide it, and what boundary to protect it with.

From the beginning, we decided to combine Azure managed services rather than directly managing VMs. This is to reduce the server operation burden, divide API and collection tasks into small functional units, and check execution status with Application Insights.

Divide functions into data flows

The dashboard is divided into three functions:

Daily dashboard
- mood quote
- todo and calendar
- real-time news

Each function has different data characteristics. TODO and Calendar are write data for each user, and news is read data collected and updated externally. Emotion-based sayings take the user's current selection as input and provide recommended results, but there is no need to store the original emotional text long-term.

User-owned data
- todos
- calendar events
- preferences

System-collected data
- news articles
- source metadata
- fetched timestamp

Generated data
- selected quote
- optional speech output
- recommendation log

If you divide it like this, the storage policy also changes. For user TODO, editing/deletion is important, and for news, deduplication and update time are important. For emotion-based phrases, how much of your input is not saved may be more important than the recommendation results.

Why Azure Static Web Apps and Functions

We thought of placing the front-end in Azure Static Web Apps and configuring the API as Azure Functions.

Browser
-> Static Web Apps
-> Functions API
-> Cosmos DB

Static Web Apps have the potential for static front-end deployment and authentication integration, and Functions are good for dividing small APIs and event-based tasks. Functions such as TODO creation, calendar search, news list search, and emotion-based quote recommendation can be divided into function units.

For example, the API path can be designed like this:

GET    /api/dashboard/today
GET    /api/todos
POST   /api/todos
PATCH  /api/todos/{id}
DELETE /api/todos/{id}
GET    /api/news
POST   /api/mood/quote
`

```/api/dashboard/today` can be set as an aggregation endpoint that downloads the data needed for the first screen at once. On the other hand, it is clear that TODO modification or deletion is divided into resource-level APIs.

## Reasons for choosing Cosmos DB MongoDB API

From the beginning, the dashboard data felt more natural in its document structure than in its strong relational model. User-specific TODOs, calendars, news documents, and quote collections can each be handled as independent documents.

```json
{
  "userId": "user-001",
  "date": "2026-07-04",
  "todos": [
    {
      "id": "todo-001",
      "title": "Review architecture diagram",
      "done": false,
      "priority": "high"
    }
  ],
  "calendar": [
    {
      "id": "event-001",
      "title": "Cloud study session",
      "startsAt": "2026-07-04T09:00:00+09:00"
    }
  ]
}

News is stored together with the point of collection and source.

{
  "articleId": "hash",
  "title": "Cloud platform update",
  "url": "https://example.com/news",
  "source": "example",
  "category": "cloud",
  "fetchedAt": "2026-07-04T07:00:00+09:00",
  "publishedAt": "2026-07-04T06:30:00+09:00"
}

Choosing a document-type DB does not mean that modeling disappears. Partition key, document size, query pattern, TTL, and RU usage must be viewed together. For example, the user dashboard can be searched based on userId and date, and news can be searched based on category and fetchedAt.

Query pattern
- today's dashboard by userId + date
- latest news by category
- incomplete todos by userId
- calendar events by date range

News gathering requires a different execution model than the API

The real-time news function is more stable as it collects and stores information periodically rather than scraping external sites every time the user opens the screen.

Timer trigger
-> fetch RSS / API
-> normalize article
-> deduplicate by URL/hash
-> save to Cosmos DB
-> record execution log

Using Timer Trigger in Azure Functions, you can run collection tasks at regular intervals. In collection work, external API failures, response format changes, duplicate articles, and network timeouts must be treated as normal failure cases.

module.exports = async function (context, myTimer) {
  context.log("News ingestion started");

  try {
    const articles = await fetchNewsSources();
    const normalized = articles.map(normalizeArticle);
    const unique = deduplicateArticles(normalized);
    await saveArticles(unique);

    context.log(`Saved ${unique.length} articles`);
  } catch (error) {
    context.log.error("News ingestion failed", error);
    throw error;
  }
};

URL alone may not be sufficient as a duplication removal criterion. This is because the same article may come in with different tracking parameters, or may come in with slightly different titles from multiple sources. You can start by simply using the normalized URL and title hash together.

function articleKey(article) {
  return hash(`${normalizeUrl(article.url)}|${article.title.trim().toLowerCase()}`);
}

Private Endpoint and Network Boundary

Even in small projects, I wanted to avoid leaving the data store open as a public endpoint. We considered the structure of attaching a private endpoint to Cosmos DB and storage.

frontend subnet
api subnet
data subnet
private endpoint subnet

Of course, how to connect Static Web Apps, Functions, and Private Endpoints requires checking the actual Azure plan and scope of network integration support. The important thing is not to consider “the front end is exposed to the outside” and “the DB is exposed to the outside” as the same problem.

Public boundary
- browser -> static app
- browser -> API endpoint

Private boundary
- API -> Cosmos DB
- API -> Storage
- ingestion function -> Cosmos DB

The security perimeter does not end with the network alone. You should also look at the Function App's managed identity, Cosmos DB permissions, secret management in application settings, and GitHub Actions deployment permissions.

GitHub Actions deployment flow

Deployment can be configured with GitHub Actions. Whether the front end and functions are managed in the same repository or separated into separate repositories depends on the team structure. For small projects, a mono-repo is simple.

repo/
  apps/
    web/
    api/
  infra/
  docs/

The workflow is roughly as follows:

name: deploy-daily-compass

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm run lint
      - run: npm run test
      - run: npm run build

For Azure deployment permissions, it is better to use OIDC-based federated credentials rather than long-term secrets if possible. You should limit which resources your deployment pipeline can modify to the minimum privileges.

Observability and Cost

Serverless means you don't have to manage the server yourself, but that doesn't mean you don't have to see how it's running. At Daily Compass, we wanted to look at the following signals, centered around Application Insights.

API
- request count
- response time
- error rate
- dependency failure

Ingestion
- scheduled run count
- fetched article count
- duplicate ratio
- source API latency
- failed source

Cosmos DB
- RU consumption
- throttled request
- query latency
- storage size

Cosmos DB costs vary depending on usage and throughput settings. Based on educational architecture, it was expected to cost around $32-45 per month, and most of it was generated from Cosmos DB throughput. In actual operation, it is necessary to check whether costs can be lowered by adjusting serverless capacity mode, autoscale, TTL, and indexing policy.

Cost levers
- Cosmos DB throughput mode
- indexing policy
- TTL for old news
- Function execution count
- Application Insights sampling
- Storage lifecycle policy

AI/Speech functions are separated into extension points

The feature that plays emotion-based sayings in a voice can be connected to Azure AI Speech. However, if AI functions are tightly tied to the core request flow, costs, latency, and failures can affect the entire screen. Therefore, it is better to separate it into optional extensions.

POST /api/mood/quote
-> select quote
-> return text immediately
-> optionally request speech audio

Voice conversion results can be cached rather than generated each time. If it is the same phrase and the same voice option, the audio URL stored in storage is reused.

{
  "quoteId": "quote-001",
  "text": "Small progress is still progress.",
  "audioUrl": "https://storage.example.com/quote-001-ko-KR.mp3"
}

When adding AI functions, you must also be careful about input data and logs. If the design does not store the user's emotional input, the original input should not be left in the Application Insights log.

If you redesign it

If I were to refine this structure again now, I would first code the infrastructure in Terraform. Leaving Cosmos DB, Function App, Static Web Apps, Application Insights, Private Endpoint, and Role Assignment as code makes it easier to reproduce the environment and review changes.

We will also create a clearer separation between news gathering and dashboard APIs.

api service
- user dashboard
- todos
- calendar
- quote

worker service
- news ingestion
- deduplication
- cleanup old articles

It is better to have an operational dashboard from the beginning. Looking at API response time, function failure rate, news collection success rate, and Cosmos DB RU usage, bottlenecks and cost structures become visible much sooner, even in small projects.

Cleanup

Daily Compass was a personal dashboard idea, but when designed, it encompasses many of the basic questions of a serverless data pipeline. You need to think about how to divide user data and externally collected data, how to separate functions into APIs and workers, how to set Cosmos DB's document model and query pattern, and what network boundary is needed if you do not want to open the DB to the public.

Serverless reduces the burden of infrastructure operation, but does not eliminate architectural concerns. Rather, as execution units become smaller, data flow, failure handling, observability, and cost must be designed more clearly. Even a small dashboard is a pretty good cloud-native design exercise by these standards.