Daily Compass: Designing a Serverless Azure Data Pipeline

Daily Compass was a team project from my cloud engineering training program. The goal was to build a personal daily dashboard that helps a user plan the day with mood-based quotes, TODO/calendar data, and real-time news.

My role focused on architecture design, data flow, database schema, UI/UX wireframes, real-time news crawling, presentation materials, and final documentation.

Why Cloud

The project was intentionally designed as a cloud-native application instead of a simple local web app.

We wanted the system to support:

• managed backend execution without server maintenance
• flexible NoSQL storage for user-specific dashboard data
• secure database and storage access through private networking
• CI/CD-driven deployment
• monitoring for API errors and performance
• optional AI/Speech integration for mood-based quotes

The core architectural decision was to use Azure managed services to reduce operational burden while still practicing production-oriented concerns such as security, availability, and observability.

High-Level Architecture

The main request flow looked like this:

User
  -> Azure Static Web Apps
  -> Azure Functions
  -> Azure Cosmos DB for MongoDB
  -> Azure AI Speech
  -> Application Insights

The frontend sends API requests to Azure Functions. The backend stores and reads user data from Cosmos DB and optionally calls Azure AI Speech to provide voice output for quotes. Application Insights collects errors and performance signals.

Functional Scope

The dashboard combined three modules:

• Mind Postbox: the user selects a mood and receives a quote. The text input is not stored.
• My Diary: TODO and calendar data are stored per user.
• Real-Time News: a Node.js crawler collects recent news and exposes it through the backend API.

The important part was not the UI itself. The interesting engineering problem was keeping these features inside a coherent data flow with reasonable security boundaries.

Security and Network Design

The network design separated responsibilities across subnets:

• frontend subnet
• backend/database subnet
• API gateway subnet

For data protection, the design applied Private Endpoint to Cosmos DB and Storage so that database and storage traffic did not rely on public access.

This shaped the project around a simple security principle:

The database should not be reachable just because the application is reachable.

That distinction is easy to miss in small training projects, but it is central to enterprise cloud architecture.

Data and Operations

Cosmos DB with the MongoDB API was selected because the dashboard data is document-shaped and user-specific:

• user profile and authentication data
• TODO items
• calendar records
• mood-based quote collections
• news documents

Application Insights was planned for:

• function success/failure rate
• API response time
• error logging
• Cosmos DB RU consumption

The expected monthly cost for the training architecture was roughly $32-45, mostly driven by Cosmos DB throughput.

What I Learned

This project made three lessons concrete for me.

First, a data pipeline is also a security design. Once user data, crawler data, and AI output share the same dashboard, the system needs clear boundaries around what is stored, what is transient, and what can be accessed publicly.

Second, serverless does not remove architecture work. Azure Functions reduces server maintenance, but it still requires careful thought around API shape, database access, network restrictions, monitoring, and deployment.

Third, documentation is part of engineering. Architecture diagrams, data-flow explanations, and operational notes are not afterthoughts. They are how a team aligns implementation decisions with reliability and security goals.

Future Improvements

If I were evolving this project today, I would focus on:

• Azure AD B2C or another managed identity provider for authentication
• event-driven processing for news ingestion and quote generation
• Terraform-based infrastructure provisioning
• stricter Private Endpoint coverage
• a production-grade observability dashboard
• a recommendation pipeline using Azure AI Language or Azure Machine Learning

Daily Compass started as a training project, but it became a useful architecture exercise: designing a small product as if it still deserved real security, deployment, and operational thinking.