Cloud-Native .NET

Modern .NET, GitHub and Azure for enterprise apps. A self-study course for senior .NET developers who built their career on .NET Framework, IIS, Remote Desktop and log files, and now want the modern stack: .NET 10, Aspire, GitHub Actions, Azure Container Apps, OpenTelemetry and the reliability habits that come with them.

📅 Written in October 2026 for .NET 10 (LTS), C# 14, Aspire 13 and the Azure services of that time. Facts marked 🔧 change often; check the linked docs.


What "cloud-native .NET" means here

You already know how to build good software. What changed is where it runs and how you know it works.

You know this This course gets you to
ASP.NET MVC / Web API 2 on IIS ASP.NET Core 10 minimal APIs in containers on Azure Container Apps
web.config and passwords in config Environment configuration, managed identities and Key Vault
Thursday-night deploys over RDP CI on every pull request, CD on every merge, with OIDC and no stored secrets
Windows Services, SQL Agent jobs, MSMQ Background workers, Azure Service Bus, the outbox pattern
Log files on three servers OpenTelemetry traces, metrics and logs, queried with KQL
"The site is up" SLIs, SLOs, error budgets and actionable alerts
Clicking in the portal Bicep in git, linted in CI

How the guide works

You join Ledgerly, a small fintech building a portfolio tracker, as a senior .NET developer. Each chapter opens with a problem the team hits, then explains the concepts, shows real code from a working sample, and ends with traps, copy-ready helpers and an optional exercise.

Section in every chapter What it gives you
📖 Story The problem, as it shows up at Ledgerly
🏚️ Old way → 🚀 New way Your old habit next to the modern practice
🧠 Concepts The ideas, in tables and decision guides
💻 In Ledgerly Real code from the sample
🗺️ Diagrams Who does what, where things live, how it flows
⚠️ Common traps Mistakes that are easy to make
🧰 Helpers Copy-ready snippets and "make it permanent" advice
🧪 Try it yourself Optional hands-on steps
✅ Recap Three lines to remember
Marker Meaning
🔧 Volatile fact (version, limit, date, SKU name). Check the docs before relying on it.
💰 Has a cost impact. The course explains cost drivers, not prices.
🧪 Not run by the author Azure steps written from the docs and validated with bicep build, but not deployed end to end.

The sample

The repository contains Ledgerly, a runnable .NET 10 system the chapters quote line by line:

Part What it is
sample/src API, background worker, fake market-data vendor, migration runner, shared core, service defaults and the Aspire AppHost
sample/tests 15 unit tests and 8 integration tests against a real SQL Server
sample/infra Bicep for Azure Container Apps, Azure SQL, Service Bus, Azure Managed Redis, Key Vault, Container Registry and Application Insights
.github/workflows CI (build, test, container images, Bicep lint), CodeQL, and a deploy workflow that stays off until you connect your own Azure subscription

Browse it on GitHub.

Tools you need

Tool Why Required?
.NET 10 SDK Build and run everything Yes
Git and a GitHub account Chapters 8, 9 and 12 Yes
An editor: Visual Studio 2026, VS Code with C# Dev Kit, or Rider Writing code Yes
Docker Desktop or Podman Aspire runs SQL Server, Redis and the Service Bus emulator as containers For running the whole system locally
Aspire CLI aspire run (or run the AppHost project with dotnet run) Recommended
Azure subscription and Azure CLI Chapters 10 to 13 and 16, if you deploy Optional
GitHub Copilot Chapter 20 Optional

The whole course can be read without an Azure subscription. Everything up to the deploy runs on your laptop.

Contents

Part 1 · Foundations

  1. From legacy to cloud-native: the mental model, the Ledgerly architecture and the course map
  2. Modern .NET in one sitting: what changed from .NET Framework 4.8 to .NET 10 and C# 14
  3. A solution set up like a pro: central package management, analyzers, warnings as errors, one-command builds

Part 2 · Build, ship, run

  1. Designing the API: minimal APIs, validation, ProblemDetails, OpenAPI, CORS, rate limiting
  2. Data: EF Core and Azure SQL: EF Core 10, migrations as a deploy step, concurrency, Azure SQL serverless
  3. Local development with Aspire: the AppHost, containers, the dashboard and service defaults
  4. Background work and messaging: workers, Service Bus, the outbox, idempotency, dead-lettering
  5. GitHub as the team's home: GitHub flow, rulesets, CODEOWNERS, Dependabot, Copilot code review
  6. CI with GitHub Actions: build, test, container images without Dockerfiles, CodeQL
  7. Choosing Azure services: App Service vs Container Apps vs AKS vs Functions, data, messaging, caching, cost
  8. Infrastructure as code: Bicep modules, role assignments, what-if, azd and aspire deploy
  9. CD to Azure with OIDC: federated credentials, environments, build once, migrations job, rollback
  10. Identity and secrets: managed identity everywhere, Entra auth to SQL and Service Bus, JWT bearer
  11. Reliability patterns: retries, timeouts, circuit breakers, graceful degradation, health probes
  12. Observability I: logging and OpenTelemetry: structured logs, traces, metrics, context through queues
  13. Observability II: Azure Monitor, KQL and SLOs: Application Insights, KQL, alerts, error budgets

Part 3 · Everyday practice

  1. Testing strategy: unit and integration tests, Testcontainers, Microsoft.Testing.Platform
  2. Performance and scale: HybridCache, async all the way, EF performance, scale rules, load tests
  3. Running production: on-call, runbooks, kill switches, safe deploys, postmortems, cost reviews
  4. AI-assisted .NET development: GitHub Copilot like a senior, custom instructions, MCP, guardrails

Part 4 · Toolkit

  1. Your toolkit: the habits notebook closed, the day-one checklist, what to learn next

Plus the toolkit (cheat sheet, checklists, templates and config files) and the glossary.

Pace

Pace Plan
Steady Two chapters a week, about 10 weeks, doing most "Try it yourself" sections
Fast One part per week, about 4 weeks, reading only, trying the sample at the end
Reference Read chapters 1, 2 and 21 first, then jump to whatever you need this week

Each chapter takes 20 to 40 minutes to read. Exercises add 15 to 60 minutes.

Run the sample in two minutes

git clone https://github.com/vygantas-meskauskis/cloud-native-dotnet.git
cd cloud-native-dotnet/sample
dotnet build Ledgerly.slnx
dotnet test --solution Ledgerly.slnx   # integration tests need Docker, or set LEDGERLY_TEST_SQL
aspire run                             # needs Docker Desktop or Podman

Start → Chapter 1: From legacy to cloud-native

Chapter 1: From legacy to cloud-native

⏱ About 25 min · You'll learn: what "cloud-native .NET" actually changes for a senior developer, the Ledgerly architecture you will build in this course, and how the 21 chapters fit together.


📖 Story: The whiteboard with no servers on it

It is your first Monday at Ledgerly. Last month you were still at Nordbank Insurance, where "production" meant six Windows Server VMs behind a hardware load balancer, a shared drive full of web.config transforms, and a Thursday-night deployment window where you copied files over Remote Desktop. When something broke, you opened three RDP sessions and searched C:\inetpub\logs and a folder of NLog text files until you found the stack trace.

Maya Okafor, Ledgerly's tech lead, hands you a coffee and walks you to a whiteboard. It shows a few boxes: an API, a worker, a database, a queue, a cache. There are no server names on it, no IP addresses, and no drive letters. "Three rules," she says. "One: there are no servers you can log in to. Two: logs are queries, not files. Three: deploys are boring. They happen on every merge, many times a day, and nobody stays late for them."

You open a new notebook and write a title on the first page: Habits I brought with me. Over the next weeks you will look at each habit honestly, keep the good ones and replace the rest. Maya smiles when she sees it. "Good. Most of your habits are right. The places where they live have changed."

Numbers in the Ledgerly story (team size, timelines, traffic, latency, costs) are illustrative. They make the story concrete; they are not benchmarks.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
.NET Framework 4.x on Windows Server and IIS .NET 10 (LTS) on Linux containers in Azure Container Apps Cross-platform runtime, much faster, a new major version every November, and no OS to patch yourself
Deploy by RDP, Web Deploy or xcopy on a Thursday night Every merge to main builds, tests and deploys through GitHub Actions Small, frequent, automated deploys are safer than big, rare, manual ones
web.config transforms with passwords in them Environment variables, Key Vault references and managed identities (no passwords at all) Secrets in files leak; identities can't be copied into a chat message
Text log files on each server, read with RDP and Notepad++ Structured logs, traces and metrics through OpenTelemetry, queried with KQL With many short-lived replicas there is no "server" to log in to, and you need to follow one request across services
Windows Services and SQL Agent jobs for background work Background services, queues (Azure Service Bus) and Container Apps jobs Work survives restarts, scales with load and retries on its own
Servers sized for peak load, running all year Autoscaling from zero (or one) replica to many, based on traffic or queue length You pay for what you use, and a traffic spike doesn't need a purchase order
Infrastructure created by hand in a console or by a ticket to the ops team Infrastructure as code (Bicep) reviewed in pull requests Reproducible environments, and the history of every change is in git
"It works on my machine" One command starts the whole system locally (Aspire), and CI proves it on a clean machine New team members are productive on day one, and environment drift shows up early

💡 Most of your instincts (clean design, careful SQL, defensive code) still apply. What changed is where the code runs, how it gets there and how you see what it is doing.


🧠 What "cloud-native" means for a .NET developer

"Cloud-native" is used for marketing, so here is a practical definition you can work with. An application is cloud-native when it is designed for an environment where machines are disposable, failures are normal and change is constant. In practice that means a handful of properties:

Property What it means in practice Where you learn it
Disposable processes Any replica can be killed at any time (scale-in, new revision, host maintenance). Startup is fast, shutdown is graceful, and no state lives on local disk. Chapters 7, 14
Configuration from the environment The same container image runs in every environment. Only configuration differs, and secrets are not in config at all. Chapters 3, 13
Everything as code Application, infrastructure, pipelines, policies and dashboards live in git and change through pull requests. Chapters 8, 9, 11
Automated delivery A merge triggers build, test, image, deploy, migration and smoke test with no manual steps. Chapters 9, 12
Designed for failure Dependencies will be slow or down. Retries, timeouts, circuit breakers and graceful degradation are part of the design. Chapter 14
Observable by default Every request produces traces, metrics and structured logs that you can query across all services. Chapters 15, 16
Elastic Capacity follows demand automatically, including scaling to zero where it makes sense. Chapters 10, 18
Identity instead of secrets Services prove who they are with managed identities and tokens, not with passwords in connection strings. Chapter 13

None of these properties needs Kubernetes, microservices or a specific vendor. Ledgerly is a small system on purpose: one API, one worker, one fake vendor service and one migration job. That is enough to practice every property above without the complexity of fifty services.

💡 Cloud-native is a set of habits, not a technology list. You can build a cloud-native monolith and a very fragile set of microservices.


🧠 What changed in .NET while you were busy

If your last big project was on .NET Framework 4.8, the gap is large but not mysterious. Microsoft rebuilt .NET as a cross-platform, open-source runtime (.NET Core, now just ".NET"), and all new investment goes there.

Topic Fact Why it matters for you
Current version .NET 10 is the current LTS release, supported until 2028-11-14 🔧 This course targets net10.0 everywhere
Older modern versions .NET 8 and .NET 9 both reach end of support on 2026-11-10 🔧 If you start something new, don't start on 8 or 9
Release rhythm A new major version every November. Even numbers are LTS (3 years), odd numbers are STS (2 years) 🔧 Plan one upgrade per year or one every two years, as normal maintenance
.NET Framework 4.8 and 4.8.1 are Windows components with no announced end date, but get no new features 🔧 Your old apps are not on fire, but new work belongs on .NET 10
Language C# 14 ships with .NET 10 Records, pattern matching, primary constructors and collection expressions change how idiomatic code looks (chapter 2)

Sources: .NET support policy and .NET Framework lifecycle.

The other big shift is the hosting model. ASP.NET on .NET Framework was tied to IIS and System.Web. ASP.NET Core runs on its own web server (Kestrel) inside your process. IIS can still front it on Windows, but in a container it is just a process listening on port 8080. That one change explains a lot of this course: once your app is a self-contained process, it can run in a container, on Linux, anywhere.

The hosting model, then and now

Concern IIS + .NET Framework Container + .NET 10
Web server IIS + System.Web pipeline Kestrel inside your process
Process lifetime App pool recycles on a schedule or memory limit The platform starts and stops replicas, you handle SIGTERM gracefully
Configuration web.config + transforms appsettings.json + environment variables + Key Vault
Health "Is the app pool running?" /alive and /health endpoints that the platform probes
Scaling Add a VM to the load balancer Change maxReplicas or add a scale rule
Logs Files on the server's disk OpenTelemetry to Application Insights

🧠 Ledgerly: the product and the team

Ledgerly is a small fintech start-up of about 25 people. It is building a personal investment portfolio tracker, also called Ledgerly. Users can:

Version 1 launches as a public beta in about 12 weeks. There is no legacy code: you start greenfield, which is exactly why the team wants to do it "the modern way" from day one.

Your teammates

Name Role What they will push you on
Maya Okafor Tech lead / architect "What happens when it fails?" She writes the ADRs and pairs with you often.
Tomás Varga Platform / DevOps engineer "If it's not in git, it doesn't exist." He owns GitHub Actions, Bicep and the Azure subscription.
Priya Raman Site reliability engineer Observability, SLOs, on-call. She hates alerts that wake people up for nothing.
Jonas Berg Security engineer Managed identities, no secrets in config, least privilege, supply chain. Friendly but strict.
Elena Rossi Product manager "What does the user see?" and "What does it cost?" She owns the beta launch.
Leo Park Frontend developer He will build the React app later, so he cares about API contracts, OpenAPI, CORS and error shapes.

And you: the senior .NET developer with nine years of IIS, WCF and MVC behind you, and a notebook of habits to examine.


🗺️ The target architecture

This is what you will have running in Azure by chapter 12. Every box is real code or real Bicep in the sample.

flowchart LR
    User["Browser or<br/>React app"] -- "HTTPS + JWT" --> Api
    subgraph ACA["Azure Container Apps environment"]
        Api["api<br/>(Container App)"]
        Worker["worker<br/>(Container App)"]
        Market["marketdata<br/>(internal Container App)"]
        Job["job-migrations<br/>(Container Apps job)"]
    end
    Api -- "EF Core" --> Sql[("Azure SQL<br/>Database")]
    Api -- "prices" --> Market
    Api -. "optional L2 cache" .-> Redis[("Azure Managed<br/>Redis")]
    Worker -- "outbox relay<br/>and imports" --> Bus["Azure Service Bus<br/>queue: imports"]
    Worker -- "EF Core" --> Sql
    Worker -- "prices" --> Market
    Job -- "schema changes" --> Sql
    ACA -- "OpenTelemetry" --> AI["Application Insights<br/>+ Log Analytics"]
    KV["Azure Key Vault"] -. "secret references" .-> Api

And this is how a change gets from your laptop to that environment:

flowchart LR
    Dev["You<br/>(Aspire locally)"] -- "push branch" --> PR["Pull request<br/>(review + checks)"]
    PR -- "merge" --> CI["GitHub Actions: ci<br/>build, test, images, Bicep lint"]
    CI -- "green" --> CD["GitHub Actions: deploy<br/>OIDC sign-in, no secrets"]
    CD -- "push images" --> ACR["Azure Container<br/>Registry"]
    CD -- "Bicep" --> Azure["Azure resources"]
    CD -- "run" --> Mig["Migration job"]
    Mig -- "success" --> Rev["New revisions of<br/>api, worker, marketdata"]
    Rev -- "telemetry" --> Obs["Application Insights<br/>dashboards and alerts"]

Who does what

Piece Responsibility Old-world equivalent
Ledgerly.Api HTTP API for portfolios, transactions, valuation, imports, alerts Your ASP.NET Web API 2 site on IIS
Ledgerly.Worker Processes imports from the queue, relays the outbox, evaluates price alerts A Windows Service plus a couple of SQL Agent jobs
Ledgerly.MarketData A fake market-data vendor with "chaos" switches to make it slow or failing The flaky third-party SOAP service everyone complained about
Ledgerly.MigrationService Applies EF Core migrations, then exits The DBA running a script on Thursday night
Ledgerly.AppHost Starts everything locally with Aspire: SQL Server, Redis, the Service Bus emulator and all services A 14-step wiki page called "Developer setup"
Ledgerly.ServiceDefaults Shared OpenTelemetry, health checks, resilience and service discovery Copy-pasted Global.asax and logging config

💻 In Ledgerly: the whole system in one file

You don't need to understand this file yet (chapter 6 explains it line by line). Just notice that the entire local topology, including containers for SQL Server, Redis and a Service Bus emulator, is a C# file you can read and review.

📄 sample/src/Ledgerly.AppHost/AppHost.cs

var builder = DistributedApplication.CreateBuilder(args);

// ── Infrastructure (containers locally; real Azure services in the cloud, see infra/) ──────────
var sql = builder.AddSqlServer("sql")
    .WithLifetime(ContainerLifetime.Persistent)   // keep the container (and data) between runs
    .WithDataVolume();
var db = sql.AddDatabase("ledgerlydb");

var cache = builder.AddRedis("cache");

var messaging = builder.AddAzureServiceBus("messaging").RunAsEmulator();
messaging.AddServiceBusQueue("imports");
// ...

Compare that with the wiki page you used to maintain about which SQL instance, which MSMQ queue and which IIS site bindings a new developer needed.


🧠 The stack you will learn, and why each piece is there

Layer Choice in this course Why this one Chapter
Runtime and language .NET 10 (LTS), C# 14, ASP.NET Core 10, EF Core 10 Current LTS, longest support window 2, 4, 5
Local orchestration Aspire (AppHost, service defaults, dashboard) 🔧 One command starts the system, with telemetry from the first run 6
Source control and collaboration GitHub: pull requests, rulesets, CODEOWNERS, Dependabot, CodeQL, GitHub Copilot Where the team works, reviews and automates 8, 20
CI/CD GitHub Actions with OIDC to Azure No stored cloud secrets, build once and deploy the same image 9, 12
Compute Azure Container Apps Containers and scale-to-zero without running Kubernetes yourself 10
Data Azure SQL Database (serverless) Your SQL Server skills transfer directly 5, 10
Messaging Azure Service Bus Durable queues with dead-lettering and duplicate detection 7
Caching HybridCache, optionally backed by Azure Managed Redis 🔧 Fast reads with stampede protection 18
Secrets and identity Microsoft Entra ID, managed identities, Azure Key Vault No passwords to leak or rotate 13
Infrastructure Bicep Readable, Azure-native infrastructure as code 11
Observability OpenTelemetry, Application Insights, Azure Monitor, KQL One standard for traces, metrics and logs, and one place to query them 15, 16

A note on two names that may surprise you:

mindmap
  root(("Cloud-native .NET"))
    Build
      ".NET 10 and C# 14"
      "Minimal APIs"
      "EF Core 10"
      "Aspire"
    Ship
      "GitHub flow"
      "GitHub Actions CI"
      "Bicep"
      "OIDC deploys"
    Run
      "Container Apps"
      "Managed identity"
      "Resilience"
      "Scaling"
    See
      "OpenTelemetry"
      "Application Insights"
      "KQL and SLOs"
      "On-call and postmortems"

🧠 How this course works

The course has four parts:

Part Chapters What you get
Part 1 · Foundations 1-3 The mental model, modern C# and .NET, and a solution set up like a pro
Part 2 · Build, ship, run 4-16 The API, data, local dev, messaging, GitHub, CI, Azure choices, Bicep, CD, identity, reliability and observability
Part 3 · Everyday practice 17-20 Testing, performance, running production and AI-assisted development
Part 4 · Toolkit 21 Cheat sheets, checklists and templates to reuse

Every chapter follows the same shape, so you always know where to look:

Section What it gives you
📖 Story The problem, as it shows up at Ledgerly
🏚️ Old way → 🚀 New way Your old habit next to the modern practice, and why it changed
🧠 Concepts The ideas, in tables and decision guides
💻 In Ledgerly Real code from the sample, with the lines that matter explained
🗺️ Diagrams Who does what, where things live and how it flows
⚠️ Common traps Mistakes that are easy to make, and what to do instead
🧰 Helpers Copy-ready snippets plus "make it permanent" advice
🧪 Try it yourself Optional hands-on steps against the sample
✅ Recap Three lines to remember

Markers you will see

Marker Meaning
🔧 Volatile fact (version, limit, date, SKU name, preview status). Correct as of October 2026; check the linked docs before relying on it.
💰 Has a cost impact. The course explains the cost driver, not exact prices.
🧪 Not run by the author Azure steps written from the docs and validated with bicep build and linting, but not deployed end to end by the author. Expect small differences.

The sample

Everything the chapters quote lives in the sample/ folder of the course repository. It builds with warnings as errors, and its 23 tests (15 unit, 8 integration against a real SQL Server) pass. The CI workflow runs on every push. The deploy workflow stays switched off until you configure your own Azure subscription in chapter 12.

Pace

Pace Plan
Steady Two chapters a week, about 10 weeks, doing most "Try it yourself" sections
Fast One part per week, about 4 weeks, reading only and trying the sample at the end
Reference Read chapters 1, 2 and 21 first, then jump to whatever you need this week

🧠 Your habits notebook

Maya was right: most of your habits are good. The notebook is a tool for sorting them. Here is the first page, as you write it on day one:

Habit Keep, change or drop? Where it is examined
"I want to see the logs" Change. You still want evidence, but you'll get it from queries over structured telemetry, not files 15, 16
"Test it in staging before Thursday" Change. Test on every pull request, deploy small changes often, and roll back fast 9, 12, 17
"The DBA runs the migration script" Change. Migrations are a pipeline step with a job that must succeed before the new code starts 5, 12
"Put the password in web.config and encrypt the section" Drop. Use managed identities, so there is no password 13
"Retry in a loop with Thread.Sleep" Change. Use standard resilience handlers with jitter, timeouts and circuit breakers 14
"Careful SQL and indexes" Keep. Azure SQL is still SQL Server underneath, and EF Core still needs your judgment 5, 18
"Read the code before you approve it" Keep. It matters even more when an AI wrote part of it 8, 20

💡 The goal is not to forget what you know. It is to move each habit to the place where it works in the cloud.


⚠️ Common traps

Trap What happens Do this instead
"Lift and shift" thinking: one container that behaves like a VM You write logs to local disk, keep session state in memory and SSH in to debug. The first scale-in or restart loses data Treat replicas as disposable from day one: state in the database or cache, telemetry out of the process
Starting with microservices Ten services, ten pipelines and distributed transactions for a product with no users yet Start with a few deployable units with clear jobs (like Ledgerly) and split only when there's a real reason
Learning Kubernetes first Weeks on cluster plumbing before shipping any feature Use a managed platform (Container Apps) and learn the concepts (replicas, probes, revisions) that transfer anywhere
Copying old patterns into new code async void, .Result, static HttpClient hacks and string-concatenated SQL show up in a modern codebase Use chapter 2 as a translation guide and let analyzers enforce it (chapter 3)
Treating "the cloud" as one big decision Analysis paralysis over every service Use the decision tables in chapter 10, and accept that most choices can be changed later through infrastructure as code
Trusting version numbers from memory Last year's blog post says a feature is preview, or a service is current, when it changed Check the 🔧 items against the linked docs before you rely on them

🧰 Helpers

Get the sample on your machine:

git clone https://github.com/vygantas-meskauskis/cloud-native-dotnet.git
cd cloud-native-dotnet/sample
dotnet --list-sdks        # you need a 10.0.x SDK
dotnet build Ledgerly.slnx

What to install before chapter 6:

Tool Why
.NET 10 SDK Build and run everything
Docker Desktop or Podman Aspire runs SQL Server, Redis and the Service Bus emulator as containers
Aspire CLI 🔧 aspire run starts the whole system (installation is covered in chapter 6)
Git + a GitHub account Chapters 8 and 9
An IDE: Visual Studio 2026, VS Code with C# Dev Kit, or Rider 🔧 Any of them works with the sample
An Azure subscription (later) Only from chapter 11 on. 💰 Most resources in the sample are serverless or have free monthly grants, but not all

Make it permanent: keep your own "Habits I brought with me" page. The toolkit has an old-to-new cheat sheet that you can extend as you go. See the toolkit and the glossary for any term you don't recognize.


🧪 Try it yourself (optional, ~15 min)

  1. Clone the repository and build the sample: bash cd cloud-native-dotnet/sample dotnet build Ledgerly.slnx
  2. Look at the solution layout. Open Ledgerly.slnx in your editor and match each project to the "Who does what" table above.
  3. Open src/Ledgerly.AppHost/AppHost.cs and find, without reading the docs, the line that tells the API to wait for the migrations to finish. (Hint: WaitForCompletion.)
  4. Run the unit tests (they don't need a database): bash dotnet test --solution Ledgerly.slnx Observe: without a container runtime or the LEDGERLY_TEST_SQL variable, the integration tests are skipped, not failed. Chapter 17 explains why.
  5. Write the first page of your own habits notebook: three habits you suspect will change, and one you are sure you'll keep.

✅ Recap

  1. Cloud-native means disposable processes, configuration from the environment, everything as code, automated delivery, designed for failure and observable by default.
  2. Ledgerly is a small, realistic system (API, worker, fake vendor, migration job) that you will build, ship and run on .NET 10, GitHub and Azure.
  3. Your experience still counts. The course moves each habit to where it works today.

Next → Chapter 2: Modern .NET in one sitting

Chapter 2: Modern .NET in one sitting

⏱ About 30 min · You'll learn: how the hosting model, DI and configuration replaced Global.asax and web.config, which C# features you will meet in every Ledgerly file, and which small modern APIs (TimeProvider, Guid.CreateVersion7(), [LoggerMessage]) replace old habits.


📖 Story: "This reads like C# 5"

Your first pull request at Ledgerly is small: a service that values a portfolio. You wrote it the way you wrote hundreds of classes at Nordbank Insurance. A constructor that assigns six private fields. A new HttpClient() inside a using block. DateTime.Now for the timestamp. A .Result on one call because the method above it wasn't async. A log line built with string.Format. And a null check at the top of every method.

Maya's review has one comment at the top, and it is friendly: "This works, and the logic is right. But it reads like C# 5. Let's pair for an hour and I'll show you what changed. Most of it is less code, not more." Below that are twelve inline suggestions, and each one deletes lines.

You open your notebook, Habits I brought with me, and write a new heading: Things the compiler and the framework now do for me. This chapter is that hour with Maya. It is a translation guide, not a tutorial: you already know how to program, so the focus is on what moved and why.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Global.asax Application_Start, HTTP modules and handlers in IIS Program.cs with WebApplication.CreateBuilder, middleware and endpoints One explicit startup file you can read top to bottom, with no IIS pipeline hidden behind it
Unity, Autofac or Ninject wired up by hand Built-in dependency injection (builder.Services) Every framework component already uses it, so your code plugs into the same container
ConfigurationManager.AppSettings["Key"] and web.config transforms IConfiguration from layered providers, bound to options classes The same build runs everywhere, and environment variables override files without transforms
new HttpClient() per call, or one static instance IHttpClientFactory with typed clients Handler lifetimes and resilience are managed for you, configured once at registration
.Result, .Wait() and async void mixed with sync code async all the way, with a CancellationToken passed down No thread-pool starvation under load, and work stops when the caller gives up
Null checks at the top of every method Nullable reference types plus required members The compiler tells you where nulls can flow, before the code runs
DateTime.Now and Guid.NewGuid() in domain code TimeProvider and Guid.CreateVersion7() Time becomes testable, and IDs sort by creation time
log.InfoFormat("Order {0} placed", id) into a text file Structured logging with [LoggerMessage] source generation Placeholders become queryable fields, and disabled log levels cost nothing

💡 The biggest change isn't a language feature. It's that the framework owns startup, DI, configuration and logging, and your code joins that model instead of building its own.


🧠 The hosting model: from IIS to a process you own

On .NET Framework, IIS owned your process. It loaded System.Web, ran Global.asax, pushed the request through HTTP modules and finally reached your controller. You configured half of the behavior in IIS Manager and the other half in web.config.

In ASP.NET Core, your app is the process. The web server, Kestrel, runs inside it. In a Linux container, that process listens on port 8080 by default 🔧 (it used to be 80; set it with ASPNETCORE_HTTP_PORTS). IIS can still sit in front on Windows, but nothing in Ledgerly needs it.

Translation table

.NET Framework concept .NET 10 equivalent Notes
Global.asax Application_Start Top of Program.cs, before builder.Build() Register services and read configuration here
HTTP modules (IHttpModule) Middleware (app.UseXyz()) Order in Program.cs is the order requests flow through
HTTP handlers and controllers Endpoints: minimal APIs or controllers Ledgerly uses minimal APIs with route groups (chapter 4)
HttpContext.Current (static) HttpContext passed or injected where needed No static ambient context, which also makes code testable
Windows Service with OnStart/OnStop BackgroundService hosted by the generic host Same host, same DI, same logging as the web app (chapter 7)
App pool recycle The platform starts and stops replicas, and you shut down gracefully HostOptions.ShutdownTimeout defaults to 30 s 🔧
Console app with a Main that news everything up Host.CreateApplicationBuilder(args) You get configuration, DI and logging even in a console tool

There are two builders, and both come from the same generic host (docs): WebApplication.CreateBuilder(args) for anything that serves HTTP (the API, the worker and the market-data service, even if only for health endpoints), and Host.CreateApplicationBuilder(args) for processes with no HTTP surface (the migration service).

💻 In Ledgerly: the API's composition root

📄 sample/src/Ledgerly.Api/Program.cs

var builder = WebApplication.CreateBuilder(args);

// ── Cross-cutting: OpenTelemetry, health checks, resilience, service discovery ────────────────
builder.AddServiceDefaults();

// ── Data: Aspire client integration = DbContext pooling + retries + health check + tracing ──────
builder.AddSqlServerDbContext<LedgerlyDbContext>("ledgerlydb");
// ...
var app = builder.Build();

app.UseExceptionHandler();
app.UseStatusCodePages();
app.UseCors(ApiPolicies.Spa);
app.UseAuthentication();
app.UseAuthorization();
app.UseRateLimiter();
// ...
app.MapPortfolioEndpoints();
app.MapImportEndpoints();
app.MapAlertEndpoints();
app.MapDefaultEndpoints();

app.Run();

Read it as two phases:

  1. Before Build(): you describe services. Nothing runs yet. This is where Application_Start code goes.
  2. After Build(): you describe the request pipeline. Each Use... line is a middleware, in order. The Map... lines attach endpoints. app.Run() starts Kestrel and blocks until shutdown.

Notice the file is short. Cross-cutting concerns (OpenTelemetry, health checks, retries) come from one call, AddServiceDefaults(), which lives in a shared project (chapter 6). That's the modern answer to the Global.asax you used to copy from project to project.

The migration runner shows the same model without HTTP:

📄 sample/src/Ledgerly.MigrationService/Program.cs

var builder = Host.CreateApplicationBuilder(args);

builder.AddServiceDefaults();
builder.AddSqlServerDbContext<LedgerlyDbContext>("ledgerlydb");
builder.Services.Configure<AppDatabaseUserOptions>(builder.Configuration.GetSection(AppDatabaseUserOptions.Section));
builder.Services.AddHostedService<Migrator>();

builder.Build().Run();

Same configuration, same DI, same telemetry, no web server. Chapter 5 explains what Migrator does.

💡 If you can find it in Program.cs, it happens. If it isn't there, it doesn't. There is no hidden IIS configuration anymore.


🗺️ How a request flows through the API

This is the middleware order from Program.cs, drawn as the path one request takes:

flowchart LR
    Client["Client<br/>browser or SPA"] -- "HTTPS" --> Kestrel["Kestrel<br/>inside the process"]
    Kestrel --> Ex["UseExceptionHandler<br/>errors become ProblemDetails"]
    Ex --> Status["UseStatusCodePages<br/>bare 4xx get a body"]
    Status --> Cors["UseCors<br/>SPA origins only"]
    Cors --> AuthN["UseAuthentication<br/>JWT bearer"]
    AuthN --> AuthZ["UseAuthorization<br/>RequireAuthorization"]
    AuthZ --> Rate["UseRateLimiter<br/>per-user policies"]
    Rate --> Endpoint["Endpoint handler<br/>services injected"]
    Endpoint -- "DbContext" --> Db[("SQL")]

Compare it with the IIS pipeline you knew: the same idea (a chain of steps around your code), but you can see every step, and you choose the order.


🧠 DI, configuration, options and HttpClient

Dependency injection is built in

You don't choose a container anymore, unless you have a strong reason. builder.Services is an IServiceCollection, and every framework feature registers itself there. Three lifetimes cover nearly everything:

Lifetime Meaning Ledgerly example
Singleton One instance for the process TimeProvider.System
Scoped One instance per request (or per scope you create) LedgerlyDbContext, ValuationService
Transient New instance every time Rarely needed in Ledgerly

The one trap that bites experienced developers: a singleton (like a BackgroundService) must not hold a scoped service such as a DbContext. Create a scope instead. AlertEvaluator takes an IServiceScopeFactory for exactly this reason (chapter 7).

Configuration is layered, not transformed

WebApplication.CreateBuilder reads configuration from several providers, and later ones override earlier ones. A key such as MarketData:BaseUrl in JSON becomes MarketData__BaseUrl as an environment variable (double underscore instead of a colon). That's how the same container image gets different settings in every environment, with no transforms and no rebuild.

flowchart LR
    A["appsettings.json<br/>defaults in git"] --> B["appsettings.Development.json<br/>local overrides"]
    B --> C["User secrets<br/>Development only"]
    C --> D["Environment variables<br/>set by Aspire or Container Apps"]
    D --> E["Command line"]
    E --> Config["IConfiguration<br/>last value wins"]
    Config --> Opt["IOptions of T<br/>typed settings"]
    Config --> Cs["GetConnectionString<br/>ConnectionStrings__ledgerlydb"]

Locally, Aspire injects connection strings as environment variables (chapter 6). In Azure, Bicep sets them on the Container App (chapter 11). Secrets don't go in any of these files at all: chapter 13 replaces them with managed identities.

Options instead of magic strings

The options pattern binds a configuration section to a class once, and code receives a typed object. No more int.Parse(ConfigurationManager.AppSettings["AlertInterval"]) scattered across the codebase.

IHttpClientFactory and typed clients

You probably remember the advice to "make HttpClient static" and then the DNS problems that came with it. The factory manages the underlying handlers for you. A typed client is a class that takes an HttpClient in its constructor, and the registration decides its base address and behavior.

💻 In Ledgerly: the worker's registrations

📄 sample/src/Ledgerly.Worker/Program.cs

builder.Services.AddHttpClient<IMarketDataClient, MarketDataClient>(client =>
    client.BaseAddress = new Uri(builder.Configuration["MarketData:BaseUrl"] ?? "https+http://marketdata"));

builder.Services.AddSingleton(TimeProvider.System);
builder.Services.Configure<AlertOptions>(builder.Configuration.GetSection("Alerts"));

builder.Services.AddHostedService<OutboxRelay>();
builder.Services.AddHostedService<ImportProcessor>();
builder.Services.AddHostedService<AlertEvaluator>();

// Give in-flight messages time to finish when Container Apps stops a replica (SIGTERM).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));
Line Old-world equivalent
AddHttpClient<IMarketDataClient, MarketDataClient> A static HttpClient plus a hand-written retry loop
"https+http://marketdata" A hard-coded URL per environment in web.config. Here service discovery resolves a logical name (chapter 6)
Configure<AlertOptions>(...GetSection("Alerts")) AppSettings["AlertIntervalSeconds"] parsed by hand
AddHostedService<...> Three Windows Services, each with its own installer
ShutdownTimeout = 25 s Nothing: IIS just recycled and in-flight work was lost

The typed client itself is now almost nothing but HTTP:

📄 sample/src/Ledgerly.Core/Market/MarketDataClient.cs

/// <summary>
/// Typed HttpClient for the market-data service. Base address, retries, timeouts and the circuit breaker
/// are configured at registration time (see ServiceDefaults), not here. This class just speaks HTTP.
/// </summary>
public sealed class MarketDataClient(HttpClient http) : IMarketDataClient
{
    public async Task<IReadOnlyDictionary<string, Quote>> GetQuotesAsync(IReadOnlyCollection<string> symbols, CancellationToken cancellationToken)
    {
        // ...
        var query = Uri.EscapeDataString(string.Join(',', symbols));
        var quotes = await http.GetFromJsonAsync<Quote[]>($"/prices?symbols={query}", cancellationToken) ?? [];
        return quotes.ToDictionary(q => q.Symbol, StringComparer.OrdinalIgnoreCase);
    }
}

Retries, timeouts and the circuit breaker are not in this class. They are added to every HttpClient by ConfigureHttpClientDefaults in the service defaults, using the standard resilience handler (chapter 14).

💡 Configuration says what and where. DI decides who gets it. Your classes just declare what they need in a constructor.


🧠 Async all the way, cancellation and nullability

Async is the default, not an optimization

At Nordbank, async was something you added to a hot path. In ASP.NET Core, the whole stack is async: EF Core, HttpClient, Service Bus, file uploads. Blocking on async code with .Result or .Wait() ties up a thread-pool thread while it waits, and under load that's how you get a service that is "up" but doesn't answer.

Old habit Modern rule
.Result or .Wait() on a Task await it, and make the caller async too
async void methods async Task, except for real event handlers
ConfigureAwait(false) everywhere Not needed in ASP.NET Core app code. The sample turns off CA2007 for this reason
No way to stop work when the client leaves Accept a CancellationToken and pass it to every async call
Thread.Sleep in a polling loop PeriodicTimer or Task.Delay with a token

CancellationToken: the parameter you'll type a thousand times

In a minimal API handler, a CancellationToken parameter is bound to the request: if the client disconnects, the token fires. In a BackgroundService, the token passed to ExecuteAsync fires on shutdown (SIGTERM from Container Apps). Pass it down to EF Core, HttpClient and the Service Bus SDK, and your app stops cleanly instead of finishing work nobody is waiting for.

💻 In Ledgerly: a loop that stops when the host stops

📄 sample/src/Ledgerly.Worker/AlertEvaluator.cs

public sealed partial class AlertEvaluator(
    IServiceScopeFactory scopes, IMarketDataClient marketData, IOptions<AlertOptions> options, TimeProvider clock,
    ILogger<AlertEvaluator> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(options.Value.IntervalSeconds), clock);
        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            try
            {
                await EvaluateAsync(stoppingToken);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                // One bad tick (market data down, DB failover) must not kill the loop.
                LogEvaluationFailed(logger, ex);
            }
        }
    }
    // ...
}

Five modern ideas in twenty lines: a primary constructor, an options object, a TimeProvider-driven PeriodicTimer, a token that ends the loop on shutdown, and an exception filter that lets cancellation through while it logs everything else.

Nullable reference types

Every Ledgerly project has <Nullable>enable</Nullable> (chapter 3). string means "never null" and string? means "might be null". The compiler warns when you dereference something that might be null, and with warnings as errors, that warning stops the build.

Before After
if (name == null) throw new ArgumentNullException(...) in every method string name in the signature, and the compiler checks the callers
Comments like "may return null" Task<PortfolioValuation?> in the return type
Guard analyzers such as CA1062 Turned off in the sample's .editorconfig, because "nullable reference types already cover argument null checks"

💡 Nullable annotations are documentation that the compiler enforces. Treat a ? like a contract, not decoration.


🧠 The C# you will see in every Ledgerly file

net10.0 projects use C# 14 by default (what's new). You skipped a lot of versions. Here are the features that change how everyday code looks, with the version that introduced them and where they show up in Ledgerly.

Feature Since What it replaces Where in Ledgerly
Records C# 9 Hand-written DTOs with Equals/GetHashCode Request and response types, Holding, Quote
init-only setters, target-typed new() C# 9 Mutable DTOs, repeated type names Entities such as Portfolio
Top-level statements C# 9 class Program { static void Main } Every Program.cs
File-scoped namespaces C# 10 One extra level of indentation in every file Every file, enforced by .editorconfig
global using, implicit usings C# 10 15 using lines at the top of each file <ImplicitUsings>enable</ImplicitUsings>
Raw string literals """...""" C# 11 Escaped SQL and JSON strings The SQL in EnsureAppUserAsync
required members C# 11 Constructors with ten parameters, or forgotten properties Portfolio.OwnerId, Transaction.Symbol
List and relational patterns C# 9 to 11 if chains with && and || file.Length is 0 or > MaxFileBytes
Primary constructors C# 12 Constructor plus private readonly fields Services, typed clients, DbContext
Collection expressions [...] and spread .. C# 12 new List<T>(), ToArray(), Concat [] defaults, [.. portfolio.Transactions, transaction]
System.Threading.Lock C# 13 lock on a private object ChaosState in the fake market-data service

Source for the versions: C# version history, C# 12 and C# 13.

💻 In Ledgerly: records, primary constructors and collection expressions

📄 sample/src/Ledgerly.Core/Domain/Holdings.cs

namespace Ledgerly.Core.Domain;

public sealed record Holding(string Symbol, decimal Quantity, decimal AverageCost)
{
    public decimal CostBasis => Quantity * AverageCost;
}

public sealed class HoldingsException(string message) : Exception(message);

A positional record gives you a constructor, init-only properties, value equality and a readable ToString() in one line. The exception is a whole class in one line too: the primary constructor passes message to the base class.

One detail that surprises people: in a class with a primary constructor, the parameters are captured, not turned into properties. Only records generate public properties. So MarketDataClient(HttpClient http) keeps http private, which is what you want.

📄 sample/src/Ledgerly.Core/Domain/Portfolio.cs

public sealed class Portfolio
{
    public Guid Id { get; init; } = Guid.CreateVersion7();
    public required string OwnerId { get; init; }
    public required string Name { get; set; }
    public string BaseCurrency { get; init; } = "EUR";
    // ...
    public List<Transaction> Transactions { get; init; } = [];
}

required means an object initializer that forgets OwnerId doesn't compile. = [] is a collection expression: the compiler picks the right construction for List<Transaction>.

💻 In Ledgerly: patterns and raw strings

Pattern matching replaces many small if chains. From the import endpoint:

📄 sample/src/Ledgerly.Api/Endpoints/ImportEndpoints.cs

        if (file.Length is 0 or > MaxFileBytes)
        {
            return TypedResults.ValidationProblem(new Dictionary<string, string[]> { ["file"] = [$"File must be between 1 byte and {MaxFileBytes} bytes."] });
        }

And a property pattern that checks two optional settings and extracts them in one expression, from the migrator:

📄 sample/src/Ledgerly.MigrationService/Program.cs

            if (appUser.Value is { Name: { Length: > 0 } name, ClientId: { Length: > 0 } clientId })
            {
                await strategy.ExecuteAsync(async () => await EnsureAppUserAsync(db, name, Guid.Parse(clientId), stoppingToken));
                LogAppUserEnsured(logger, name);
            }

The same file builds a SQL batch with a raw string literal, so the T-SQL reads like T-SQL:

        var sql = $"""
            IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE name = N'{user.Replace("'", "''", StringComparison.Ordinal)}')
                CREATE USER [{user}] WITH SID = {sid}, TYPE = E;
            ALTER ROLE db_datareader ADD MEMBER [{user}];
            ALTER ROLE db_datawriter ADD MEMBER [{user}];
            """;

Two C# 14 features worth knowing

The sample doesn't use these yet, but you will see them in new code. Both are from the C# 14 notes.

// illustrative, not in the sample
public sealed class Position
{
    // `field` is the compiler-generated backing field: no private _symbol needed.
    public required string Symbol
    {
        get;
        set => field = value.Trim().ToUpperInvariant();
    }
}

public static class EnumerableExtensions
{
    // Extension members: properties (and static members) on types you don't own.
    extension<T>(IEnumerable<T> source)
    {
        public bool IsEmpty => !source.Any();
    }
}

Use field when a property needs one small rule, and extension members when you want items.IsEmpty instead of items.IsEmpty(). If the logic grows beyond a line, make it a method or a value object instead.

💡 Modern C# mostly removes ceremony. If a feature makes the line shorter and the intent clearer, use it. If it only makes the line shorter, don't.


🧠 Small APIs that replace big habits

TimeProvider: time as a dependency

DateTime.Now in domain code makes tests depend on the clock. TimeProvider is an abstract class in the base library. Production registers TimeProvider.System, and tests pass a FakeTimeProvider that you can move forward.

📄 sample/tests/Ledgerly.UnitTests/PriceTests.cs

        // TimeProvider makes "time" a dependency you can control in tests. No more DateTime.Now in domain code.
        var clock = new FakeTimeProvider(new DateTimeOffset(2026, 10, 28, 12, 0, 0, TimeSpan.Zero));
        var generator = new PriceGenerator(clock);

        var first = generator.GetQuote("aapl");
        clock.Advance(TimeSpan.FromSeconds(20));
        var sameMinute = generator.GetQuote("AAPL");
        clock.Advance(TimeSpan.FromHours(3));
        var later = generator.GetQuote("AAPL");

No Thread.Sleep, no flaky test that fails at midnight.

Guid.CreateVersion7(): IDs that sort

Every Ledgerly entity gets its ID in C#, not from the database: Guid.CreateVersion7(). A version 7 GUID starts with a timestamp, so new IDs are roughly in creation order. You get IDs you can generate before saving (useful for outbox messages and Location headers) without the random insert pattern that made you avoid GUID keys in SQL Server.

There is one sharp edge with client-generated keys in EF Core, and you will hit it in chapter 5.

[LoggerMessage]: structured logs, generated

Old habit: log.InfoFormat("Price for {0} unavailable", symbol). That produces a string, and strings are hard to query. Modern logging uses message templates with named placeholders, and the [LoggerMessage] source generator writes the fast code for you (docs).

📄 sample/src/Ledgerly.Api/Services/ValuationService.cs

public sealed partial class ValuationService(
    LedgerlyDbContext db, IMarketDataClient marketData, HybridCache cache, TimeProvider clock, ILogger<ValuationService> logger)
{
    // ...
    [LoggerMessage(Level = LogLevel.Warning, Message = "Price for {Symbol} unavailable ({Reason}); returning partial valuation")]
    private static partial void LogPriceUnavailable(ILogger logger, string symbol, string reason);
}
What you get Why it matters
Symbol and Reason become separate, queryable fields In chapter 16 you filter on them with KQL instead of grepping text
No allocations when the level is disabled Debug logging in hot paths stops costing you
Compile-time checks A method parameter that isn't referenced in the template is flagged (SYSLIB1015), and with warnings as errors, that's a broken build

The class must be partial so the generator can add the method body. Chapter 15 goes deeper.

What's gone, and where it went

Some things you relied on simply don't exist in .NET 10. This list is from experience, not an exhaustive compatibility matrix. Before you plan a real port of a Nordbank-style app, check the docs for each technology.

What you used In .NET 10 Where Ledgerly does it instead
System.Web: HttpContext.Current, HTTP modules, Global.asax Not part of ASP.NET Core Middleware and endpoints in Program.cs
WCF services (server side) Not part of .NET 10. Check the docs for current options before porting a WCF service HTTP APIs with OpenAPI for request/response, Azure Service Bus for messages
AppDomains for isolation and unloading You can't create extra AppDomains for isolation Separate processes or containers
.NET Remoting Not available HTTP APIs or messaging
web.config for app settings Only relevant if you host behind IIS appsettings.json, environment variables, Key Vault (chapter 13)
Windows-only assumptions (registry, event log, C:\ paths) The app runs on Linux in a container Configuration, OpenTelemetry, and no local disk state

.NET Framework 4.8 and 4.8.1 have no announced end date 🔧: they are supported as long as they're installed on a supported Windows version. They just aren't where new features land (lifecycle). Your old apps are fine where they are. New work, like Ledgerly, belongs on .NET 10, which is supported until 2028-11-14 🔧 (support policy).

💡 Most "missing" features were ways to share one big process. In the cloud you share nothing in-process: you split by process, and talk over HTTP or messages.


⚠️ Common traps

Trap What happens Do this instead
Injecting a DbContext into a BackgroundService constructor A scoped service is captured by a singleton. You get stale tracking and threading bugs, or a startup error from scope validation Inject IServiceScopeFactory and create a scope per unit of work, as AlertEvaluator does
.Result "just this once" in a request path Thread-pool starvation under load, and the service hangs while health checks still pass Make the call chain async. Analyzers and code review catch the rest
Forgetting to pass the CancellationToken Shutdown waits for work nobody needs, and requests keep running after the client left Accept CancellationToken ct in every async method and pass it on
Reading IConfiguration["Some:Key"] all over the code Typos fail at runtime, and there's no single place that lists the settings Bind a section to an options class once, inject IOptions<T>
new HttpClient() per call Socket and DNS problems under load, and no retries or timeouts Typed clients via AddHttpClient<TClient, TImpl> and the service defaults
DateTime.Now in domain logic Tests that depend on the time of day, and local-time bugs on servers in UTC TimeProvider (inject it) and DateTimeOffset
String-interpolated log messages Every value is baked into one string, so nothing is queryable Message templates, ideally [LoggerMessage]

🧰 Helpers

Your translation checklist for code review (copy it into your notes):

A minimal modern host, if you want to see the bare shape without Ledgerly around it:

// illustrative, not in the sample
var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddHostedService<Heartbeat>();
builder.Build().Run();

internal sealed partial class Heartbeat(TimeProvider clock, ILogger<Heartbeat> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(5), clock);
        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            LogTick(logger, clock.GetUtcNow());
        }
    }

    [LoggerMessage(Level = LogLevel.Information, Message = "Tick at {Now}")]
    private static partial void LogTick(ILogger logger, DateTimeOffset now);
}

Make it permanent: don't rely on memory. Chapter 3 turns most of this checklist into build rules: nullable enabled everywhere, file-scoped namespaces and primary constructors in .editorconfig, analyzers at latest-recommended, and warnings as errors. The toolkit's cheat-sheet-old-to-new.md has the old-to-new tables from this chapter on one page (see the toolkit), and the glossary explains terms like generic host and typed client.


🧪 Try it yourself (optional, ~20 min)

  1. Read the composition root and map it to your old world: bash cd sample Open src/Ledgerly.Api/Program.cs. For each line before builder.Build(), write down what you would have done in Global.asax or web.config. For each Use... line after it, name the IIS module or web.config section it replaces.
  2. Find the modern C# features from the table in the real code. Open src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs and find a record with validation attributes, a collection expression with a spread, and a handler that takes a CancellationToken.
  3. Try the C# 14 features in a file-based app. In a scratch folder outside sample/ (so the sample's strict build settings don't apply), create scratch.cs with the Position class and the extension block from this chapter, plus these top-level lines at the top of the file: csharp // illustrative, not in the sample var p = new Position { Symbol = " vwce " }; Console.WriteLine(p.Symbol); int[] none = []; Console.WriteLine(none.IsEmpty); Console.WriteLine(Guid.CreateVersion7()); Then run it: bash dotnet run scratch.cs Observe: VWCE, True, and a GUID. Run it twice and compare the first characters of the two GUIDs: version 7 GUIDs created later sort later.
  4. Build the sample and check that it's clean: bash cd sample dotnet build Ledgerly.slnx

✅ Recap

  1. Your app is now the process: Program.cs builds the host (services first, then the middleware pipeline), and DI, configuration and logging are built in.
  2. Async with cancellation, nullable reference types, records, primary constructors and collection expressions are the everyday shape of modern C#, and they mostly remove code.
  3. Small APIs retire old habits: typed HttpClients, options classes, TimeProvider, Guid.CreateVersion7() and [LoggerMessage].

Next → Chapter 3: A solution set up like a pro

Chapter 3: A solution set up like a pro

⏱ About 30 min · You'll learn: how Ledgerly's repo-wide files (Ledgerly.slnx, Directory.Build.props, Directory.Packages.props, global.json, nuget.config, .editorconfig, local tools) make a fresh clone build the same way everywhere, and why a vulnerable package you never referenced can fail the build.


📖 Story: "Clone it and build it, or it's broken"

On your second day, Tomás hands you a brand-new laptop that has nothing on it but the .NET 10 SDK, Git and Docker. "Ledgerly has one rule," he says. "Clone it and build it on a fresh laptop in one command, or it's broken. No wiki page with twelve setup steps. No 'works on my machine.' If you need a step, it goes in the repo."

You think of Nordbank. The build server had a hand-installed SDK nobody dared to upgrade. Each project had its own packages.config with slightly different versions of Newtonsoft.Json. The analyzers were "on" in one project and silently off in the others. Getting a new developer productive took two days and a senior engineer.

You clone, cd sample, run dotnet build Ledgerly.slnx, and it's green. Then Tomás tells you about the morning it wasn't: nobody had touched a single package, and the build went red with NU1903. A package Ledgerly never referenced directly had a known high-severity vulnerability. "That's the setup working," he says. "I'd rather the build tell me than an auditor." You write in your habits notebook: Build settings are code. They live in the repo, and they fail loudly.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
A .sln file full of GUIDs that merges badly Ledgerly.slnx: short, readable XML Starting with .NET 10, slnx is the default format 🔧, and diffs make sense
The same settings copy-pasted into every .csproj One Directory.Build.props at the root Every project inherits nullable, analyzers and warnings as errors, and none can drift
packages.config or versions in every project file Central Package Management in Directory.Packages.props One version per package for the whole solution, upgraded in one place
Whatever SDK the build server happened to have global.json pins the SDK band Your laptop, CI and teammates use the same compiler
Machine-wide NuGet feeds nobody remembers adding nuget.config with <clear /> and package source mapping Restores only from the sources the repo names, which blocks dependency confusion
Style rules in a wiki, enforced in code review .editorconfig plus analyzers that run on build The build rejects style violations, so reviews talk about design
"Install dotnet-ef globally, version whatever" .config/dotnet-tools.json and dotnet tool restore Tool versions are pinned in git like packages
Security scans once a quarter NuGet audit on every restore, with warnings as errors A vulnerable transitive package fails the build the day it's disclosed

💡 A pro setup isn't about clever MSBuild. It's about making the right thing the default, so nobody has to remember it.


🧠 The solution layout and .slnx

Ledgerly has nine projects. The layout is deliberately boring: src/ for things that ship, tests/ for things that verify them, and a handful of repo-wide files at the root.

sample/
├── Ledgerly.slnx
├── Directory.Build.props        compiler and quality settings for every project
├── Directory.Packages.props     every NuGet version, in one place
├── global.json                  SDK pin and test runner
├── nuget.config                 package sources
├── .editorconfig                code style and analyzer severities
├── .config/dotnet-tools.json    local tools (dotnet-ef)
├── src/
│   ├── Ledgerly.AppHost/            Aspire orchestration (chapter 6)
│   ├── Ledgerly.ServiceDefaults/    telemetry, health, resilience
│   ├── Ledgerly.Core/               domain, DbContext, migrations, contracts
│   ├── Ledgerly.Api/                minimal API (chapter 4)
│   ├── Ledgerly.Worker/             background processing (chapter 7)
│   ├── Ledgerly.MarketData/         fake price service
│   └── Ledgerly.MigrationService/   runs migrations, then exits (chapter 5)
└── tests/
    ├── Ledgerly.UnitTests/
    └── Ledgerly.IntegrationTests/

💻 In Ledgerly: the whole solution file

📄 sample/Ledgerly.slnx

<Solution>
  <Folder Name="/src/">
    <Project Path="src/Ledgerly.AppHost/Ledgerly.AppHost.csproj" />
    <Project Path="src/Ledgerly.ServiceDefaults/Ledgerly.ServiceDefaults.csproj" />
    <Project Path="src/Ledgerly.Core/Ledgerly.Core.csproj" />
    <Project Path="src/Ledgerly.Api/Ledgerly.Api.csproj" />
    <Project Path="src/Ledgerly.Worker/Ledgerly.Worker.csproj" />
    <Project Path="src/Ledgerly.MarketData/Ledgerly.MarketData.csproj" />
    <Project Path="src/Ledgerly.MigrationService/Ledgerly.MigrationService.csproj" />
  </Folder>
  <Folder Name="/tests/">
    <Project Path="tests/Ledgerly.UnitTests/Ledgerly.UnitTests.csproj" />
    <Project Path="tests/Ledgerly.IntegrationTests/Ledgerly.IntegrationTests.csproj" />
  </Folder>
  <Folder Name="/Solution Items/">
    <File Path=".editorconfig" />
    <File Path="Directory.Build.props" />
    <File Path="Directory.Packages.props" />
    <File Path="global.json" />
    <File Path="nuget.config" />
  </Folder>
</Solution>

That's the entire file. No project-type GUIDs, no per-configuration platform matrix, no GlobalSection blocks. When two people add a project on the same day, Git can merge it. dotnet new sln creates .slnx by default from .NET 10 🔧, and Visual Studio 2026 opens it (SDK what's new). To convert an existing .sln, the SDK has a dotnet sln migrate command 🔧 (check the docs for the exact syntax on your SDK).

Project references: who may depend on whom

The layout also encodes the architecture. Ledgerly.Core has no web dependencies, so the API, the worker and the migration runner can all share the domain model and the DbContext.

flowchart TD
    AppHost["Ledgerly.AppHost<br/>Aspire orchestration"]
    Api["Ledgerly.Api"]
    Worker["Ledgerly.Worker"]
    Market["Ledgerly.MarketData"]
    Mig["Ledgerly.MigrationService"]
    Defaults["Ledgerly.ServiceDefaults<br/>telemetry, health, resilience"]
    Core["Ledgerly.Core<br/>domain, DbContext, contracts"]
    Unit["Ledgerly.UnitTests"]
    Int["Ledgerly.IntegrationTests"]
    AppHost --> Api
    AppHost --> Worker
    AppHost --> Market
    AppHost --> Mig
    Api --> Defaults
    Api --> Core
    Worker --> Defaults
    Worker --> Core
    Mig --> Defaults
    Mig --> Core
    Market --> Defaults
    Unit --> Core
    Unit --> Market
    Int --> Api
Rule the graph enforces Why it matters
Nothing references Ledgerly.Api except the AppHost and the integration tests The API is an edge, not a library. Shared logic goes to Core
Ledgerly.MarketData doesn't reference Core It plays an external vendor, so it may share nothing but HTTP
Every service references ServiceDefaults Telemetry, health checks and resilience are identical everywhere (chapter 6)

💡 A project reference is an architectural decision. If a new arrow in this graph surprises you, it should surprise the reviewer too.


🧠 Directory.Build.props: one place for compiler and quality settings

MSBuild automatically imports Directory.Build.props from the project folder upward, and the first one it finds wins (docs). Put one at the solution root and every project gets its settings. You never see <Nullable> or <TargetFramework> in a Ledgerly .csproj.

💻 In Ledgerly: the quality gates

📄 sample/Directory.Build.props

<Project>
  <!-- Applies to every project under this folder. One place for compiler and quality settings. -->
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <LangVersion>latest</LangVersion>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>

    <!-- Quality gates: warnings fail the build, analyzers and code style run in every build (also in CI). -->
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <AnalysisLevel>latest-recommended</AnalysisLevel>
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
    <!-- Needed so IDE0005 (unused usings) runs on build. CS1591 = "missing XML comment", not wanted for apps. -->
    <GenerateDocumentationFile>true</GenerateDocumentationFile>
    <NoWarn>$(NoWarn);CS1591</NoWarn>

    <!-- Reproducible, CI-friendly builds. -->
    <Deterministic>true</Deterministic>
    <ContinuousIntegrationBuild Condition="'$(GITHUB_ACTIONS)' == 'true'">true</ContinuousIntegrationBuild>
    <RestorePackagesWithLockFile>false</RestorePackagesWithLockFile>
  </PropertyGroup>
</Project>
Setting What it does Old-world equivalent
TargetFramework net10.0 Every project targets .NET 10 LTS A different TargetFrameworkVersion in each project
Nullable enable Nullable reference types everywhere (chapter 2) Null checks by hand
ImplicitUsings enable Common namespaces are imported globally 15 using lines per file
TreatWarningsAsErrors Every warning, including analyzer and NuGet audit warnings, fails the build A warnings list that grew to 400 and was ignored
AnalysisLevel latest-recommended Turns on the recommended set of .NET analyzer rules for the latest SDK FxCop, if someone remembered to install it
EnforceCodeStyleInBuild Runs the IDE code-style rules (IDExxxx) during dotnet build, not just in the editor Style rules that only showed up as squiggles
GenerateDocumentationFile Produces the XML doc file. Also required for IDE0005 to run on build Nothing
ContinuousIntegrationBuild Normalizes paths in builds on GitHub Actions Build-server-specific hacks

Analyzers ship with the SDK and are on by default for .NET 5 and later, but only a few rules are warnings out of the box (docs). latest-recommended raises a curated set to warnings, and TreatWarningsAsErrors turns those into errors. If you ever need CA rules to warn without blocking, CodeAnalysisTreatWarningsAsErrors=false exempts them. Ledgerly doesn't use it.

The IDE0005 lesson

When the team first set this up, the .editorconfig said unused usings (IDE0005) should be a warning. In the editor, they were. On dotnet build, nothing happened. The rule depends on information the compiler only produces when it generates the XML documentation file, so IDE0005 only runs on build with GenerateDocumentationFile=true.

Turning that on fixed IDE0005 and broke everything else: now every public type without an XML comment raised CS1591, "missing XML comment", and with warnings as errors, that's an error. Ledgerly is an app, not a library, so the fix was <NoWarn>$(NoWarn);CS1591</NoWarn>. The comment in the file records why, so the next person doesn't "clean it up."

💡 Every line in Directory.Build.props that isn't obvious gets a comment. Build settings are code that people will be tempted to delete.


🧠 Central Package Management, transitive pinning and NuGet audit

With Central Package Management (CPM), Directory.Packages.props holds every package version, and project files only name the package (docs). Upgrading EF Core is now a one-line change, and two projects can't end up on different versions by accident.

💻 In Ledgerly: one file for every version

📄 sample/Directory.Packages.props

<Project>
  <!-- Central Package Management: every NuGet version lives here, projects only name the package. -->
  <PropertyGroup>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    <!-- Lets a PackageVersion below also override a TRANSITIVE dependency (used for security patches). -->
    <CentralPackageTransitivePinningEnabled>true</CentralPackageTransitivePinningEnabled>
  </PropertyGroup>

  <ItemGroup Label="Security pins for transitive dependencies (NuGet audit, NU1902/NU1903)">
    <PackageVersion Include="System.Security.Cryptography.Pkcs" Version="10.0.14" />
  </ItemGroup>
  <!-- ... -->
  <ItemGroup Label="ASP.NET Core and EF Core">
    <PackageVersion Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.14" />
    <PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="10.0.14" />
    <PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.14" />
    <PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.14" />
    <!-- ... -->
  </ItemGroup>
  <!-- ... -->
</Project>

And the project side, with no versions at all:

📄 sample/src/Ledgerly.Core/Ledgerly.Core.csproj

  <ItemGroup>
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" PrivateAssets="all" />
  </ItemGroup>

PrivateAssets="all" keeps the EF design-time package (used by dotnet ef) from flowing to projects that reference Core. The ItemGroup labels group packages by purpose, which makes upgrade pull requests easier to review: everything Aspire moves together, and everything ASP.NET Core and EF Core moves together.

Two CPM features Ledgerly doesn't need, but you should know exist:

<!-- illustrative, not in the sample -->
<!-- In one .csproj: an approved exception to the central version -->
<PackageReference Include="Some.Package" VersionOverride="2.0.0" />

<!-- In Directory.Packages.props: added to every project, e.g. an analyzer package -->
<GlobalPackageReference Include="Some.Analyzers" Version="1.0.0" />

Only the nearest Directory.Packages.props applies to a project. If you nest one deeper in the tree, it replaces the root one for that subtree instead of adding to it.

The day NuGet audit failed the build

Here is what Tomás meant. During restore, NuGet checks your packages, including transitive ones (the packages your packages depend on), against known vulnerabilities and reports them as warnings. One morning a transitive dependency, System.Security.Cryptography.Pkcs (versions 9.0.11 and 10.0.12 were in the graph), was flagged: NU1902 (moderate) and NU1903 (high). With TreatWarningsAsErrors, the build failed.

Nobody at Ledgerly had referenced that package. It came in underneath other packages. Without CPM, the usual fix is to add a direct PackageReference to a patched version in every affected project, and to remember to remove it later. With CPM, CentralPackageTransitivePinningEnabled lets a PackageVersion entry override a transitive dependency too. One line in Directory.Packages.props, under a label that says why it's there, and the whole graph moves to the patched version.

flowchart TD
    Restore["dotnet restore"] --> Graph["Resolve package graph<br/>direct and transitive"]
    Graph --> Audit{"NuGet audit<br/>known vulnerabilities?"}
    Audit -- "no" --> Build["Compile, analyzers, code style"]
    Audit -- "yes" --> Warn["NU1902 moderate<br/>NU1903 high"]
    Warn --> Twae["TreatWarningsAsErrors<br/>build fails"]
    Twae --> Pin["Pin patched version<br/>in Directory.Packages.props"]
    Pin --> Transitive["Transitive pinning<br/>overrides the old version"]
    Transitive --> Restore
    Build --> Green["Green build"]
Without the setup With the setup
The vulnerable package ships, and an auditor or attacker finds it months later The first build after disclosure fails, with the package name and advisory
The fix is a direct reference copied into several projects The fix is one pinned line, labeled with its reason
Nobody remembers why that odd reference is there The ItemGroup label documents it, and Dependabot proposes upgrades later (chapter 9)

🔧 NuGet audit has settings for severity levels and audit modes. Ledgerly uses the defaults, and this course doesn't depend on any specific value. Check the docs before you tune them. To see what audit sees, list vulnerable packages yourself (the command is in Try it).

💡 A red build caused by a vulnerability is a feature. Pin the fix centrally, label it, and let your dependency bot remove the pin when upstream catches up.


🧠 global.json and nuget.config: same SDK, same sources

Pinning the SDK

📄 sample/global.json

{
  "sdk": {
    "version": "10.0.100",
    "rollForward": "latestFeature"
  },
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}

version sets the minimum SDK and rollForward decides how far the CLI may move from it (docs). latestFeature means "any .NET 10 SDK at or above 10.0.100, preferring the newest installed feature band." That's a pragmatic middle: everyone builds with .NET 10, but you don't have to install one exact patch to work.

rollForward value Behavior in short When to use it
latestFeature Newest installed feature band and patch of the same major and minor Ledgerly's choice for app repos
latestPatch Same feature band, newest patch You want tighter control and install SDKs deliberately
disable Exactly the version in the file Rarely: reproducing a specific compiler bug
latestMajor Whatever newest SDK is installed Almost never in a team repo

The test section is new in the .NET 10 SDK. It switches dotnet test to Microsoft.Testing.Platform, which xUnit v3 test projects need. Without it, dotnet test on the .NET 10 SDK failed with "Testing with VSTest target is no longer supported by Microsoft.Testing.Platform on .NET 10 SDK." Chapter 17 tells that story in full.

CI uses the same file: the actions/setup-dotnet step reads sample/global.json through its global-json-file input, so the build server can't drift from your laptop (chapter 8).

Locking down package sources

📄 sample/nuget.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- Only nuget.org, no machine-wide feeds leaking in. Package source mapping blocks dependency-confusion attacks. -->
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
  </packageSourceMapping>
</configuration>

Jonas from security cares about this file more than any other in the repo:

💡 global.json decides which compiler builds your code, and nuget.config decides where the code you didn't write comes from. Both belong in git.


🧠 Style, analyzers and local tools

.editorconfig: decisions, not opinions

The .editorconfig at the root sets formatting, naming and analyzer severities for every file. Because EnforceCodeStyleInBuild is on, rules set to warning are checked during build, and with warnings as errors, they block it.

📄 sample/.editorconfig

[*.cs]
# Namespaces and usings
csharp_style_namespace_declarations = file_scoped:warning
dotnet_sort_system_directives_first = true

# Prefer modern syntax
csharp_style_var_for_built_in_types = true:suggestion
csharp_style_var_when_type_is_apparent = true:suggestion
csharp_prefer_simple_using_statement = true:suggestion
csharp_style_prefer_primary_constructors = true:suggestion
dotnet_style_prefer_collection_expression = when_types_loosely_match:suggestion
# ...
# Analyzer tuning (each line is a deliberate team decision)
dotnet_diagnostic.CA1848.severity = suggestion   # LoggerMessage delegates: we use [LoggerMessage] where it matters
dotnet_diagnostic.CA1515.severity = none         # 'make types internal' is noisy for app code
dotnet_diagnostic.CA1062.severity = none         # nullable reference types already cover argument null checks
dotnet_diagnostic.CA2007.severity = none         # ConfigureAwait(false) is not needed in ASP.NET Core apps
dotnet_diagnostic.CA1716.severity = none
dotnet_diagnostic.IDE0005.severity = warning     # remove unnecessary usings

[**/Migrations/*.cs]
generated_code = true
dotnet_diagnostic.IDE0005.severity = none

[tests/**.cs]
dotnet_diagnostic.CA1707.severity = none         # underscores in test method names are fine
dotnet_diagnostic.CA2000.severity = none

Notice the pattern. Each rule that's turned off has a reason in a comment, so turning a rule off is a reviewed team decision, not a quiet #pragma. Generated code (EF migrations) is marked as generated, so it doesn't fight the style rules. Tests get their own section, because Stale_version_gets_409_conflict is a better test name than StaleVersionGets409Conflict.

Severity Effect in the editor Effect on dotnet build here
none Hidden Nothing
suggestion Dots under the code, with a quick fix Nothing
warning Green squiggle Error, because of TreatWarningsAsErrors
error Red squiggle Error

Suggestions are the gentle nudge for modern syntax (primary constructors, collection expressions). Warnings are the rules the team agreed are non-negotiable.

dotnet format

dotnet format applies .editorconfig style and analyzer fixes to the code, and dotnet format --verify-no-changes fails if anything would change (docs). Ledgerly's CI doesn't run it as a separate step, because EnforceCodeStyleInBuild already fails the build on the rules that matter. Run it locally to fix a batch of style issues in one go.

Local tools: pinned like packages

dotnet ef is a .NET tool. Installing it globally means every developer has a different version, and some have none. A tool manifest pins it in the repo.

📄 sample/.config/dotnet-tools.json

{
  "version": 1,
  "isRoot": true,
  "tools": {
    "dotnet-ef": {
      "version": "10.0.14",
      "commands": [
        "dotnet-ef"
      ],
      "rollForward": false
    }
  }
}

dotnet tool restore installs exactly these versions for this repo, and dotnet ef then works inside sample/. The tool version matches the EF Core package version in Directory.Packages.props (10.0.14 🔧), which avoids "tools version is older than runtime" warnings. Keep them in sync when you upgrade.

For a one-off tool you don't want in the manifest, the .NET 10 SDK can run a tool without installing it: dnx <tool> or dotnet tool exec 🔧, for example dnx dotnet-counters when you need runtime counters once.

💡 If a command is part of the workflow, its tool belongs in the manifest. If it's a one-off, use dnx and leave no trace.


🗺️ A fresh clone, end to end

This is Tomás's "one command" rule as a flow. Every box is driven by a file in the repo, not by the machine.

flowchart TD
    Clone["git clone<br/>cd sample"] --> Sdk["global.json<br/>selects a .NET 10 SDK"]
    Sdk --> Sources["nuget.config<br/>nuget.org only"]
    Sources --> Versions["Directory.Packages.props<br/>one version per package"]
    Versions --> Audit["NuGet audit<br/>direct and transitive"]
    Audit --> Props["Directory.Build.props<br/>net10.0, nullable, analyzers"]
    Props --> Style[".editorconfig<br/>style and rule severities"]
    Style --> Compile["Compile all 9 projects"]
    Compile -- "any warning" --> Red["Build fails<br/>fix before merge"]
    Compile -- "clean" --> Green["Green build<br/>same as CI"]

The CI workflow runs exactly the same steps. Its comment says it plainly: "Warnings are errors, analyzers + code style run, NuGet audit fails on vulnerable packages." There is no separate "CI configuration" for quality, which is why a green build on your laptop predicts a green build in CI (chapter 8).


⚠️ Common traps

Trap What happens Do this instead
Adding Version="..." to a PackageReference in a project With CPM on, restore reports an error instead of quietly using two versions Add or change the PackageVersion in Directory.Packages.props. Use VersionOverride only for a reviewed exception
Fixing a vulnerable transitive package with direct references in each project Pins scatter, nobody removes them, and versions drift Use CentralPackageTransitivePinningEnabled and one labeled pin
Setting IDE0005 to warning and expecting the build to enforce it It only shows in the editor Turn on GenerateDocumentationFile (and suppress CS1591 for apps)
Silencing a rule with #pragma warning disable in code The decision is invisible and spreads by copy-paste Change severity in .editorconfig with a comment, or fix the code. Keep #pragma for narrow, commented cases (the migrator's raw SQL is one)
Nesting a second Directory.Packages.props or Directory.Build.props The nearest one wins, and the root settings silently stop applying to that subtree Keep one of each at the root, or import the parent explicitly
Installing dotnet-ef globally Version mismatches between developers and CI Use the tool manifest and dotnet tool restore
A nuget.config without <clear /> Machine-level feeds join the restore, and results differ per laptop <clear />, then list exactly the sources you need, with source mapping

🧰 Helpers

A starter Directory.Build.props for a new app repo:

<!-- illustrative, not in the sample -->
<Project>
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <AnalysisLevel>latest-recommended</AnalysisLevel>
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
    <GenerateDocumentationFile>true</GenerateDocumentationFile>
    <NoWarn>$(NoWarn);CS1591</NoWarn>
  </PropertyGroup>
</Project>

Commands you'll use for repo hygiene (run from sample/):

Task Command
Which SDK does this folder use? dotnet --version
Build everything with all gates dotnet build Ledgerly.slnx
Restore pinned local tools dotnet tool restore
Show vulnerable packages, including transitive dotnet list package --vulnerable --include-transitive
Fix style issues in bulk dotnet format
Check style without changing files dotnet format --verify-no-changes

Checklist for a new repo:

Make it permanent: copy these files from sample/ into your next repo on day one, before the first feature, because tightening rules later means fixing hundreds of warnings at once. Add a Dependabot config (chapter 9) so pinned versions don't rot. The toolkit has ready-to-copy Directory.Build.props, Directory.Packages.props, .editorconfig and dependabot.yml files, and cheat-sheet-old-to-new.md repeats the command table (see the toolkit). Terms like CPM and transitive dependency are in the glossary.


🧪 Try it yourself (optional, ~20 min)

  1. Check which SDK the repo selects, and build: bash cd sample dotnet --version dotnet build Ledgerly.slnx The version should be a 10.0 SDK at or above 10.0.100, and the build should end with 0 warnings and 0 errors.
  2. Watch a quality gate work. Open src/Ledgerly.Core/Domain/Holdings.cs and add an unused using System.Text; below the namespace line. Build again: bash dotnet build Ledgerly.slnx You should get error IDE0005 (unnecessary using). It's a warning in .editorconfig, and warnings are errors here. Remove the line.
  3. Restore the pinned tools and check the EF tool version: bash dotnet tool restore dotnet ef --version It should report 10.0.14, the version in .config/dotnet-tools.json.
  4. See what NuGet audit sees: bash dotnet list package --vulnerable --include-transitive With the Pkcs pin in place, you should see no vulnerable packages, unless something new was disclosed since this was written 🔧. If something shows up, you know what to do.
  5. Open Ledgerly.slnx and an old .sln from one of your own projects side by side. Count the lines. Write the difference in your habits notebook.

✅ Recap

  1. Repo-wide files (Directory.Build.props, Directory.Packages.props, global.json, nuget.config, .editorconfig, the tool manifest) make every clone build the same way on any machine and in CI.
  2. Warnings as errors plus latest-recommended analyzers and code style in build turn team agreements into rules the compiler enforces, with every exception commented.
  3. CPM with transitive pinning lets NuGet audit fail the build on a vulnerable package you never referenced, and lets you fix it with one labeled line.

Next → Chapter 4: Designing the API

Chapter 4: Designing the API

⏱ About 35 min · You'll learn: how to build a clear HTTP contract with minimal APIs, route groups and TypedResults, how validation, ProblemDetails and OpenAPI 3.1 come almost for free in ASP.NET Core 10, and how CORS, rate limiting, concurrency, async imports and feature flags each map to a status code the frontend can rely on.


📖 Story: Leo needs a contract

Leo, the frontend developer, sits down next to you with a coffee and a list. "I'm starting the React app next sprint. I don't need your code. I need a contract. What are the URLs? What do I send? What comes back when it works, and, more important, what comes back when it doesn't? If one endpoint returns { error: "..." }, another returns plain text and a third returns an HTML error page, I'll write three error handlers and hate you."

At Nordbank, your contract was a Word document with screenshots, and it was out of date the week after it was written. Errors were whatever the controller happened to throw. Leo wants something different: a machine-readable document generated from the code, one error format for everything, and status codes that mean the same thing on every endpoint.

You open your habits notebook and write: The API is a product. Its users are developers, and status codes are part of the UI. Maya adds one rule before you start: "Every response the client can get must be visible in the handler's signature. If it isn't in the type, it doesn't exist."


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Web API controllers with attribute routing, one class per resource Minimal APIs: route groups and static handler methods Less ceremony, and shared prefixes, tags and policies are set once per group
IHttpActionResult and return Ok(x) with no compile-time contract TypedResults and Results<Ok<T>, NotFound, ...> The possible responses are in the method signature, so OpenAPI and tests can see them
if (!ModelState.IsValid) in every action AddValidation() with DataAnnotations on records One registration validates every handler's input and returns a standard 400
Custom error JSON per team, or HTML error pages ProblemDetails (application/problem+json, RFC 9457) everywhere One error shape for every failure, including unhandled exceptions
A Word document or Swashbuckle bolted on Built-in OpenAPI 3.1 at /openapi/v1.json, Scalar as a dev UI The contract is generated from the code, so it can't drift
No protection against a client hammering one endpoint Built-in rate limiting with per-user partitions and 429 Expensive endpoints stay usable for everyone
"Last write wins" on updates Optimistic concurrency with a version and 409 Conflict Two users editing the same thing can't silently overwrite each other
Long uploads processed inside the request 202 Accepted plus a status URL, work done by the worker Requests stay fast, and the heavy work survives a restart

💡 A good API contract is mostly about the unhappy paths. The happy path is easy to agree on.


🧠 Minimal APIs, route groups and typed results

Minimal APIs map a route straight to a method. There's no controller base class and no attribute routing. What you lose in convention you gain in clarity: every endpoint and every policy on it is visible in one place (docs).

A route group applies a prefix, OpenAPI tags, authorization and filters to every endpoint in it. Ledgerly keeps each group in its own file, as an extension method that Program.cs calls.

💻 In Ledgerly: the portfolio routes

📄 sample/src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs

public static class PortfolioEndpoints
{
    public static IEndpointRouteBuilder MapPortfolioEndpoints(this IEndpointRouteBuilder app)
    {
        var group = app.MapGroup("/api/portfolios")
            .WithTags("Portfolios")
            .RequireAuthorization();

        group.MapGet("/", ListPortfolios);
        group.MapPost("/", CreatePortfolio);
        group.MapGet("/{id:guid}", GetPortfolio).WithName("GetPortfolio");
        group.MapPut("/{id:guid}", RenamePortfolio);
        group.MapPost("/{id:guid}/transactions", RecordTransaction);
        group.MapGet("/{id:guid}/valuation", GetValuation);

        return app;
    }
    // ...
}

You can read the whole resource from this block, like a table of contents. RequireAuthorization() on the group means no endpoint in it can be forgotten. {id:guid} is a route constraint: /api/portfolios/abc doesn't match, so it returns 404 without reaching your code.

Handlers are plain static methods. Parameters are bound by type and source: route values, the JSON body, ClaimsPrincipal for the user, services from DI and the request's CancellationToken.

    private static async Task<Results<Ok<PortfolioDetails>, NotFound>> GetPortfolio(
        Guid id, ClaimsPrincipal user, LedgerlyDbContext db, CancellationToken ct)
    {
        var ownerId = user.GetOwnerId();
        var portfolio = await db.Portfolios.AsNoTracking()
            .Include(p => p.Transactions)
            .SingleOrDefaultAsync(p => p.Id == id && p.OwnerId == ownerId, ct);

        if (portfolio is null)
        {
            return TypedResults.NotFound();
        }
        // ...
    }

Why Results<...> matters

The return type Results<Ok<PortfolioDetails>, NotFound> is Maya's rule in code. The compiler only allows those two outcomes, OpenAPI documents both automatically, and a unit test can assert on the result type without an HTTP server. Compare it with the old IHttpActionResult, which could be anything at all.

Return type Meaning for Leo
Ok<PortfolioSummary[]> Always 200 with an array
CreatedAtRoute<PortfolioSummary> 201 with a Location header pointing at GetPortfolio
Results<Ok<PortfolioDetails>, NotFound> 200, or 404 if it doesn't exist or isn't yours
Results<Ok<PortfolioSummary>, NotFound, ProblemHttpResult> 200, 404, or a ProblemDetails error (409 here)
Results<Created<TransactionResponse>, NotFound, ValidationProblem, ProblemHttpResult> 201, 404, 400 with field errors, or 409

Notice the ownership check: p.Id == id && p.OwnerId == ownerId. Another user's portfolio returns 404, not 403, so the API doesn't even confirm that the ID exists. An integration test (Users_cannot_see_each_others_portfolios) guards this.

The owner ID comes from the token, through one small extension method:

📄 sample/src/Ledgerly.Api/ApiConventions.cs

    /// <summary>
    /// Stable user id. Entra ID puts the object id in "oid"; dev tokens from `dotnet user-jwts` use "sub"
    /// (mapped to NameIdentifier by the JWT handler).
    /// </summary>
    public static string GetOwnerId(this ClaimsPrincipal user) =>
        user.FindFirstValue("oid")
        ?? user.FindFirstValue("http://schemas.microsoft.com/identity/claims/objectidentifier")
        ?? user.FindFirstValue(ClaimTypes.NameIdentifier)
        ?? user.FindFirstValue("sub")
        ?? throw new InvalidOperationException("Authenticated user has no id claim.");

The same code works with local dev tokens and with Entra ID tokens in Azure. Chapter 13 covers the identity side.

💡 If a status code isn't in the handler's return type, the client should never see it, except for the cross-cutting ones (401, 429, 500) that middleware owns.


🧠 Validation and ProblemDetails: one error shape for everything

Validation with AddValidation()

In ASP.NET Core 10, one line turns on validation for minimal APIs: builder.Services.AddValidation(). A source generator finds the types your handlers take, validates DataAnnotations attributes (and IValidatableObject), and returns 400 with a validation ProblemDetails when something fails. Records are supported, and you can opt an endpoint out with .DisableValidation() (release notes). The validation APIs now live in the Microsoft.Extensions.Validation namespace, so you can reuse them outside ASP.NET Core.

💻 In Ledgerly: requests are records with attributes

📄 sample/src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs

public sealed record CreatePortfolioRequest(
    [Required, StringLength(100, MinimumLength = 1)] string Name,
    [RegularExpression("^[A-Z]{3}$")] string? BaseCurrency);

public sealed record RenamePortfolioRequest(
    [Required, StringLength(100, MinimumLength = 1)] string Name,
    [Required] string Version);
// ...
public sealed record RecordTransactionRequest(
    [Required, StringLength(12, MinimumLength = 1)] string Symbol,
    TransactionType Type,
    [Range(typeof(decimal), "0.000001", "1000000000")] decimal Quantity,
    [Range(typeof(decimal), "0.000001", "1000000000")] decimal Price,
    DateOnly TradeDate);

The handler never sees an invalid CreatePortfolioRequest. There's no ModelState check, because validation runs before the handler.

Attribute validation covers the shape of input. Business rules still live in the domain. "You cannot sell what you do not hold" needs the portfolio's history, so RecordTransaction checks it and returns the same 400 format by hand:

        try
        {
            // Domain rule: you cannot sell what you do not hold (at that date).
            HoldingCalculator.Calculate([.. portfolio.Transactions, transaction]);
        }
        catch (HoldingsException ex)
        {
            return TypedResults.ValidationProblem(new Dictionary<string, string[]> { ["quantity"] = [ex.Message] });
        }

For Leo, both kinds of failure look identical: a 400 with an errors object keyed by field name. He writes one function that shows field errors next to inputs.

ProblemDetails for every other error

Three registrations in Program.cs give you RFC 9457 application/problem+json responses everywhere (docs):

📄 sample/src/Ledgerly.Api/Program.cs

// ── API surface: validation, ProblemDetails, OpenAPI, JSON ─────────────────────────────────────
builder.Services.AddValidation();
builder.Services.AddProblemDetails();
builder.Services.AddOpenApi();
builder.Services.ConfigureHttpJsonOptions(o => o.SerializerOptions.Converters.Add(new JsonStringEnumConverter()));
// ...
app.UseExceptionHandler();
app.UseStatusCodePages();
Registration What it covers
AddProblemDetails() Registers the service that writes ProblemDetails bodies
UseExceptionHandler() An unhandled exception becomes a 500 ProblemDetails, with no stack trace leaked to the client
UseStatusCodePages() A bare 401 or 404 with no body gets a ProblemDetails body too
JsonStringEnumConverter Enums travel as "Buy" and "Sell", not 0 and 1, which is friendlier for a TypeScript client

The integration tests pin this down, so nobody can break it by accident:

📄 sample/tests/Ledgerly.IntegrationTests/PortfolioApiTests.cs

    [Fact]
    public async Task Invalid_input_returns_problem_details()
    {
        var client = ClientFor("alice");

        var response = await client.PostAsJsonAsync("/api/portfolios", new { name = "", baseCurrency = "euro" }, Ct);

        Assert.Equal(HttpStatusCode.BadRequest, response.StatusCode);
        Assert.Equal("application/problem+json", response.Content.Headers.ContentType?.MediaType);
    }

💡 Errors are part of the contract. One format (ProblemDetails), one place to produce it (the framework), and a test that asserts the content type.


🗺️ The status code map Leo gets

Every Ledgerly endpoint follows the same map. Leo prints it and pins it above his desk.

flowchart TD
    Req["Request"] --> Auth{"Valid bearer token?"}
    Auth -- "no" --> R401["401 Unauthorized"]
    Auth -- "yes" --> Rate{"Within rate limit?<br/>imports only"}
    Rate -- "no" --> R429["429 Too Many Requests"]
    Rate -- "yes" --> Flag{"Feature enabled?<br/>alerts only"}
    Flag -- "no" --> R404a["404 Not Found"]
    Flag -- "yes" --> Valid{"Input valid?"}
    Valid -- "no" --> R400["400 validation ProblemDetails"]
    Valid -- "yes" --> Own{"Exists and yours?"}
    Own -- "no" --> R404b["404 Not Found"]
    Own -- "yes" --> Ver{"Version current?"}
    Ver -- "no" --> R409["409 Conflict"]
    Ver -- "yes" --> R2xx["200 OK, 201 Created<br/>or 202 Accepted"]

Anything unexpected becomes a 500 ProblemDetails from the exception handler. Chapter 14 covers the other kind of failure, when a dependency rather than your code is the problem.


🧠 OpenAPI 3.1, Scalar and the versioning decision

The contract, generated

builder.Services.AddOpenApi() plus app.MapOpenApi() serves an OpenAPI document at /openapi/v1.json, and .NET 10 generates OpenAPI 3.1 by default (docs). Your Results<...> return types become the documented responses, the route group's WithTags becomes the grouping, and XML doc comments flow into descriptions. That's one more reason chapter 3 turned on GenerateDocumentationFile: a /// <summary> on a handler like RenamePortfolio ("Optimistic concurrency: the client sends back the Version it read; a stale version gets 409.") shows up in the docs Leo reads.

Swashbuckle is no longer in the project templates 🔧. For a browsable UI, Ledgerly uses Scalar, a third-party package that renders the document at /scalar.

💻 In Ledgerly: docs in development only

📄 sample/src/Ledgerly.Api/Program.cs

if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();                 // /openapi/v1.json
    app.MapScalarApiReference();      // /scalar  (interactive API explorer)
}

The interactive explorer stays off in production: there's no reason to hand an attacker a map with a "try it" button. Leo doesn't need production to get the contract. He can generate a TypeScript client from the JSON on a dev machine, or the team can generate the document at build time with Microsoft.Extensions.ApiDescription.Server and commit it.

Option Pros Cons
Serve /openapi/v1.json and /scalar in Development only (Ledgerly) Zero setup, always current Clients need a running dev instance
Generate the document at build time Reviewable in pull requests, contract diffs are visible One more package and build step
Serve docs in production behind auth Partners can explore the live API Extra surface to protect

Versioning: a decision, not a package

Leo asks, "Is it /api/v1/portfolios?" It isn't, and that's deliberate. ASP.NET Core has no built-in API versioning. The common choice is the .NET Foundation package Asp.Versioning.Http (10.2.3 🔧) for minimal APIs.

Ledgerly's only client is its own SPA, deployed from the same repo. The team decided to not version yet, keep changes backward compatible (add fields, never rename or remove them), and add Asp.Versioning.Http the day a third party depends on the API. The sample uses no versioning package. Maya asks you to write that down as an architecture decision record using the toolkit's ADR template, so the "why" survives the people who made it.

💡 Versioning has a cost. Pay it when you have consumers you can't redeploy, not before.


🧠 Protecting the API: CORS and rate limiting

CORS for the SPA

Leo's React dev server runs at http://localhost:5173, and the API runs on a different origin. Browsers block that kind of cross-origin call unless the API says the origin is allowed. Ledgerly reads the allowed origins from configuration, so each environment can list its own front end.

📄 sample/src/Ledgerly.Api/Program.cs

// ── Ready for a React SPA later: allowed origins come from configuration ────────────────────────
var spaOrigins = builder.Configuration.GetSection("Cors:AllowedOrigins").Get<string[]>() ?? [];
builder.Services.AddCors(o => o.AddPolicy(ApiPolicies.Spa, p => p.WithOrigins(spaOrigins).AllowAnyHeader().AllowAnyMethod()));

appsettings.json has an empty list, and appsettings.Development.json adds http://localhost:5173. An empty list by default means "no browser origin is allowed" until someone decides otherwise, which is the safe default. In Azure, the origin comes from an environment variable (Cors__AllowedOrigins__0) set by Bicep (chapter 11).

CORS is a browser rule, not a security boundary for the API. A script or curl ignores it. Authentication (the bearer token) is what protects the data.

Rate limiting the expensive endpoint

Importing a broker CSV is the most expensive thing a user can do. Ledgerly limits it per user with the built-in rate-limiting middleware (docs).

// ── Protection: per-user rate limit for the expensive import endpoint ──────────────────────────
builder.Services.AddRateLimiter(options =>
{
    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    options.AddPolicy(ApiPolicies.Imports, http => RateLimitPartition.GetFixedWindowLimiter(
        partitionKey: http.User.Identity?.Name ?? http.Connection.RemoteIpAddress?.ToString() ?? "anonymous",
        factory: _ => new FixedWindowRateLimiterOptions { PermitLimit = 5, Window = TimeSpan.FromMinutes(1) }));
});

Three decisions are hiding in those lines:

Decision Why
RejectionStatusCode = 429 The default is 503 Service Unavailable. A 503 tells clients and monitors that the server is in trouble, which triggers retries and alerts. 429 says "you, slow down."
Partition by user name, then IP, then "anonymous" Each user gets their own five imports per minute. One noisy user can't use up everyone's budget
Fixed window, 5 per minute The simplest of the four built-in algorithms (fixed window, sliding window, token bucket, concurrency). Good enough for an upload endpoint

The policy is attached to just the import endpoint, with .RequireRateLimiting(ApiPolicies.Imports). Policy names live in ApiConventions.cs as constants ("imports", "spa"), so a typo is a compile error, not a policy that silently doesn't apply.

💡 Always set RejectionStatusCode explicitly. A rate limiter that answers 503 makes your own monitoring think the service is down.


🧠 Concurrency, async imports and feature flags

Three endpoints show three more contract patterns Leo will meet.

409 Conflict: optimistic concurrency

Every portfolio response includes a Version string: the database's rowversion, Base64-encoded. To rename, the client sends that version back. If someone else changed the portfolio in between, the update affects zero rows, EF Core throws, and the API returns 409.

        db.Entry(portfolio).Property(p => p.RowVersion).OriginalValue = Convert.FromBase64String(request.Version);
        portfolio.Name = request.Name.Trim();
        portfolio.UpdatedAt = DateTimeOffset.UtcNow;

        try
        {
            await db.SaveChangesAsync(ct);
        }
        catch (DbUpdateConcurrencyException)
        {
            return TypedResults.Problem("Portfolio was changed by someone else. Reload and try again.", statusCode: StatusCodes.Status409Conflict);
        }

For Leo, the rule is simple: on 409, reload and show the user the current data. Chapter 5 explains the database side, and why recording a trade uses the same mechanism to stop a double "sell all."

202 Accepted: work that takes a while

A CSV import can take seconds or minutes. Holding an HTTP request open that long is fragile: a deployment or a scale-in kills it halfway. Instead, the API stores the file, queues the work and answers immediately.

📄 sample/src/Ledgerly.Api/Endpoints/ImportEndpoints.cs

        group.MapPost("/portfolios/{id:guid}/imports", StartImport)
            .RequireRateLimiting(ApiPolicies.Imports)
            // Bearer-token APIs are not vulnerable to CSRF (browsers never attach the token automatically).
            .DisableAntiforgery();
        // ...
    private static async Task<Results<AcceptedAtRoute<ImportStatusResponse>, NotFound, ValidationProblem>> StartImport(
        Guid id, IFormFile file, ClaimsPrincipal user, LedgerlyDbContext db, CancellationToken ct)
    {
        if (file.Length is 0 or > MaxFileBytes)
        {
            return TypedResults.ValidationProblem(new Dictionary<string, string[]> { ["file"] = [$"File must be between 1 byte and {MaxFileBytes} bytes."] });
        }
        // ...
        db.Imports.Add(import);
        db.EnqueueOutboxMessage(Queues.Imports, new ImportRequested(import.Id, id));
        await db.SaveChangesAsync(ct); // one transaction: both rows or neither

        return TypedResults.AcceptedAtRoute(ToResponse(import), "GetImport", new { importId = import.Id });
    }
Detail Why
IFormFile parameter Minimal APIs bind a multipart upload directly
MaxFileBytes = 1_000_000 Rejects empty and oversized files with the standard 400
.DisableAntiforgery() Opts the upload out of antiforgery checks. As the comment says, browsers never attach a bearer token automatically, so CSRF doesn't apply
Import row and outbox message in one SaveChangesAsync Both rows commit or neither does. Chapter 7 explains the outbox
AcceptedAtRoute(..., "GetImport", ...) 202 with a Location header pointing at /api/imports/{importId}
sequenceDiagram
    participant SPA as "React SPA"
    participant Api as "Ledgerly.Api"
    participant Sql as "Azure SQL"
    participant Worker as "Ledgerly.Worker"
    participant Bus as "Service Bus"
    SPA->>Api: POST /api/portfolios/id/imports with CSV
    Api->>Sql: Insert Import and outbox row in one transaction
    Api-->>SPA: 202 Accepted with Location /api/imports/importId
    Worker->>Sql: Read pending outbox row
    Worker->>Bus: Send ImportRequested
    Bus->>Worker: Deliver ImportRequested
    Worker->>Sql: Save trades and set status Completed
    SPA->>Api: GET /api/imports/importId
    Api-->>SPA: 200 OK with status and row counts

Leo polls the Location URL every couple of seconds until the status is Completed or Failed. Later the team may push a notification instead, but the contract stays the same.

404 when a feature is off

Price alerts are new, and Elena wants to turn them off without a deployment if something goes wrong. Ledgerly uses Microsoft.FeatureManagement.AspNetCore (4.8.0 🔧), with flags read from the FeatureManagement configuration section. appsettings.json has "PriceAlerts": true.

For MVC, the library has a [FeatureGate] attribute that returns 404 when a feature is off. Minimal APIs have no attribute, so Ledgerly wrote a small endpoint filter that does the same for the whole group:

📄 sample/src/Ledgerly.Api/Endpoints/AlertEndpoints.cs

        var alerts = app.MapGroup("/api/alerts")
            .WithTags("Alerts")
            .RequireAuthorization()
            .AddEndpointFilter(RequireFeature(Features.PriceAlerts));
        // ...
    /// <summary>Feature flag gate: when the flag is off the endpoints behave as if they do not exist.</summary>
    private static Func<EndpointFilterInvocationContext, EndpointFilterDelegate, ValueTask<object?>> RequireFeature(string feature) =>
        async (context, next) =>
        {
            var features = context.HttpContext.RequestServices.GetRequiredService<IFeatureManager>();
            return await features.IsEnabledAsync(feature) ? await next(context) : TypedResults.NotFound();
        };

404 is the honest answer: with the flag off, the feature doesn't exist. Leo's app hides the alerts menu on 404. Chapter 19 uses flags like this one as kill switches during an incident.

💡 Every pattern in this section is a status code with a documented client reaction: 409 means reload, 202 means poll, 404 on a flagged feature means hide it.


⚠️ Common traps

Trap What happens Do this instead
Returning IResult or Task<IResult> from handlers OpenAPI can't see the responses, and nothing stops a handler from returning anything Use TypedResults and Results<...> so the contract is in the signature
Leaving the rate limiter's rejection code at the default Clients get 503, retry harder, and your alerts say the service is down Set RejectionStatusCode = StatusCodes.Status429TooManyRequests
Returning 403 for another user's resource You confirm the ID exists, which leaks information Filter by owner in the query and return 404
WithOrigins("*") or AllowAnyOrigin() "just for now" Any website can call your API from a user's browser Read allowed origins from configuration, empty by default
Mapping Scalar or OpenAPI in every environment A browsable attack map in production Map them inside IsDevelopment(), or protect them
Doing the CSV import inside the request Timeouts, half-finished imports when a replica stops, and angry retries 202 plus a status URL, with the work done by the worker via the outbox
UseCors after UseAuthorization, or UseRateLimiter before authentication Preflight requests fail, or per-user partitions see no user Keep the order from Program.cs: exception handler, status pages, CORS, authentication, authorization, rate limiter

🧰 Helpers

A route group template to copy for a new resource:

// illustrative, not in the sample
public static class WatchlistEndpoints
{
    public static IEndpointRouteBuilder MapWatchlistEndpoints(this IEndpointRouteBuilder app)
    {
        var group = app.MapGroup("/api/watchlists").WithTags("Watchlists").RequireAuthorization();
        group.MapGet("/{id:guid}", GetWatchlist).WithName("GetWatchlist");
        return app;
    }

    private static async Task<Results<Ok<WatchlistResponse>, NotFound>> GetWatchlist(
        Guid id, ClaimsPrincipal user, LedgerlyDbContext db, CancellationToken ct)
    {
        // Filter by owner in the query, return NotFound otherwise.
        await Task.CompletedTask;
        return TypedResults.NotFound();
    }
}

public sealed record WatchlistResponse(Guid Id, string Name);

Status codes and what the client should do (share this with your frontend team):

Code When Ledgerly returns it Client reaction
200 / 201 Success, with Location on 201 Show the result
202 Import accepted Poll the Location URL
400 Validation or business rule failed Show errors next to the fields
401 Missing or expired token Sign in again
404 Not found, not yours, or feature off Show "not found" or hide the feature
409 Stale version Reload, then let the user retry
429 Rate limit hit Wait, then retry, and tell the user
500 Bug or unexpected failure Generic error, and report it

API review checklist:

Make it permanent: add the checklist to your pull request template (the toolkit has pr-review-checklist.md and pull_request_template.md, see the toolkit), and back each rule with an integration test like Invalid_input_returns_problem_details and Stale_version_gets_409_conflict. Record the versioning decision with adr-template.md. Terms like ProblemDetails and route group are in the glossary.


🧪 Try it yourself (optional, ~30 min)

You need a container runtime (Docker Desktop or Podman) for Aspire.

  1. Start everything: bash cd sample aspire run (Without the Aspire CLI: dotnet run --project src/Ledgerly.AppHost.) Open the dashboard link from the console, and find the API's HTTPS URL on the resources page.
  2. Browse the contract: open <api-url>/scalar and <api-url>/openapi/v1.json. Find the responses listed for PUT /api/portfolios/{id} and match them to the Results<...> type in PortfolioEndpoints.cs.
  3. Create a dev token in a second terminal: bash cd sample dotnet user-jwts create --project src/Ledgerly.Api Paste the token into src/Ledgerly.Api/Ledgerly.Api.http (the @token line) and set @api to the API URL from the dashboard.
  4. Send the requests in the .http file from Visual Studio, Rider or VS Code with an HTTP client extension: create a portfolio (201 with a Location header), then send it again with an empty name (400, application/problem+json).
  5. Trigger a 409: get the portfolio, copy its version, and send the same rename twice with that version: ```http # illustrative, not in the sample PUT {{api}}/api/portfolios/ Authorization: Bearer {{token}} Content-Type: application/json

{ "name": "Renamed", "version": "" } `` The first returns 200, the second 409. 6. Trigger a 429: send the "Import a broker CSV" request from the.httpfile six times within a minute. The first five return 202 (or 404 if you didn't replace the portfolio ID), the sixth returns 429. 7. Turn the feature off: stop the app, set"PriceAlerts": falseinsrc/Ledgerly.Api/appsettings.json, run it again, and callPOST /api/alerts. You get 404. Set it back totrue`.


✅ Recap

  1. Minimal APIs with route groups and TypedResults put the contract in the code: every possible response is in the handler's Results<...> type, and OpenAPI 3.1 documents it.
  2. AddValidation(), ProblemDetails and a fixed status code map (400, 401, 404, 409, 429, 202) give the frontend one error shape and predictable reactions.
  3. Protection and evolution are explicit decisions: CORS from configuration, rate limiting with 429 instead of the default 503, Scalar only in development, feature flags as 404, and no versioning until a real consumer needs it.

Next → Chapter 5: Data: EF Core and Azure SQL

Chapter 5: Data: EF Core and Azure SQL

⏱ About 35 min · You'll learn: how Ledgerly wires EF Core 10 through Aspire, how a rowversion turns a double "sell all" into a 409, the EF trap with client-generated keys that the team really hit, why migrations run as their own short-lived process, and how Azure SQL serverless with Entra-only auth removes both idle cost and passwords.


📖 Story: "We sold twice?"

Elena forwards a support ticket with the subject line in capitals. A beta tester had Ledgerly open on her phone and her laptop, tapped Sell All on one, got impatient and clicked it on the other. Her portfolio now shows minus 40 shares of VWCE. "Two clicks at once and we sold twice?" Elena asks. "How is that even possible? We check that you own the shares before selling."

You know exactly how. Both requests read the portfolio at the same moment. Both saw 40 shares. Both passed the check. Both inserted a sell. Each request was correct on its own, and together they were wrong. At Nordbank you would have fixed it with a stored procedure and WITH (UPDLOCK), or by accepting "last write wins" and hoping nobody noticed.

Maya has a different plan: let the database tell you when someone else got there first, and turn that into a clear answer for the second request. In your habits notebook, under Data, you write: Never trust a check you did before someone else's write. This chapter is the fix, plus everything around it: the DbContext, migrations as a deploy step, the EF Core trap that cost the team an afternoon, and the Azure SQL setup that pauses when nobody is using it.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
EDMX designer or hand-written ADO.NET, stored procedures for writes EF Core 10 code-first model, configured in OnModelCreating The model, the schema and the migrations live in one codebase and one pull request
Pessimistic locks (UPDLOCK) or "last write wins" Optimistic concurrency with a rowversion column, conflicts become 409 No locks held across requests, and the loser gets a clear answer instead of silent data loss
IDENTITY integer keys from the database Guid.CreateVersion7() keys generated in C# IDs exist before the insert (for outbox messages and Location headers) and still sort by time
A DBA runs a script by hand, or Database.Migrate() at startup Migrations run in a separate process that runs to completion before the apps start Ten replicas don't race to change the schema, and a failed migration stops the deployment
Hand-rolled retry loops, or none An execution strategy that retries transient faults A cloud database that fails over or wakes from pause is a normal event, not an outage
A SQL Server VM running 24/7 Azure SQL serverless that auto-pauses, on the free offer 💰 You pay for compute when it's used, and a beta with little traffic can cost very little
SQL logins with passwords in web.config Microsoft Entra-only authentication with managed identities There's no password to leak, rotate or paste into Slack

💡 Most of what changed is about time and concurrency: many replicas, many requests and a database that can briefly disappear. The SQL itself is the part you already know.


🧠 The DbContext, the model and the indexes

Registering the DbContext through Aspire

In the old world, you'd pass a connection string name to a DbContext constructor and hope web.config had it. In Ledgerly, the API, the worker and the migration service each register the same DbContext with one line:

📄 sample/src/Ledgerly.Api/Program.cs

// ── Data: Aspire client integration = DbContext pooling + retries + health check + tracing ──────
builder.AddSqlServerDbContext<LedgerlyDbContext>("ledgerlydb");

AddSqlServerDbContext comes from the Aspire client integration Aspire.Microsoft.EntityFrameworkCore.SqlServer. It reads the connection string named ledgerlydb (locally from Aspire, in Azure from an environment variable), and it adds retries, a health check and telemetry 🔧. You get a retrying execution strategy without writing EnableRetryOnFailure() yourself, and every query shows up as a span in the traces (chapter 15).

The model, in one place

📄 sample/src/Ledgerly.Core/Data/LedgerlyDbContext.cs

public sealed class LedgerlyDbContext(DbContextOptions<LedgerlyDbContext> options) : DbContext(options)
{
    public DbSet<Portfolio> Portfolios => Set<Portfolio>();
    public DbSet<Transaction> Transactions => Set<Transaction>();
    // ...
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Portfolio>(e =>
        {
            e.Property(p => p.OwnerId).HasMaxLength(128);
            e.Property(p => p.Name).HasMaxLength(100);
            e.Property(p => p.BaseCurrency).HasMaxLength(3).IsFixedLength();
            e.Property(p => p.RowVersion).IsRowVersion();
            e.HasIndex(p => p.OwnerId);
            e.HasMany(p => p.Transactions).WithOne().HasForeignKey(t => t.PortfolioId);
        });

        modelBuilder.Entity<Transaction>(e =>
        {
            e.Property(t => t.Symbol).HasMaxLength(12);
            e.Property(t => t.Type).HasConversion<string>().HasMaxLength(4);
            e.Property(t => t.Quantity).HasPrecision(18, 6);
            e.Property(t => t.Price).HasPrecision(18, 6);
            e.Property(t => t.ExternalRef).HasMaxLength(64);
            e.HasIndex(t => new { t.PortfolioId, t.ExternalRef })
                .IsUnique()
                .HasFilter("[ExternalRef] IS NOT NULL");
        });
        // ...
    }
}

The domain classes stay free of EF attributes (they live in Ledgerly.Core/Domain). Everything about storage is here, in fluent configuration a DBA can review:

Configuration What it becomes in SQL Why
HasMaxLength(128), IsFixedLength() nvarchar(128), nchar(3) No accidental nvarchar(max) columns
HasPrecision(18, 6) decimal(18,6) Money and quantities never use floating point
HasConversion<string>() nvarchar(4) holding Buy or Sell Readable data, and reordering the enum can't corrupt rows
IsRowVersion() rowversion The concurrency token, next section
HasIndex(p => p.OwnerId) A nonclustered index Every query filters by owner

A filtered unique index for idempotent imports

The last index deserves a second look. An imported broker row carries an ExternalRef (the broker's own reference, like BRK-1001), and the domain comment says why it matters: "Unique per portfolio, so a re-imported file cannot double-count." Trades typed in by hand have no reference, so the index only applies where one exists. EF Core turns HasFilter into exactly that in the migration:

📄 sample/src/Ledgerly.Core/Data/Migrations/20261028152017_Initial.cs

            migrationBuilder.CreateIndex(
                name: "IX_Transactions_PortfolioId_ExternalRef",
                table: "Transactions",
                columns: new[] { "PortfolioId", "ExternalRef" },
                unique: true,
                filter: "[ExternalRef] IS NOT NULL");

If a user uploads the same CSV twice, the database rejects the duplicates, however the code above it behaves. Chapter 7 shows how the worker uses this for idempotent imports.

💡 Put invariants in the database when you can. Code checks can race, but a unique index can't.


🧠 Optimistic concurrency: how one "sell all" loses

The rowversion column

SQL Server's rowversion is a value the database changes on every update of the row. EF Core includes the value it originally read in the WHERE clause of each UPDATE. If another write got there first, the row no longer matches, zero rows are affected, and EF Core throws DbUpdateConcurrencyException (docs).

📄 sample/src/Ledgerly.Core/Domain/Portfolio.cs

    // Touched on every change (including new trades) so the rowversion check below protects the whole aggregate.
    public DateTimeOffset UpdatedAt { get; set; } = DateTimeOffset.UtcNow;

    // SQL Server rowversion: EF Core uses it for optimistic concurrency.
    public byte[] RowVersion { get; set; } = [];

Here's the subtle part. A new trade is an INSERT into Transactions. Nothing about the Portfolios row changes, so on its own a rowversion on Portfolios would protect nothing. That's why recording a trade also touches UpdatedAt on the portfolio. Now every trade is also an UPDATE of the portfolio row, guarded by its rowversion, in the same transaction.

💻 In Ledgerly: recording a trade

📄 sample/src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs

        // EF trap: the Id is already set (Guid v7, generated in C#). Adding via portfolio.Transactions would make EF
        // assume the row EXISTS and send an UPDATE (0 rows -> DbUpdateConcurrencyException). Add it explicitly.
        db.Transactions.Add(transaction);
        portfolio.UpdatedAt = DateTimeOffset.UtcNow; // bumps rowversion: two parallel "sell all" requests cannot both win

        try
        {
            await db.SaveChangesAsync(ct);
        }
        catch (DbUpdateConcurrencyException)
        {
            return TypedResults.Problem("Portfolio changed while recording the trade. Retry the request.", statusCode: StatusCodes.Status409Conflict);
        }

Here is the race from Elena's ticket, with the fix in place:

sequenceDiagram
    participant Phone as "Phone"
    participant Laptop as "Laptop"
    participant Api as "Ledgerly.Api"
    participant Sql as "Azure SQL"
    Phone->>Api: POST sell 40 VWCE
    Laptop->>Api: POST sell 40 VWCE
    Api->>Sql: Read portfolio, RowVersion v1, holds 40
    Api->>Sql: Read portfolio, RowVersion v1, holds 40
    Note over Api: Both requests pass the holdings check
    Api->>Sql: Insert trade and update portfolio where RowVersion is v1
    Sql-->>Api: 1 row, commit, RowVersion is now v2
    Api-->>Phone: 201 Created
    Api->>Sql: Insert trade and update portfolio where RowVersion is v1
    Sql-->>Api: 0 rows, EF rolls back the transaction
    Api-->>Laptop: 409 Conflict, retry

When the laptop retries, it reads the new state (0 shares), and the holdings check rejects the sell with a 400. The user sees "you don't hold these shares", which is the truth.

Two flavors of the same mechanism

Ledgerly uses the rowversion in two ways, and it helps to name them:

Scenario Where the expected version comes from Endpoint
Stale client: the user edited an old copy The client sends back the Version it read (Base64 of the rowversion) PUT /api/portfolios/{id} (rename) sets OriginalValue from the request
Racing requests: two writes in flight at once The version EF read at the start of this request POST .../transactions relies on the tracked value

Both end in DbUpdateConcurrencyException and a 409 ProblemDetails, the contract from chapter 4. If you ever need to show the user what changed, entry.GetDatabaseValues() gives you the current row.

💡 Optimistic concurrency doesn't prevent the race. It makes sure exactly one side wins and the other side knows it lost.


🧠 The EF trap you will hit: client keys added through a navigation

This one is a real story from building Ledgerly, and it will probably happen to you.

Every entity gets its ID in C#: public Guid Id { get; init; } = Guid.CreateVersion7();. The first version of RecordTransaction added the new trade the natural object-oriented way:

// illustrative, not in the sample: the version that failed
portfolio.Transactions.Add(transaction);
portfolio.UpdatedAt = DateTimeOffset.UtcNow;
await db.SaveChangesAsync(ct);

Unit tests were green. Then the integration test against a real SQL Server failed: the buy returned 409, on a portfolio nobody else was touching.

What happened: when EF Core finds a new object through a navigation of a tracked entity, it has to guess whether that object is new or already in the database. For a key that EF expects to generate, a key that's already set means "this row exists." So EF tracked the transaction as an existing row and sent an UPDATE Transactions ... WHERE Id = @id. No such row existed, the update affected 0 rows, and EF reported that the same way it reports a concurrency conflict: DbUpdateConcurrencyException. The endpoint caught it and returned 409, so the bug looked like a concurrency problem when it was really a tracking problem.

flowchart TD
    New["new Transaction<br/>Id = Guid.CreateVersion7"] --> How{"How is it added?"}
    How -- "portfolio.Transactions.Add" --> Guess["EF discovers it via navigation<br/>key already set"]
    Guess --> Existing["Tracked as an existing row"]
    Existing --> Update["UPDATE Transactions<br/>WHERE Id = new id"]
    Update --> Zero["0 rows affected"]
    Zero --> Ex["DbUpdateConcurrencyException<br/>API returns 409"]
    How -- "db.Transactions.Add" --> Added["Tracked as Added"]
    Added --> Insert["INSERT Transactions"]
    Insert --> Ok["201 Created"]

The fix is one line: add the entity through its DbSet, which tells EF explicitly that it's new. The comment in PortfolioEndpoints.cs (in the excerpt above) records why, so nobody "simplifies" it back.

Option Effect Ledgerly's choice
db.Transactions.Add(transaction) Explicitly Added, so EF sends an INSERT ✅ Used, with a comment
Configure the key as never generated by EF EF stops treating a set key as "exists" (check the docs for how this changes graph handling) Not used
Let the database generate keys No client-side IDs, so you lose IDs before insert Not used: the outbox and Location headers need them

If you want to try the second option, it looks like this:

// illustrative, not in the sample
modelBuilder.Entity<Transaction>().Property(t => t.Id).ValueGeneratedNever();

The bigger lesson is about tests. Only a test that talked to a real SQL Server could catch this, because the in-memory unit tests never sent SQL. Chapter 17 builds those tests with Testcontainers.

💡 With client-generated keys, add new entities through their DbSet. And when a 409 shows up with no second user in sight, suspect tracking before concurrency.


🧠 Migrations as a deploy step

Adding a migration

Migrations live in Ledgerly.Core/Data/Migrations, next to the model. The dotnet ef tool comes from the repo's tool manifest (chapter 3), so everyone uses the same version:

cd sample
dotnet tool restore
dotnet ef migrations add <Name> --project src/Ledgerly.Core --output-dir Data/Migrations

Adding a migration doesn't need a database. The tool builds the DbContext through a design-time factory with a placeholder connection string:

📄 sample/src/Ledgerly.Core/Data/DesignTimeDbContextFactory.cs

/// <summary>
/// Used only by the `dotnet ef` tool at design time (adding migrations, building bundles).
/// The connection string is a placeholder: no database is contacted when adding a migration.
/// </summary>
public sealed class DesignTimeDbContextFactory : IDesignTimeDbContextFactory<LedgerlyDbContext>

Review the generated migration like any other code. The Initial migration shows the rowversion column exactly as configured: RowVersion = table.Column<byte[]>(type: "rowversion", rowVersion: true, nullable: false).

Why not Database.Migrate() at startup?

It's tempting: the app starts, migrates and serves. With one instance on one server, it even works. In the cloud, the API might start as ten replicas at once, and the docs advise against migrating at startup in multi-instance production (docs). Even though EF9 and later take a migration lock, you still have problems: every replica needs schema-change rights, a slow migration delays every replica's readiness, and a failed migration leaves you with half-started apps.

Ledgerly runs migrations in their own process, Ledgerly.MigrationService, which migrates and then exits. The apps don't start until it has finished successfully.

📄 sample/src/Ledgerly.MigrationService/Program.cs

            // The execution strategy retries transient failures (e.g. Azure SQL serverless waking up).
            var strategy = db.Database.CreateExecutionStrategy();
            await strategy.ExecuteAsync(async () => await db.Database.MigrateAsync(stoppingToken));
            LogMigrated(logger);
            // ...
        catch (Exception ex)
        {
            activity?.AddException(ex);
            activity?.SetStatus(ActivityStatusCode.Error);
            LogMigrationFailed(logger, ex);
            Environment.ExitCode = 1; // a failed job execution blocks the deployment pipeline
        }

        lifetime.StopApplication();

Three details matter:

  1. The execution strategy. A retrying strategy can't safely replay work it didn't start, so work you run yourself must go through CreateExecutionStrategy().ExecuteAsync(...) (docs). Here it covers the migration and the database-user step, so a paused Azure SQL database that takes a moment to resume isn't a failed deployment.
  2. The exit code. On failure, the process exits with code 1. Locally, Aspire shows the resource as failed. In Azure, the migration runs as a Container Apps job, and a failed job execution stops the pipeline before new app revisions roll out (chapter 12).
  3. StopApplication(): success or failure, the process ends. A migration runner that keeps running is a bug.

💻 In Ledgerly: the apps wait for completion

📄 sample/src/Ledgerly.AppHost/AppHost.cs

var migrations = builder.AddProject<Projects.Ledgerly_MigrationService>("migrations")
    .WithReference(db)
    .WaitFor(db);

builder.AddProject<Projects.Ledgerly_Api>("api")
    .WithReference(db)
    .WithReference(cache)
    .WithReference(marketData)
    .WaitFor(cache)
    .WaitForCompletion(migrations)
    .WithHttpHealthCheck("/health")
    .WithExternalHttpEndpoints();

WaitFor(db) means "start when the database is healthy." WaitForCompletion(migrations) means "start only after the migrator has exited successfully." The worker has the same line. Locally and in Azure, the order is the same:

flowchart LR
    Sql["SQL database<br/>container or Azure SQL"] -- "healthy" --> Mig["Migration service<br/>run to completion"]
    Mig -- "exit code 0" --> Api["Ledgerly.Api"]
    Mig -- "exit code 0" --> Worker["Ledgerly.Worker"]
    Mig -- "exit code 1" --> Stop["Apps not started<br/>deployment stops"]

Other ways to ship migrations

Approach How When it fits
Migration service (Ledgerly) A small app that calls MigrateAsync, run as a job You want the same path locally and in Azure, plus extra steps like creating the DB user
Migration bundle dotnet ef migrations bundle --self-contained -r linux-x64 -o efbundle, then run ./efbundle --connection "<cs>" A single executable in CI, with no SDK or source needed
Idempotent SQL script dotnet ef migrations script --idempotent -o migrate.sql A DBA must review and run the SQL

Whatever you choose, add dotnet ef migrations has-pending-model-changes to CI. It fails when someone changed the model and forgot to add a migration.

💡 A schema change is a deployment step with its own success or failure, not a side effect of an app starting up.


🧠 Azure SQL: serverless, free to start, and no passwords

Serverless and auto-pause

For the beta, Ledgerly uses an Azure SQL serverless database in the General Purpose tier (docs). Compute scales between a minimum (as low as 0.5 vCore) and a maximum, and you're billed per vCore-second of use 💰. When nobody connects for the auto-pause delay (default 60 minutes, minimum 15, -1 disables it), the database pauses and compute billing stops. The first connection after a pause triggers a resume with a short delay, which is why every data access in Ledgerly goes through a retrying execution strategy.

On top of that, the Azure SQL free offer 🔧 gives each database, per month, 100,000 vCore-seconds, 32 GB of data and 32 GB of backup storage, for up to 10 serverless databases per subscription (docs). When the monthly allowance is used up, the database either auto-pauses until next month or continues and bills the overage, depending on a setting 💰.

📄 sample/infra/modules/sql.bicep

    administrators: {
      administratorType: 'ActiveDirectory'
      principalType: 'Application'
      login: adminIdentityName
      sid: adminIdentityClientId // 🔧 for a managed identity, the Learn Bicep sample uses the client id
      tenantId: subscription().tenantId
      azureADOnlyAuthentication: true
    }
// ...
  sku: {
    name: 'GP_S_Gen5_2'
    tier: 'GeneralPurpose'
    family: 'Gen5'
    capacity: 2
  }
  properties: {
    autoPauseDelay: 60
    minCapacity: json('0.5')
    useFreeLimit: useFreeLimit // 💰 100,000 vCore-seconds per month, then pause until next month
    freeLimitExhaustionBehavior: useFreeLimit ? 'AutoPause' : null
  }
Setting Value Effect
GP_S_Gen5_2 General Purpose, serverless, up to 2 vCores Small, and scales down when idle
minCapacity 0.5 The lowest compute while running
autoPauseDelay 60 minutes Pauses after an idle hour
useFreeLimit and AutoPause on exhaustion On by default A runaway month pauses instead of producing a surprise bill 💰

For a production system with steady traffic, auto-pause's resume delay may be unacceptable. Chapter 10 compares the options.

Entra-only authentication

azureADOnlyAuthentication: true disables SQL authentication on the server. There are no SQL logins and no passwords: only Microsoft Entra identities can connect (docs). That's Jonas's favorite line in the whole infrastructure.

The apps connect with their managed identity. This is the connection string the Bicep sets on the Container Apps:

📄 sample/infra/modules/apps.bicep

var sqlConnection = 'Server=tcp:${sqlServerFqdn},1433;Database=${databaseName};Authentication=Active Directory Managed Identity;User Id=${identityClientId};Encrypt=True;Connect Timeout=60'
Part Meaning
Authentication=Active Directory Managed Identity Microsoft.Data.SqlClient gets a token for the app's managed identity, with no secret (docs)
User Id=<client id> Which user-assigned identity to use
Connect Timeout=60 Gives a paused database time to resume on the first connection
No Password There's nothing to leak

You may see Authentication=Active Directory Default in tutorials. It tries a chain of credentials, which is convenient on a laptop, but the docs note it can be slow and isn't recommended for production. Use the explicit managed identity form in Azure.

Who creates the database user?

An identity that can sign in still needs a user in the database with the right roles. Ledgerly uses two identities: the migration job runs as the SQL admin identity, and the apps run as a separate identity with read and write rights only. After migrating, the migration service creates the app's database user idempotently (EnsureAppUserAsync, with CREATE USER ... WITH SID = ..., TYPE = E) when the Database:AppUser settings are present. Locally those settings aren't set, so the step is skipped. Chapter 11 explains why it uses a SID instead of FROM EXTERNAL PROVIDER, and chapter 13 covers the identities themselves.

💡 The cheapest database is the one that's paused, and the safest password is the one that doesn't exist. Azure SQL serverless with Entra-only auth gives you both.


⚠️ Common traps

Trap What happens Do this instead
Adding a new entity with a client-generated key through a navigation EF assumes the row exists, sends an UPDATE, affects 0 rows and throws DbUpdateConcurrencyException Add it via its DbSet (db.Transactions.Add), and keep the comment that explains why
A rowversion on the parent, but child inserts don't touch the parent Two requests both insert children, and nothing conflicts Touch the parent (UpdatedAt) in the same SaveChanges, so the parent's rowversion guards the aggregate
Database.Migrate() in Program.cs of every app Replicas race, every app needs DDL rights, and a failed migration half-starts the system A run-to-completion migration process, with the apps waiting for its success
Your own transaction or multi-step work with a retrying strategy An exception explaining that the strategy doesn't support user-initiated transactions, or a half-retried operation Wrap the whole unit in db.Database.CreateExecutionStrategy().ExecuteAsync(...)
No retries against serverless Azure SQL The first request after an idle hour fails while the database resumes Use the Aspire integration (retries included) and a generous connect timeout
Authentication=Active Directory Default in production Slow connections while it walks the credential chain Active Directory Managed Identity with User Id=<client id>
Testing data access only with in-memory fakes Tracking and SQL-translation bugs like the one in this chapter reach production Integration tests against a real SQL Server (chapter 17)

🧰 Helpers

EF Core commands (from sample/, after dotnet tool restore):

Task Command
Add a migration dotnet ef migrations add <Name> --project src/Ledgerly.Core --output-dir Data/Migrations
List migrations dotnet ef migrations list --project src/Ledgerly.Core (connects to the database to show which are applied)
Check for a forgotten migration dotnet ef migrations has-pending-model-changes --project src/Ledgerly.Core
SQL script for review dotnet ef migrations script --idempotent --project src/Ledgerly.Core -o migrate.sql
Self-contained bundle dotnet ef migrations bundle --self-contained -r linux-x64 -o efbundle --project src/Ledgerly.Core

A concurrency-safe write, the pattern from this chapter in one place:

// illustrative, not in the sample
var order = await db.Orders.SingleOrDefaultAsync(o => o.Id == id && o.OwnerId == ownerId, ct);
if (order is null)
{
    return TypedResults.NotFound();
}

db.Entry(order).Property(o => o.RowVersion).OriginalValue = Convert.FromBase64String(request.Version);
db.OrderLines.Add(new OrderLine { OrderId = order.Id, Sku = request.Sku }); // via the DbSet, not order.Lines
order.UpdatedAt = clock.GetUtcNow();                                         // touch the parent

try
{
    await db.SaveChangesAsync(ct);
}
catch (DbUpdateConcurrencyException)
{
    return TypedResults.Problem("Changed by someone else. Reload and try again.", statusCode: StatusCodes.Status409Conflict);
}

Data checklist for a new service:

Make it permanent: add dotnet ef migrations has-pending-model-changes as a CI step, and keep at least one integration test that records a trade against real SQL (chapter 17), because that's the test that caught the trap. The toolkit's production-readiness-checklist.md repeats the key ones (migrations as a separate step, managed identities for Azure SQL, 409 on conflicts) (see the toolkit). Terms like rowversion, execution strategy and auto-pause are in the glossary.


🧪 Try it yourself (optional, ~25 min)

None of these steps need a running database, except step 5.

  1. Restore the EF tool and look at the migrations: bash cd sample dotnet tool restore dotnet ef --version Open src/Ledgerly.Core/Data/Migrations/20261028152017_Initial.cs and find the rowversion column and the filtered index IX_Transactions_PortfolioId_ExternalRef.
  2. Generate the SQL a DBA would review: bash dotnet ef migrations script --idempotent --project src/Ledgerly.Core -o migrate.sql Open migrate.sql. Find CREATE UNIQUE INDEX ... WHERE [ExternalRef] IS NOT NULL and the IF NOT EXISTS checks that make the script safe to run twice. Delete the file when you're done.
  3. Catch a forgotten migration. Add a property to Portfolio in src/Ledgerly.Core/Domain/Portfolio.cs, for example public string? Notes { get; set; }, then run: bash dotnet ef migrations has-pending-model-changes --project src/Ledgerly.Core It reports pending changes (a CI step would fail here). Remove the property and run it again: no changes.
  4. Read the trap: open src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs, find the "EF trap" comment in RecordTransaction, and follow the sequence diagram above against the code.
  5. Optional, needs Docker: reproduce the trap. Change db.Transactions.Add(transaction); to portfolio.Transactions.Add(transaction); and run the tests: bash dotnet test --solution Ledgerly.slnx The integration tests that record a trade fail, because the setup buy returns 409 instead of 201. Revert the line and run the tests again: all green.

✅ Recap

  1. EF Core 10, registered through Aspire's AddSqlServerDbContext, gives you a configured model, retries, health checks and tracing, and invariants like duplicate imports belong in (filtered) unique indexes.
  2. A rowversion plus a touched parent turns racing writes into exactly one success and one 409, and client-generated keys must be added via their DbSet to avoid a fake concurrency error.
  3. Migrations run as a separate process that the apps wait for, against an Azure SQL serverless database that auto-pauses, uses the free offer, and only accepts Entra identities.

Next → Chapter 6: Local development with Aspire

Chapter 6: Local development with Aspire

⏱ About 30 min · You'll learn: how one C# file (the AppHost) starts SQL Server, Redis, the Service Bus emulator and every Ledgerly service, how connection strings and service addresses reach your code as environment variables, and what the Aspire dashboard and service defaults give you from the first run.


📖 Story: The 14-step wiki page

Your import feature needs everything at once: SQL Server for the data, Redis for the cache, a Service Bus queue for the work, the fake market-data vendor for prices, the API, the worker, and the migrations in front of all of them. At Nordbank Insurance this was the moment you opened the wiki page called "Developer setup". It had 14 steps. Step 4 was "Ask the DBA for a login on DEVSQL02", the shared dev server. Step 9 was "Enable MSMQ in Windows Features and create the private queues by hand". Step 12 said "If the service doesn't start, ask Lars". New joiners lost two days to it, and because everyone shared one SQL Server, one person's half-finished migration broke everybody's afternoon.

Tomás Varga sees you open a blank text file to start your own list. "Don't," he says. "Clone the repo, run aspire run, and the whole system is up. If it takes more than that, it's a bug, and we fix it in the repo, not on a wiki." Maya Okafor pulls up a chair. "And once it's up, you'll see every log, trace and metric from every service in one place. Then we can talk about what happens when the market-data vendor falls over."

In your notebook, under Habits I brought with me, you write: "The dev environment is a document." Then you add a question mark.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
A wiki page with 14 manual setup steps An AppHost project in the repo, started with aspire run Code is reviewed and versioned, so it can't drift from reality the way a wiki does
A shared dev SQL Server that everyone migrated against A SQL Server container per developer, with a data volume that survives restarts Your experiments don't break your teammates, and a reset is one command
MSMQ enabled by hand in Windows Features The Azure Service Bus emulator as a container, with the queue declared in C# Same SDK and same queue semantics as production, no machine setup
Connection strings copied into web.config on every machine Aspire injects connection strings and service addresses as environment variables Nothing to copy or commit, and the same code reads Azure settings later
Starting four Visual Studio instances in the right order WaitFor and WaitForCompletion express the start order in code The migrations finish before the API starts, every time
Reading NLog files in three folders The Aspire dashboard: console logs, structured logs, traces and metrics for every service One request is one trace across the API, the database and the vendor
Copy-pasted logging and retry setup in each project One AddServiceDefaults() call from a shared project OpenTelemetry, health checks, resilience and service discovery are the same everywhere

🧠 What Aspire is, and what it is not

Aspire is a set of tools for describing, running and observing a distributed application. It started as a .NET-only stack and, since version 13, also hosts JavaScript, Python, Java and Rust apps. You will use three parts every day:

Part What it is In Ledgerly
AppHost A small .NET project that describes your resources (containers, projects, cloud services) and how they connect, in C# src/Ledgerly.AppHost
Service defaults A shared project with one extension method that adds OpenTelemetry, health checks, resilience and service discovery src/Ledgerly.ServiceDefaults
Aspire dashboard A web UI with resources, console logs, structured logs, traces and metrics Opens when you run aspire run

A few facts changed recently, so check them before you rely on them (all 🔧):

# Windows (PowerShell)
irm https://aspire.dev/install.ps1 | iex
# macOS or Linux
curl -sSL https://aspire.dev/install.sh | bash

# Alternatives: winget install Microsoft.Aspire   or   dotnet tool install -g Aspire.Cli

Source: Install the Aspire CLI.

You also need a container runtime: Docker Desktop or Podman. Aspire runs SQL Server, Redis and the Service Bus emulator as containers. Without a runtime, your projects can start, but their dependencies can't.

Local orchestration first, deployment optional

Aspire can do more than run things on your laptop. aspire publish emits deployment artifacts such as Bicep, and aspire deploy is now the recommended way to deploy an Aspire app to Azure 🔧 (Deploy with Aspire). The Azure Developer CLI (azd) still works, but it is no longer the recommended default for Aspire.

This course uses Aspire for local development only and deploys with hand-written Bicep (chapter 11). That is a teaching choice, not a verdict on aspire deploy:

Use aspire deploy when Use hand-written Bicep (this course) when
You want the fastest path from AppHost to Azure Container Apps You want to see and review every Azure resource, role assignment and setting
The team is small and the generated defaults fit A platform team owns infrastructure separately from application code
You are prototyping or running a demo environment A security reviewer needs every line explained, or you need settings the generator doesn't cover

💡 The AppHost describes your system for development. It is not your production configuration, even though it can generate one.


🧠 The AppHost, line by line

The AppHost is an ordinary .NET project with a special SDK. The version lives in the Sdk attribute, and the four service projects are plain project references:

📄 sample/src/Ledgerly.AppHost/Ledgerly.AppHost.csproj

<Project Sdk="Aspire.AppHost.Sdk/13.6.4">
  <!-- ... -->
  <ItemGroup>
    <ProjectReference Include="..\Ledgerly.Api\Ledgerly.Api.csproj" />
    <ProjectReference Include="..\Ledgerly.Worker\Ledgerly.Worker.csproj" />
    <ProjectReference Include="..\Ledgerly.MarketData\Ledgerly.MarketData.csproj" />
    <ProjectReference Include="..\Ledgerly.MigrationService\Ledgerly.MigrationService.csproj" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Aspire.Hosting.SqlServer" />
    <PackageReference Include="Aspire.Hosting.Redis" />
    <PackageReference Include="Aspire.Hosting.Azure.ServiceBus" />
  </ItemGroup>
</Project>

The Aspire.Hosting.* packages are hosting integrations: they teach the AppHost how to run a resource. The services use separate client integrations (for example Aspire.Microsoft.EntityFrameworkCore.SqlServer) to consume it. The versions come from Directory.Packages.props (chapter 3).

💻 In Ledgerly: the infrastructure

📄 sample/src/Ledgerly.AppHost/AppHost.cs

// The whole Ledgerly system, described in C#. `aspire run` (or F5 on this project) starts everything.
var builder = DistributedApplication.CreateBuilder(args);

// ── Infrastructure (containers locally; real Azure services in the cloud, see infra/) ──────────
var sql = builder.AddSqlServer("sql")
    .WithLifetime(ContainerLifetime.Persistent)   // keep the container (and data) between runs
    .WithDataVolume();
var db = sql.AddDatabase("ledgerlydb");

var cache = builder.AddRedis("cache");

var messaging = builder.AddAzureServiceBus("messaging").RunAsEmulator();
messaging.AddServiceBusQueue("imports");
Line What it does Why it matters
AddSqlServer("sql") Runs a SQL Server container named sql No shared dev server, no install, and no password written in the repo
.WithLifetime(ContainerLifetime.Persistent) Keeps the container running after you stop the AppHost, and reuses it next time SQL Server is slow to start. The second aspire run is much faster
.WithDataVolume() Stores the database files in a named container volume Your data survives even if the container is recreated. Delete the volume to reset
sql.AddDatabase("ledgerlydb") Declares a database on that server ledgerlydb is the name the services ask for, here and in Azure
AddRedis("cache") Runs a Redis container The API uses it as the shared second level of HybridCache (chapter 18)
AddAzureServiceBus("messaging").RunAsEmulator() Declares a Service Bus namespace and, locally, runs the official emulator container instead Same SDK and same peek-lock semantics as production (chapter 7)
messaging.AddServiceBusQueue("imports") Declares the imports queue The queue is code. Nobody creates it by hand

The Service Bus emulator is honest about what it is. It is a Linux container for local development and testing only, it brings its own SQL backing store, and it has limits 🔧: 50 entities, 100 MB per entity, no Entra ID authentication (it uses a connection string with UseDevelopmentEmulator=true) and no persistence, so messages are lost when it restarts (Service Bus emulator overview).

Notice the asymmetry: SQL Server is persistent, the queue is not. If you stop the AppHost while messages are still waiting in imports, they are gone and those imports stay Pending. That is fine for development, and a good reminder that the emulator proves your code paths, not your durability.

💻 In Ledgerly: the services

// ── Services ────────────────────────────────────────────────────────────────────────────────────
var marketData = builder.AddProject<Projects.Ledgerly_MarketData>("marketdata")
    .WithHttpHealthCheck("/health");

var migrations = builder.AddProject<Projects.Ledgerly_MigrationService>("migrations")
    .WithReference(db)
    .WaitFor(db);

builder.AddProject<Projects.Ledgerly_Api>("api")
    .WithReference(db)
    .WithReference(cache)
    .WithReference(marketData)
    .WaitFor(cache)
    .WaitForCompletion(migrations)
    .WithHttpHealthCheck("/health")
    .WithExternalHttpEndpoints();

builder.AddProject<Projects.Ledgerly_Worker>("worker")
    .WithReference(db)
    .WithReference(messaging)
    .WithReference(marketData)
    .WaitFor(messaging)
    .WaitForCompletion(migrations)
    .WithHttpHealthCheck("/health");

builder.Build().Run();
Call What it means
AddProject<Projects.Ledgerly_Api>("api") Projects.Ledgerly_Api is generated from the project reference, so a renamed or missing project is a compile error, not a runtime surprise
.WithReference(db) Give this project the connection information for ledgerlydb
.WithReference(marketData) Give this project the address of marketdata, for service discovery
.WaitFor(cache) Don't start until the dependency is running and reports healthy
.WaitForCompletion(migrations) Don't start until the migrations process has exited successfully
.WithHttpHealthCheck("/health") The AppHost polls this endpoint. The dashboard shows the result, and anything that waits for this resource waits for "healthy"
.WithExternalHttpEndpoints() Marks the API's endpoints as public. It matters when Aspire generates deployment artifacts. Locally everything is on localhost anyway

Two details are worth a second look.

First, the reference graph is your architecture. The API references the cache but not Service Bus. The worker references Service Bus but not the cache. That is not an accident: the API never talks to the queue directly; it writes an outbox row in SQL, and the worker publishes it (chapter 7). Anyone reviewing a pull request that adds .WithReference(messaging) to the API should ask why.

Second, WaitForCompletion turns the migration service into a gate. The migrator is a run-to-completion process. If it fails, it sets a non-zero exit code, and the API and worker never start:

📄 sample/src/Ledgerly.MigrationService/Program.cs

        catch (Exception ex)
        {
            activity?.AddException(ex);
            activity?.SetStatus(ActivityStatusCode.Error);
            LogMigrationFailed(logger, ex);
            Environment.ExitCode = 1; // a failed job execution blocks the deployment pipeline
        }

        lifetime.StopApplication();

This is the same contract the Container Apps job follows in Azure (chapter 12): new code never runs against an old schema. You practice the production deployment order every time you press F5.

💡 WaitFor means "is it up?". WaitForCompletion means "did it finish, and did it succeed?". Use the second one for anything that is supposed to exit.


🗺️ The Ledgerly resource graph

This is AppHost.cs drawn as a picture. Solid arrows are references (configuration flows along them). Dotted arrows are start-order gates.

flowchart LR
    subgraph Containers["Containers, started by Aspire"]
        Sql["sql<br/>SQL Server, persistent"]
        Db["ledgerlydb<br/>database"]
        Cache["cache<br/>Redis"]
        Bus["messaging<br/>Service Bus emulator"]
        Queue["imports<br/>queue"]
    end
    subgraph Projects["Your .NET projects"]
        Market["marketdata"]
        Mig["migrations<br/>runs once, exits"]
        Api["api<br/>external endpoints"]
        Worker["worker"]
    end
    Sql --- Db
    Bus --- Queue
    Mig -- "reference" --> Db
    Api -- "reference" --> Db
    Api -- "reference" --> Cache
    Api -- "reference" --> Market
    Worker -- "reference" --> Db
    Worker -- "reference" --> Bus
    Worker -- "reference" --> Market
    Mig -. "must complete first" .-> Api
    Mig -. "must complete first" .-> Worker

🧠 How configuration flows: environment variables, not files

WithReference doesn't do anything magical. When the AppHost starts a project, it sets environment variables on that process. Your code reads them through the normal .NET configuration system, where a double underscore __ in a variable name means : in a configuration key. So ConnectionStrings__ledgerlydb is the same key as "ConnectionStrings": { "ledgerlydb": ... } in appsettings.json.

AppHost line What the service receives Code that reads it
.WithReference(db) ConnectionStrings__ledgerlydb builder.AddSqlServerDbContext<LedgerlyDbContext>("ledgerlydb")
.WithReference(cache) ConnectionStrings__cache GetConnectionString("cache"), then AddRedisDistributedCache("cache")
.WithReference(messaging) ConnectionStrings__messaging, an emulator connection string with UseDevelopmentEmulator=true builder.AddAzureServiceBusClient("messaging")
.WithReference(marketData) Endpoint variables named like services__marketdata__https__0 Service discovery resolves https+http://marketdata
Every project OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME and related OTLP settings ConfigureOpenTelemetry() in service defaults

You don't need to memorize the exact names. Click a resource in the dashboard and its environment variables are listed there, which is the fastest way to answer "what does this service actually see?".

The address https+http://marketdata reads as: "find the service called marketdata, prefer its HTTPS endpoint, fall back to HTTP". The handler that resolves it is added once, in service defaults, for every HttpClient in the process.

💻 In Ledgerly: the same code reads Azure settings

The services never mention containers, ports or passwords. They ask for names:

📄 sample/src/Ledgerly.Api/Program.cs

// ── Data: Aspire client integration = DbContext pooling + retries + health check + tracing ──────
builder.AddSqlServerDbContext<LedgerlyDbContext>("ledgerlydb");
// ...
// ── Outbound HTTP: typed client. Resilience + discovery come from ConfigureHttpClientDefaults. ───
builder.Services.AddHttpClient<IMarketDataClient, MarketDataClient>(client =>
    client.BaseAddress = new Uri(builder.Configuration["MarketData:BaseUrl"] ?? "https+http://marketdata"));

In Azure there is no AppHost. Bicep sets the same variable names on the Container App instead:

📄 sample/infra/modules/apps.bicep

          env: concat(commonEnv, [
            { name: 'OTEL_SERVICE_NAME', value: 'worker' }
            { name: 'ConnectionStrings__ledgerlydb', value: sqlConnection }
            { name: 'ConnectionStrings__messaging', value: serviceBusFqns } // a namespace, not a key: Entra auth
            { name: 'MarketData__BaseUrl', value: marketDataUrl }
          ])
Setting Locally (AppHost) In Azure (Bicep)
ledgerlydb Connection string to the local container, supplied by the AppHost Azure SQL with Authentication=Active Directory Managed Identity, no password (chapter 13)
messaging Emulator connection string The namespace host name only. The sample's comment says it: "a namespace, not a key: Entra auth"
Market data address Service discovery from Aspire's endpoint variables MarketData__BaseUrl, the internal HTTPS address of the marketdata Container App
Telemetry OTLP to the Aspire dashboard APPLICATIONINSIGHTS_CONNECTION_STRING to Application Insights (chapter 16)

💡 Code asks for names. Environments supply values. That is how the same container image runs on your laptop and in production.


🗺️ What happens when you type aspire run

sequenceDiagram
    participant You
    participant CLI as Aspire CLI and AppHost
    participant Runtime as Container runtime
    participant Mig as migrations
    participant Svc as api and worker
    participant Dash as Aspire dashboard
    You->>CLI: aspire run
    CLI->>Dash: start dashboard and OTLP receiver
    CLI->>Runtime: start or reuse sql, cache, messaging
    Runtime-->>CLI: containers healthy
    CLI->>Mig: start with ConnectionStrings__ledgerlydb
    Mig->>Runtime: apply EF Core migrations
    Mig-->>CLI: exit code 0
    CLI->>Svc: start with connection strings and service addresses
    Svc->>Dash: logs, traces and metrics over OTLP
    CLI-->>You: dashboard URL with login token

🧠 The dashboard: your new log folder

At Nordbank, "looking at the logs" meant RDP, a file share and Notepad++. The Aspire dashboard is the local version of what Application Insights will give you in Azure, and it receives the same OpenTelemetry data.

View What you see Old-world equivalent Use it when
Resources Every container and project, its state, health, endpoints and environment variables Services console plus IIS Manager Something didn't start, or you want a service's URL
Console logs Raw stdout and stderr per resource Tailing a log file A process crashes on startup before telemetry is wired
Structured logs OpenTelemetry log records with their properties, filterable by resource, level and value grep over three servers "Show me every warning for import X"
Traces One request as a tree of spans across services, with timings and errors Nothing; you correlated IDs by hand "Why was this valuation slow?"
Metrics Counters and histograms, including ASP.NET Core's built-in http.server.request.duration and Ledgerly's own Ledgerly meter Performance Monitor "Is the import duration growing?"

Two things to know 🔧:

The data on the dashboard is only as good as what your services emit. That is the job of service defaults.


🧠 Service defaults: one line of cross-cutting code

Every Ledgerly service starts with builder.AddServiceDefaults(). The method lives in a shared project that started from the Aspire template and was then adapted.

💻 In Ledgerly: what one line adds

📄 sample/src/Ledgerly.ServiceDefaults/Extensions.cs

    public static TBuilder AddServiceDefaults<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
    {
        builder.ConfigureOpenTelemetry();
        builder.AddDefaultHealthChecks();

        builder.Services.AddServiceDiscovery();

        builder.Services.ConfigureHttpClientDefaults(http =>
        {
            // Retries with jittered back-off, per-attempt + total timeouts, circuit breaker, rate limiter.
            http.AddStandardResilienceHandler();

            // Resolves "https+http://marketdata" to a real address (Aspire locally, config or DNS in Azure).
            http.AddServiceDiscovery();
        });

        return builder;
    }
Piece What it gives you Where you go deeper
ConfigureOpenTelemetry() Logs, metrics and traces for ASP.NET Core, HttpClient, the runtime and Ledgerly's own ActivitySource and Meter. Export goes to OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set (locally) and to Azure Monitor when APPLICATIONINSIGHTS_CONNECTION_STRING is set (in Azure) Chapter 15
AddDefaultHealthChecks() A self check tagged live. Aspire client integrations add their own checks (database, Redis, Service Bus) Chapter 14
AddServiceDiscovery() Logical names such as marketdata become real addresses This chapter
AddStandardResilienceHandler() For every HttpClient: rate limiter, total timeout (30 s), retries (3, exponential back-off with jitter), circuit breaker, attempt timeout (10 s) 🔧 Chapter 14

The tracing setup also filters out the health endpoints, because probes call them every few seconds and would drown the real traffic:

                .AddAspNetCoreInstrumentation(options =>
                    // Probes hit these every few seconds: keep them out of traces.
                    options.Filter = context =>
                        !context.Request.Path.StartsWithSegments(ReadinessPath)
                        && !context.Request.Path.StartsWithSegments(LivenessPath))

One deliberate difference from the template: the template maps /health and /alive only in Development. Ledgerly maps them in every environment, because Container Apps probes need them, and they return a status without details:

    public static WebApplication MapDefaultEndpoints(this WebApplication app)
    {
        // Readiness: all checks (including DB, Redis, Service Bus registered by Aspire integrations).
        app.MapHealthChecks(ReadinessPath).DisableHttpMetrics();

        // Liveness: only checks tagged "live".
        app.MapHealthChecks(LivenessPath, new HealthCheckOptions
        {
            Predicate = r => r.Tags.Contains("live"),
        }).DisableHttpMetrics();

        return app;
    }

💻 In Ledgerly: the chaos switches

Ledgerly.MarketData is the fake vendor. It starts from the Chaos section in its appsettings.json (failure rate 0, latency 0), and in Development it exposes an endpoint that changes those values at runtime:

📄 sample/src/Ledgerly.MarketData/Program.cs

// Chaos controls. Only exposed in Development: never ship a "break me" button to production.
if (app.Environment.IsDevelopment())
{
    app.MapGet("/chaos", (ChaosState chaos) => chaos.Current);
    app.MapPost("/chaos", (ChaosOptions options, ChaosState chaos) =>
    {
        chaos.Set(options);
        return Results.Ok(chaos.Current);
    });
}

ChaosState.Set clamps the values (failure rate between 0 and 1, latency up to 60,000 ms). With chaos on, the /prices endpoint adds latency and returns 503 for a share of requests. Because the API's HttpClient has the standard resilience handler, you can watch the retries happen in the Traces view. Priya will use this switch in chapter 14 to prove the API degrades gracefully instead of failing.

💡 Service defaults make "observable, resilient and discoverable" the default state of a new service instead of a task someone has to remember.


⚠️ Common traps

Trap What happens Do this instead
No container runtime running Containers never start, and the API and worker wait forever behind them Start Docker Desktop or Podman first, then check the Resources view for the stuck resource
Editing a migration that the persistent SQL container already applied The database keeps the old schema, and you get errors that don't match the code Add a new migration instead. To start clean, stop the AppHost and delete the SQL container and its data volume
Hard-coding https://localhost:7180 in another service It works on your machine, breaks for a teammate whose port differs, and breaks in Azure Use a reference plus a logical name (https+http://marketdata) or a configuration key such as MarketData:BaseUrl
Using WaitFor(migrations) instead of WaitForCompletion(migrations) The API starts while the migration is still running and hits missing tables Use WaitForCompletion for anything that is supposed to exit
Tagging a database check as live One database blip fails liveness, and the platform restarts every replica Keep only self in liveness. Dependencies belong in readiness (chapter 14)
Treating the emulator as production-like You assume messages survive restarts and that Entra ID auth works, and both assumptions fail Use the emulator for code paths. Prove durability and identity in Azure (chapters 12 and 13)
Putting real cloud secrets in the AppHost "just for local" They end up in git history Let the AppHost supply local connection strings, and keep cloud credentials out of the repo entirely (chapter 13)

🧰 Helpers

Install and run:

# once: install the Aspire CLI (see the PowerShell variant above)
curl -sSL https://aspire.dev/install.sh | bash

# every day
cd sample
aspire run                                   # finds the AppHost through aspire.config.json
dotnet run --project src/Ledgerly.AppHost    # same thing without the CLI

Checklist: adding a new service to the system

  1. Reference Ledgerly.ServiceDefaults, call builder.AddServiceDefaults() and app.MapDefaultEndpoints().
  2. Add the project to AppHost.cs with AddProject<Projects.X>("name").
  3. Add .WithReference(...) for each dependency and .WaitFor(...) or .WaitForCompletion(...) for each one that must be ready first.
  4. Add .WithHttpHealthCheck("/health").
  5. In the service, read connection strings and addresses by name only.
// illustrative, not in the sample: a hypothetical notifications service
builder.AddProject<Projects.Ledgerly_Notifications>("notifications")
    .WithReference(db)
    .WaitForCompletion(migrations)
    .WithHttpHealthCheck("/health");

Reset local data: stop the AppHost, then remove the SQL Server container and its volume with your container tool (docker ps -a, docker volume ls, then docker rm and docker volume rm, or the Podman equivalents). The next aspire run creates a fresh database and the migrator rebuilds the schema.

Make it permanent:


🧪 Try it yourself (optional, ~25 min)

  1. Start Docker Desktop or Podman, install the Aspire CLI, then run: bash cd sample aspire run Open the dashboard link the CLI prints.
  2. In Resources, watch the order: sql, cache and messaging start first, migrations runs and finishes, and only then do api and worker start. That is WaitForCompletion at work.
  3. Click api and look at its environment variables. Find ConnectionStrings__ledgerlydb and the services__marketdata__... entries.
  4. Create a development token and copy it: bash dotnet user-jwts create --project src/Ledgerly.Api
  5. Open the API's URL from the dashboard and add /scalar (or use src/Ledgerly.Api/Ledgerly.Api.http in your IDE). Create a portfolio, record a buy, then request /api/portfolios/{id}/valuation.
  6. In Traces, open the valuation request. Find the span for the call from api to marketdata.
  7. Turn on chaos. Take the marketdata URL from the dashboard and send: bash curl -X POST https://localhost:<marketdata-port>/chaos -H "Content-Type: application/json" -d '{"failureRate":0.7,"latencyMs":500}' Quotes are cached for 30 seconds, so wait a moment (or value a portfolio with a different symbol), then request the valuation again. Observe the retry spans in Traces, and isComplete: false when retries run out. Set both values back to 0 when you are done.
  8. Stop the AppHost with Ctrl+C and run aspire run again. The SQL container is reused and your portfolio is still there.

✅ Recap

  1. The AppHost describes the whole local system in C#: containers, projects, references and start order, so aspire run replaces the setup wiki.
  2. References become environment variables, and services read connection strings and addresses by name, which is why the same code runs against containers locally and Azure services in production.
  3. Service defaults and the dashboard make every service observable, healthy-checked and resilient from its first run.

Next → Chapter 7: Background work and messaging

Chapter 7: Background work and messaging

⏱ About 35 min · You'll learn: how to move slow work out of the HTTP request with hosted services and an Azure Service Bus queue, how the transactional outbox and an idempotent consumer make "at-least-once" safe, what peek-lock, dead-lettering and duplicate detection do for you, and how a worker shuts down without losing work.


📖 Story: The import that timed out

Elena Rossi forwards a support ticket on Monday morning. A beta user tried to import five years of broker statements, a file of about 15,000 rows (numbers in this story are illustrative). The first prototype parsed the CSV inside the HTTP request, the request ran past the 30-second timeout, and the browser showed an error. So the user clicked Upload again. And again. "What does the user see?" Elena asks. "Right now: an error, and then three copies of some of their trades."

Maya Okafor draws two boxes on the whiteboard, "API" and "worker", and a pipe between them. "The API accepts the file and answers in milliseconds. The worker does the heavy lifting. Now tell me what happens when it fails halfway." You start to answer, and she keeps going: "When the worker crashes after writing half the trades. When the queue delivers the same message twice. When the platform stops the worker in the middle of a file. Every one of those happens in the cloud, and every one has to be boring."

At Nordbank Insurance, background work meant a Windows Service with a System.Timers.Timer polling a "jobs" table, plus a SQL Agent job at 02:00 that nobody dared to touch. In your Habits I brought with me notebook, you write: "Background work runs on one special server that never restarts." Then you cross out "never".


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Long work inside the HTTP request, with a bigger timeout The API stores the request and returns 202 Accepted, and a worker processes it Users get an answer immediately, and the slow part can retry without the user
A Windows Service with a timer on one server, plus a SQL Agent job at 02:00 BackgroundService classes in a worker container with PeriodicTimer, or a scheduled Container Apps job Same hosting, DI, configuration and telemetry as the API, and it lives in the repo
MSMQ on the same box, or a "jobs" table polled every few seconds An Azure Service Bus queue with peek-lock, retries and a dead-letter queue Failed messages are retried and then parked for a human, not lost or retried forever
"Save to the database, then send the message" in two steps A transactional outbox: the business row and the message row in one transaction No more "saved but message lost" or "message sent but save rolled back"
Hoping a message is processed only once An idempotent consumer: a ProcessedMessages table and a unique index At-least-once delivery is safe when processing twice changes nothing
Stopping the service and hoping the current job finished Graceful shutdown: stop taking new work, finish in-flight work within a timeout Containers are stopped all the time (scale-in, new revisions), so stopping must be safe

🧠 Hosted services: the Windows Service, rebuilt

💡 A BackgroundService is a Windows Service that shares hosting, DI, logging and telemetry with the rest of your app, and that can be stopped at any time.

.NET's generic host runs hosted services (IHostedService) next to everything else in the process. BackgroundService is the convenient base class: you override ExecuteAsync, and the host starts it, passes it a CancellationToken, and signals that token when the process must stop. Ledgerly's worker is three of them in one process:

💻 In Ledgerly: the worker's composition root

📄 sample/src/Ledgerly.Worker/Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();
builder.AddSqlServerDbContext<LedgerlyDbContext>("ledgerlydb");
builder.AddAzureServiceBusClient("messaging");
// ...
builder.Services.AddHostedService<OutboxRelay>();
builder.Services.AddHostedService<ImportProcessor>();
builder.Services.AddHostedService<AlertEvaluator>();

// Give in-flight messages time to finish when Container Apps stops a replica (SIGTERM).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));

var app = builder.Build();
app.MapDefaultEndpoints();
app.Run();

Notice WebApplication, not Host: the worker has no API, but it exposes /health and /alive for Container Apps probes (chapter 14). AddAzureServiceBusClient("messaging") is the Aspire client integration. It registers one ServiceBusClient as a singleton, which is what the SDK wants: clients, senders and processors are thread-safe and meant to be reused. Compared with the Windows Service, the biggest change is scaling: there can be more than one replica, so you must design for "more than one".

💻 In Ledgerly: a periodic job with PeriodicTimer

AlertEvaluator checks price alerts every 30 seconds (AlertOptions.IntervalSeconds defaults to 30). It is the replacement for the SQL Agent job:

📄 sample/src/Ledgerly.Worker/AlertEvaluator.cs

/// <summary>
/// Periodically checks active price alerts against live quotes.
/// Run the worker with ONE replica (or move this to a scheduled Container Apps job) so alerts are not evaluated twice.
/// </summary>
public sealed partial class AlertEvaluator(
    IServiceScopeFactory scopes, IMarketDataClient marketData, IOptions<AlertOptions> options, TimeProvider clock,
    ILogger<AlertEvaluator> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(options.Value.IntervalSeconds), clock);
        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            try
            {
                await EvaluateAsync(stoppingToken);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                // One bad tick (market data down, DB failover) must not kill the loop.
                LogEvaluationFailed(logger, ex);
            }
        }
    }

Four habits are packed into those lines:

  1. PeriodicTimer instead of System.Timers.Timer. You await the next tick, so a slow tick can't overlap the next one, and cancellation ends the loop cleanly.
  2. TimeProvider clock is injected, so tests can control time instead of waiting 30 seconds (chapter 17).
  3. A new DI scope per tick (scopes.CreateAsyncScope() inside EvaluateAsync). A hosted service is a singleton, and a DbContext must not live that long.
  4. Catch everything except cancellation. An exception that escapes ExecuteAsync ends the loop for good (and by default the host stops too), and one failed market-data call must not end alert checking.

Two replicas, two evaluations

Read the comment again: a timer loop runs in every replica, so two replicas can trigger the same alert at nearly the same moment and create two notifications. The Windows Service never had this problem because there was only one of it. In Azure, the worker has minReplicas: 1 (so the timers always run) and scales out on queue length, which is exactly when a second AlertEvaluator appears. You have three options:

Option How Trade-off
Keep one replica maxReplicas: 1 for the worker Simple, but imports can't scale out
Split the periodic job out A Container Apps job with a Schedule (cron) trigger Runs once per schedule wherever it lands, and finishes. Cron can't express "every 30 seconds"
Make the job safe to run twice A concurrency token on the alert, or a unique constraint on the notification The most robust choice, at the price of more code

Container Apps jobs run finite tasks and support three trigger types: Manual, Schedule (cron) and Event (a KEDA scaler such as a Service Bus queue) (Jobs in Azure Container Apps). Ledgerly already runs its migrations as a job with triggerType: 'Manual' (📄 sample/infra/modules/migration-job.bicep). A scheduled alert job would change only the trigger:

// illustrative, not in the sample: run the alert check every 5 minutes as a job
    configuration: {
      triggerType: 'Schedule'
      replicaTimeout: 300
      replicaRetryLimit: 0
      scheduleTriggerConfig: { cronExpression: '*/5 * * * *', parallelism: 1, replicaCompletionCount: 1 }
    }

💰 A job that runs for a few seconds every few minutes costs less than a replica that is always on. It also needs its own entry point (a console app that runs one evaluation and exits), which is the price of the split.


🧠 Azure Service Bus queues: peek-lock, retries and the dead-letter queue

💡 A queue doesn't make failures go away. It gives every failure a defined next step: try again, or park the message for a human.

A Service Bus queue is point-to-point: senders put messages in, and competing consumers take them out, so you can add workers to drain it faster. That is load leveling, and it is why a burst of imports doesn't hurt the API.

The important part is how a message is received. In PeekLock mode (the default) the receiver gets the message with a lock, and must then settle it:

Settle action Meaning Ledgerly uses it when
Complete "Done, delete it." The import was applied, or the message was a duplicate
Abandon "I failed, give it to someone else." The delivery count goes up An unexpected exception (database blip, bug)
Dead-letter "This will never work. Park it." The import row doesn't exist: a poison message

(A fourth action, Defer, sets a message aside to fetch later. Ledgerly doesn't use it.)

That gives you at-least-once delivery: a message is only removed when someone completes it. The alternative, ReceiveAndDelete, removes the message on receipt (at-most-once), so a crash loses it. For imports, losing work is worse than doing it twice, as long as doing it twice is harmless.

Three queue settings turn this into a safety net:

💻 In Ledgerly: the queue, as code

📄 sample/infra/modules/servicebus.bicep

// Service Bus Standard (needed for duplicate detection). Local (SAS key) auth is disabled:
// only Entra identities with a data role can send or receive.
// ...
  sku: { name: 'Standard', tier: 'Standard' }
  properties: {
    disableLocalAuth: true
    // ...
resource imports 'Microsoft.ServiceBus/namespaces/queues@2024-01-01' = {
  parent: namespace
  name: 'imports'
  properties: {
    lockDuration: 'PT1M'
    maxDeliveryCount: 10 // after 10 failed attempts the message moves to imports/$deadletterqueue
    requiresDuplicateDetection: true // the same MessageId within the window is dropped
    duplicateDetectionHistoryTimeWindow: 'PT10M'
    deadLetteringOnMessageExpiration: true
  }
}

lockDuration: 'PT1M' is how long the worker holds a message before Service Bus assumes it died and offers the message to someone else. disableLocalAuth: true means no connection-string keys at all: the worker's managed identity gets the Azure Service Bus Data Sender and Data Receiver roles further down the same file (chapter 13).

The tier matters 🔧 💰. Basic has no duplicate detection, sessions or transactions, so Ledgerly uses Standard (shared capacity, pay per operation). Premium adds dedicated Messaging Units, VNet and Private Link, and messages up to 100 MB, at a fixed cost even when idle.

One honest gap: locally, the AppHost declares the queue with messaging.AddServiceBusQueue("imports") and no extra settings, so the emulator queue has no duplicate detection window. The idempotent consumer below is what keeps local runs correct, which is a good argument for never relying on duplicate detection alone.


🗺️ The life of an import message

flowchart TD
    Send["Relay sends message<br/>MessageId = outbox id"]
    Dup{"MessageId seen in<br/>last 10 minutes?"}
    Drop["Dropped by<br/>duplicate detection"]
    Queue["Waiting in<br/>imports queue"]
    Locked["Delivered with a lock<br/>to one worker"]
    Done["Completed<br/>message deleted"]
    Retry{"Delivery count<br/>below 10?"}
    Dlq["Dead-letter queue<br/>waits for a human"]
    Send --> Dup
    Dup -- "yes, in Azure" --> Drop
    Dup -- "no" --> Queue
    Queue --> Locked
    Locked -- "complete" --> Done
    Locked -- "abandon or lock expires" --> Retry
    Retry -- "yes" --> Queue
    Retry -- "no" --> Dlq
    Locked -- "dead-letter: ImportNotFound" --> Dlq

🧠 The transactional outbox: never lose the message

💡 If the database and the queue can't share a transaction, put the message in the database and send it later.

Here is the bug you would write by habit:

// illustrative, not in the sample: the dual-write bug
db.Imports.Add(import);
await db.SaveChangesAsync(ct);                                   // 1. committed
await sender.SendMessageAsync(new ServiceBusMessage(json), ct);  // 2. what if this throws?

If step 2 fails (a network blip, a throttled namespace, a deployment that kills the process), the import row exists but no message does. The import stays Pending forever. Swap the order, and you get the opposite bug: a message about an import that was rolled back. SQL Server and Service Bus can't take part in one transaction, so you need a different shape.

The transactional outbox writes the message as a row in the same database, in the same transaction as the business change. A separate relay reads unsent rows and publishes them. EnqueueOutboxMessage only adds the row: it is saved by the caller's next SaveChangesAsync.

💻 In Ledgerly: the outbox row

📄 sample/src/Ledgerly.Core/Messaging/Outbox.cs

public sealed class OutboxMessage
{
    public Guid Id { get; init; } = Guid.CreateVersion7();
    public required string Type { get; init; }
    public required string Destination { get; init; }
    public required string Payload { get; init; }
    public DateTimeOffset OccurredAt { get; init; } = DateTimeOffset.UtcNow;
    public DateTimeOffset? ProcessedAt { get; set; }
    public int Attempts { get; set; }
    public string? LastError { get; set; }

    // W3C trace context of the request that created the message, so the trace continues in the worker.
    public string? TraceParent { get; init; }
}
// ...
    public static void EnqueueOutboxMessage<T>(this LedgerlyDbContext db, string destination, T message)
    {
        db.OutboxMessages.Add(new OutboxMessage
        {
            Type = typeof(T).Name,
            Destination = destination,
            Payload = JsonSerializer.Serialize(message),
            TraceParent = System.Diagnostics.Activity.Current?.Id,
        });
    }

Guid.CreateVersion7() gives time-ordered IDs, which index well. TraceParent captures the current trace, so the worker's spans appear under the original HTTP request in the dashboard (chapter 15).

💻 In Ledgerly: the API side, one SaveChanges

📄 sample/src/Ledgerly.Api/Endpoints/ImportEndpoints.cs

        db.Imports.Add(import);
        db.EnqueueOutboxMessage(Queues.Imports, new ImportRequested(import.Id, id));
        await db.SaveChangesAsync(ct); // one transaction: both rows or neither

        return TypedResults.AcceptedAtRoute(ToResponse(import), "GetImport", new { importId = import.Id });

EF Core wraps one SaveChangesAsync in one transaction, so either both rows exist or neither does. The endpoint returns 202 Accepted with a Location header pointing at GET /api/imports/{importId}, which the client polls. The endpoint is also rate-limited (five uploads per user per minute) and rejects files over 1,000,000 bytes, so the "click Upload three times" problem can't turn into a flood.

💻 In Ledgerly: the relay

📄 sample/src/Ledgerly.Worker/OutboxRelay.cs

    private const int BatchSize = 20;
    private const int MaxAttempts = 10;
    // ...
        var pending = await db.OutboxMessages
            .Where(m => m.ProcessedAt == null && m.Attempts < MaxAttempts)
            .OrderBy(m => m.OccurredAt)
            .Take(BatchSize)
            .ToListAsync(ct);
        // ...
                var sender = _senders.GetOrAdd(message.Destination, serviceBus.CreateSender);
                await sender.SendMessageAsync(
                    new ServiceBusMessage(message.Payload)
                    {
                        MessageId = message.Id.ToString(),
                        Subject = message.Type,
                        ContentType = "application/json",
                    },
                    ct);

                message.ProcessedAt = clock.GetUtcNow();

Every 2 seconds (PeriodicTimer, again), the relay takes up to 20 unsent rows in order, sends each one, and marks it processed. A failed send increments Attempts and stores LastError. After 10 attempts the row is skipped, so one broken message can't block the rest, and the row stays in the table for someone to investigate.

The class comment names the weak spot: delivery is at-least-once, because if the worker crashes after SendMessageAsync but before SaveChangesAsync, the next tick sends the same row again. The same happens if two worker replicas pick up the same batch. That is why MessageId = message.Id.ToString() matters: the duplicate carries the same ID, so Service Bus duplicate detection drops it in Azure, and the consumer recognizes it everywhere.


🗺️ From upload to processed import

sequenceDiagram
    participant User
    participant Api
    participant Sql as Azure SQL
    participant Relay as OutboxRelay
    participant Bus as Service Bus imports
    participant Proc as ImportProcessor
    User->>Api: POST imports with CSV
    Api->>Sql: insert Import and OutboxMessage in one transaction
    Api-->>User: 202 Accepted with Location
    Relay->>Sql: read unsent outbox rows
    Relay->>Bus: send with MessageId = outbox id
    Relay->>Sql: set ProcessedAt
    Bus->>Proc: deliver with peek-lock
    Proc->>Sql: already in ProcessedMessages?
    Proc->>Sql: trades, import status and ProcessedMessage in one SaveChanges
    Proc->>Bus: complete
    User->>Api: GET import status
    Api-->>User: Completed, rows imported and skipped

🧠 The idempotent consumer: processing twice changes nothing

💡 You can't get exactly-once delivery, but you can get exactly-once effects.

At-least-once delivery means your handler will see some messages twice: relay re-sends, abandoned messages, a lock that expired while a slow import was still running. The handler's job is to make the second time a no-op.

💻 In Ledgerly: the handler

📄 sample/src/Ledgerly.Worker/ImportProcessor.cs

    private async Task HandleAsync(ProcessMessageEventArgs args)
    {
        var messageId = args.Message.MessageId;
        // ...
        if (await db.ProcessedMessages.AnyAsync(p => p.MessageId == messageId, ct))
        {
            LogDuplicate(logger, messageId);
            await args.CompleteMessageAsync(args.Message, ct);
            return;
        }
        // ...
        if (import is null)
        {
            // Poison message: retrying will never help. Park it in the DLQ for a human to look at.
            await args.DeadLetterMessageAsync(args.Message, "ImportNotFound", $"No import for message {messageId}", ct);
            return;
        }

        try
        {
            var outcome = await ApplyImportAsync(db, import, ct);
            db.ProcessedMessages.Add(new ProcessedMessage { MessageId = messageId, Handler = HandlerName });
            await db.SaveChangesAsync(ct); // trades + import status + idempotency record in ONE transaction
            await args.CompleteMessageAsync(args.Message, ct);
            // ...
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            // ...
            await args.AbandonMessageAsync(args.Message, cancellationToken: ct); // Service Bus will redeliver
        }
    }

Walk through it the way Maya would:

What happens What the code does Result
A duplicate arrives (relay re-send, emulator without duplicate detection) ProcessedMessages already has this MessageId Complete it, log it, do nothing else
The import row doesn't exist DeadLetterMessageAsync(..., "ImportNotFound", ...) Retrying can't help, so it goes straight to the DLQ
The database blips during SaveChangesAsync Nothing was committed, then AbandonMessageAsync Delivered again, up to 10 times, then the DLQ
The worker crashes after SaveChangesAsync, before Complete The lock expires and the message comes back The ProcessedMessages check finds it: complete, no second import
The file breaks a business rule (selling more than you hold) ApplyImportAsync marks the import Failed and returns "rejected" Saved and completed. A business failure is an outcome, not a retry

Three design choices make this work:

  1. AutoCompleteMessages = false in the ServiceBusProcessorOptions. The processor never completes a message on your behalf. You complete it only after the transaction committed.
  2. The idempotency record is in the same transaction as the work. Trades, import status and the ProcessedMessage row are saved by one SaveChangesAsync. There is no moment where the trades exist but the record doesn't.
  3. The database enforces it, not just the code. ProcessedMessage uses MessageId as its primary key. If two replicas handle the same message at the same moment, both pass the AnyAsync check, but only one insert can succeed. The other gets an exception, abandons, and on redelivery completes as a duplicate.

There is a third layer, in the data model itself. Each broker row's reference is stored as Transaction.ExternalRef, with a filtered unique index on (PortfolioId, ExternalRef) (chapter 5), and ApplyImportAsync skips references the portfolio already has. Re-uploading the same file imports zero new rows and reports them as skipped. That fixes Elena's ticket from the user's side.

The processor also sets MaxConcurrentCalls = 4, so one replica handles four imports in parallel. Each call gets its own DI scope and DbContext, because a DbContext is not thread-safe.


🧠 Graceful shutdown and boring parsing

💡 In the cloud, your process is stopped on purpose many times a day. Make stopping a normal code path.

Container Apps stops replicas when it scales in and when it rolls out a new revision. It sends SIGTERM, and the .NET host then stops accepting new work, signals the CancellationToken passed to ExecuteAsync, and calls StopAsync on every hosted service. HostOptions.ShutdownTimeout caps how long that may take: the default is 30 seconds, and Ledgerly sets 25 so it finishes inside the platform's termination grace period (❓ Container Apps' default is believed to be 30 seconds but this course hasn't re-verified it) (.NET Generic Host).

💻 In Ledgerly: stop taking messages, finish the ones in flight

📄 sample/src/Ledgerly.Worker/ImportProcessor.cs

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        // Graceful shutdown: stop taking new messages, let in-flight handlers finish.
        if (_processor is not null)
        {
            await _processor.StopProcessingAsync(cancellationToken);
            await _processor.DisposeAsync();
        }

        await base.StopAsync(cancellationToken);
    }

If an import is still running when the timeout hits, its cancellation token fires, SaveChangesAsync never commits, and the message is never completed. Peek-lock then gives the message to another replica after the lock expires, and the idempotent handler starts from a clean state. A Windows Service killed mid-job left half a batch behind. This one leaves nothing behind, which is the point of putting the whole effect in one transaction.

OutboxRelay.StopAsync does the same in its own way: it lets the loop end, then disposes its cached senders.

💻 In Ledgerly: a parser that doesn't care where it runs

Background work runs on machines you never see, with cultures you didn't choose. BrokerCsvParser parses every number and date with the invariant culture, and it reports bad rows instead of throwing:

📄 sample/src/Ledgerly.Core/Imports/BrokerCsvParser.cs

/// <summary>
/// Parses the broker export format:
/// <c>date,type,symbol,quantity,price,reference</c> (header required, ISO dates, invariant decimals).
/// Bad rows are reported, not thrown: one typo should not reject a 5 000-row file.
/// </summary>
public static class BrokerCsvParser
{
    // ...
            if (!decimal.TryParse(cells[4], NumberStyles.Number, CultureInfo.InvariantCulture, out var price) || price <= 0)
            {
                errors.Add(new ParseError(lineNumber, $"Invalid price '{cells[4]}'."));
                continue;
            }

A unit test proves it by switching the thread to lt-LT, where the decimal separator is a comma:

📄 sample/tests/Ledgerly.UnitTests/BrokerCsvParserTests.cs

    public void Decimal_parsing_ignores_server_culture()
    {
        // The old IIS box had lt-LT culture and "190,10" broke imports. Invariant culture, always.
        var previous = Thread.CurrentThread.CurrentCulture;
        Thread.CurrentThread.CurrentCulture = new System.Globalization.CultureInfo("lt-LT");
        try
        {
            var result = BrokerCsvParser.Parse($"{Header}\n2026-09-01,buy,AAPL,5,190.10,BRK-1");
            Assert.Equal(190.10m, Assert.Single(result.Trades).Price);
        }
        // ...

"Reported, not thrown" also matters for the queue. A parse error is a business result, saved on the import (completed_with_errors, with the first five errors in Error). If it were an exception, the message would be abandoned and retried 10 times, with the same result every time, and then dead-lettered. Only throw for things a retry can fix.


⚠️ Common traps

Trap What happens Do this instead
Saving to the database, then sending the message A crash between the two loses the message, and the work never happens Use a transactional outbox: both rows in one SaveChanges, and a relay sends later
Leaving AutoCompleteMessages on and doing work after an await The message is completed even if your save failed, so the work is lost Set AutoCompleteMessages = false and complete only after the commit
Checking for duplicates in code only Two replicas pass the check at the same time and both apply the import Back the check with a primary key or unique index in the same transaction
Nobody watches the dead-letter queue Messages pile up silently, because DLQ messages are never auto-deleted Alert on DLQ depth and write a runbook for replaying them (chapters 16 and 19)
A timer job in a worker that scales out Every replica runs the job, so alerts fire twice Keep that worker at one replica, move the job to a scheduled Container Apps job, or make it idempotent
Ignoring the CancellationToken in ExecuteAsync Shutdown waits for the timeout and then the process is killed mid-work Pass stoppingToken everywhere, and keep the shutdown timeout below the platform's grace period

🧰 Helpers

Checklist: a new message type

  1. Define the message as a small record with IDs, not whole entities (like ImportRequested(ImportId, PortfolioId)).
  2. Enqueue it with db.EnqueueOutboxMessage(...) before the one SaveChangesAsync of the request.
  3. In the consumer: check ProcessedMessages, do the work, add the ProcessedMessage, save once, then complete.
  4. Decide, for each failure, which settle action it gets: abandon for transient faults (timeouts, throttling), dead-letter for messages that can never succeed, complete for business outcomes you recorded.
  5. Add a test that delivers the same message twice and asserts that nothing changes the second time (chapter 17).

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

  1. Start the system and create a token: bash cd sample aspire run dotnet user-jwts create --project src/Ledgerly.Api
  2. Open src/Ledgerly.Api/Ledgerly.Api.http in your IDE, set @api to the API's URL from the dashboard and @token to your token, and create a portfolio. Copy its id.
  3. Save this as broker.csv: text date,type,symbol,quantity,price,reference 2026-09-01,buy,AAPL,5,190.10,BRK-1001 2026-09-15,buy,MSFT,3,410.00,BRK-1002 2026-09-20,buy,MSFT,abc,410.00,BRK-1003
  4. Upload it (replace the URL, ID and token): bash curl -i -X POST https://localhost:7180/api/portfolios/<id>/imports -H "Authorization: Bearer <token>" -F "file=@broker.csv" You get 202 Accepted and a Location header right away.
  5. Request the Location URL (with the same Authorization header) until the status is Completed. Expect 2 rows imported and 1 skipped, with "Invalid quantity" in error.
  6. Upload the same file again. The second import completes with 0 rows imported and all rows skipped: idempotency by ExternalRef.
  7. In the dashboard's Traces view, find the POST request. The worker's outbox publish imports span appears in the same trace: that is the TraceParent column at work. Then find the import process span for the same import (chapter 15 explains how trace context crosses the queue).
  8. Stop the AppHost and run the unit tests, including the lt-LT parser test: bash dotnet test --solution Ledgerly.slnx

✅ Recap

  1. Slow work leaves the HTTP request: the API returns 202, and hosted services in the worker do the work with PeriodicTimer loops and a Service Bus processor.
  2. The transactional outbox guarantees the message is sent if and only if the data was saved, and the idempotent consumer (one transaction plus a primary key) makes at-least-once delivery safe.
  3. Peek-lock, MaxDeliveryCount, the dead-letter queue and graceful shutdown give every failure a defined next step, so stopping or crashing a worker never leaves half an import behind.

Next → Chapter 8: GitHub as the team's home

Chapter 8: GitHub as the team's home

⏱ About 30 min · You'll learn: how a change travels from a short-lived branch to main through a pull request, how rulesets, required checks and CODEOWNERS turn team rules into settings GitHub enforces, and how Dependabot, CodeQL, secret scanning and Copilot code review keep watch without a meeting.


📖 Story: Your first pull request

The background import from chapter 7 works on your machine. The Service Bus emulator drains the queue, the duplicate test passes, and Elena's support ticket can finally be closed. At Nordbank Insurance, this is where the waiting began. Your work would sit on a branch called release/2019-Q3 for weeks. Someone would zip a diff and email it to two senior developers, who would print it and bring it to the Thursday review meeting. Then came the merge weekend, when four long-lived branches met for the first time and nobody went home before midnight.

You ask Maya who you should email for a review.

"Nobody," she says. "Push your branch and open a pull request. CI will build it, CODEOWNERS will ask the right people, and Copilot will leave comments before any of us look. If main is red or nobody approved, the merge button stays grey. Not because we're strict: because GitHub won't let it."

Tomás adds from the next desk: "And keep it small. If your PR is open longer than two days, it's two PRs."

Jonas Berg forwards you a link. A Dependabot pull request bumped the OpenTelemetry packages overnight, CI is already green, and it's waiting for one approval. "That's the other half of the job," he says. "The robots open PRs too. We review them like anyone else's."

In your notebook, under Habits I brought with me, you write: "Code review is a meeting." Then: "Merging is an event." You suspect both are about to be crossed out.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Long-lived release branches, merged on a "merge weekend" Short-lived branches off main, merged through a pull request within a day or two (GitHub flow) Small diffs are easy to review and easy to revert, and conflicts stay small
Reviews by email and in a Thursday meeting Reviews in the pull request, with CODEOWNERS requesting the right people automatically The discussion lives next to the code, forever, and nobody has to remember who owns what
"Never push to trunk" as a line in the wiki A ruleset on main: pull request required, checks required, no force pushes A rule GitHub enforces can't be forgotten on a Friday evening
A build server someone checked "now and then" Required status checks from GitHub Actions on every PR (chapter 9) Red means "can't merge", not "someone should look at that"
Library upgrades once a year, as a project Dependabot opens grouped update PRs every week, and CI tests each one Small, frequent upgrades are cheap. A three-year jump is a migration
A security review by another team, just before release CodeQL on every PR, secret scanning and push protection on every push Findings arrive while the code is still fresh in your head
Commit messages like "fix" and "wip 2" Conventional commits and PR titles such as feat(imports): process CSV files in the background History you can scan, and release notes you can generate

🧠 GitHub flow: one main branch, many short ones

💡 main is always releasable. Every change reaches it through a pull request, and nothing else does.

GitHub flow is deliberately simple:

  1. Branch from main with a descriptive name (fix/import-timeout, feat/price-alerts).
  2. Commit small steps and push early. A pushed branch is backed up and visible.
  3. Open a pull request, as a draft if you want feedback before it's finished.
  4. CI runs and reviewers comment. You push fixes to the same branch, and the PR updates.
  5. Merge when the checks are green and the required approvals are in. Delete the branch.
  6. Deploy from main. At Ledgerly that happens automatically after CI passes on main (chapter 12).

There is no develop branch, no release/* branch and no merge weekend. Unfinished work either stays on its branch, or it's merged behind a feature flag (the alerts endpoint already does this, see chapter 19). The flag decides what users see. The branch decides nothing.

💻 In Ledgerly: what "merged to main" sets in motion

The CI workflow runs on every pull request and every push to main that touches the sample:

📄 .github/workflows/ci.yml

# CI: every push and pull request is built and tested. A red build blocks the merge (see branch protection, chapter 8).
name: ci

on:
  push:
    branches: [main]
    paths: ["sample/**", ".github/workflows/ci.yml"]
  pull_request:
    paths: ["sample/**", ".github/workflows/ci.yml"]
  workflow_dispatch:

The deploy workflow doesn't watch branches at all. It waits for CI to finish on main:

📄 .github/workflows/deploy.yml

on:
  workflow_run:
    workflows: [ci]
    types: [completed]
    branches: [main]
  workflow_dispatch:

So the only road to production is: pull request, green CI, merge, green CI on main, deploy. Chapter 9 walks through the CI file line by line, and chapter 12 does the same for deploy.

Issues, projects and commit messages

Work starts as an issue, not as an email. Elena writes the "CSV imports time out" issue with the user's words and a sample file. You link your PR to it, and the issue closes when the PR merges. A project board (Elena's has the columns Todo, In progress, In review and Done) shows the team's work without a status meeting. Leo uses the same board for frontend tasks, so the API change and the UI change sit next to each other.

Ledgerly writes commit messages and PR titles as conventional commits: a type, an optional scope and a short summary.

feat(imports): process CSV files in a background worker
fix(worker): complete duplicate messages instead of abandoning them
docs(readme): replace the setup wiki with aspire run
chore(deps): bump the opentelemetry group

The team squash-merges, so the PR title becomes the one commit on main. Scan git log --oneline and you can tell features from fixes from chores at a glance, and a tool can turn the history into release notes later.


🗺️ The life of a change

flowchart LR
    Issue["Issue<br/>CSV imports time out"] --> Branch["Branch<br/>fix/import-timeout"]
    Branch -- "push" --> PR["Pull request"]
    PR --> CI["CI checks<br/>build-test, containers, infra"]
    PR --> Scan["CodeQL<br/>analyze"]
    PR --> Review["Reviews<br/>Copilot, then CODEOWNERS"]
    CI --> Gate{"Ruleset on main<br/>all green and approved?"}
    Scan --> Gate
    Review --> Gate
    Gate -- "no" --> Branch
    Gate -- "yes, squash merge" --> Main["main"]
    Main -- "CI passes on main" --> Deploy["deploy.yml<br/>production environment"]

🧠 Protecting main: rulesets, required checks and environments

💡 A team rule that lives in a wiki is a suggestion. A team rule that lives in a ruleset is a fact.

GitHub has two ways to protect a branch. Classic branch protection rules are the older one. Rulesets are the newer mechanism, and the one to reach for today (About rulesets):

The rules you can turn on include (Available rules for rulesets): require a pull request (with a number of approvals, review from code owners, and dismissing stale approvals when new commits arrive), require status checks to pass, require linear history, block force pushes, require signed commits, require deployments to succeed and require code scanning results.

Ledgerly's ruleset on main is short:

Rule Setting Why
Require a pull request 1 approval, review from code owners, dismiss stale approvals Nobody merges their own unreviewed code, and approving an old version doesn't count
Require status checks The CI and CodeQL jobs (below), branch up to date main stays green
Block force pushes On History on main is never rewritten
Require linear history On Squash merges keep one commit per PR
Bypass list Empty Not even Maya skips review

💻 In Ledgerly: where the check names come from

A required status check must have passed on the latest commit of the PR, and its name must match the job name exactly. The names come straight from the workflow files:

📄 .github/workflows/ci.yml

jobs:
  build-test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    # ...
  containers:
    runs-on: ubuntu-latest
    needs: build-test
    timeout-minutes: 20
    strategy:
      matrix:
        project: [Ledgerly.Api, Ledgerly.Worker, Ledgerly.MarketData, Ledgerly.MigrationService]
    # ...
  infra:
    runs-on: ubuntu-latest
    timeout-minutes: 10

📄 .github/workflows/codeql.yml

jobs:
  analyze:
    runs-on: ubuntu-latest
    timeout-minutes: 30

None of the jobs has a name:, so the job ID is the check name: build-test, infra and analyze. A matrix job reports one check per matrix value, so containers shows up four times, as containers (Ledgerly.Api), containers (Ledgerly.Worker) and so on. Rename a job, and the required check waits forever for a name that no longer reports. Change the workflow and the ruleset in the same PR.

Environments: the gate before production

Rulesets guard main. Environments guard deployments. Ledgerly's deploy job runs in an environment called production:

📄 .github/workflows/deploy.yml

    environment:
      name: production # protection rules (reviewers, branch filter) live on the environment
      url: ${{ steps.apps.outputs.apiUrl }}

An environment can require reviewers (up to 6 users or teams, and only one of them needs to approve, optionally never the person who triggered the run), a wait timer, deployment branch restrictions and custom protection rules. It also holds its own secrets and variables (Deployments and environments).

🔧 💰 Read the plan small print: on Free, Pro and Team plans, required reviewers and wait timers work only for public repositories. For a private repository you need Enterprise. Ledgerly's beta repo is private on Team, so the team relies on the main ruleset plus a branch filter on the environment, and Priya gets a notification for every deploy.

Tomás has one complaint about all of this: rulesets and environments are settings, not files, so they don't show up in a diff. "If it's not in git, it doesn't exist," he says, and writes the ruleset down as an ADR (the toolkit has adr-template.md) so the next person can see what was decided and why.


🧠 Reviews that find the right people: CODEOWNERS, templates and Copilot

💡 Let GitHub decide who reviews and what the author must say. Spend human attention on the design.

CODEOWNERS

A CODEOWNERS file maps paths to the people or teams who own them. When a PR touches a path, its owners are requested for review automatically. With the "require review from code owners" rule, their approval becomes mandatory (About code owners).

Two details matter:

The sample repository doesn't contain a CODEOWNERS file. This is what Ledgerly's looks like:

# illustrative, not in the sample: .github/CODEOWNERS
# The last matching pattern wins, so the catch-all goes first.
*                                   @ledgerly/developers

# Infrastructure and pipelines: Tomás's team reviews every change.
/sample/infra/                      @ledgerly/platform
/.github/workflows/                 @ledgerly/platform

# Supply chain: package sources and versions need a security review.
/sample/nuget.config                @ledgerly/security
/sample/Directory.Packages.props    @ledgerly/security

# Schema changes are hard to undo: the tech lead looks at every migration.
/sample/src/Ledgerly.Core/Data/Migrations/   @maya-okafor

Use teams rather than individuals where you can, so a vacation doesn't block a merge. Note what the supply-chain lines mean: every Dependabot NuGet PR touches Directory.Packages.props, so Jonas's team approves each one. That's intended.

The pull request template

A pull request template pre-fills the description of every new PR, so the reviewer always gets the same questions answered. The sample doesn't include one, but the toolkit has pull_request_template.md ready to copy to .github/ (see the toolkit). Its shape:

<!-- illustrative, not in the sample: .github/pull_request_template.md -->
## What and why
Fixes #

## How I tested it
- [ ] Unit or integration tests added or updated
- [ ] Ran it locally with `aspire run`

## What happens when it fails?
<!-- Retries, timeouts, the dead-letter queue, rollback -->

## Checklist
- [ ] Database migration is backward compatible (old code still runs against the new schema)
- [ ] New settings are documented, and no secrets are in code or config
- [ ] Logs, metrics or traces added for the new path

The "What happens when it fails?" heading is Maya's. Every PR answers it before she has to ask.

Copilot code review and the Copilot cloud agent

Copilot code review reviews a pull request the way a teammate would: it leaves comments on the diff, often with a suggested change. You request it like any other reviewer, or make it automatic for a repository or organization through settings or a ruleset. Business and Enterprise organizations can enable it even for members who don't have a Copilot license, billed to the organization 💰 (Copilot on GitHub).

At Ledgerly, Copilot reviews first and humans second. It's good at the things people skim: a missing CancellationToken, a DbContext captured in a singleton, a null path nobody tested. Its comment is not an approval, and it doesn't count toward the ruleset. A person who understands the design still signs off.

🔧 The Copilot cloud agent (GitHub's docs renamed it from "Copilot coding agent") goes one step further. You assign an issue to Copilot, it works in an environment powered by GitHub Actions, and it opens a pull request for review. It's available on paid plans, and for Business and Enterprise an org admin has to enable it. Its PR goes through exactly the same gates as yours: CI, CODEOWNERS, a human approval. Chapter 20 covers how to write issues it can finish, and the .github/copilot-instructions.md file that teaches it Ledgerly's conventions.


🗺️ A pull request meets the gates

sequenceDiagram
    participant You as You
    participant GH as GitHub
    participant CI as Actions ci and codeql
    participant Bot as Copilot code review
    participant Owner as Code owner
    You->>GH: Push branch and open pull request
    GH->>CI: Start build-test, containers, infra, analyze
    GH->>Bot: Request automatic review
    GH->>Owner: Request review from CODEOWNERS
    Bot-->>GH: Comments and suggested changes
    CI-->>GH: build-test failed
    GH-->>You: Merge blocked by required checks
    You->>GH: Push a fix to the same branch
    GH->>CI: Run all checks again
    CI-->>GH: All checks passed
    Owner-->>GH: Approve
    GH-->>You: Ruleset satisfied, merge enabled

🧠 Supply chain on autopilot: Dependabot, CodeQL and secret scanning

💡 The safest dependency update is a small one that CI tested last week, not a big one somebody postponed for a year.

Dependabot version updates

Dependabot reads .github/dependabot.yml and opens pull requests when a newer version of a package or an action exists. Every one of those PRs runs CI like yours does, so "is this upgrade safe?" is answered by your tests, not by a hunch.

💻 In Ledgerly: grouped updates for NuGet and actions

📄 .github/dependabot.yml

# Dependabot opens pull requests for outdated packages and actions. CI tests every one of them.
version: 2
updates:
  - package-ecosystem: nuget
    directory: /sample
    schedule:
      interval: weekly
    groups:
      # One PR per family instead of twenty. Versions inside a family must move together.
      aspire:
        patterns: ["Aspire.*"]
      dotnet-platform:
        patterns: ["Microsoft.AspNetCore.*", "Microsoft.EntityFrameworkCore*", "Microsoft.Extensions.*"]
      opentelemetry:
        patterns: ["OpenTelemetry*", "Azure.Monitor.OpenTelemetry*"]
      testing:
        patterns: ["xunit*", "Microsoft.Testing.*", "Testcontainers*"]

  - package-ecosystem: github-actions
    directory: /
    schedule:
      interval: weekly
    groups:
      actions:
        patterns: ["*"]

Read it top to bottom:

Two more ecosystems worth knowing: dotnet-sdk updates the SDK version in global.json, and docker updates base images in Dockerfiles (Ledgerly has none, see chapter 9) (Dependabot options reference).

Dependabot is half of the supply-chain story. The other half runs at build time: NuGet audit fails the build when a package has a known vulnerability, which is exactly what happened while building the sample (chapter 9).

CodeQL code scanning

CodeQL is GitHub's static analysis engine. It looks for security bugs (SQL injection, path traversal, unsafe deserialization) and reports them under Security → Code scanning and as comments on the pull request (Configuring code scanning).

💻 In Ledgerly: the CodeQL workflow

📄 .github/workflows/codeql.yml

# Static security analysis (SAST). Findings show up under Security > Code scanning and on pull requests.
name: codeql

on:
  push:
    branches: [main]
    paths: ["sample/**"]
  pull_request:
    paths: ["sample/**"]
  schedule:
    - cron: "17 3 * * 1" # weekly, catches new queries against unchanged code

permissions:
  contents: read
  security-events: write

jobs:
  analyze:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - uses: github/codeql-action/init@v4
        with:
          languages: csharp
          build-mode: none # no build needed for C#, analysis reads the source
      - uses: github/codeql-action/analyze@v4

Three choices stand out:

This is the advanced setup: a workflow file you control. The default setup is a one-click setting with no file at all, and for C# it also uses build-mode: none. It scans pushes to the default branch, pull requests and weekly. 🔧 💰 CodeQL is free for public repositories. Private and internal repositories in an organization need GitHub Code Security, sold separately since 2025.

Secret scanning and push protection

Secret scanning finds credentials committed to the repository, and push protection blocks a push that contains one before it ever reaches GitHub. Both are free and automatic for public repositories, where push protection for users is on by default. 🔧 For private repositories and repo-level push protection you need GitHub Secret Protection (About push protection).

Ledgerly's best defense is having almost nothing to leak: no SAS keys (chapter 7), no deployment secrets thanks to OIDC (chapter 12), and managed identities everywhere (chapter 13). Jonas still turns push protection on, because a developer's personal token in a test script is enough to ruin a week.


⚠️ Common traps

Trap What happens Do this instead
Renaming a CI job without updating the ruleset The required check never reports under the old name, and every PR waits forever Change the job name and the required checks in the same change, and tell the team
Making a path-filtered workflow a required check A PR that doesn't touch sample/** never runs CI, so the required check stays pending Require only checks that run on every PR, or remove the path filter from the required workflow
Expecting environment reviewers on a private repo on Team The production approval gate silently doesn't apply 🔧 Check your plan. Use branch filters and the main ruleset, or move to Enterprise
A broad CODEOWNERS rule at the bottom of the file It matches everything and overrides the specific owners above it Catch-all first, specific paths last, because the last match wins
Treating Copilot's review as the approval Nobody who understands the design looked at the change Copilot reviews first, a human approves. Its comments don't satisfy the ruleset
Ignoring Dependabot PRs for months They pile up, conflict with each other, and the upgrade becomes a project again Review the grouped PRs weekly. Green CI plus a release-notes skim is usually enough
A long bypass list "just in case" Anyone on it can merge without checks, and eventually someone does Keep the bypass list empty or limited to a bot, and use Evaluate mode to trial new rules

🧰 Helpers

The pull request loop from the terminal, with the GitHub CLI:

git switch -c fix/import-timeout          # branch from an up-to-date main
git commit -am "fix(imports): move CSV processing to the worker"
git push -u origin HEAD
gh pr create --fill                       # title and body from your commits
gh pr checks --watch                      # wait for CI and CodeQL
gh pr view --web                          # read the review comments
gh pr merge --squash --delete-branch      # when the ruleset allows it

Reviewing someone else's PR locally:

gh pr list                                # open PRs in this repo
gh pr checkout 42                         # check out PR #42 as a local branch
cd sample
dotnet test --solution Ledgerly.slnx      # run the tests yourself

A conventional commit cheat sheet: feat (a new feature), fix (a bug fix), docs, test, refactor (no behavior change), ci (workflows), chore (dependencies, tooling). Add a scope in parentheses when it helps: fix(worker): ....

Make it permanent:


🧪 Try it yourself (optional, ~25 min)

  1. Fork the course repository and clone your fork, then make your fork the default for gh: bash gh repo fork vygantas-meskauskis/cloud-native-dotnet --clone cd cloud-native-dotnet gh repo set-default Pick your fork when gh repo set-default asks. On GitHub, open your fork's Actions tab and enable workflows if GitHub asks you to.
  2. Create a branch and make a small change under sample/ so CI runs, for example improve a comment in sample/src/Ledgerly.Worker/AlertEvaluator.cs: bash git switch -c docs/alert-evaluator-comment git commit -am "docs(worker): explain why alerts need a single replica" git push -u origin HEAD
  3. Open the pull request against your fork and watch the checks: bash gh pr create --fill gh pr checks --watch Note the check names: build-test, containers (Ledgerly.Api) and its siblings, infra, analyze.
  4. In your fork, go to Settings → Rules → Rulesets and create a branch ruleset for the default branch. Require a pull request and the build-test status check, block force pushes, and set enforcement to Evaluate. (Your fork is public, so rulesets work on the Free plan 🔧.)
  5. Push another commit to the PR. Then open Settings → Rules → Insights and see what the ruleset would have blocked, without anything being blocked.
  6. If your plan includes Copilot code review, add Copilot as a reviewer on the PR and read its comments. Decide which ones you'd accept.
  7. Add an illustrative .github/CODEOWNERS with yourself as the owner of /sample/infra/, open a second PR that touches a Bicep comment, and watch GitHub request your review automatically.
  8. Merge with gh pr merge --squash --delete-branch and run git log --oneline on main: one commit, with your conventional title.

✅ Recap

  1. GitHub flow keeps main releasable: short branches, pull requests for every change, and deployment only after CI passes on main.
  2. Rulesets, required status checks, CODEOWNERS and environments turn the team's rules into settings GitHub enforces, so nothing depends on someone remembering.
  3. Dependabot, CodeQL, secret scanning and Copilot code review run on every PR, and humans still give the approval.

Next → Chapter 9: CI with GitHub Actions

Chapter 9: CI with GitHub Actions

⏱ About 35 min · You'll learn: how Ledgerly's CI workflow builds, tests, packages and lints every pull request on a fresh machine, why it runs the same commands you run locally, how container images get built without a Dockerfile, and the real failures that shaped the workflow.


📖 Story: Green on my machine, red in CI

Your first week includes a turn on "CI duty", and Tomás starts it by opening the run history. He scrolls back a month to a red cross next to build-test on a pull request from Leo, and opens the log:

error NU1903: Package 'System.Security.Cryptography.Pkcs' 10.0.12 has a known high severity vulnerability

"Leo's laptop was green," Tomás says. "He hadn't touched a package. CI was red anyway, and CI was right. It's the only machine that counts. Your laptop has a warm cache, an SDK you installed in March and a bin folder full of history. The runner starts from nothing, every time."

The cause was a package nobody referenced directly, pulled in by another one, with a newly published vulnerability. NuGet audit flagged it, and because warnings are errors at Ledgerly, the build stopped. One line in Directory.Packages.props pinned the patched version. Then Priya tells you about the week the team moved to the .NET 10 SDK and dotnet test refused to run at all.

At Nordbank Insurance, "the build" was BUILD01, a tower PC under a desk in the server room. Somebody installed SDKs on it by hand, a scheduled batch file ran every night, and when it broke, the person who knew the batch file was on holiday. In your notebook, under Habits I brought with me, you write: "The build server is a pet." You don't need a question mark this time.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
BUILD01 under a desk, with SDKs installed by hand A fresh ubuntu-latest runner per job, with the SDK version taken from global.json Nothing is left over from the last build, so a green build means the repo is enough
A build definition clicked together in a UI A YAML workflow in .github/workflows, reviewed in pull requests The build changes with the code, in the same PR, with the same review
A nightly build A run on every push and pull request, with outdated runs cancelled You find out in minutes which commit broke it, not the next morning
Test results in a shared folder, or nowhere TRX results and coverage uploaded as an artifact, even when tests fail The evidence for a red build is one click away for 14 days
A hand-written Dockerfile one person maintains dotnet publish /t:PublishContainer, no Dockerfile The SDK picks a secure base image, a non-root user and the right port
Infrastructure scripts tested by running them in production az bicep lint and az bicep build in CI Mistakes in infrastructure code fail the PR, not the deployment
A build service account with admin rights permissions: contents: read for the workflow token A compromised step can read the repo, and that's all

🧠 Anatomy of a workflow

💡 A workflow is a file that answers four questions: when does it run, with which permissions, in what order, and where.

GitHub Actions runs workflows (YAML files in .github/workflows). A workflow has triggers (on:), one or more jobs that each run on a fresh runner machine, and steps inside each job that either run a shell command (run:) or call a reusable action (uses:). Jobs run in parallel unless one needs another.

💻 In Ledgerly: the top of ci.yml

📄 .github/workflows/ci.yml

# CI: every push and pull request is built and tested. A red build blocks the merge (see branch protection, chapter 8).
name: ci

on:
  push:
    branches: [main]
    paths: ["sample/**", ".github/workflows/ci.yml"]
  pull_request:
    paths: ["sample/**", ".github/workflows/ci.yml"]
  workflow_dispatch:

# Least privilege: this workflow only reads the repo.
permissions:
  contents: read

# A new push to the same branch cancels the run that is now outdated.
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

defaults:
  run:
    working-directory: sample

env:
  DOTNET_NOLOGO: true
  DOTNET_CLI_TELEMETRY_OPTOUT: true

Line by line:

Compare the deploy workflow's concurrency block:

📄 .github/workflows/deploy.yml

# Never run two deployments at once, and never cancel one halfway.
concurrency:
  group: deploy-production
  cancel-in-progress: false

Cancelling an outdated build is free. Cancelling a deployment halfway leaves production in a state nobody tested. One fixed group name, no cancellation: deployments wait in line (chapter 12).


🧠 The build-test job: the commands you already know

💡 CI should run the same commands you run locally. If the workflow needs a trick your laptop doesn't, one of them is lying.

💻 In Ledgerly: build-test

📄 .github/workflows/ci.yml

  build-test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7

      - uses: actions/setup-dotnet@v6
        with:
          global-json-file: sample/global.json

      - name: Restore
        run: dotnet restore Ledgerly.slnx

      # Warnings are errors, analyzers + code style run, NuGet audit fails on vulnerable packages.
      - name: Build
        run: dotnet build Ledgerly.slnx -c Release --no-restore

      # Integration tests start a throwaway SQL Server container with Testcontainers (Docker is on the runner).
      - name: Test
        run: >
          dotnet test --solution Ledgerly.slnx -c Release --no-build
          --coverage --coverage-output-format cobertura
          --report-xunit-trx

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: test-results
          path: sample/TestResults/
          retention-days: 14

The steps, in order:

  1. actions/checkout@v7 clones the commit under test.
  2. actions/setup-dotnet@v6 installs the SDK. Instead of a version number in YAML, it reads global.json, the same file your local dotnet reads. Upgrade the SDK in one place and both follow.
  3. Restore, then Build with --no-restore, then Test with --no-build. Three steps instead of one so the log shows exactly which phase failed, and nothing is done twice.
  4. timeout-minutes: 20. A hung test (a container that never starts) fails after 20 minutes instead of burning runner time for hours.
  5. if: always() on the upload. Without it, a failed test step skips every later step, and you lose the results exactly when you need them.

The > after run: is YAML's folded style: the three lines become one command, which keeps a long command readable.

The SDK pin itself:

📄 sample/global.json

{
  "sdk": {
    "version": "10.0.100",
    "rollForward": "latestFeature"
  },
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}

rollForward: latestFeature means "10.0.100 or any newer 10.0 SDK", so a patched SDK on the runner is used without a commit. The test section is the second real lesson below.

Lesson 1: NuGet audit failed the build

The Build step's comment promises that warnings are errors and NuGet audit fails on vulnerable packages. Both come from the repo-wide build settings (chapter 3):

📄 sample/Directory.Build.props

    <!-- Quality gates: warnings fail the build, analyzers and code style run in every build (also in CI). -->
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <AnalysisLevel>latest-recommended</AnalysisLevel>
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>

NuGet audit checks every package in the graph during restore, including the ones you never referenced yourself. While the sample was being built, a transitive dependency, System.Security.Cryptography.Pkcs (versions 9.0.11 and 10.0.12 came in through other packages), had known vulnerabilities: NU1902 (moderate) and NU1903 (high). With TreatWarningsAsErrors, those warnings stopped the build.

The tempting fix is to turn audit off or downgrade the warning. The right fix is to pin the patched version with Central Package Management's transitive pinning:

📄 sample/Directory.Packages.props

  <PropertyGroup>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    <!-- Lets a PackageVersion below also override a TRANSITIVE dependency (used for security patches). -->
    <CentralPackageTransitivePinningEnabled>true</CentralPackageTransitivePinningEnabled>
  </PropertyGroup>

  <ItemGroup Label="Security pins for transitive dependencies (NuGet audit, NU1902/NU1903)">
    <PackageVersion Include="System.Security.Cryptography.Pkcs" Version="10.0.14" />
  </ItemGroup>

The label says why the pin exists, so whoever removes it later (when the parent packages ship the fix) knows what to check. This is why a red build with no code change is a feature: the vulnerability was found by a robot during a pull request, not by an attacker.

Lesson 2: VSTest is gone, Microsoft.Testing.Platform is in

The second failure happened when the team moved to the .NET 10 SDK with xUnit v3 4.x. dotnet test stopped with:

Testing with VSTest target is no longer supported by Microsoft.Testing.Platform on .NET 10 SDK

Some background. dotnet test now has two modes. VSTest mode is the old one and still the default. Microsoft.Testing.Platform (MTP) mode is new in the .NET 10 SDK: the test runner is embedded in the test project, which is an executable. xUnit v3 supports MTP natively, and running MTP test projects through the VSTest bridge is now legacy (MTP v2 drops it on the .NET 10 SDK).

The fix had three parts:

  1. Opt in to MTP with "test": { "runner": "Microsoft.Testing.Platform" } in global.json (above).
  2. Use the new command line: dotnet test --solution Ledgerly.slnx. In MTP mode the solution is passed with --solution.
  3. Drop the VSTest packages. Microsoft.NET.Test.Sdk, xunit.runner.visualstudio and coverlet.collector were removed. Coverage now comes from Microsoft.Testing.Extensions.CodeCoverage, which provides --coverage:

📄 sample/tests/Ledgerly.UnitTests/Ledgerly.UnitTests.csproj

  <PropertyGroup>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
    <OutputType>Exe</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="xunit.v3" />
    <PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" />
    <PackageReference Include="Microsoft.Extensions.TimeProvider.Testing" />
  </ItemGroup>

In the Test step, --coverage --coverage-output-format cobertura writes a Cobertura coverage file, and --report-xunit-trx writes xUnit's TRX results file. The suite has 23 tests: 15 unit tests and 8 integration tests against a real SQL Server that Testcontainers starts on the runner (Docker is already installed there). Locally, the integration tests skip themselves when no SQL Server is available (chapter 17).

One more MTP detail bit the team: when zero tests run, MTP returns exit code 8, and CI fails. That's usually right, because a filter that matches nothing or a project that isn't discovered should be loud. If a run with no tests is genuinely expected, MTP's --ignore-exit-code 8 says so explicitly.

Test results without a test UI

GitHub Actions has no built-in test results page beyond job summaries and artifacts. You have three options: upload the TRX files with actions/upload-artifact (what Ledgerly does), write a Markdown summary to $GITHUB_STEP_SUMMARY, or use a community action that renders TRX files as a check. The artifact keeps the evidence for retention-days: 14, long enough to investigate a flaky test without paying to store every run forever 💰.


🗺️ What one pull request runs

flowchart LR
    PR["Pull request<br/>touches sample/**"] --> BT["build-test<br/>restore, build, test"]
    PR --> Infra["infra<br/>bicep lint and build"]
    PR --> CQ["analyze<br/>codeql.yml"]
    BT -- "needs" --> C1["containers<br/>Ledgerly.Api"]
    BT -- "needs" --> C2["containers<br/>Ledgerly.Worker"]
    BT -- "needs" --> C3["containers<br/>Ledgerly.MarketData"]
    BT -- "needs" --> C4["containers<br/>Ledgerly.MigrationService"]
    BT -- "always" --> Art["Artifact<br/>test-results, 14 days"]
    C1 --> Checks["Required status checks<br/>ruleset on main"]
    C2 --> Checks
    C3 --> Checks
    C4 --> Checks
    Infra --> Checks
    CQ --> Checks

build-test, infra and analyze start in parallel on separate runners. The four containers jobs wait for build-test, because there's no point packaging code whose tests fail.


🧠 Container images without a Dockerfile

💡 The .NET SDK knows your app better than a Dockerfile does. Let it build the image.

The .NET SDK can build an OCI container image directly: dotnet publish with the PublishContainer target. No Dockerfile, no docker build. Without a registry it loads the image into the local Docker or Podman daemon. With -p:ContainerRegistry=... it pushes straight to a registry such as Azure Container Registry (SDK container publishing). Aspire and azd use the same mechanism for .NET projects.

💻 In Ledgerly: the containers matrix

📄 .github/workflows/ci.yml

  # Proves the deployable images build. Pushing happens only in deploy.yml.
  containers:
    runs-on: ubuntu-latest
    needs: build-test
    timeout-minutes: 20
    strategy:
      matrix:
        project: [Ledgerly.Api, Ledgerly.Worker, Ledgerly.MarketData, Ledgerly.MigrationService]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          global-json-file: sample/global.json
      - name: Build container image (no Dockerfile needed)
        run: dotnet publish src/${{ matrix.project }} -c Release /t:PublishContainer -p:ContainerImageTag=ci-${{ github.sha }}

A matrix turns one job definition into one job per value: four projects, four parallel jobs, four checks. Each image is tagged ci-<commit SHA>, so you can tell exactly which commit produced it. The image name comes from the project file:

📄 sample/src/Ledgerly.Api/Ledgerly.Api.csproj

  <PropertyGroup>
    <UserSecretsId>ledgerly-api-6f1d2c</UserSecretsId>
    <!-- SDK container support: `dotnet publish /t:PublishContainer` builds an image without a Dockerfile. -->
    <ContainerRepository>ledgerly-api</ContainerRepository>
  </PropertyGroup>

The other three set ledgerly-worker, ledgerly-marketdata and ledgerly-migrations. Without ContainerRepository, the name would default to the assembly name.

CI builds the images but never pushes them: it has no registry credentials and only contents: read. The deploy workflow runs the same command with a registry, after signing in to Azure with OIDC:

📄 .github/workflows/deploy.yml

          for project in Ledgerly.Api Ledgerly.Worker Ledgerly.MarketData Ledgerly.MigrationService; do
            dotnet publish "src/$project" -c Release /t:PublishContainer \
              -p:ContainerRegistry="$LOGIN_SERVER" -p:ContainerImageTag="$SHA"
          done

What you get for free

The SDK makes choices that a hand-written Dockerfile often gets wrong:

📄 sample/infra/modules/apps.bicep

var probes = [
  {
    type: 'Startup'
    httpGet: { path: '/alive', port: 8080 }
    // ...
  }
  // ...
]

Going smaller: chiseled images

Chiseled images (Ubuntu's distroless flavor) have no shell, no package manager, run non-root by default and contain only the packages the runtime needs. Less inside the image means fewer CVEs to patch. You choose them with ContainerFamily, which is appended to the tag (noble-chiseled gives mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled):

<!-- illustrative, not in the sample: smaller, shell-less base images -->
<PropertyGroup>
  <ContainerFamily>noble-chiseled-extra</ContainerFamily>
</PropertyGroup>

Plain chiseled images have no ICU (globalization) or time zone data, so culture-aware formatting and time zone conversions stop behaving like they do on your laptop. Ledgerly would use the -extra variant, which adds both. The alternative is InvariantGlobalization=true, which is only right for apps that never need culture-specific behavior. 🔧 And no shell means no docker exec ... bash for debugging, which is exactly the point.

One escaping trap: to set several tags, use ContainerImageTags (plural), which takes a semicolon-separated list. In a shell, an unquoted semicolon ends the command, so quote the value carefully (Container publish configuration).


🧠 Infrastructure is code, so CI lints it

💡 Bicep is a compiler. Use it on every pull request, not just on deployment day.

💻 In Ledgerly: the infra job

📄 .github/workflows/ci.yml

  # Infrastructure is code, so it gets linted like code. (Deploying it happens in deploy.yml.)
  infra:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
      - name: Lint and build Bicep
        run: |
          az bicep install
          az bicep lint --file infra/main.bicep
          az bicep build --file infra/main.bicep --stdout > /dev/null
          az bicep build-params --file infra/main.bicepparam --stdout > /dev/null

The job needs no Azure login: linting and compiling are local operations, so it runs with the same read-only token as everything else. lint applies the linter rules, build compiles main.bicep and all its modules to an ARM template (thrown away with > /dev/null, since only success matters), and build-params checks that main.bicepparam matches the template's parameters.

This job exists because of a real lesson. While writing the infrastructure, the Bicep compiler caught two mistakes before Azure did:

Neither needed a subscription to find. Chapter 11 goes through the Bicep itself.


🗺️ From push to green

sequenceDiagram
    participant Dev as You
    participant GH as GitHub
    participant R as Runners
    participant RS as Ruleset on main
    Dev->>GH: Push commit A to the PR branch
    GH->>R: Start ci run for commit A
    Dev->>GH: Push commit B a minute later
    GH->>R: Cancel run A, same concurrency group
    GH->>R: Start ci run for commit B
    R->>R: build-test, infra and analyze in parallel
    R->>R: containers matrix after build-test
    R-->>GH: Upload test-results artifact
    R-->>GH: Report checks for commit B
    GH->>RS: Are all required checks green on the latest commit
    RS-->>Dev: Merge allowed after approval

🧠 Keeping CI fast, current and trustworthy

💡 Your workflow runs third-party code with access to your repository. Treat it like a dependency, because it is one.

Action versions and pinning

Every uses: line runs code someone else wrote. Ledgerly uses these versions 🔧:

Action Version in the sample
actions/checkout v7
actions/setup-dotnet v6
actions/upload-artifact v7
github/codeql-action (init and analyze) v4
azure/login (deploy only) v3

These are major version tags: @v7 moves to each new 7.x release. You get fixes without editing the file, and you trust the publisher not to break or compromise the tag. The alternative is pinning to a full commit SHA, which can never change under you, at the price of less readable files and more update PRs. There's no single right answer. A common split is major tags for actions published by GitHub and Microsoft, and SHA pins for smaller third-party actions. Never use a branch ref like @master: for azure/login, branch refs aren't supported at all.

Either way, Dependabot keeps them current. The github-actions ecosystem in .github/dependabot.yml (chapter 8) groups every action update into one weekly PR, and CI tests it like any other change.

Caching and lock files

The sample doesn't cache NuGet packages, and RestorePackagesWithLockFile is false in Directory.Build.props. On a fresh runner, restore downloads everything each time. For a solution this size that's a small part of the run, and it keeps the workflow simple. If restore time grows, measure first, then add a cache. A cache needs a stable key, which is the usual reason teams turn on NuGet lock files (packages.lock.json) at the same moment. Treat it as an optimization you add with numbers, not a default.

The rest of the safety net


⚠️ Common traps

Trap What happens Do this instead
Running VSTest-style dotnet test on the .NET 10 SDK with xUnit v3 4.x "Testing with VSTest target is no longer supported by Microsoft.Testing.Platform on .NET 10 SDK" Opt in to MTP in global.json, use dotnet test --solution, and remove Microsoft.NET.Test.Sdk, xunit.runner.visualstudio and coverlet.collector
Silencing NuGet audit when it breaks the build The vulnerable package ships to production Pin the patched version with CPM transitive pinning, with a label that says why
A test filter or project setup that runs zero tests MTP exits with code 8 and CI goes red Fix the filter or discovery. Use --ignore-exit-code 8 only when zero tests is truly expected
Uploading test results without if: always() The upload is skipped when tests fail, the one time you need it Add if: always() to every evidence-gathering step
No permissions: block The token gets the repository's default permissions, possibly write access Start every workflow with contents: read and widen per job
cancel-in-progress: true on a deployment A newer push kills a deployment halfway Cancel outdated builds, queue deployments (cancel-in-progress: false)
Passing several ContainerImageTags unquoted in a shell The shell treats ; as the end of the command Quote the property value, or use a single ContainerImageTag per publish

🧰 Helpers

Run what CI runs, from your machine:

cd sample
dotnet restore Ledgerly.slnx
dotnet build Ledgerly.slnx -c Release --no-restore
dotnet test --solution Ledgerly.slnx -c Release --no-build --coverage --coverage-output-format cobertura --report-xunit-trx
dotnet publish src/Ledgerly.Api -c Release /t:PublishContainer -p:ContainerImageTag=local

Watch and debug runs with the GitHub CLI:

gh workflow run ci                 # start CI by hand (workflow_dispatch)
gh run list --workflow ci          # recent runs
gh run watch                       # follow a run live
gh run view --log-failed           # only the logs of failed steps
gh run download <run-id> -n test-results   # get the TRX and coverage files
gh pr checks                       # check status for the current PR

A job summary, for when the artifact is too far away:

# illustrative, not in the sample: a one-line summary on the run page
      - name: Summary
        if: always()
        run: echo "Tests finished for commit ${{ github.sha }}. Results are in the test-results artifact." >> "$GITHUB_STEP_SUMMARY"

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

  1. Run the CI commands locally, in order: bash cd sample dotnet restore Ledgerly.slnx dotnet build Ledgerly.slnx -c Release --no-restore dotnet test --solution Ledgerly.slnx -c Release --no-build --coverage --coverage-output-format cobertura --report-xunit-trx With Docker running, all 23 tests run. Without it, notice which integration tests skip.
  2. Build a container image without a Dockerfile: bash dotnet publish src/Ledgerly.Api -c Release /t:PublishContainer -p:ContainerImageTag=local docker image inspect ledgerly-api:local In the output, find the user (Config.User) and the exposed port (Config.ExposedPorts). Compare them with the probes in apps.bicep.
  3. In your fork from chapter 8, start CI by hand and follow it: bash gh workflow run ci gh run watch
  4. Break a test on purpose (change an expected value in a unit test), push to a PR branch, and read the failure with gh run view --log-failed. Download the test-results artifact and open the TRX file.
  5. Push two commits quickly to the same branch. In the Actions tab, watch the first run get cancelled by the concurrency group.
  6. Edit only a file under chapters/ in a separate branch and open a PR. CI doesn't start, because of the paths filter. If you made build-test a required check in chapter 8, notice that the PR now waits for a check that will never run.
  7. Revert the broken test and watch the PR turn green.

✅ Recap

  1. Ledgerly's CI runs the same restore, build and test commands you run locally, on a fresh runner, with least-privilege permissions and cancellation of outdated runs.
  2. Real failures shaped it: NuGet audit stopped a vulnerable transitive package, the .NET 10 SDK required Microsoft.Testing.Platform, and Bicep lint catches infrastructure mistakes before Azure does.
  3. The SDK builds secure, non-root container images on port 8080 without a Dockerfile, and CI proves every image builds before deploy ever pushes one.

Next → Chapter 10: Choosing Azure services

Chapter 10: Choosing Azure services

⏱ About 30 min · You'll learn: how to choose compute, data, messaging, caching, secrets and observability services on Azure with decision tables, why Ledgerly runs on Azure Container Apps, and which cost drivers actually move the monthly bill.


📖 Story: The Azure menu

Elena Rossi walks into standup with a spreadsheet open on her laptop. "The beta opens in eight weeks. Finance wants to know what it costs per month. I don't need a number to the cent. I need a range, and I need to know what makes it go up."

Maya Okafor books the small meeting room for an hour and writes AZURE MENU across the top of the whiteboard. "Azure has a couple of hundred services. We need six kinds." She draws six columns: compute, data, messaging, caching, secrets, observability. "For each column we pick one dish, write down why, and write down what makes it expensive. Elena gets cost drivers, not a guess."

You open your notebook. At Nordbank Insurance, choosing infrastructure meant a ticket, a three-week wait and a VM sized for the one busy week in January. Under Habits I brought with me you write: "Size for peak, then forget about it."

The numbers in this chapter's story (weeks, users, percentages) are illustrative. The service limits and free grants are real and marked 🔧.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Order a VM sized for peak load from the ops team Pick a managed service that bills for use and scales with rules Idle capacity costs money every hour, and a scale rule reacts faster than a purchase order
One IIS site per app on a shared server farm One Container App per service in a shared Container Apps environment Same "shared host" idea, but the platform owns the OS, patching, TLS and scaling
SQL Server on a VM, looked after by a DBA Azure SQL Database, serverless tier Same engine and T-SQL, with backups and patching managed and compute that pauses when idle
MSMQ installed on each server Azure Service Bus queues One durable queue shared by every replica, with dead-lettering and duplicate detection built in
MemoryCache per server, or a self-hosted cache cluster HybridCache in memory, plus Azure Managed Redis only when replicas must share a cache Two levels with stampede protection, and you pay for a shared cache only when you need one
Passwords in web.config, encrypted with aspnet_regiis Managed identities for almost everything, Key Vault (RBAC mode) for the few secrets left A password can leak, an identity can't be pasted into a chat
Perfmon counters and log files on each server Application Insights backed by a Log Analytics workspace One place to query every replica, with costs you can cap

💡 In the cloud, the size of the bill is decided by your architecture choices, not by a hardware order.


🧠 How the menu is organized

Before choosing dishes, learn how the menu is laid out. Four concepts decide where a resource lives, who can touch it and who pays for it.

Concept What it is Old-world equivalent Ledgerly's choice
Subscription A billing and access boundary. Quotas and several free grants are counted per subscription A cost center with its own budget line One subscription for the beta
Resource group A folder of resources that share a lifecycle. You deploy, grant access to and delete them together "All the servers for app X" in the CMDB One resource group per environment, for example rg-ledgerly-prod
Region The datacenter location of a resource. It affects latency, data residency and which SKUs exist Your own datacenter The resource group's region. Everything stays in one region
Tags Key/value labels on resources. Cost analysis can group and filter by them A column in the asset spreadsheet app and managed-by on every resource

Pick a region close to your users and keep every service in it, so calls between them stay short (not every SKU exists in every region, so check before you commit). Put tags in code so they can't be forgotten: cost analysis can group by them (cost management).

💻 In Ledgerly: one region, one name, one set of tags

📄 sample/infra/main.bicep

param location string = resourceGroup().location
// ...
var name = '${appName}-${take(uniqueString(resourceGroup().id), 6)}'
var tags = {
  app: appName
  'managed-by': 'bicep'
}

💡 A resource group is a lifecycle boundary: choose its contents by asking "would I delete these together?"


🧠 Compute: where your code runs

This is the biggest decision on the menu, and the one with the most opinions attached. Microsoft publishes a comparison of its container and compute options (compare options). Here it is as a decision table.

Option What it is Use it when Avoid it when
Azure App Service Fully managed hosting optimized for web apps, from code or a container. The closest thing to your IIS mental model You have one or a few web apps and want "a site" with as little platform thinking as possible You have several cooperating services, background workers scaled by queue length, and run-to-completion jobs
Azure Container Apps (ACA) Serverless containers built on Kubernetes, KEDA, Dapr and Envoy, with no direct Kubernetes API APIs, microservices, event-driven workers and jobs, including scale to zero You need the Kubernetes API itself: custom operators, CRDs or cluster-level control
Azure Kubernetes Service (AKS) Full Kubernetes API control. Offered as Standard or AKS Automatic 🔧 A platform team already runs Kubernetes, or you need what only the Kubernetes API offers A small team without Kubernetes skills that just wants to ship an API
Azure Functions, Flex Consumption 🔧 Functions as a service. Flex Consumption is the recommended serverless plan: Linux, scale to zero, VNet integration and "always ready" instances. It replaces Linux Consumption for new apps Small event handlers and glue code: react to a blob, a queue message or a timer A normal ASP.NET Core API you also run locally with Aspire

Sources: compare options and Flex Consumption plan.

Why Ledgerly chose Container Apps

Maya writes the requirements on the left of the board and the matching ACA feature on the right. The result becomes an ADR (use the ADR template from the toolkit).

Ledgerly needs Container Apps gives
Three long-running services (api, worker, marketdata) and one run-to-completion task (migrations) Container apps for the services and a Container Apps job with a manual trigger for migrations
The worker must scale with the import queue, not with HTTP traffic KEDA scale rules, including a Service Bus queue-length scaler
The beta API is idle at night Scale to zero: with min replicas 0 and no traffic, no usage charges 🔧
The fake market-data vendor must not be reachable from the internet Internal ingress, visible only inside the environment
Deploys without downtime Revisions: in single revision mode the new revision replaces the old one with zero downtime
A small team with no Kubernetes specialists Kubernetes underneath, but no cluster to run or upgrade

The trade-off Maya writes underneath: cold starts. A service scaled to zero takes a moment to wake up. The sample makes it a parameter (apiMinReplicas) so you can change your mind without changing code (overview, scaling).

💻 In Ledgerly: one environment, the serverless profile

📄 sample/infra/modules/environment.bicep

resource environment 'Microsoft.App/managedEnvironments@2025-01-01' = {
  name: 'cae-${name}'
  location: location
  tags: tags
  properties: {
    appLogsConfiguration: {
      destination: 'log-analytics'
      // ...
    }
    workloadProfiles: [
      { name: 'Consumption', workloadProfileType: 'Consumption' } // 💰 serverless, pay per use, free monthly grant
    ]
  }
}

The environment is the shared boundary: one network and one Log Analytics workspace for all apps and jobs inside it. Think of it as the IIS server farm, except that you never see the servers. The Consumption workload profile is the serverless option; Dedicated profiles reserve hardware when you need it.

💡 Choose the platform that matches the shape of your work: one web app, many event handlers, or a handful of services, workers and jobs.


🗺️ Picking compute in four questions

flowchart TB
    Start["A new service"] --> K8s{"Do you need the<br/>Kubernetes API itself?"}
    K8s -- "yes" --> AKS["AKS<br/>Standard or Automatic"]
    K8s -- "no" --> Shape{"What shape<br/>is the work?"}
    Shape -- "one web app,<br/>IIS-like" --> AppSvc["App Service"]
    Shape -- "small event<br/>handlers" --> Func["Functions<br/>Flex Consumption"]
    Shape -- "APIs, workers<br/>and jobs" --> ACA["Container Apps"]
    ACA --> Idle{"Idle for<br/>long periods?"}
    Idle -- "yes" --> Zero["min replicas 0<br/>accept cold starts"]
    Idle -- "no, or must<br/>poll on a timer" --> One["min replicas 1"]

🧠 Data, messaging and caching

Data: Azure SQL Database, serverless

Your SQL Server knowledge transfers directly, which is the main reason Ledgerly didn't look further. The interesting decision is the purchasing model.

Feature What it means 🔧 Why it matters
Serverless compute Compute scales between a min and max vCore count (min can be 0.5 vCore), billed per vCore-second Light traffic costs little, and there's no instance to size
Auto-pause General Purpose serverless databases pause after an idle delay. The default is 60 minutes, the minimum is 15, and -1 disables it 💰 Nights and weekends cost only storage. The first connection after a pause waits for a resume, so the app needs retries
Free offer Per database, per month: 100,000 vCore-seconds, 32 GB of data and 32 GB of backup storage, for the life of the subscription. Up to 10 free databases per subscription 💰 A beta can run inside it. When the monthly allowance is used up, the database is either paused until next month or billed for the overage, depending on a setting
Entra-only authentication Disables SQL logins, so only Microsoft Entra ID principals can sign in No database password exists to leak (chapter 13)

Sources: serverless tier, free offer, Entra-only authentication. Serverless fits low or bursty traffic. A database that is busy around the clock is more predictable on a provisioned tier.

📄 sample/infra/modules/sql.bicep

  sku: {
    name: 'GP_S_Gen5_2'
    tier: 'GeneralPurpose'
    family: 'Gen5'
    capacity: 2
  }
  properties: {
    autoPauseDelay: 60
    minCapacity: json('0.5')
    useFreeLimit: useFreeLimit // 💰 100,000 vCore-seconds per month, then pause until next month
    freeLimitExhaustionBehavior: useFreeLimit ? 'AutoPause' : null
  }

GP_S_Gen5_2 reads as General Purpose, Serverless, Gen5, max 2 vCores. Elena spots the catch: "Pause until next month means the beta is down until next month?" Yes. Fine for learning, bad once real users depend on it, so Maya adds a launch-checklist item: set sqlUseFreeLimit = false before the public beta. The migration service already copes with resume delays: "The execution strategy retries transient failures (e.g. Azure SQL serverless waking up)."

Messaging: Azure Service Bus tiers

Chapter 7 built the import pipeline on a queue. Here is how to pick the tier.

Basic Standard Premium
Queues ✅ ✅ ✅
Topics and subscriptions ❌ ✅ ✅
Sessions, transactions, duplicate detection ❌ ✅ ✅
Capacity Shared Shared, pay per operation Dedicated messaging units
Max message size 🔧 256 KB 256 KB Up to 100 MB
VNet / Private Link, geo-replication ❌ ❌ ✅

Source: Service Bus Premium and tiers.

Ledgerly's outbox relay sets MessageId to the outbox row id, and the queue drops any message with an id it has seen in the last ten minutes (duplicate detection). That one feature rules out Basic. Nothing in the beta needs dedicated capacity, private networking or 100 MB messages, which rules out Premium for now. Standard it is.

📄 sample/infra/modules/servicebus.bicep

// Service Bus Standard (needed for duplicate detection). Local (SAS key) auth is disabled:
// only Entra identities with a data role can send or receive.
// ...
  sku: { name: 'Standard', tier: 'Standard' }
  properties: {
    disableLocalAuth: true
    minimumTlsVersion: '1.2'
  }

Duplicate detection is configured on the entity when it is created, so this is a decision you want right on day one.

Caching: HybridCache first, Azure Managed Redis when you need it

Two facts shape this decision.

First, Azure Cache for Redis is being retired 🔧 (retirement FAQ):

Azure Cache for Redis tiers Retirement date 🔧 Instances disabled from 🔧
Enterprise and Enterprise Flash 2027-03-31 2027-04-01
Basic, Standard and Premium 2028-09-30 2028-10-01

New instance creation is being restricted before those dates. Don't start new work on it, even if a blog post from 2023 tells you to.

Second, the replacement is Azure Managed Redis 🔧 (overview). It is built on Redis Enterprise, comes in Memory Optimized, Balanced, Compute Optimized and Flash Optimized SKUs, is clustered by default (a non-clustered option exists up to 25 GB), is zone-redundant by default and supports Microsoft Entra ID authentication. StackExchange.Redis works with it without special cluster configuration.

But the first question isn't "which Redis?" It's "do we need a shared cache at all?"

Option What it is Use it when Avoid it when
HybridCache, L1 only In-process memory cache with stampede protection (HybridCache) Few replicas, short expirations, and a little staleness between replicas is fine Replicas must agree on cached values, or the cache is expensive to rebuild
HybridCache + Azure Managed Redis as L2 The same API, with a shared distributed cache behind the memory cache Many replicas, expensive values, or cache data that should survive a restart You're adding it "just in case" (💰 it is a paid resource even at its smallest size)
Azure Cache for Redis The retiring service 🔧 Only to keep an existing instance alive while you migrate Any new work

💻 In Ledgerly: Redis is optional by design

📄 sample/src/Ledgerly.Api/Program.cs

// ── Caching: HybridCache (in-memory L1). Redis becomes the shared L2 when a "cache" connection exists.
if (!string.IsNullOrEmpty(builder.Configuration.GetConnectionString("cache")))
{
    builder.AddRedisDistributedCache("cache");
}

builder.Services.AddHybridCache(options => options.DefaultEntryOptions = new HybridCacheEntryOptions
{
    Expiration = TimeSpan.FromSeconds(30),
    LocalCacheExpiration = TimeSpan.FromSeconds(10),
});

The code doesn't care which mode it runs in. Locally, Aspire starts a Redis container and passes a cache connection string, so you get both levels. In Azure, enableRedis defaults to false in main.bicep: the API uses memory only, and prices are cached for 30 seconds per replica. If the beta grows, you flip one parameter and the infrastructure creates Azure Managed Redis (Balanced_B0, the smallest size) and wires the connection string through Key Vault.

💡 Start with the cheapest option that is correct, and make the expensive option a parameter rather than a rewrite.


🧠 Secrets and observability

Secrets: Azure Key Vault in RBAC mode

The best secret is one that doesn't exist. Chapter 13 shows how managed identities remove passwords from SQL, Service Bus and the container registry entirely. Key Vault is for what's left: in Ledgerly, exactly one secret, the Redis access key, and only when Redis is enabled.

Decision Choice Why 🔧
Permission model Azure RBAC RBAC is the recommended model and vault access policies are legacy. Starting with ARM API version 2026-02-01, new vaults default to RBAC
Who reads secrets Key Vault Secrets User for the app identity only Reads secret values and nothing else
Who manages secrets Key Vault Secrets Officer or Administrator, for people or pipelines that need it Separate "read" from "manage"
Scope The vault, or narrower Never the whole subscription

Sources: RBAC vs access policies, Key Vault RBAC guide.

Observability: Application Insights and Log Analytics

There is only one shape left for Application Insights: workspace-based. Classic Application Insights resources are retired, and all telemetry is stored in a Log Analytics workspace and billed through it (workspace-based resources). Chapters 15 and 16 cover the telemetry itself. The menu decision here is about cost: Log Analytics ingestion is often the surprise on the bill. Sampling, a daily cap and a sensible retention period are the levers (sampling).

💻 In Ledgerly: a hard daily cap

📄 sample/infra/modules/monitoring.bicep

resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' = {
  name: 'log-${name}'
  location: location
  tags: tags
  properties: {
    sku: { name: 'PerGB2018' }
    retentionInDays: 30
    workspaceCapping: { dailyQuotaGb: 1 } // 💰 a hard daily cap keeps a learning subscription safe
  }
}

resource appInsights 'Microsoft.Insights/components@2020-02-02' = {
  // ...
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: workspace.id
  }
}

💡 Telemetry is a product with a price per GB: decide what you keep, for how long and at what sample rate.


🧠 What actually drives the bill

Now you can answer Elena. Not with a number, but with a table of drivers, where each one lives in Ledgerly's code, and the lever that controls it. Prices change and differ by region, so the source of truth is each service's pricing page and the Azure Pricing Calculator 🔧.

Service What you pay for 💰 Where it is in Ledgerly Lever
Container Apps (Consumption) vCPU-seconds, GiB-seconds and requests, after a monthly free grant per subscription of 180,000 vCPU-seconds, 360,000 GiB-seconds and 2 million requests 🔧. Replicas held warm by min replicas > 0 are billed at a reduced idle rate apps.bicep: api min 0, marketdata min 0, worker min 1 Scale to zero where you can accept a cold start (apiMinReplicas)
Azure SQL serverless vCore-seconds while running, plus storage. Free offer covers 100,000 vCore-seconds a month 🔧 sql.bicep: autoPauseDelay: 60, minCapacity: 0.5, useFreeLimit Auto-pause, free offer, max vCores
Service Bus Standard Shared capacity, pay per operation servicebus.bicep Fewer, larger messages, and no polling loops against the queue
Log Analytics + App Insights GB ingested and retention monitoring.bicep: 1 GB daily cap, 30 days Sampling, daily cap, retention, and not logging every probe
Azure Managed Redis A fixed size that runs all month, even when idle redis.bicep, off unless enableRedis = true Leave it off until you need a shared cache
Container Registry, Key Vault Small, mostly fixed or per-operation charges registry.bicep (Basic), keyvault.bicep Check the pricing pages, then move on

Sources: Container Apps billing, Azure SQL free offer.

Two details hide in that table:

Budgets: an alarm, not a brake

Maya's last item: a budget on the resource group, with alerts on actual or forecasted cost. It's cheap insurance, but budgets don't stop resources: they notify, and stopping things needs automation wired to an action group. Cost data typically lands within 8 to 24 hours and budgets are evaluated every 24 hours, so a runaway loop can run for a day before anyone hears about it (create budgets).

Elena leaves with what she needed: "Most of the beta fits in free grants and serverless pauses. The fixed costs are the worker replica, the registry and Redis if we turn it on. The surprise risk is logs, and that's capped."

💡 You can't predict a cloud bill to the cent, but you can name every driver and put a lever next to it.


🗺️ Ledgerly on Azure, with its cost drivers

flowchart LR
    subgraph RG["Resource group rg-ledgerly-prod, one region"]
        subgraph ENV["Container Apps environment, Consumption profile"]
            Api["api<br/>min 0, max 10"]
            Worker["worker<br/>min 1, max 5"]
            Market["marketdata<br/>internal, min 0, max 3"]
            Job["job-migrations<br/>runs once per deploy"]
        end
        Sql[("Azure SQL serverless<br/>auto-pause, free offer")]
        Bus["Service Bus Standard<br/>pay per operation"]
        Redis[("Azure Managed Redis<br/>optional, off by default")]
        KV["Key Vault<br/>RBAC mode"]
        Logs["Log Analytics + App Insights<br/>1 GB daily cap, 30 days"]
        ACR["Container Registry<br/>Basic"]
    end
    Api -- "EF Core" --> Sql
    Worker -- "EF Core" --> Sql
    Worker -- "imports queue" --> Bus
    Api -. "L2 cache if enabled" .-> Redis
    KV -. "Redis secret reference" .-> Api
    ACR -- "images" --> ENV
    ENV -- "telemetry and logs" --> Logs

⚠️ Common traps

Trap What happens Do this instead
Choosing AKS because "everyone uses Kubernetes" Weeks of cluster plumbing, upgrades and node pools before the first feature ships Start on Container Apps. The concepts (replicas, probes, revisions, KEDA) transfer if you ever need AKS
Creating Azure Cache for Redis for a new project You build on a service with published retirement dates 🔧 and migrate within two years Use Azure Managed Redis, or no shared cache at all until you need one
Picking Service Bus Basic to save money Duplicate detection, topics and sessions are missing, and duplicate detection is set when the entity is created Use Standard when your design depends on duplicate detection or topics
Leaving the SQL free offer on "auto-pause when exhausted" in production The allowance runs out mid-month and the database pauses until next month Use the free offer for learning and early testing, then switch to billed overage before real users arrive
Min replicas 1 on every app "to be safe" Every app is billed at least at the idle rate, all month Use 0 where a cold start is acceptable, 1 only where it isn't or where a timer must run
No daily cap, no sampling, verbose logging in production Log ingestion becomes the largest line on the bill Cap, sample and filter (probes, chatty categories) from day one
Treating a budget as a spending limit The alert email arrives, the resources keep running Wire an action group to automation if you need a brake

🧰 Helpers

A one-page "Azure menu" for any new service. Copy it into the ADR for each choice:

## Service choice: <compute | data | messaging | caching | secrets | observability>
- Requirement in one sentence:
- Options considered: (from the decision table)
- Chosen, and why:
- What makes it expensive, and the lever in Bicep:
- What would make us change our mind:
- 🔧 Volatile facts we relied on, with doc links:

Monthly cost review (15 minutes):

Make it permanent:


🧪 Try it yourself (optional, ~25 min)

No Azure subscription is needed for these steps.

  1. Install the Bicep CLI. If you have the Azure CLI, run az bicep install. Otherwise install the standalone bicep CLI from the Bicep docs.
  2. Find every cost lever in the parameters: bash cd sample grep -n "param " infra/main.bicep Mark the ones that change the bill: enableRedis, sqlUseFreeLimit and apiMinReplicas. Read their @description text.
  3. Find the scale settings for each app: bash grep -n -A2 "scale:" infra/modules/apps.bicep Observe: the API uses apiMinReplicas, the worker is pinned to minReplicas: 1, and marketdata scales from 0 to 3.
  4. Compile the Bicep to the ARM JSON that Azure would receive, and look for every SKU: bash bicep build infra/main.bicep --stdout | grep -n '"sku"' (With the Azure CLI: az bicep build --file infra/main.bicep --stdout.)
  5. Open the Azure Pricing Calculator. Add Container Apps, Azure SQL Database (serverless), Service Bus Standard and Log Analytics. Model a quiet beta month and a busy one. Note which line moves the most between the two.
  6. Write a short ADR: "Why Ledgerly runs on Azure Container Apps", with the cold-start trade-off and the condition that would make you switch apiMinReplicas to 1.

✅ Recap

  1. Pick compute by the shape of the work: App Service for an IIS-like web app, Functions Flex Consumption for event handlers, Container Apps for services, workers and jobs, AKS only when you need the Kubernetes API.
  2. Ledgerly uses Azure SQL serverless, Service Bus Standard (for duplicate detection), HybridCache with optional Azure Managed Redis, Key Vault in RBAC mode and workspace-based Application Insights.
  3. Cost is a design property: scale to zero, free grants, auto-pause and a log ingestion cap are levers in Bicep, and budgets warn but never stop spending.

Next → Chapter 11: Infrastructure as code

Chapter 11: Infrastructure as code

⏱ About 35 min · You'll learn: Bicep basics (parameters, variables, modules, outputs, conditions, loops, existing, .bicepparam), how Ledgerly's main.bicep deploys in three phases, least-privilege role assignments for managed identities, how lint and what-if catch mistakes before Azure does, and how the database user for a managed identity gets created without anyone clicking.


📖 Story: "Show me the pull request"

Maya's Azure menu from chapter 10 is on the wiki. Priya has a free afternoon and offers to "just click it together in the portal, it's maybe twenty resources." Tomás Reyes, who owns the platform, doesn't even look up from his screen. "No."

He turns around. "If I click it, there is no record of what I clicked. Next month someone changes a setting during an incident, nobody remembers, and staging and production quietly drift apart. If it isn't in a pull request, it doesn't exist." He opens an empty file called main.bicep. "We write it down once. Azure reads it. Anyone can review it, rerun it or delete everything and get it back."

You think of the server build document at Nordbank Insurance: a 40-page Word file with screenshots, version 7, last updated by someone who left in 2019. In your Habits I brought with me notebook you write: "The runbook is the documentation." Next to it, Tomás writes in pencil: "The code is the runbook."


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
A Word document with screenshots of every setting A .bicep file in the same repo as the app Text can be reviewed, diffed and versioned, and it can't fall out of date with itself
Click through the portal, then hope staging matches production Deploy the same template with a different .bicepparam file Environments differ only where the parameters say so
"Who changed this?" answered by guessing git log infra/ and the deployment history Every change is a commit with an author and a reason
Ask a DBA to create logins and grant rights by hand Managed identities and role assignments declared in Bicep, the database user created by the migration job Permissions are reviewed in a pull request like any other code
Find typos when the deployment fails at 2 a.m. bicep lint and bicep build in CI, what-if before applying The compiler and a preview catch mistakes before Azure does
One big script that only works on a clean server Idempotent, incremental deployments Running the same template twice changes nothing the second time

🧠 Bicep in ten minutes

Bicep is a declarative language for Azure resources. You describe the end state; Azure Resource Manager (ARM) works out what to create or change. bicep build compiles it to an ARM JSON template, and that JSON is what Azure actually receives. For a C# developer, it reads like a typed configuration file with functions.

Concept What it does Ledgerly example
targetScope Where the template deploys: a resource group, subscription, management group or tenant targetScope = 'resourceGroup'
param + decorators An input, with type, default and validation @minLength(3) @maxLength(12) param appName string = 'ledgerly'
var A computed value, like a const var name = '${appName}-${take(uniqueString(resourceGroup().id), 6)}'
resource One Azure resource: symbolic name, type and API version resource registry 'Microsoft.ContainerRegistry/registries@2023-07-01'
module Another Bicep file, called like a function with params module sql 'modules/sql.bicep' = { ... }
output A value returned to the caller or the CLI output loginServer string = registry.properties.loginServer
Condition Deploy only if an expression is true module redis 'modules/redis.bicep' = if (enableRedis) { ... }
Loop Create one item per element of an array [for principalId in pullPrincipalIds: { ... }]
existing Reference a resource you didn't create in this file, to read its properties or nest children resource vault '...' existing = { name: keyVaultName }
parent Declare a child resource without spelling out the full type path The imports queue has parent: namespace
.bicepparam The modern parameter file, type-checked against the template using 'main.bicep' then param appName = 'ledgerly'

Two details trip up every newcomer:

💻 In Ledgerly: parameters that explain themselves

📄 sample/infra/main.bicep

@description('Short name used in every resource name.')
@minLength(3)
@maxLength(12)
param appName string = 'ledgerly'

param location string = resourceGroup().location

@description('Container image tag (the git commit SHA). Empty = deploy the platform only.')
param imageTag string = ''

@description('Deploy the long-running apps. The pipeline sets this after the migration job succeeded.')
param deployApps bool = false
// ...
@description('0 = scale to zero (cheapest), 1 = always warm (no cold start).')
@minValue(0)
param apiMinReplicas int = 0

📄 sample/infra/main.bicepparam

// Values for the "production" environment. Secrets never go here: there are none to pass,
// because every service authenticates with a managed identity.
using 'main.bicep'

param appName = 'ledgerly'
param jwtAuthority = 'https://login.microsoftonline.com/<your-tenant-id>/v2.0'
param jwtAudience = 'api://ledgerly-api'
param corsOrigins = []
param enableRedis = false
param apiMinReplicas = 0

// imageTag and deployApps are passed by the pipeline on the command line.

using 'main.bicep' binds the file to the template, so a typo like param apiMinReplica = 0 is a compile error, not a silently ignored value. A second environment is a second file (staging.bicepparam), not a second template. Before your first real deployment, replace <your-tenant-id> with your tenant id (chapter 12).

💡 A good parameter file reads like the environment's spec sheet: everything that differs, nothing that doesn't, and no secrets.


🧠 One template, three phases

A naive template deploys everything at once. That fails on day one (the apps reference an image tag that doesn't exist yet in a registry the same deployment creates), and it's wrong after day one (the new API would start before the migrations ran).

Ledgerly's main.bicep solves both with two parameters and two conditions. The header comment is the design document.

💻 In Ledgerly: phases as conditions

📄 sample/infra/main.bicep

// Ledgerly on Azure. Resource-group scope; the deploy workflow runs it in three phases:
//   1. imageTag = ''                       -> platform only (registry, data, messaging, environment)
//   2. imageTag = <sha>, deployApps = false -> + the migration job with the new image (pipeline runs it)
//   3. imageTag = <sha>, deployApps = true  -> + api, worker, marketdata on the new image
// Deployments are incremental and idempotent: re-running a phase changes only what differs.
targetScope = 'resourceGroup'
// ...
module migrationJob 'modules/migration-job.bicep' = if (!empty(imageTag)) {
  name: 'migration-job'
  params: {
    // ...
    image: '${registry.outputs.loginServer}/ledgerly-migrations:${imageTag}'
    identityId: identities.outputs.migratorId
    // ...
  }
}

module apps 'modules/apps.bicep' = if (deployApps && !empty(imageTag)) {
  name: 'apps'
  params: {
    // ...
    cacheSecretUri: enableRedis ? redis!.outputs.secretUri : ''
    // ...
  }
}
// ...
output apiUrl string = (deployApps && !empty(imageTag)) ? apps!.outputs.apiUrl : ''
Phase Parameters What Azure has afterwards What the pipeline does next
1 imageTag = '' Monitoring, identities, registry, Key Vault, Service Bus, SQL, the Container Apps environment (and Redis if enabled) Builds the four images and pushes them, tagged with the commit SHA
2 imageTag = <sha>, deployApps = false Phase 1, plus job-migrations on the new image Starts the job and waits for success
3 imageTag = <sha>, deployApps = true Everything, with api, worker and marketdata on the new image Smoke-tests the API

Three Bicep details make this work:

Faster paths, and why the course doesn't take them

You don't have to write Bicep by hand. Know the alternatives:

Tool What it does 🔧 Why Ledgerly hand-writes instead
aspire deploy Provisions Azure resources and deploys containers from the AppHost model. Now the recommended path for Aspire apps. aspire publish emits the generated artifacts, including Bicep. AddAzureContainerAppEnvironment and PublishAsAzureContainerApp customize the output (deploy with Aspire, Container Apps) Great for getting started. The course wants you to see every resource, role and setting, so you can review what a generator gives you
Azure Developer CLI (azd), version 1.37.1 azd up provisions (infra/) and deploys from azure.yaml (host: containerapp). azd pipeline config creates OIDC credentials and repository variables. For Aspire it is "still supported but no longer the recommended default" (azd overview) Same reason. azd pipeline config is still a handy shortcut for the setup in chapter 12
Azure Verified Modules (AVM) Microsoft-maintained modules in the public registry, for example br/public:avm/res/storage/storage-account:0.18.0. Resource modules live under avm/res, patterns under avm/ptn. Non-AVM modules have been retired from the public registry (modules) A good option for a team template. Ledgerly's modules are small on purpose, so every line can be read in a course

A practical path: start with aspire publish to see what it generates, then own the Bicep once you need to change it.

💡 Ordering belongs in the template's design, not in a human's memory: conditions plus a pipeline turn "deploy in the right order" into code.


🗺️ The module graph

flowchart TB
    Main["main.bicep<br/>params and phases"]
    Mon["monitoring<br/>Log Analytics, App Insights"]
    Id["identity<br/>app and migrator"]
    Reg["registry<br/>AcrPull for both"]
    KV["keyvault<br/>Secrets User for app"]
    SB["servicebus<br/>Sender and Receiver for app"]
    Sql["sql<br/>migrator is Entra admin"]
    Redis["redis<br/>if enableRedis"]
    Env["environment<br/>Container Apps"]
    Job["migration-job<br/>if imageTag set"]
    Apps["apps<br/>if deployApps and imageTag"]
    Main --> Mon
    Main --> Id
    Id -- "principal ids" --> Reg
    Id -- "principal id" --> KV
    Id -- "principal id" --> SB
    Id -- "migrator name and client id" --> Sql
    KV -- "vault name" --> Redis
    Mon -- "workspace name" --> Env
    Env -- "environment id" --> Job
    Env -- "environment id" --> Apps
    Redis -. "secret URI" .-> Apps

🧠 Identities and least-privilege role assignments

On your old servers, the app pool ran as a domain account and someone added that account to a SQL login and an MSMQ ACL. In Azure, the equivalent is a managed identity: an identity in Microsoft Entra ID whose credentials Azure manages for you. A user-assigned managed identity is a resource you declare and then attach to apps; it survives redeploys and can be granted rights before any app exists. That's why Ledgerly uses user-assigned identities instead of system-assigned ones, which are created and deleted with each app.

💻 In Ledgerly: two identities, two jobs

📄 sample/infra/modules/identity.bicep

// Two user-assigned managed identities. No passwords or connection-string secrets anywhere:
//  - "app": what api, worker and marketdata run as (least privilege: read/write data, send/receive messages).
//  - "migrator": what the migration job runs as. It is the SQL Entra admin, so it can change the schema.
// ...
resource app 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'id-${name}-app'
  location: location
  tags: tags
}

The identity is a principal. Rights come from role assignments: a principal, a role definition and a scope. Ledgerly grants the smallest role at the narrowest scope:

Principal Role Scope Lets it
app, migrator AcrPull The registry Pull images. Nothing else
app Key Vault Secrets User The vault Read secret values. Not list keys, not manage the vault
app Azure Service Bus Data Sender + Data Receiver The namespace Send and receive messages. Not create queues or change settings
app Azure Service Bus Data Owner 🔧 The imports queue only Lets the KEDA scale rule read the queue length (see below)
migrator SQL Entra admin The SQL server Change the schema and create database users
app db_datareader + db_datawriter The ledgerly database Read and write rows. Not alter tables

💻 In Ledgerly: role assignments with guid() names

📄 sample/infra/modules/registry.bicep

var acrPullRoleId = '7f951dda-4ed3-4680-a7ca-43fe172d538d'
// ...
resource acrPull 'Microsoft.Authorization/roleAssignments@2022-04-01' = [for principalId in pullPrincipalIds: {
  name: guid(registry.id, principalId, acrPullRoleId)
  scope: registry
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', acrPullRoleId)
    principalId: principalId
    principalType: 'ServicePrincipal'
  }
}]

Read it line by line:

Service Bus needs two roles per principal, so it builds the list with map and flatten:

📄 sample/infra/modules/servicebus.bicep

resource roles 'Microsoft.Authorization/roleAssignments@2022-04-01' = [for item in flatten(map(dataPrincipalIds, p => [
  { principalId: p, roleId: senderRoleId }
  { principalId: p, roleId: receiverRoleId }
])): {
  name: guid(namespace.id, item.principalId, item.roleId)
  scope: namespace
  // ...
}]

If you know LINQ, this is dataPrincipalIds.SelectMany(p => new[] { (p, sender), (p, receiver) }).

The same file has one more assignment, and it's there because of a scaling detail. The worker's KEDA scale rule (chapter 18) reads the imports queue's message count through the Service Bus management API, and KEDA's docs say that needs Manage rights. Sender and Receiver don't include Manage, so without another role the worker would quietly never scale out. The fix grants Data Owner, but scoped to the one queue rather than the namespace:

📄 sample/infra/modules/servicebus.bicep

// 🔧 KEDA reads the queue's message count through the management API, which needs Manage rights.
// Sender and Receiver don't include them, so grant Data Owner on this one queue only, not the namespace.
resource scalerRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(imports.id, scalerPrincipalId, dataOwnerRoleId)
  scope: imports
  // ...
}

scope: imports works because a queue is a child resource with its own id, and Azure RBAC can scope a role to any resource. 🔧 Verify the scaler after your first deploy: if the worker stays at one replica while the queue grows, check the scale rule's events in the portal.

Jonas reviews the pull request: the API, worker and marketdata share the app identity, so the API could receive from the imports queue even though it never does. A per-app identity is stricter and costs a few more resources. They keep one identity for the beta and record it in an ADR. Least privilege is a dial; know where yours is set.

💡 A role assignment is code like any other: reviewed in a pull request, named deterministically, scoped as narrowly as the work allows.


🧠 Catch it before Azure does: lint, build and what-if

Bicep has a compiler and a linter, and you should treat their output exactly like C# compiler warnings: fix them, then make CI enforce it.

💻 In Ledgerly: infrastructure gets a CI job

📄 .github/workflows/ci.yml

  infra:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
      - name: Lint and build Bicep
        run: |
          az bicep install
          az bicep lint --file infra/main.bicep
          az bicep build --file infra/main.bicep --stdout > /dev/null
          az bicep build-params --file infra/main.bicepparam --stdout > /dev/null

No Azure login is needed: the linter and compiler run offline. build-params checks the parameter file against the template's types.

Lesson 7: two mistakes Bicep caught first

This really happened while building the sample.

BCP138. The first version of the API's environment variables put the CORS loop directly inside concat():

// illustrative, not in the sample: the first attempt, rejected by bicep build with BCP138
env: concat(commonEnv, [ /* ... */ ], [for (origin, i) in corsOrigins: { name: 'Cors__AllowedOrigins__${i}', value: origin }])

bicep build refused it: a for expression can't be nested inside a function call in that position. The fix was to move the loop into a variable and pass the variable:

📄 sample/infra/modules/apps.bicep

var corsEnv = [for (origin, i) in corsOrigins: { name: 'Cors__AllowedOrigins__${i}', value: origin }]
// ...
          ], empty(cacheSecretUri) ? [] : [
            { name: 'ConnectionStrings__cache', secretRef: 'cache-connection' }
          ], corsEnv)

BCP037. The container apps were first declared with API version 2024-03-01, and the linter warned that identity was not a known property on the worker's KEDA scale rule. On that schema, the managed-identity setting for a custom scaler didn't exist, so Azure would likely have ignored it and the scaler would have had no way to authenticate. Moving to Microsoft.App/containerApps@2025-01-01 fixed it:

📄 sample/infra/modules/apps.bicep

            name: 'imports-queue'
            custom: {
              type: 'azure-servicebus'
              metadata: {
                namespace: serviceBusNamespace
                queueName: 'imports'
                messageCount: '20'
              }
              identity: identityId
            }

The lesson: a warning about an unknown property is not noise. It usually means "this setting will be silently dropped." Since then, lint is a CI job.

What-if: the diff before the deploy

Lint proves the template is valid. What-if shows what it would change in a real resource group: creates, modifies and deletes, without applying anything (what-if).

# 🧪 needs an Azure login and an existing resource group
az deployment group what-if -g <rg> -f infra/main.bicep -p infra/main.bicepparam

🧪 Not run by the author: this step was written from the docs and validated with bicep build only. Expect small differences.

Related options from the docs: az deployment group create ... --confirm-with-what-if (-c) shows the preview and asks before applying, and --no-pretty-print returns JSON, which a PR workflow can post as a comment. Ledgerly doesn't run what-if in CI yet, because that would need Azure credentials on pull requests. It's a good next step once the OIDC setup from chapter 12 exists, with a read-only identity.

💡 Lint answers "is this valid?", what-if answers "what will this change?". You want both before anything touches production.


🗺️ From edit to Azure

flowchart LR
    Edit["Edit a .bicep file"] --> Local["bicep lint<br/>bicep build"]
    Local --> PR["Pull request"]
    PR --> CI["CI infra job<br/>lint, build, build-params"]
    CI -- "fails" --> Edit
    CI -- "passes" --> Review["Review<br/>roles, SKUs, cost"]
    Review --> WhatIf["what-if<br/>optional, needs Azure"]
    WhatIf --> Merge["Merge to main"]
    Merge --> Deploy["deploy.yml<br/>three phases"]

🧠 The database user nobody has to click

There's one permission Bicep can't grant: a database user inside Azure SQL. Role assignments live in Azure Resource Manager. Database users live inside the database, and only a principal that can sign in to SQL can create them.

Ledgerly's design:

  1. The SQL server has Entra-only authentication, and its Entra admin is the migrator identity.
  2. The migration job runs as the migrator, so it can change the schema.
  3. After migrating, the same job creates a database user for the app identity and gives it read and write rights, nothing more.

📄 sample/infra/modules/sql.bicep

    administrators: {
      administratorType: 'ActiveDirectory'
      principalType: 'Application'
      login: adminIdentityName
      sid: adminIdentityClientId // 🔧 for a managed identity, the Learn Bicep sample uses the client id
      tenantId: subscription().tenantId
      azureADOnlyAuthentication: true
    }

Lesson 8: FROM EXTERNAL PROVIDER needs Graph, WITH SID doesn't

The documented way to create a user for an Entra principal is CREATE USER [name] FROM EXTERNAL PROVIDER. To resolve the name, the SQL server looks the principal up in Microsoft Entra ID, which requires the server itself to have permission to read the directory through Microsoft Graph. Granting that is a tenant-level step that many organizations reserve for administrators, and it would have been the one manual step in an otherwise automated setup.

The alternative: tell SQL the identity's id directly. CREATE USER [name] WITH SID = <id>, TYPE = E maps an external (Entra) principal without a directory lookup. For a managed identity, the SID is the client id as bytes.

📄 sample/src/Ledgerly.MigrationService/Program.cs

    /// <summary>
    /// CREATE USER ... WITH SID, TYPE = E maps a managed identity (by its client id) without needing Microsoft Graph
    /// permissions on the SQL server. Idempotent, so it is safe on every deployment.
    /// </summary>
    private static async Task EnsureAppUserAsync(LedgerlyDbContext db, string name, Guid clientId, CancellationToken ct)
    {
        var user = name.Replace("]", "]]", StringComparison.Ordinal);
        var sid = "0x" + Convert.ToHexString(clientId.ToByteArray());
        var sql = $"""
            IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE name = N'{user.Replace("'", "''", StringComparison.Ordinal)}')
                CREATE USER [{user}] WITH SID = {sid}, TYPE = E;
            ALTER ROLE db_datareader ADD MEMBER [{user}];
            ALTER ROLE db_datawriter ADD MEMBER [{user}];
            """;

#pragma warning disable EF1002 // values come from deployment configuration, are escaped above, and DDL cannot be parameterized
        await db.Database.ExecuteSqlRawAsync(sql, ct);
#pragma warning restore EF1002
    }

Elena asks why this matters for a PM. Tomás: "It means a new environment is git push plus one-time OIDC setup. No ticket to the identity team, no DBA."

💡 When the "last manual step" needs elevated rights, look for a design that removes the step rather than a person who can do it.


⚠️ Common traps

Trap What happens Do this instead
Fixing a production setting in the portal "just this once" The next deployment silently reverts it, or worse, staging and production drift Change the Bicep, open a PR, deploy. Use the portal to look, not to change
newGuid() or a random name for role assignments Every rerun tries to create a duplicate and fails guid(scope.id, principalId, roleId)
Roles at resource-group or subscription scope "to make it work" The identity can do far more than its job Assign at the resource (scope: registry) with the narrowest data role
Ignoring BCP037 "unknown property" warnings The property is dropped and the feature silently doesn't work Check the API version, upgrade it, and keep lint in CI
Deploying apps and schema changes in one step New code runs against the old schema for a few minutes Separate phases: platform, migration job, then apps
Secrets in .bicepparam They end up in git history forever Managed identities first, Key Vault for what's left, never secrets in parameter files
Granting the SQL server Graph permissions to create users A tenant-level permission for one CREATE USER CREATE USER ... WITH SID = <client id>, TYPE = E from the migrator

🧰 Helpers

A minimal module template (copy for each new resource type):

// illustrative, not in the sample: a starting point for a new module
param location string
param name string
param tags object

resource thing '<Provider/type>@<api-version>' = {
  name: '<prefix>-${name}'
  location: location
  tags: tags
  properties: {}
}

output id string = thing.id

Add role assignments by copying the acrPull loop from registry.bicep and changing the scope and role id.

PR review checklist for Bicep:

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

Steps 1 to 4 run locally without Azure.

  1. Install Bicep: az bicep install if you have the Azure CLI, otherwise the standalone bicep CLI.
  2. Build and lint: bash cd sample bicep lint infra/main.bicep bicep build infra/main.bicep --stdout > main.json bicep build-params infra/main.bicepparam --stdout Open main.json and find the role assignment names: they are guid(...) expressions that ARM evaluates at deploy time. Delete main.json when you're done.
  3. Reproduce lesson 7. In infra/modules/apps.bicep, temporarily replace corsEnv in the API's env with the loop itself: [for (origin, i) in corsOrigins: { name: 'Cors__AllowedOrigins__${i}', value: origin }]. Run bicep build infra/main.bicep and read the error. Then revert with git checkout -- infra/modules/apps.bicep.
  4. Add a tag. In main.bicep, add environment: 'production' to var tags, build again and confirm every resource in main.json picks it up. Revert.
  5. 🧪 With an Azure subscription and a resource group, run what-if: bash az deployment group what-if -g <rg> -f infra/main.bicep -p infra/main.bicepparam

    🧪 Not run by the author: this step was written from the docs and validated with bicep build only. Expect small differences.

On an empty group, everything shows as "create". Run phase 1 for real (chapter 12 does this in the pipeline), then run what-if again: it should show almost no changes. That's idempotency.


✅ Recap

  1. Bicep turns infrastructure into reviewed code: parameters and .bicepparam files per environment, modules for structure, conditions and loops for variation, and lint plus build in CI.
  2. Ledgerly's main.bicep deploys in three phases (platform, migration job, apps), and every identity gets the narrowest role at the narrowest scope, with guid() names so reruns are idempotent.
  3. The migration job creates the app's database user with CREATE USER ... WITH SID, TYPE = E, so no portal clicks and no Graph permissions are needed.

Next → Chapter 12: CD to Azure with OIDC

Chapter 12: CD to Azure with OIDC

⏱ About 40 min · You'll learn: how GitHub Actions signs in to Azure with OIDC and no stored secret, how to set up the federated credential and the deployer's roles, how Ledgerly's deploy.yml builds once and deploys in three phases with a migration job and a smoke test, and how to roll back and change schemas without downtime.


📖 Story: The first real deploy

The Bicep from chapter 11 is merged, the CI infra job is green and Elena has a date for the beta. Tomás asks the question that decides the whole design: "How does GitHub get into our Azure subscription?"

You know the answer from Nordbank Insurance. A service account with a password that never expires, pasted into the build server's credential store in 2017. Everyone knew it was there. Nobody knew who else had copied it. When the auditor asked, the answer was a long silence. In your Habits I brought with me notebook you write: "The deploy account has a password. Rotation is a project."

Jonas shakes his head before you finish describing it. "No password. No client secret. GitHub proves who it is with a token that lives for a few minutes, and Azure decides whether to trust it. If someone steals that token tomorrow, it's already useless." Maya pulls up deploy.yml. "And the deploy is a workflow anyone can read. If it isn't in the file, it doesn't happen in production."


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
A deploy account with a long-lived password in the build server OIDC: a short-lived GitHub token exchanged for an Azure token through a federated credential Nothing long-lived to steal, rotate or leak
Right-click, Publish from Visual Studio, or Web Deploy to IIS A workflow that runs after CI passes on main, with an environment approval Every deploy is reviewed, logged and repeatable
Build on the build server, build again for production Build once per commit, tag images with the commit SHA, run that exact image everywhere What you tested is what you run
Run migration scripts by hand before or after the deploy A migration job that must succeed before the apps get the new image The order is code, and a failed migration stops the pipeline
"The deploy succeeded" means the copy finished A smoke test calls the new revision before the pipeline goes green Accepted by Azure isn't the same as working
Rollback means restoring last night's backup of the site folder Redeploy an earlier commit, or point traffic back at an earlier revision Images and revisions are immutable, so the old version still exists
Schema changes need a maintenance window Expand and contract migrations across releases Old and new code can both run against the schema during the switch

🧠 OIDC: signing in without a secret

OpenID Connect (OIDC) lets GitHub Actions prove its identity to Azure with a token that GitHub signs and Azure verifies. Nothing secret is stored in GitHub. The setup has two halves:

  1. In GitHub, the workflow asks for an ID token. That needs permissions: id-token: write.
  2. In Azure, an identity has a federated credential that says: "trust tokens from this issuer, for this audience, with exactly this subject." The identity can be an Entra app registration or a user-assigned managed identity (GitHub OIDC in Azure, Azure Login with OIDC).
Federated credential field Value What it pins down
Issuer https://token.actions.githubusercontent.com Only GitHub Actions tokens
Audience api://AzureADTokenExchange Tokens minted for Azure token exchange
Subject (sub) Must match the token's sub claim exactly Which repository, and which branch, tag or environment

The subject is where the security lives (OIDC reference):

Workflow context Subject format
Job with environment: production repo:ORG/REPO:environment:production
Branch repo:ORG/REPO:ref:refs/heads/main
Tag repo:ORG/REPO:ref:refs/tags/v1
Pull request repo:ORG/REPO:pull_request

If the job declares an environment:, the subject is the environment form, not the branch form. Ledgerly's deploy job uses environment: production, so its federated credential uses repo:<owner>/<repo>:environment:production. That's a strong pin: only jobs that pass the production environment's protection rules can sign in.

🔧 The subject format is changing for new repositories. Repositories created after 2026-07-15 get an immutable subject that includes the owner and repository ids, for example repo:OWNER@OWNER-ID/REPO@REPO-ID:ref:refs/heads/main. Older repositories keep the classic format unless they opt in, and a rename or transfer after that date switches a repository to the new format. A fork is a new repository. The environment form follows the same pattern for the prefix; check the docs and the actual claim before you create the credential, because a single character of difference means "no".

💻 In Ledgerly: permissions and the login step

📄 .github/workflows/deploy.yml

permissions:
  id-token: write # request the OIDC token for azure/login
  contents: read
# ...
      - name: Sign in to Azure (OIDC, no secret)
        uses: azure/login@v3
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}

azure/login@v3 🔧 requests the GitHub token, sends it to Entra ID and gets back an Azure access token for the identity. Every az command after it runs as that identity. There is no creds input with a JSON secret: that's the legacy way.

Why these are variables, not secrets

Ledgerly stores four repository variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID and AZURE_RESOURCE_GROUP. Many examples use secrets for the first three, and that works too. Ledgerly uses variables on purpose:

What the deployer identity may do

The deployer creates resources and role assignments (chapter 11's guid() assignments), and pushes images. That needs three things on the resource group:

Role Why Note
Contributor Create and update every resource in main.bicep Can't create role assignments
Role Based Access Control Administrator (or User Access Administrator) Create the AcrPull, Key Vault Secrets User and Service Bus data role assignments Prefer the narrower RBAC Administrator role, ideally with a condition limiting which roles it can assign. Check the Azure RBAC docs
AcrPush az acr login plus pushing the images Contributor may not include registry data-plane push, depending on the registry's permission settings, so grant it explicitly

Scope all three to the resource group, never the subscription. The identity can then do anything inside rg-ledgerly-prod and nothing outside it.

💡 OIDC moves trust from "who knows the password" to "which repository, which environment": there's nothing in GitHub worth stealing.


🗺️ The token exchange

sequenceDiagram
    participant Job as "Deploy job in production"
    participant GH as "GitHub OIDC provider"
    participant Entra as "Microsoft Entra ID"
    participant ARM as "Azure Resource Manager"
    Job->>GH: Request an ID token for the Azure token exchange audience
    GH-->>Job: Signed JWT with the environment production subject
    Job->>Entra: Exchange the JWT, client id from repository vars
    Entra->>Entra: Match issuer, audience and subject on the federated credential
    Entra-->>Job: Short-lived Azure access token
    Job->>ARM: az deployment group create as the deployer identity
    ARM-->>Job: Allowed by the roles on the resource group

🧠 One-time setup

This is the only manual part, and you do it once per environment. It's written as Azure CLI commands so you can keep it in your repo as a script. You could also let azd pipeline config create the credential and variables for you 🔧 (azd pipelines).

🧪 Not run by the author: this step was written from the docs and validated with bicep build only. Expect small differences.

# illustrative, not in the sample: one-time setup for the production environment
# Prerequisites: az login (as someone who can assign roles), gh auth login, and the repo cloned.
RG=rg-ledgerly-prod
LOCATION=westeurope            # pick your region (chapter 10)
REPO=<owner>/<repo>            # check the real sub claim format (classic or immutable)

az group create --name "$RG" --location "$LOCATION"

# A user-assigned managed identity for the pipeline (not the app identities from Bicep).
az identity create --name id-github-deployer --resource-group "$RG"
CLIENT_ID=$(az identity show --name id-github-deployer --resource-group "$RG" --query clientId -o tsv)
PRINCIPAL_ID=$(az identity show --name id-github-deployer --resource-group "$RG" --query principalId -o tsv)

# Trust GitHub tokens from the production environment of this repository only.
az identity federated-credential create \
  --name github-production \
  --identity-name id-github-deployer \
  --resource-group "$RG" \
  --issuer https://token.actions.githubusercontent.com \
  --subject "repo:$REPO:environment:production" \
  --audiences api://AzureADTokenExchange

# Roles on the resource group only.
RG_ID=$(az group show --name "$RG" --query id -o tsv)
for role in "Contributor" "Role Based Access Control Administrator" "AcrPush"; do
  az role assignment create \
    --assignee-object-id "$PRINCIPAL_ID" \
    --assignee-principal-type ServicePrincipal \
    --role "$role" \
    --scope "$RG_ID"
done

# Identifiers, not secrets: store them as repository variables.
gh variable set AZURE_CLIENT_ID --body "$CLIENT_ID"
gh variable set AZURE_TENANT_ID --body "$(az account show --query tenantId -o tsv)"
gh variable set AZURE_SUBSCRIPTION_ID --body "$(az account show --query id -o tsv)"
gh variable set AZURE_RESOURCE_GROUP --body "$RG"

Then, in GitHub:

  1. Create the production environment (Settings → Environments). Add required reviewers (up to 6 people or teams, one approval is enough), turn on prevent self-review, and restrict deployment branches to main (environments). 🔧 On Free, Pro and Team plans, required reviewers and wait timers only work for public repositories.
  2. Edit sample/infra/main.bicepparam: replace <your-tenant-id> in jwtAuthority and set jwtAudience to your API's app registration (chapter 13).
  3. Push to main. Watch CI, then the deploy waits for approval.

One caution on the setup itself: the deployer identity lives in the resource group it deploys to. Don't delete that group to "start fresh" unless you're ready to repeat this section. Teams with several environments usually keep pipeline identities in a separate resource group.

💡 Keep the one-time setup as a script in the repo: "manual" should mean "run once", not "remember how".


🧠 The pipeline, step by step

Trigger, gate and concurrency

📄 .github/workflows/deploy.yml

on:
  workflow_run:
    workflows: [ci]
    types: [completed]
    branches: [main]
  workflow_dispatch:
# ...
# Never run two deployments at once, and never cancel one halfway.
concurrency:
  group: deploy-production
  cancel-in-progress: false

env:
  # ...
  RESOURCE_GROUP: ${{ vars.AZURE_RESOURCE_GROUP }}
  # The exact commit CI tested. Every image is tagged with it, so a revision maps back to a commit.
  SHA: ${{ github.event.workflow_run.head_sha || github.sha }}

jobs:
  deploy:
    # workflow_run also fires for pull requests (even from forks whose branch is named main), so only
    # deploy CI runs that came from a push to this repository, never someone else's code.
    if: >
      vars.AZURE_CLIENT_ID != '' &&
      (github.event_name == 'workflow_dispatch' ||
        (github.event.workflow_run.conclusion == 'success' &&
         github.event.workflow_run.event == 'push' &&
         github.event.workflow_run.head_repository.full_name == github.repository))
    runs-on: ubuntu-latest
    timeout-minutes: 45
    environment:
      name: production # protection rules (reviewers, branch filter) live on the environment
      url: ${{ steps.apps.outputs.apiUrl }}

Three phases, built once

Step What it runs Why
Phase 1: platform az deployment group create --name "platform-$SHA" with only the parameter file Creates or confirms the registry, data, messaging and environment. Its outputs (registry name, login server, job name) feed the next steps
Build and push az acr login, then dotnet publish ... /t:PublishContainer -p:ContainerImageTag="$SHA" for four projects No Dockerfile (chapter 9). Each image is tagged with the commit
Phase 2: migration job Same template, imageTag="$SHA" deployApps=false Points job-migrations at the new migrations image
Run migrations and wait Start the job, poll its status The apps must not change until the schema is ready
Phase 3: apps Same template, imageTag="$SHA" deployApps=true New revisions of api, worker and marketdata on the same SHA
Smoke test curl -fsS "$apiUrl/alive" until it answers Proves the new revision runs

📄 .github/workflows/deploy.yml

      - name: Build and push images (no Dockerfile)
        env:
          LOGIN_SERVER: ${{ steps.platform.outputs.loginServer }}
        run: |
          az acr login --name "${{ steps.platform.outputs.registry }}"
          for project in Ledgerly.Api Ledgerly.Worker Ledgerly.MarketData Ledgerly.MigrationService; do
            dotnet publish "src/$project" -c Release /t:PublishContainer \
              -p:ContainerRegistry="$LOGIN_SERVER" -p:ContainerImageTag="$SHA"
          done

Build once, promote the tag. CI builds the images too, but tags them ci-<sha> and never pushes them: it proves they build. The deploy builds once per commit and pushes once. After that, the migration job and all three apps use the same tag, and nothing is rebuilt. When you add a staging environment, keep that rule: build and push in the first environment, then deploy the existing tag to the next one. A revision's image tag always maps back to one commit.

The migration gate

📄 .github/workflows/deploy.yml

      - name: Run migrations and wait
        env:
          JOB: ${{ steps.platform.outputs.job }}
        run: |
          execution=$(az containerapp job start -n "$JOB" -g "$RESOURCE_GROUP" --query name -o tsv)
          echo "Started $execution"
          for i in $(seq 1 60); do
            status=$(az containerapp job execution show -n "$JOB" -g "$RESOURCE_GROUP" \
              --job-execution-name "$execution" --query properties.status -o tsv)
            echo "[$i] $status"
            case "$status" in
              Succeeded) exit 0 ;;
              Failed|Stopped|Degraded) echo "::error::Migration $execution ended as $status. See its logs in Log Analytics."; exit 1 ;;
            esac
            sleep 10
          done
          echo "::error::Migration did not finish within 10 minutes"; exit 1

az containerapp job start returns immediately with an execution name, so the step polls: up to 60 times, every 10 seconds. The job itself (chapter 11) has replicaTimeout: 600 and one retry. The migration service sets Environment.ExitCode = 1 on any failure, so a failed migration becomes a failed execution, the step exits 1 and phase 3 never runs. The old apps keep serving on the old, untouched schema.

The smoke test

📄 .github/workflows/deploy.yml

      # The deployment "succeeding" only means Azure accepted it. This proves the new revision answers.
      - name: Smoke test
        run: |
          for i in $(seq 1 30); do
            if curl -fsS "${{ steps.apps.outputs.apiUrl }}/alive"; then echo; echo "API is up"; exit 0; fi
            sleep 10
          done
          echo "::error::API did not become healthy"; exit 1

With apiMinReplicas = 0, the first call may hit a cold start, so the loop allows up to five minutes. /alive is mapped in every environment by the service defaults (unlike the Aspire template's development-only default), because the Container Apps probes need it too.

💡 A pipeline is a sequence of gates: tests, approval, migrations, smoke test. Each one is a place where a bad change stops before users see it.


🗺️ From merge to running revision

flowchart TB
    Merge["Merge to main"] --> CI["ci workflow<br/>build, test, containers, infra lint"]
    CI -- "success" --> Gate{"AZURE_CLIENT_ID set<br/>and CI succeeded?"}
    Gate -- "no" --> Skip["Job skipped"]
    Gate -- "yes" --> Approve["Environment production<br/>reviewer approves"]
    Approve --> Login["azure/login<br/>OIDC"]
    Login --> P1["Phase 1<br/>platform"]
    P1 --> Build["Build and push<br/>tag = commit SHA"]
    Build --> P2["Phase 2<br/>migration job image"]
    P2 --> Run{"Migration job<br/>succeeded?"}
    Run -- "no" --> Stop["Fail, old apps keep running"]
    Run -- "yes" --> P3["Phase 3<br/>new app revisions"]
    P3 --> Smoke["Smoke test /alive"]

🧠 Revisions, rollback and zero-downtime schema changes

Revisions in single mode

Every change to an app's template (a new image, an environment variable, a scale rule) creates a new revision, an immutable snapshot (revisions). Ledgerly's apps use activeRevisionsMode: 'Single', the default: the new revision replaces the old one with zero downtime, once its startup and readiness probes pass. If the new revision never becomes ready, it doesn't get traffic, and the smoke test fails the pipeline.

Multiple mode keeps several revisions active and lets you split traffic by percentage, for canary or blue/green releases, with labels that give a revision a stable URL for testing (traffic splitting). Ledgerly starts with single mode because it's simpler; Priya has multiple mode on her list for after the beta.

Rollback

Situation What to do
The deploy failed before phase 3 Nothing. The old revisions are still serving
A bad change is live, and you have time Roll forward: revert the commit on main. CI and deploy ship the revert like any change
A bad change is live, and you need it gone now Re-run the earlier deploy run in the Actions tab. It reuses that run's head_sha, so it redeploys the previous images. Migrations have nothing to apply, because the schema is already ahead
You use multiple revision mode Shift traffic back to the previous revision

Break glass, when even a pipeline run is too slow:

# illustrative, not in the sample: point the API back at the previous image (🧪 check the az containerapp docs)
az containerapp update -n api -g rg-ledgerly-prod --image <loginServer>/ledgerly-api:<previous-sha>

That creates a new revision outside the pipeline. The next deploy will overwrite it, so follow up with a revert on main.

The rollback rows have one hidden requirement: the previous code must still work against the current schema. That's what expand and contract guarantees.

Expand and contract

Suppose you rename Portfolio.Name to Portfolio.DisplayName. A single migration that renames the column breaks the old revision for the seconds (or minutes) between phase 2 and phase 3, and it makes rollback impossible. Split it across releases:

Release Migration (phase 2) Code (phase 3) Old code still works?
1. Expand Add DisplayName, nullable. Copy existing values Write both columns, read DisplayName with a fallback ✅ It ignores the new column
2. Switch Make DisplayName required Read and write only DisplayName ✅ Release 1 code still writes both
3. Contract Drop Name No change ✅ Release 2 code doesn't use Name

Each release is safe to roll back by one step. It's slower than one big migration, and that's the price of no maintenance window.

💡 Rollback is only real if the previous version can run against today's schema: design migrations so it always can.


🗺️ Expand and contract as states

stateDiagram-v2
    [*] --> OldOnly: schema has Name
    OldOnly --> Expanded: release 1 adds DisplayName
    Expanded --> Switched: release 2 code uses DisplayName only
    Switched --> Contracted: release 3 drops Name
    Contracted --> [*]
    Expanded --> OldOnly: rollback is safe, Name still exists
    Switched --> Expanded: rollback is safe, both columns exist

⚠️ Common traps

Trap What happens Do this instead
A client secret in AZURE_CREDENTIALS "because the tutorial did it" A long-lived secret that can leak and must be rotated OIDC with a federated credential, no secret at all
Federated subject ...:ref:refs/heads/main while the job uses environment: Login fails: the token's subject is the environment form Match the subject to the job, and check the real claim (classic or immutable 🔧)
Deployer with Owner on the subscription A compromised workflow can do anything, anywhere Contributor + RBAC Administrator + AcrPush, on the resource group only
cancel-in-progress: true on a deploy with migrations A newer push cancels a run halfway through a migration cancel-in-progress: false: queue, never cancel
A workflow_run gate that only checks conclusion branches: [main] matches the head branch name, so a pull request from a fork whose branch is called main can also trigger it after CI Also require github.event.workflow_run.event == 'push' and github.event.workflow_run.head_repository.full_name == github.repository
Renaming or dropping a column in the same release as the code change Old revision errors during the switch, and rollback breaks Expand and contract across releases
"Deploy succeeded" as the definition of done A revision that crashes on startup ships green A smoke test against the real URL

🧰 Helpers

Hardened gate for any workflow_run deploy (the same condition the sample uses, ready to paste into your own workflows):

    if: >
      vars.AZURE_CLIENT_ID != '' &&
      (github.event_name == 'workflow_dispatch' ||
       (github.event.workflow_run.conclusion == 'success' &&
        github.event.workflow_run.event == 'push' &&
        github.event.workflow_run.head_repository.full_name == github.repository))

Pre-flight checklist for the first deploy:

Make it permanent:


🧪 Try it yourself (optional, ~45 min)

  1. Locally, no Azure: read .github/workflows/deploy.yml and trace one commit SHA through it: where is it set, which steps use it, and which image tags and deployment names it ends up in? Then check that bicep build-params infra/main.bicepparam still passes from cd sample.
  2. Locally: run the migration service against your Aspire SQL with aspire run and find its "Database migrated" log line in the dashboard. That line is what the job execution logs show in Azure.
  3. 🧪 With Azure: run the one-time setup script for your fork, create the production environment and push a commit to main. Approve the deployment and watch the three phases.

    🧪 Not run by the author: this step was written from the docs and validated with bicep build only. Expect small differences.

  4. 🧪 With Azure: make a visible change (for example, a new field in an endpoint response), deploy it, then roll back by re-running the previous deploy run. Confirm with az containerapp revision list -n api -g <rg> -o table that a new revision appeared.

✅ Recap

  1. OIDC replaces deploy secrets: GitHub issues a short-lived token, and Azure trusts it through a federated credential pinned to issuer, audience and an exact subject (the environment form for Ledgerly, and the immutable format for new repos 🔧).
  2. The deployer needs Contributor, a role-assignment role and AcrPush on the resource group only, and its identifiers live in repository variables because they aren't secrets.
  3. deploy.yml builds once per SHA and deploys in three gated phases (platform, migration job, apps), then smoke-tests, while revisions plus expand-and-contract migrations make rollback safe.

Next → Chapter 13: Identity and secrets

Chapter 13: Identity and secrets

⏱ About 40 min · You'll learn: how Ledgerly reaches SQL, Service Bus, Key Vault and the registry with managed identities instead of passwords, when to use DefaultAzureCredential and when to use ManagedIdentityCredential, how the API authenticates users with JWT bearer tokens and keeps each user's data private, and which few secrets remain and where they live.


📖 Story: The connection string in Slack

It's Tuesday morning. Jonas, the security engineer, sends you a screenshot without a greeting. It's a message from the team's Slack channel, eight weeks old, from the time before you joined. Someone was debugging a prototype and pasted a full connection string: Endpoint=sb://...;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=....

"Is that ours?" he asks.

You check. It belongs to a prototype namespace that nobody deleted. The key still works. Jonas rotates it, deletes the namespace, and then asks the question that matters: "Could the same thing happen with Ledgerly?"

You open servicebus.bicep together. One line answers him: disableLocalAuth: true. Ledgerly's namespace doesn't accept keys at all. Even if someone pasted the root key in a channel, it would be rejected. The worker connects with a managed identity that has two data roles, and there's nothing to paste.

"Good," Jonas says. "Now show me everything else."

That becomes the afternoon. Maya joins, and the three of you walk through every arrow in the architecture. For each one, Jonas asks the same two questions: "Who is calling?" and "What would an attacker need to steal to pretend to be them?"

In your notebook, under Habits I brought with me, you write: "Encrypt the connection strings section of web.config." Then you cross out the word "encrypt" and write "delete".


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
SQL logins with passwords in web.config, sometimes encrypted with aspnet_regiis Authentication=Active Directory Managed Identity and Entra-only auth on the server There's no password to leak, rotate or forget in a backup
One shared domain service account for every app User-assigned managed identities with narrow data-plane roles Each workload gets only the access it needs, and you can see it in Bicep
Service Bus or storage keys in config files Namespace name only, local (key) auth disabled A leaked key is worthless when the service refuses keys
Secrets typed into release variables by an admin Key Vault references resolved by the platform with the app's identity The value never appears in the pipeline, the logs or the template outputs
Windows auth or forms-auth cookies JWT bearer tokens from Microsoft Entra ID, dotnet user-jwts locally The API is stateless and serves a SPA and other clients the same way
A security review the week before release Dependabot, CodeQL and push protection on every pull request Problems surface while the change is small and the author still remembers it

🧠 Managed identity everywhere

A managed identity is a Microsoft Entra ID identity for a workload. Azure creates and stores its credentials, and the code asks the platform for a token when it needs one. You never see a password, so you can't paste it anywhere.

Container Apps supports two kinds (facts from the Container Apps managed identity docs):

Ledgerly uses two user-assigned identities, created before the apps exist so that role assignments and the SQL admin can refer to them (chapter 11 explains why the order matters).

💻 In Ledgerly: two identities, two jobs

📄 sample/infra/modules/identity.bicep

// Two user-assigned managed identities. No passwords or connection-string secrets anywhere:
//  - "app": what api, worker and marketdata run as (least privilege: read/write data, send/receive messages).
//  - "migrator": what the migration job runs as. It is the SQL Entra admin, so it can change the schema.
// ...
resource app 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'id-${name}-app'
  location: location
  tags: tags
}

resource migrator 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'id-${name}-migrator'
  location: location
  tags: tags
}

Splitting the identities is the important design choice. The apps can read and write rows, but they can't drop a table. Only the short-lived migration job can change the schema.

Now follow each arrow Jonas asked about:

Arrow How the app proves who it is Permission it gets Where it's set
Container Apps → Container Registry registries entry with the identity, admin user disabled AcrPull registry.bicep, apps.bicep
api, worker → Azure SQL Authentication=Active Directory Managed Identity;User Id=<client id> db_datareader and db_datawriter apps.bicep, MigrationService
migration job → Azure SQL Same keyword, the migrator's client id Entra admin of the server sql.bicep
worker → Service Bus Namespace FQDN, no key Azure Service Bus Data Sender and Data Receiver servicebus.bicep
KEDA scaler → Service Bus identity: identityId on the scale rule Reads the queue length (see the traps) apps.bicep
api → Key Vault Key Vault reference with the identity Key Vault Secrets User keyvault.bicep

💻 In Ledgerly: SQL without a password

📄 sample/infra/modules/apps.bicep

var sqlConnection = 'Server=tcp:${sqlServerFqdn},1433;Database=${databaseName};Authentication=Active Directory Managed Identity;User Id=${identityClientId};Encrypt=True;Connect Timeout=60'

var commonEnv = [
  { name: 'AZURE_CLIENT_ID', value: identityClientId } // makes Azure SDK credentials pick the user-assigned identity
  { name: 'APPLICATIONINSIGHTS_CONNECTION_STRING', value: appInsightsConnectionString }
]

Read the connection string closely. There's a server, a database, an authentication mode and a User Id that is the identity's client id, not a user name. Microsoft.Data.SqlClient asks the platform for a token for that identity and sends it to SQL. The SqlClient Entra authentication docs describe the options:

On the server side, sql.bicep sets azureADOnlyAuthentication: true, which disables SQL authentication entirely (Entra-only authentication). The database user for the app identity is created by the migration service with CREATE USER ... WITH SID = ..., TYPE = E, so the SQL server doesn't need Microsoft Graph permissions to look the identity up. Chapter 11 tells the story of how the team found that out.

💻 In Ledgerly: Service Bus without a key

📄 sample/infra/modules/servicebus.bicep

// Service Bus Standard (needed for duplicate detection). Local (SAS key) auth is disabled:
// only Entra identities with a data role can send or receive.
// ...
  properties: {
    disableLocalAuth: true
    minimumTlsVersion: '1.2'
  }

📄 sample/infra/modules/apps.bicep

            { name: 'ConnectionStrings__messaging', value: serviceBusFqns } // a namespace, not a key: Entra auth
// ...
            custom: {
              type: 'azure-servicebus'
              metadata: {
                namespace: serviceBusNamespace
                queueName: 'imports'
                messageCount: '20'
              }
              identity: identityId
            }

The worker's code doesn't change between your laptop and Azure: it calls builder.AddAzureServiceBusClient("messaging"). Locally, Aspire hands it an emulator connection string. In Azure, it gets only the namespace name, and the client authenticates with a token credential. The KEDA scale rule uses the same identity, so the scaler doesn't need a key either.

The registry follows the same pattern: adminUserEnabled: false in registry.bicep, and var registries = [ { server: registryServer, identity: identityId } ] in apps.bicep. Container Apps pulls images with the identity and the AcrPull role, so no registry password exists.

💡 For every arrow in your architecture, ask "what would an attacker need to steal?" With managed identities the answer becomes "control of the running workload", which is a much higher bar than "one line from a Slack message".


🗺️ Who can reach what

flowchart LR
    API["api"] -- "runs as" --> APPID["app identity<br/>user-assigned"]
    WORKER["worker"] -- "runs as" --> APPID
    MD["marketdata"] -- "runs as" --> APPID
    JOB["migrations job"] -- "runs as" --> MIGID["migrator identity<br/>user-assigned"]
    APPID -- "AcrPull" --> ACR["Container Registry<br/>admin user off"]
    MIGID -- "AcrPull" --> ACR
    APPID -- "Data Sender and Receiver" --> SB["Service Bus<br/>local auth off"]
    APPID -- "Key Vault Secrets User" --> KV["Key Vault<br/>RBAC mode"]
    APPID -- "db_datareader and db_datawriter" --> SQL["Azure SQL<br/>Entra-only auth"]
    MIGID -- "Entra admin, changes schema" --> SQL

Every arrow on the right is a role assignment or a database role in the Bicep or the migration service. If it isn't in the code, the access doesn't exist.


🧠 DefaultAzureCredential or ManagedIdentityCredential?

When Azure SDK clients need a token, they take a TokenCredential. The one you see in every quick start is DefaultAzureCredential. It tries a chain of credentials in order and uses the first that works. In Azure.Identity 1.21.0 🔧 the order is (credential chains):

  1. Environment
  2. Workload Identity
  3. Managed Identity
  4. Visual Studio
  5. Visual Studio Code
  6. Azure CLI
  7. Azure PowerShell
  8. Azure Developer CLI
  9. Interactive browser (excluded by default)
  10. Broker

That chain is why the same code works on your laptop (Visual Studio or Azure CLI sign-in) and in Azure (managed identity). It's also why it can surprise you. The SDK best practices are direct about it:

💻 In Ledgerly: no credential code at all

Ledgerly has no new DefaultAzureCredential() anywhere in its application code. Two things make that possible:

The AZURE_CLIENT_ID environment variable in commonEnv ties it together. A container app can have several user-assigned identities, and the variable tells Azure SDK credentials which one to use. Forget it, and authentication fails in Azure even though everything worked locally.

If you write your own SDK client registration, follow the best-practice pattern:

// illustrative, not in the sample
TokenCredential credential = builder.Environment.IsDevelopment()
    ? new DefaultAzureCredential()
    : new ManagedIdentityCredential(ManagedIdentityId.FromUserAssignedClientId(builder.Configuration["AZURE_CLIENT_ID"]));

builder.Services.AddSingleton(credential); // one instance, so its token cache is shared

🔧 The ManagedIdentityId factory is in recent Azure.Identity versions; check the API reference for the version you use.

💡 DefaultAzureCredential is a great developer experience and a poor production contract. In Azure, say exactly which identity you mean.


🧠 Users: JWT bearer and per-user data

Managed identities answer "which workload is calling Azure?". Users are a different question: "which person is calling the API, and which portfolios are theirs?"

The API uses JWT bearer authentication. A client gets an access token from an identity provider and sends it in the Authorization header. The API checks the signature, issuer, audience and lifetime, and then trusts the claims inside. No session, no cookie, no server-side state.

💻 In Ledgerly: two lines and a config section

📄 sample/src/Ledgerly.Api/Program.cs

// ── Security: JWT bearer. Settings come from "Authentication:Schemes:Bearer" ───────────────────
// Locally: `dotnet user-jwts create`. In Azure: Authority = your Entra ID tenant, audience = the API app.
builder.Services.AddAuthentication().AddJwtBearer();
builder.Services.AddAuthorization();

AddJwtBearer() with no arguments reads its settings from the Authentication:Schemes:Bearer configuration section (configure JWT bearer authentication). That's what lets the same code accept two kinds of tokens:

📄 sample/infra/modules/apps.bicep

            { name: 'Authentication__Schemes__Bearer__Authority', value: jwtAuthority }
            { name: 'Authentication__Schemes__Bearer__ValidAudiences__0', value: jwtAudience }

The authority is your Entra ID tenant, and the audience is the API's app registration. The handler downloads the tenant's signing keys from the authority's metadata, so you never configure a key.

Microsoft.Identity.Web (4.16.1 🔧) is the other option. AddMicrosoftIdentityWebApi reads an AzureAd section and adds Entra-specific validation and helpers (protected web API configuration). Delegated scopes arrive in the scp claim and app roles in the roles claim (scopes and app roles). Ledgerly keeps plain AddJwtBearer so local and Entra tokens share one code path. When Leo builds the React SPA, it signs users in with MSAL.js and the authorization code flow with PKCE; the implicit flow isn't supported.

💻 In Ledgerly: a stable user id

📄 sample/src/Ledgerly.Api/ApiConventions.cs

    /// <summary>
    /// Stable user id. Entra ID puts the object id in "oid"; dev tokens from `dotnet user-jwts` use "sub"
    /// (mapped to NameIdentifier by the JWT handler).
    /// </summary>
    public static string GetOwnerId(this ClaimsPrincipal user) =>
        user.FindFirstValue("oid")
        ?? user.FindFirstValue("http://schemas.microsoft.com/identity/claims/objectidentifier")
        ?? user.FindFirstValue(ClaimTypes.NameIdentifier)
        ?? user.FindFirstValue("sub")
        ?? throw new InvalidOperationException("Authenticated user has no id claim.");

Each portfolio stores this id as OwnerId. Not the email, not the display name: those change, and an id that changes orphans data. The fallbacks exist because the JWT handler can map claim names, so oid may arrive under its long URI form and sub as NameIdentifier.

💻 In Ledgerly: authorization is a WHERE clause

📄 sample/src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs

        var group = app.MapGroup("/api/portfolios")
            .WithTags("Portfolios")
            .RequireAuthorization();
// ...
        var ownerId = user.GetOwnerId();
        var portfolio = await db.Portfolios.AsNoTracking()
            .Include(p => p.Transactions)
            .SingleOrDefaultAsync(p => p.Id == id && p.OwnerId == ownerId, ct);

        if (portfolio is null)
        {
            return TypedResults.NotFound();
        }

RequireAuthorization() on the group means "you must be signed in", and an anonymous call gets 401. The second check is per row: the query filters on both the id and the owner. If you ask for someone else's portfolio, the database returns nothing, and you get 404, not 403. A 403 would confirm that the id exists. Jonas calls this "don't answer questions nobody should be asking".

💡 Authentication is configuration. Authorization is code, and in a multi-user API most of it is a WHERE OwnerId = @me that you never forget.


🗺️ One request, two kinds of identity

sequenceDiagram
    participant SPA as "React SPA"
    participant Entra as "Microsoft Entra ID"
    participant Api as "Ledgerly.Api"
    participant Sql as "Azure SQL"
    SPA->>Entra: Sign in with authorization code and PKCE
    Entra-->>SPA: Access token for the API audience
    SPA->>Api: GET /api/portfolios/id with the bearer token
    Api->>Api: Validate signature, issuer, audience and lifetime
    Api->>Api: GetOwnerId reads the oid claim
    Api->>Entra: Token request as the app managed identity
    Entra-->>Api: Token for Azure SQL
    Api->>Sql: Select where Id and OwnerId match
    Sql-->>Api: No row, the portfolio belongs to someone else
    Api-->>SPA: 404 Not Found

Two identities take part in every request: the user, proven by the bearer token, and the workload, proven by the managed identity. The user never talks to SQL, and the API never sees the user's password.


🧠 The secrets that remain: Key Vault, CORS and Data Protection

After the identity work, Jonas asks for the list of real secrets in Ledgerly. It's short:

💻 In Ledgerly: a Key Vault reference

The Redis connection string is built inside Bicep, written straight into Key Vault and never returned as an output. The API gets it through a Key Vault reference:

📄 sample/infra/modules/apps.bicep

      // A Key Vault reference: Container Apps fetches the value with the managed identity.
      secrets: empty(cacheSecretUri) ? [] : [
        { name: 'cache-connection', keyVaultUrl: cacheSecretUri, identity: identityId }
      ]
// ...
            { name: 'ConnectionStrings__cache', secretRef: 'cache-connection' }

Container Apps reads the secret with the app identity, which needs Key Vault Secrets User (Container Apps secrets). The value never passes through the GitHub workflow or the deployment outputs.

📄 sample/infra/modules/keyvault.bicep

// Key Vault in RBAC mode (access policies are legacy). Apps read secrets via Key Vault references
// in Container Apps, so the secret value never appears in the pipeline or in app settings.
// ...
    enableRbacAuthorization: true
    enableSoftDelete: true
    softDeleteRetentionInDays: 7

Key Vault facts worth knowing (Key Vault RBAC guide):

The header of redis.bicep admits the gap: "Entra auth is the stronger option, see chapter 13." Azure Managed Redis supports Microsoft Entra ID authentication (Azure Managed Redis overview), and Aspire's Aspire.Microsoft.Azure.StackExchangeRedis package is the client side of it. With that in place you could turn off accessKeysAuthentication and delete the last secret. Redis is optional, so the team keeps the key for now, and Jonas puts the switch on the backlog with a date.

CORS for the SPA

Chapter 4 added the CORS policy. Here's how it reaches production: the allowed origins come from corsOrigins in main.bicepparam, and Bicep turns them into indexed environment variables.

📄 sample/infra/modules/apps.bicep

var corsEnv = [for (origin, i) in corsOrigins: { name: 'Cors__AllowedOrigins__${i}', value: origin }]

📄 sample/src/Ledgerly.Api/Program.cs

app.UseExceptionHandler();
app.UseStatusCodePages();
app.UseCors(ApiPolicies.Spa);
app.UseAuthentication();
app.UseAuthorization();
app.UseRateLimiter();

UseCors runs before authentication, because a browser's preflight OPTIONS request carries no token and must still get CORS headers. Remember what CORS is: a browser rule about which sites may call your API from JavaScript. It isn't authentication. A script or curl ignores it, which is why the bearer token and the owner check do the real work.

Data Protection

ASP.NET Core Data Protection encrypts cookies and antiforgery tokens. Ledgerly's API uses bearer tokens with no cookies, so nothing depends on the key ring today. If you add cookie authentication or a backend-for-frontend later, all replicas must share one key ring, or a cookie written by one replica can't be read by another. Check the ASP.NET Core Data Protection docs for key storage options before you add cookies.

💡 Count your secrets. The goal isn't a better vault; it's a shorter list. Every secret you can replace with an identity is one less thing to rotate, scan for and explain.


🧠 The supply chain is part of security

Jonas's last stop isn't Azure at all. "Half of your code is other people's packages," he says. "Who checks them?"

💻 In Ledgerly: one package source

📄 sample/nuget.config

  <!-- Only nuget.org, no machine-wide feeds leaking in. Package source mapping blocks dependency-confusion attacks. -->
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
  </packageSourceMapping>

<clear /> ignores feeds configured on the machine, and source mapping says which feed each package may come from. When you add a private feed, map your company's prefix to it, so a public package with the same name can't sneak in. NuGet audit proved its worth in chapter 3: it failed the build on a vulnerable transitive System.Security.Cryptography.Pkcs, fixed with transitive pinning.

💻 In Ledgerly: Dependabot and CodeQL

📄 .github/dependabot.yml

  - package-ecosystem: nuget
    directory: /sample
    schedule:
      interval: weekly
    groups:
      # One PR per family instead of twenty. Versions inside a family must move together.
      aspire:
        patterns: ["Aspire.*"]
# ...
  - package-ecosystem: github-actions
    directory: /

Dependabot opens pull requests for outdated NuGet packages and GitHub Actions, grouped by family so Aspire packages move together (Dependabot options). 🔧 Version updates have a default cooldown of 3 days, so brand-new releases aren't proposed until they've been out for a few days. CI tests every Dependabot PR like any other.

📄 .github/workflows/codeql.yml

  schedule:
    - cron: "17 3 * * 1" # weekly, catches new queries against unchanged code
# ...
      - uses: github/codeql-action/init@v4
        with:
          languages: csharp
          build-mode: none # no build needed for C#, analysis reads the source
      - uses: github/codeql-action/analyze@v4

CodeQL is static analysis. build-mode: none analyzes C# without a build, which keeps the workflow fast (configure code scanning). The weekly run matters: new queries find problems in code that hasn't changed. 🔧 Code scanning is free for public repositories; private and internal ones need GitHub Code Security.

Secret scanning and push protection

Secret scanning looks for known secret formats in the repository, and push protection blocks a push that contains one. Both are free and automatic for public repositories, and push protection for users is on by default there. 🔧 Repository-level push protection and private repositories need GitHub Secret Protection.

Jonas points out the limit: push protection guards Git, not Slack. The only complete fix for the story at the top of this chapter is the one you already made: have nothing worth pasting.

💡 You own every package and action in your build. Pin the sources, let a bot propose updates, let CI test them, and let a scanner read the code every week.


⚠️ Common traps

Trap What happens Do this instead
Authentication=Active Directory Default in production The full credential chain is probed on connect: slow, and the docs advise against it Active Directory Managed Identity;User Id=<client id>
A user-assigned identity without AZURE_CLIENT_ID It works locally, then fails in Azure because the SDK can't tell which identity to use Set AZURE_CLIENT_ID per app, as commonEnv does
Leaving local auth on "just in case" Any leaked key still works, as the Slack story shows disableLocalAuth: true on Service Bus, admin user off on the registry
Giving apps Contributor or Key Vault Administrator A compromised app can change infrastructure or read every secret Data-plane roles only: Secrets User, Data Sender, Data Receiver, AcrPull
Returning 403 for another user's portfolio Attackers learn which ids exist Filter by owner in the query and return 404
Keying data on email or display name Data is orphaned when a user's email changes Store the oid claim through GetOwnerId
The worker never scales out The KEDA scaler's identity can't read queue metrics: that needs Manage rights, which Sender and Receiver don't include Grant Azure Service Bus Data Owner on the queue only, as servicebus.bicep does (chapter 11), and watch the scale events after deploying 🔧
Treating CORS as security curl and scripts ignore it Rely on the bearer token and per-user checks
A secret pasted in chat Push protection can't see it Rotate it immediately, then remove the need for it

🧰 Helpers

The "no secrets" review for a new arrow (copy into your design notes):

See what an identity can do 🧪 (illustrative, not in the sample):

# illustrative, not in the sample
# principal id of the app identity, from the Bicep output appPrincipalId or the portal
az role assignment list --assignee <app-principal-id> --all -o table
az servicebus namespace show -g <rg> -n <namespace> --query disableLocalAuth

🧪 Not run by the author: this step was written from the docs and validated with bicep build only. Expect small differences.

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

  1. Locally, two users: start everything with cd sample and aspire run. In a second terminal, create two tokens: bash cd sample dotnet user-jwts create --project src/Ledgerly.Api --name alice dotnet user-jwts create --project src/Ledgerly.Api --name bob Open src/Ledgerly.Api/Ledgerly.Api.http, set @api to the API URL from the dashboard and @token to Alice's token. Create a portfolio and copy its id from the Location header.
  2. Switch @token to Bob's token and request GET {{api}}/api/portfolios/<alice's id>. You get 404. List portfolios as Bob: the list is empty.
  3. Remove the Authorization header and send the list request again. You get 401.
  4. Locally, no Azure: from cd sample, run bicep build infra/main.bicep --stdout and search the output for password and Active Directory. The only password= sits inside the Redis expression that writes straight into Key Vault. Active Directory Managed Identity shows up in both SQL connection strings: the apps' and the migration job's.
  5. 🧪 With Azure: after a deployment, run the two commands from the helpers. Match each role assignment to an arrow in the "who can reach what" diagram, and confirm disableLocalAuth is true.

    🧪 Not run by the author: this step was written from the docs and validated with bicep build only. Expect small differences.


✅ Recap

  1. Managed identities replace passwords and keys for every Ledgerly arrow into Azure (SQL, Service Bus, KEDA, the registry and Key Vault), and disabling local auth makes leaked keys worthless.
  2. Use a deterministic credential in production (ManagedIdentityCredential with the client id, or the SQL Active Directory Managed Identity keyword) and keep DefaultAzureCredential for developer machines.
  3. Users authenticate with JWT bearer tokens (user-jwts locally, Entra ID in Azure), every query filters by the owner's oid, and the few remaining secrets live in Key Vault while Dependabot, CodeQL and push protection guard the supply chain.

Next → Chapter 14: Reliability patterns

Chapter 14: Reliability patterns

⏱ About 30 min · You'll learn: what the standard resilience handler really does with its defaults, how to degrade gracefully and wire health probes without restarting healthy replicas, and how idempotency, EF retries, rate limits and bulkheads stop one failure from becoming many.


📖 Story: Priya flips the switch

It is demo day. Elena has invited two advisors to see the beta build, and you are driving. You open a portfolio, click Valuation, and the numbers appear in a few hundred milliseconds. Elena nods. Then Priya, sitting at the back with her laptop, says "one second" and types something. You refresh. The spinner turns, and keeps turning. After about fifteen seconds the page fills in: holdings and cost basis are there, but every price says "unavailable" and the total market value is empty.

"I set the market-data failure rate to 100%," Priya says. "That's what happens when the vendor goes down at 9 a.m. on a Monday." Elena asks the question she always asks: "What does the user see?" Priya answers honestly: "A spinner for fifteen seconds, then half a page. That's better than a 500. It's not good enough."

Afterwards Maya pulls you into a room. "At Nordbank, how did you handle the flaky SOAP vendor?" You admit it: a for loop, Thread.Sleep(1000), three attempts, and a WCF sendTimeout nobody remembered setting. She smiles. "Same instinct, better tools. Let's see what's already protecting us, and what those fifteen seconds are made of." You open your notebook to Habits I brought with me and write: "Retry in a loop with Thread.Sleep."

Numbers in the Ledgerly story (latency, request counts, timings) are illustrative. The resilience defaults quoted from the docs are real and marked 🔧.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
A for loop with Thread.Sleep around a SOAP or HTTP call AddStandardResilienceHandler(): retries with exponential backoff and jitter, bounded by timeouts Fixed delays from many callers arrive in waves and hammer a recovering dependency; jitter spreads them out
WCF binding timeouts in web.config, set once and forgotten Per-attempt and total timeouts in the pipeline, plus a CancellationToken that flows from the HTTP request to EF and HttpClient A slow dependency must not hold your threads and your user longer than the user is willing to wait
Catch the exception and show an error page Graceful degradation: return what you have and say it is partial (IsComplete = false) Most screens have parts that don't depend on the failing service
The load balancer pings default.aspx or checks "is the app pool running" Startup, liveness and readiness probes mapped to /alive and /health The platform restarts or removes replicas for you, so the checks must mean the right thing
MSDTC and the dream of exactly-once delivery At-least-once delivery plus idempotent consumers (ProcessedMessages, Service Bus duplicate detection) Distributed transactions don't exist across cloud services; duplicates must be harmless instead
Deadlock-retry code copied into every repository class The EF Core execution strategy retries transient SQL errors in one place Azure SQL has transient faults (failovers, serverless resume) that are normal, not exceptional
One slow dependency starved the IIS thread pool Bulkheads: concurrency limits, MaxConcurrentCalls, rate limiting with 429 Isolating a failure keeps the rest of the system responsive

💡 Reliability is not about preventing failure. It is about deciding, in advance, what happens when each dependency fails.


🧠 The failures you design for

Before choosing patterns, name the failures. Maya keeps this table on the team wiki, and each row is a question for every new dependency.

Failure mode What it looks like Ledgerly example Pattern
Transient error A single request fails, the next one works MarketData returns 503 for one call Retry with backoff and jitter
Slow dependency Requests succeed, but take seconds MarketData adds 3 s of latency Timeouts per attempt and in total, cancellation
Hard outage Every call fails for minutes The vendor is down Circuit breaker, then graceful degradation
Overload Too many requests for the capacity A user scripts 200 CSV uploads Rate limiting, bulkheads, queue-based load leveling
Duplicate delivery The same message arrives twice The outbox relay crashes after Send but before SaveChanges Idempotent consumer, duplicate detection
Poison message A message fails every time An import message for an import row that no longer exists Dead-letter queue, alert a human
Process death A replica is killed mid-request Scale-in or a new revision sends SIGTERM Graceful shutdown, probes, durable queues

The first four are about calls, the last three are about work. Chapter 7 covered the work side in depth (outbox, dead-letter, graceful shutdown). This chapter connects both sides and adds the HTTP pipeline, probes and limits.

💡 A retry is a bet that the next attempt will succeed. It is a good bet for transient errors and a terrible bet for outages and overload.


🧠 The standard resilience handler, defaults and all

The package Microsoft.Extensions.Http.Resilience 🔧 is built on Polly v8. One call, AddStandardResilienceHandler(), wraps every outgoing HttpClient request in a pipeline of five strategies. The older Microsoft.Extensions.Http.Polly (AddPolicyHandler, Polly v7) is deprecated, so if you find it in old code, migrate.

Order (outer → inner) Strategy Default 🔧 What it protects
1 Rate limiter Concurrency limiter, permit limit 1,000, queue 0 Your own process: a cap on concurrent outbound calls
2 Total request timeout 30 s The caller: no request waits longer than this, retries included
3 Retry 3 retries, exponential backoff with jitter, base delay 2 s. Handles 5xx, 408, 429 and transient exceptions Transient errors
4 Circuit breaker Opens at a 10% failure ratio, minimum throughput 100, sampling duration 30 s, break duration 5 s The dependency: stop calling it while it's down, and fail fast
5 Attempt timeout 10 s per attempt One hung attempt can't eat the whole budget

Source: Build resilient HTTP apps.

💻 In Ledgerly: one line for every HttpClient

📄 sample/src/Ledgerly.ServiceDefaults/Extensions.cs

        builder.Services.ConfigureHttpClientDefaults(http =>
        {
            // Retries with jittered back-off, per-attempt + total timeouts, circuit breaker, rate limiter.
            http.AddStandardResilienceHandler();

            // Resolves "https+http://marketdata" to a real address (Aspire locally, config or DNS in Azure).
            http.AddServiceDiscovery();
        });

ConfigureHttpClientDefaults applies to every client created by IHttpClientFactory in that service, including the typed MarketDataClient. The client itself stays boring on purpose:

📄 sample/src/Ledgerly.Core/Market/MarketDataClient.cs

/// <summary>
/// Typed HttpClient for the market-data service. Base address, retries, timeouts and the circuit breaker
/// are configured at registration time (see ServiceDefaults), not here. This class just speaks HTTP.
/// </summary>
public sealed class MarketDataClient(HttpClient http) : IMarketDataClient
{
    // ...
        var quotes = await http.GetFromJsonAsync<Quote[]>($"/prices?symbols={query}", cancellationToken) ?? [];
    // ...
}

No retry loop, no timeout constant, no try/catch. Policy lives in one place, code lives in another. That separation is the biggest change from the Thread.Sleep habit.

What the defaults did during the demo

Now you can explain Priya's fifteen seconds:

  1. Every call to /prices got a 503 from the chaos switch.
  2. The retry strategy treated 503 as transient and tried again three times, waiting a little longer each time (starting around 2 s, growing exponentially, with jitter). Those waits add up to roughly the fifteen seconds you saw.
  3. The circuit breaker never opened. It needs at least 100 calls in its 30-second sampling window before it judges the failure ratio. A demo with three people clicking produces a handful of calls.
  4. After the last retry, the handler returned the final 503, GetFromJsonAsync threw HttpRequestException, and the valuation degraded (next section).

Point 3 is the one that surprises people. The defaults are tuned for services with real traffic. At beta volume, the breaker protects you only when many users hit the outage at once. In production with real load, the breaker opens after enough failures, every call fails immediately with BrokenCircuitException for 5 seconds, and the page degrades in milliseconds instead of seconds.

flowchart LR
    App["ValuationService<br/>calls MarketDataClient"] --> RL["1 Rate limiter<br/>1,000 concurrent"]
    RL --> TT["2 Total timeout<br/>30 s"]
    TT --> RT["3 Retry<br/>3 times, backoff + jitter"]
    RT --> CB["4 Circuit breaker<br/>10% of at least 100 calls"]
    CB --> AT["5 Attempt timeout<br/>10 s"]
    AT --> MD["MarketData<br/>/prices"]
    CB -- "open: fail fast" --> X["BrokenCircuitException"]
    TT -- "budget spent" --> Y["TimeoutRejectedException"]

Tuning it for a user-facing call

For a valuation screen, fifteen seconds is too long. You can tune the standard handler with an options callback:

// illustrative, not in the sample
http.AddStandardResilienceHandler(options =>
{
    options.Retry.MaxRetryAttempts = 2;
    options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(3);
    options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(8);
});

Two rules keep tuning safe. Keep the total timeout shorter than your caller's patience (the browser, Leo's SPA, or an upstream service). And change the pipeline in one place: adding a second AddStandardResilienceHandler() on a typed client that already gets one from ConfigureHttpClientDefaults stacks two pipelines, and the retries multiply.

Use retries when Avoid retries when
The operation is a read (GET) or is idempotent It is a POST that creates something and has no idempotency key. Use o.Retry.DisableForUnsafeHttpMethods()
Errors are transient (503, 408, timeouts, connection resets) Errors are permanent (400, 401, 404): retrying can't help
The dependency is shared and recovers on its own You are already inside a retry loop higher up (queue redelivery, the caller's retry)

For non-HTTP work (a database call, a file share, an SDK without built-in retries), Microsoft.Extensions.Resilience gives you the same building blocks with AddResiliencePipeline("name", ...).

💡 The defaults are a good starting point, not a design. Read them, do the math for your traffic, and tune the call paths that users wait on.


🧠 Graceful degradation, timeouts and cancellation

Retries and breakers decide when to give up. Degradation decides what to return when you do. Ledgerly's valuation has two parts: the holdings and cost basis come from your own database, the prices come from the vendor. If the vendor is down, half the answer is still true.

💻 In Ledgerly: a partial valuation instead of a 500

📄 sample/src/Ledgerly.Api/Services/ValuationService.cs

    private async Task<decimal?> GetPriceAsync(string symbol, CancellationToken ct)
    {
        try
        {
            return await cache.GetOrCreateAsync(
                $"quote:{symbol}",
                async token =>
                {
                    var quotes = await marketData.GetQuotesAsync([symbol], token);
                    return quotes.TryGetValue(symbol, out var q) ? q.Price : (decimal?)null;
                },
                tags: ["quotes"],
                cancellationToken: ct);
        }
        catch (Exception ex) when (ex is HttpRequestException or BrokenCircuitException or TimeoutRejectedException)
        {
            LogPriceUnavailable(logger, symbol, ex.GetType().Name);
            return null;
        }
    }

And in GetAsync:

        var complete = lines.All(l => l.Price is not null);
        return new PortfolioValuation(
            portfolio.Id,
            portfolio.BaseCurrency,
            lines,
            lines.Sum(l => l.CostBasis),
            complete ? lines.Sum(l => l.MarketValue!.Value) : null,
            complete,
            clock.GetUtcNow());

What matters here:

Line Why
The catch names three exception types These are the three ways the resilience pipeline gives up: final HTTP failure, open circuit, spent timeout. Anything else (a bug) still fails loudly
OperationCanceledException is not caught If the user closes the tab, the request is aborted and the work should stop, not degrade
HybridCache.GetOrCreateAsync A price cached a few seconds ago is still served during an outage, and stampede protection means 50 concurrent requests for AAPL make one vendor call (chapter 18)
IsComplete and a null TotalMarketValue The API is honest. Leo's SPA can show "prices delayed" instead of a wrong total

When you showed Elena the JSON, she asked for one change in the SPA backlog: a banner that says "Live prices are temporarily unavailable" when isComplete is false. That is graceful degradation done properly. It is a product decision, not only a technical one.

Cancellation is part of the timeout story

In minimal APIs, a CancellationToken parameter is bound to the request's abort signal. Ledgerly passes ct from the endpoint into ValuationService, into EF Core and into HttpClient. When the user gives up, everything downstream stops. At Nordbank most of your call stacks had no tokens at all, so an abandoned request kept a thread busy until the WCF timeout fired.

sequenceDiagram
    participant U as "Browser"
    participant A as "Api ValuationService"
    participant C as "HybridCache"
    participant P as "Resilience pipeline"
    participant M as "MarketData"
    U->>A: GET valuation
    A->>C: GetOrCreate quote AAPL
    C->>P: cache miss, fetch price
    P->>M: attempt 1
    M-->>P: 503 chaos
    P->>M: retries with backoff and jitter
    M-->>P: 503 again
    P-->>C: HttpRequestException
    C-->>A: exception
    A-->>U: 200 with IsComplete false

💡 Every dependency needs a written answer to "what does the user see when this is down?" Even "a 500" is a decision, so make it on purpose.


🧠 Health probes: liveness, readiness and startup

In IIS, health was "is the app pool running?". In Azure Container Apps, the platform asks your replica three different questions, and it takes different actions for each answer. Getting them confused is one of the most common outages in container platforms.

Probe Question Action on failure Ledgerly path
Startup Has the process finished starting? Keep waiting, other probes are paused. After too many failures, restart /alive
Liveness Is the process responsive at all? Restart the container /alive
Readiness Can this replica serve traffic right now? Remove it from load balancing until it recovers. No restart /health

💻 In Ledgerly: two endpoints with different jobs

📄 sample/src/Ledgerly.ServiceDefaults/Extensions.cs

    public static TBuilder AddDefaultHealthChecks<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
    {
        builder.Services.AddHealthChecks()
            // Liveness: "the process is responsive". Never check dependencies here, or a DB blip restarts every replica.
            .AddCheck("self", () => HealthCheckResult.Healthy(), ["live"]);

        return builder;
    }

    public static WebApplication MapDefaultEndpoints(this WebApplication app)
    {
        // Readiness: all checks (including DB, Redis, Service Bus registered by Aspire integrations).
        app.MapHealthChecks(ReadinessPath).DisableHttpMetrics();

        // Liveness: only checks tagged "live".
        app.MapHealthChecks(LivenessPath, new HealthCheckOptions
        {
            Predicate = r => r.Tags.Contains("live"),
        }).DisableHttpMetrics();

        return app;
    }

Read the comment on the self check twice. If liveness checked the database, a 30-second Azure SQL failover would fail liveness on every replica at once, the platform would restart all of them, and you would turn a short database blip into a full outage with cold starts on top. Liveness answers only "is this process stuck?".

Readiness (/health) runs all registered checks. You didn't write a database check: Aspire's AddSqlServerDbContext registers one, and the other Aspire client integrations do the same for their resources. By default, Healthy and Degraded return HTTP 200 and Unhealthy returns 503.

The Aspire service defaults template maps these endpoints only in Development. Ledgerly maps them in every environment because Container Apps probes need them, and the default response is a single word ("Healthy") that reveals no details.

📄 sample/infra/modules/apps.bicep

var probes = [
  {
    type: 'Startup'
    httpGet: { path: '/alive', port: 8080 }
    initialDelaySeconds: 2
    periodSeconds: 3
    failureThreshold: 20
  }
  {
    type: 'Liveness'
    httpGet: { path: '/alive', port: 8080 }
    periodSeconds: 10
    failureThreshold: 3
  }
  {
    type: 'Readiness'
    httpGet: { path: '/health', port: 8080 }
    periodSeconds: 10
    failureThreshold: 3
  }
]

The startup probe gives a replica about a minute (20 checks, 3 seconds apart) to start before anything else is judged. After that, three failed liveness checks in a row (about 30 seconds) trigger a restart, and three failed readiness checks take the replica out of rotation.

flowchart TB
    Start["Replica starts"] --> SP{"Startup probe<br/>/alive ok?"}
    SP -- "not yet" --> SP
    SP -- "20 failures" --> Restart["Container restarted"]
    SP -- "yes" --> Run["Running"]
    Run --> LP{"Liveness<br/>/alive ok?"}
    LP -- "3 failures" --> Restart
    Run --> RP{"Readiness<br/>/health ok?"}
    RP -- "yes" --> Traffic["Receives traffic"]
    RP -- "3 failures" --> Out["Out of rotation<br/>no restart"]
    Out -- "checks pass again" --> Traffic

A trade-off worth discussing

Readiness that checks a shared dependency has its own risk: if the database is down, every API replica becomes unready at once, and callers get an error from the platform instead of your nicely shaped ProblemDetails. Maya's position, which the team wrote down in an ADR: readiness checks only dependencies without which this replica can do nothing useful. For the API, that is SQL. MarketData is not a readiness dependency, because valuation degrades without it. Never add the vendor to a health check.

💡 Liveness asks "should I be restarted?", readiness asks "should I get traffic?". A dependency failure is almost never a reason to restart.


🧠 Idempotency, EF retries, rate limits and bulkheads

Retries create duplicates. Every layer that retries (the HTTP pipeline, Service Bus redelivery, the outbox relay, a user clicking twice) needs a partner that makes the duplicate harmless.

💻 In Ledgerly: an idempotent consumer

📄 sample/src/Ledgerly.Worker/ImportProcessor.cs

        if (await db.ProcessedMessages.AnyAsync(p => p.MessageId == messageId, ct))
        {
            LogDuplicate(logger, messageId);
            await args.CompleteMessageAsync(args.Message, ct);
            return;
        }
        // ...
        try
        {
            var outcome = await ApplyImportAsync(db, import, ct);
            db.ProcessedMessages.Add(new ProcessedMessage { MessageId = messageId, Handler = HandlerName });
            await db.SaveChangesAsync(ct); // trades + import status + idempotency record in ONE transaction
            await args.CompleteMessageAsync(args.Message, ct);
            // ...
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            // ...
            await args.AbandonMessageAsync(args.Message, cancellationToken: ct); // Service Bus will redeliver
        }

The trades, the import status and the "I processed this message" record are saved in one SaveChangesAsync, so they commit together or not at all. MessageId is the primary key of ProcessedMessages, so even two replicas racing on the same message can't both commit. There are three layers of protection:

Layer Mechanism Catches
Service Bus Duplicate detection: the same MessageId within the window is dropped. Ledgerly sets MessageId = outbox row id The relay re-sending after a crash, within 10 minutes
Database ProcessedMessages keyed by message id, written in the same transaction Redelivery after the window, or after an abandon
Domain ExternalRef from the broker CSV is unique per portfolio (filtered unique index, chapter 5) The user uploading the same file twice

📄 sample/infra/modules/servicebus.bicep

  properties: {
    lockDuration: 'PT1M'
    maxDeliveryCount: 10 // after 10 failed attempts the message moves to imports/$deadletterqueue
    requiresDuplicateDetection: true // the same MessageId within the window is dropped
    duplicateDetectionHistoryTimeWindow: 'PT10M'
    deadLetteringOnMessageExpiration: true
  }

Duplicate detection needs the Standard or Premium tier 🔧 and is configured when the queue is created. The 10-minute window is also the default (range 20 s to 7 days) 🔧.

EF Core: the execution strategy

Azure SQL has transient faults that are part of normal operation: a failover, a connection reset, a serverless database resuming from auto-pause. EF Core's SQL Server provider can retry them through an execution strategy (EnableRetryOnFailure). Aspire's AddSqlServerDbContext turns retries on for you. The catch: once retries are on, a transaction you start must run inside the strategy, so the whole unit can be replayed.

📄 sample/src/Ledgerly.MigrationService/Program.cs

            // The execution strategy retries transient failures (e.g. Azure SQL serverless waking up).
            var strategy = db.Database.CreateExecutionStrategy();
            await strategy.ExecuteAsync(async () => await db.Database.MigrateAsync(stoppingToken));

A single SaveChangesAsync is already one transaction, which is why the import handler doesn't need this wrapper. Reach for CreateExecutionStrategy().ExecuteAsync(...) when you call BeginTransactionAsync yourself or group several saves.

Rate limiting and bulkheads

A bulkhead (named after ship compartments) limits how much of your capacity one kind of work can take, so a flood in one place can't sink everything. Ledgerly has several, some of which you might not have noticed:

Bulkhead Where Limit Protects
Per-user rate limit on imports Program.cs 5 uploads per minute per user (fixed window) API and worker from one noisy user
Outbound concurrency limiter Standard resilience handler 1,000 concurrent calls, queue 0 🔧 The process from unbounded outbound calls
MaxConcurrentCalls = 4 ImportProcessor 4 imports per worker replica SQL from too many parallel imports
maxReplicas: 5 on the worker apps.bicep KEDA adds replicas, up to 5 SQL again: scale is bounded on purpose
CPU and memory per container apps.bicep 0.5 vCPU, 1 GiB for api and worker Neighbors in the same environment

📄 sample/src/Ledgerly.Api/Program.cs

builder.Services.AddRateLimiter(options =>
{
    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    options.AddPolicy(ApiPolicies.Imports, http => RateLimitPartition.GetFixedWindowLimiter(
        partitionKey: http.User.Identity?.Name ?? http.Connection.RemoteIpAddress?.ToString() ?? "anonymous",
        factory: _ => new FixedWindowRateLimiterOptions { PermitLimit = 5, Window = TimeSpan.FromMinutes(1) }));
});

Note the first line inside the lambda. The rate limiting middleware's default RejectionStatusCode is 503, not 429. A 503 tells clients and monitoring "the server is broken", while 429 says "you are sending too much". Leo's SPA should back off on 429; your alerts should not page Priya for it. Set it explicitly, as the rate limiting docs samples do.

💡 Every retry needs an idempotency partner, and every queue or scale rule needs an upper bound that your database can survive.


🗺️ How a duplicate becomes harmless

flowchart LR
    API["Api<br/>import + outbox row<br/>one transaction"] --> OB[("OutboxMessages")]
    OB --> Relay["OutboxRelay<br/>MessageId = outbox id"]
    Relay -- "send, maybe twice" --> SB["Service Bus queue imports<br/>duplicate detection 10 min"]
    SB -- "deliver, at least once" --> W["ImportProcessor"]
    W --> Q{"MessageId in<br/>ProcessedMessages?"}
    Q -- "yes" --> Done["Complete, do nothing"]
    Q -- "no" --> Apply["Apply trades + record id<br/>one SaveChanges"]
    Apply -- "crash or error" --> Ab["Abandon, redelivered"]
    Ab --> SB
    SB -- "after 10 deliveries" --> DLQ["imports dead-letter queue"]

⚠️ Common traps

Trap What happens Do this instead
Liveness probe checks the database A short SQL failover fails liveness everywhere, every replica restarts, and the outage gets longer Liveness checks only the process (self). Dependencies go in readiness, and only the essential ones
Retrying a non-idempotent POST A timeout after the server committed leads to a second order, a second trade, a second import o.Retry.DisableForUnsafeHttpMethods(), or an idempotency key the server checks
Stacking two resilience handlers ConfigureHttpClientDefaults adds one, a typed client adds another, and 3 retries become many Configure the pipeline in one place and read the effective defaults
Catching Exception around a call, including cancellation Aborted requests "degrade" instead of stopping, and real bugs return 200 with missing data Catch the specific resilience outcomes, as ValuationService does, and let OperationCanceledException flow
Trusting the circuit breaker at low traffic With a minimum throughput of 100 in 30 s 🔧, a quiet service never trips it, and every user pays the full retry time Do the math for your traffic and tune attempt and total timeouts for user-facing calls
Leaving the rate limiter on its default status Rejections show up as 503s, alerts fire, and clients retry harder Set RejectionStatusCode = 429 and let clients back off
Unbounded scale-out on a queue KEDA adds 50 workers, each opens SQL connections, and the database becomes the outage Set maxReplicas and MaxConcurrentCalls to what the database can take

🧰 Helpers

The "new dependency" checklist (paste into your ADR or PR description):

- [ ] What does the user see when this dependency is down? (degrade, queue, or fail)
- [ ] Timeouts: attempt and total, shorter than the caller's patience
- [ ] Retries: only for idempotent operations, with backoff and jitter
- [ ] Circuit breaker: does our traffic reach the minimum throughput?
- [ ] Health: readiness only if this replica is useless without it, never liveness
- [ ] Duplicates: what makes a second delivery harmless?
- [ ] Limits: concurrency, rate and scale bounds the dependency can survive

Run chaos against your local system (MarketData's /chaos endpoint exists only in Development):

# 50% failures and 2 s of latency on every /prices call
curl -X POST "$MARKETDATA_URL/chaos" -H "Content-Type: application/json" -d '{"failureRate":0.5,"latencyMs":2000}'
# back to normal
curl -X POST "$MARKETDATA_URL/chaos" -H "Content-Type: application/json" -d '{"failureRate":0,"latencyMs":0}'

Make it permanent:


🧪 Try it yourself (optional, ~25 min)

You need a container runtime for Aspire.

  1. Start the system and open the Aspire dashboard link from the console: bash cd sample aspire run
  2. Create a dev token and use src/Ledgerly.Api/Ledgerly.Api.http (or curl) to create a portfolio, record a buy, and call the valuation endpoint. Note the response time and "isComplete": true. bash dotnet user-jwts create --project src/Ledgerly.Api
  3. In the dashboard, find the marketdata resource and copy its URL into MARKETDATA_URL. Turn on full failure with the curl command from Helpers, using "failureRate":1.0. Wait about 30 seconds so the cached quote expires (HybridCache expiration is 30 s in Program.cs).
  4. Call the valuation again. Observe: it takes many seconds, returns 200, and isComplete is false. In the dashboard Traces view, open the trace and count the HTTP client attempts to marketdata. In Structured logs, find "Price for ... unavailable (HttpRequestException)".
  5. Now set "failureRate":0,"latencyMs":12000. Call the valuation again. Observe: each attempt hits the 10 s attempt timeout, the 30 s total timeout ends the retries, and the log reason becomes TimeoutRejectedException.
  6. Reset chaos to zero. If the /chaos route isn't reachable on your machine, set FailureRate and LatencyMs in the Chaos section of src/Ledgerly.MarketData/appsettings.json instead and restart.

✅ Recap

  1. The standard resilience handler gives every HttpClient a rate limiter, total timeout, jittered retries, a circuit breaker and attempt timeouts, but you must check its defaults against your traffic and your users' patience.
  2. Degrade on purpose (IsComplete = false), flow CancellationToken everywhere, keep dependencies out of liveness, and keep only essential ones in readiness.
  3. Retries create duplicates and load: pair them with idempotent consumers, the EF execution strategy, a 429 rate limiter and bounded concurrency and scale.

Next → Chapter 15: Observability I: logging and OpenTelemetry

Chapter 15: Observability I: logging and OpenTelemetry

⏱ About 30 min · You'll learn: structured logging with message templates and [LoggerMessage], the OpenTelemetry model of traces, metrics and logs, and how Ledgerly follows one request from the browser through the outbox and Service Bus into the worker.


📖 Story: the log file that doesn't exist

The first real bug report from a beta tester arrives on a Tuesday. "I uploaded my broker CSV. It said the import finished, but three trades are missing." Elena forwards it with one line: "Can we tell her what happened today, not next week?"

Your hands know what to do. At Nordbank you would RDP into the three IIS servers, open D:\Logs\Policy\, and grep each day's log4net file for the customer number. So you ask Tomás, half joking: "Where's the log file for the worker?" He doesn't laugh. "Which replica? The one KEDA started at 10:02 and removed at 10:09? Its file system is gone. There are no files, and there's no box to log in to."

Maya slides her laptop over. The Aspire dashboard is open on a local run, and she types the import id into the structured logs filter. One line appears: Import 0193... completed_with_errors: 47 imported, 3 skipped. She clicks the trace id next to it, and the whole story unfolds on one screen: the upload request in the API, the outbox publish in the worker, the Service Bus processing and every SQL call. "Same thing in Application Insights for the beta environment," she says. "Logs aren't files anymore. They're events with properties, and they belong to a trace." In your notebook under Habits I brought with me: "Grep log files on each server."

The ids, row counts and timings in this chapter are illustrative. Package versions and defaults from the docs are marked 🔧.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
log4net or NLog files on each server, RDP and grep Logs exported as structured events (OTLP) to the Aspire dashboard locally and Application Insights in Azure Containers are short-lived and have no persistent disk; there is no server to log in to
logger.Info("Import " + id + " done") Message templates: "Import {ImportId} {Outcome}: ..." with named properties, generated by [LoggerMessage] Properties are queryable fields, not text you parse with regex
A correlation id copied by hand into WCF headers W3C trace context (traceparent) propagated by OpenTelemetry, and carried through the outbox in a TraceParent column One trace id links every span and log line across services and queues
Performance counters and perfmon on the box Meter counters and histograms, plus built-in ASP.NET Core, HttpClient and runtime metrics Metrics are cheap aggregates that survive sampling and replica churn
The Application Insights 2.x SDK with TelemetryClient.TrackEvent everywhere Vendor-neutral APIs (ILogger, ActivitySource, Meter) and an exporter chosen at startup Code doesn't change when the backend changes, and the classic SDK is now built on OpenTelemetry anyway
"Turn on Debug in production" when something breaks Traces show the path; logs at Information and above carry the facts; sampling keeps the cost bounded Debug-level volume in the cloud is a cost and a privacy problem

💡 You don't search logs to find out what happened anymore. You find one event, then follow its trace.


🧠 Structured logging that you can query

The single most important habit to drop is building log messages with string concatenation or interpolation. A message template keeps the text and the values separate. The text stays constant (so you can group by it), and each {Placeholder} becomes a named property on the log record. The official guidance is direct about it: use templates, never interpolation.

Style Example What the backend stores
Interpolation ❌ logger.LogInformation($"Import {id} done") One unique string per import. Nothing to filter on, and the string is built even when the level is off
Template ✅ logger.LogInformation("Import {ImportId} done", id) A constant template plus a property ImportId you can query
Source-generated ✅✅ [LoggerMessage(...)] static partial void LogImportDone(...) Same as the template, with no boxing, no template parsing at runtime, and no allocation when the level is disabled

💻 In Ledgerly: every log call is generated

Ledgerly uses [LoggerMessage] everywhere except one deliberate line in the chaos service. The worker's import processor declares its messages at the bottom of the class:

📄 sample/src/Ledgerly.Worker/ImportProcessor.cs

    [LoggerMessage(Level = LogLevel.Information, Message = "Message {MessageId} already processed, completing duplicate")]
    private static partial void LogDuplicate(ILogger logger, string messageId);

    [LoggerMessage(Level = LogLevel.Information, Message = "Import {ImportId} {Outcome}: {RowsImported} imported, {RowsSkipped} skipped")]
    private static partial void LogImportDone(ILogger logger, Guid importId, string outcome, int rowsImported, int rowsSkipped);

    [LoggerMessage(Level = LogLevel.Error, Message = "Import {ImportId} failed on delivery {DeliveryCount}; abandoning for retry")]
    private static partial void LogImportCrashed(ILogger logger, Guid importId, int deliveryCount, Exception ex);

Three details are worth copying:

  1. The class is sealed partial class ImportProcessor. The generator writes the method bodies into the other half of the partial class, which is why the methods must be partial.
  2. The Exception parameter isn't in the template. The generator recognizes it and attaches it to the log record as the exception, with its stack trace.
  3. If you add a parameter and forget to use it in the template, the build warns with SYSLIB1015. With warnings as errors (chapter 3), that is a build break, which is exactly what you want.

Compare it with the one hand-written call, in the market-data simulator:

📄 sample/src/Ledgerly.MarketData/Program.cs

        logger.LogWarning("Chaos: failing /prices request for {Symbols}", symbols);

It still uses a template, so Symbols is a property. For a throwaway simulator that's fine. In code that runs on every request, prefer the generator.

Scopes: properties for a whole block

IncludeScopes = true (set in the service defaults) means properties from ILogger.BeginScope are attached to every log record written inside the scope. Ledgerly doesn't use scopes, because spans already group the work, but they're handy when several log lines need the same business key:

// illustrative, not in the sample
using (logger.BeginScope(new Dictionary<string, object?>
{
    ["ImportId"] = import.Id,
    ["PortfolioId"] = import.PortfolioId,
}))
{
    // every log written here carries ImportId and PortfolioId
}

💡 Write log messages for the query you'll run at 2 a.m.: a constant template, the business ids as properties, and the exception as an exception.


🧠 OpenTelemetry: one model for traces, metrics and logs

OpenTelemetry (OTel) is a vendor-neutral standard for telemetry. In .NET, the beautiful part is that the APIs are already in the runtime. Your code doesn't reference OpenTelemetry at all; only the composition root does.

Signal .NET API Question it answers Ledgerly example
Traces (spans) ActivitySource and Activity in System.Diagnostics "What happened in this one request, and where did the time go?" outbox publish imports, import process
Metrics Meter, Counter<T>, Histogram<T> in System.Diagnostics.Metrics "How much, how often, how fast, in aggregate?" ledgerly.imports.processed, http.server.request.duration
Logs ILogger "What fact did the code want to record?" Import {ImportId} {Outcome}: ...

A span in OTel is an Activity in .NET (the names differ for historical reasons). Each span has a trace id shared by the whole operation, its own span id and a parent span id. Log records written while a span is active automatically get its trace id and span id. That's how Maya jumped from one log line to the whole trace.

Three more concepts finish the model:

flowchart LR
    subgraph Code["Your code: no OTel references"]
        L["ILogger<br/>LoggerMessage"]
        A["ActivitySource<br/>Ledgerly"]
        M["Meter<br/>Ledgerly"]
    end
    subgraph SDK["ServiceDefaults: OTel SDK"]
        I["Instrumentation<br/>AspNetCore, HttpClient, Runtime"]
        P["Pipeline<br/>sources, meters, filters"]
        E{"Which exporter?"}
    end
    L --> P
    A --> P
    M --> P
    I --> P
    P --> E
    E -- "OTEL_EXPORTER_OTLP_ENDPOINT set" --> D["Aspire dashboard<br/>or any OTLP backend"]
    E -- "APPLICATIONINSIGHTS_CONNECTION_STRING set" --> AI["Application Insights<br/>Log Analytics workspace"]

💡 Instrument with the .NET APIs, export with OpenTelemetry. The backend becomes a startup decision, not a code change.


🧠 How ServiceDefaults wires it

Every Ledgerly service calls builder.AddServiceDefaults(). That one line is where all the OTel decisions live.

💻 In Ledgerly: signals, sources and filters

📄 sample/src/Ledgerly.ServiceDefaults/Extensions.cs

    public static TBuilder ConfigureOpenTelemetry<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
    {
        builder.Logging.AddOpenTelemetry(logging =>
        {
            logging.IncludeFormattedMessage = true;
            logging.IncludeScopes = true;
        });

        builder.Services.AddOpenTelemetry()
            .WithMetrics(metrics => metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddMeter(TelemetryName))
            .WithTracing(tracing => tracing
                .AddSource(builder.Environment.ApplicationName)
                .AddSource(TelemetryName)
                .AddSource("Azure.*")
                .AddAspNetCoreInstrumentation(options =>
                    // Probes hit these every few seconds: keep them out of traces.
                    options.Filter = context =>
                        !context.Request.Path.StartsWithSegments(ReadinessPath)
                        && !context.Request.Path.StartsWithSegments(LivenessPath))
                .AddHttpClientInstrumentation());

        builder.AddOpenTelemetryExporters();
        return builder;
    }

Line by line:

Line What it does
IncludeFormattedMessage = true Sends the rendered text ("Import 0193... completed: 47 imported") alongside the template and properties
AddMeter(TelemetryName) / AddSource(TelemetryName) Subscribes to Ledgerly's own meter and activity source, both named "Ledgerly". Without these lines, custom telemetry is silently dropped
AddSource("Azure.*") Collects spans from Azure SDK clients such as Service Bus
The Filter on ASP.NET Core instrumentation Drops /health and /alive requests. Container Apps probes call them every few seconds on every replica, and you don't want them in traces or in your bill
.DisableHttpMetrics() on the health endpoints (in MapDefaultEndpoints) The same idea for metrics: probes don't skew request-duration percentiles

Choosing the exporter at startup

📄 sample/src/Ledgerly.ServiceDefaults/Extensions.cs

    private static TBuilder AddOpenTelemetryExporters<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
    {
        // Locally Aspire sets this and the dashboard receives OTLP. Any OTLP backend works the same way.
        if (!string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]))
        {
            builder.Services.AddOpenTelemetry().UseOtlpExporter();
        }

        // In Azure the Container App gets this setting and telemetry goes to Application Insights.
        if (!string.IsNullOrWhiteSpace(builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
        {
            builder.Services.AddOpenTelemetry().UseAzureMonitor();
        }

        return builder;
    }

The code never asks "am I in Azure?". It asks "where am I told to send telemetry?". Locally, Aspire sets the OTLP endpoint and the dashboard receives everything. In Azure, apps.bicep puts APPLICATIONINSIGHTS_CONNECTION_STRING into commonEnv for every container app (chapter 16 follows it from there). With neither variable, as in integration tests, nothing is exported.

Two distros, pick one 🔧

UseAzureMonitor() comes from the Azure Monitor OpenTelemetry Distro (Azure.Monitor.OpenTelemetry.AspNetCore, 1.6.0 in the sample 🔧). The current docs recommend a newer option for new apps, the Microsoft OpenTelemetry Distro (package Microsoft.OpenTelemetry, 1.2.0 🔧), which can export to Azure Monitor, any OTLP endpoint or Agent 365:

// illustrative, not in the sample
using Microsoft.OpenTelemetry;

builder.UseMicrosoftOpenTelemetry(o => o.Exporters = ExportTarget.AzureMonitor);

The docs say existing users of the Azure Monitor distro need to take no action, which is why Ledgerly stays on UseAzureMonitor() for now. The one hard rule: don't initialize both distros in the same app. And if you inherit an older app on the classic Application Insights SDK, know that SDK 3.x (Microsoft.ApplicationInsights.AspNetCore 3.2.0 🔧) is itself built on OpenTelemetry, that many 2.x packages were removed, and that 2.x and 3.x packages must not be mixed. See Enable Azure Monitor OpenTelemetry and the migration guide.

The Aspire dashboard

Locally you don't need any cloud resource to see all three signals. aspire run starts the dashboard with an OTLP receiver, and it shows resources, console output, structured logs (filterable by property), traces (a waterfall per trace) and metrics. It is the fastest feedback loop you'll have: add a span, run, look. Chapter 6 covers the dashboard's other views.

💡 Configure telemetry in one shared place, choose the exporter from configuration, and filter out probe noise before it costs you anything.


🧠 Custom spans, custom metrics and one trace through the queue

Built-in instrumentation covers HTTP in and out, and the Azure SDKs. Your business operations need their own names.

💻 In Ledgerly: one source, one meter

📄 sample/src/Ledgerly.Core/Diagnostics/LedgerlyTelemetry.cs

public static class LedgerlyTelemetry
{
    public const string SourceName = "Ledgerly";

    public static readonly ActivitySource ActivitySource = new(SourceName);

    private static readonly Meter Meter = new(SourceName);

    public static readonly Counter<long> TransactionsRecorded =
        Meter.CreateCounter<long>("ledgerly.transactions.recorded", unit: "{transaction}", description: "Trades recorded via API or import.");

    public static readonly Counter<long> ImportsProcessed =
        Meter.CreateCounter<long>("ledgerly.imports.processed", unit: "{import}", description: "Broker imports processed, tagged by outcome.");

    public static readonly Histogram<double> ImportDuration =
        Meter.CreateHistogram<double>("ledgerly.imports.duration", unit: "s", description: "Time to process one broker import.");
    // ...
}

Ledgerly.Core references no OpenTelemetry package. ActivitySource and Meter are plain .NET types, and the service defaults subscribe to the name "Ledgerly". Two notes on choices you might make differently:

Using them is one line each, with tags as key/value pairs:

📄 sample/src/Ledgerly.Worker/ImportProcessor.cs

        var request = args.Message.Body.ToObjectFromJson<ImportRequested>();
        using var activity = LedgerlyTelemetry.ActivitySource.StartActivity("import process");
        activity?.SetTag("ledgerly.import.id", request?.ImportId);
        // ...
            LedgerlyTelemetry.ImportsProcessed.Add(1, new KeyValuePair<string, object?>("outcome", outcome));
            LedgerlyTelemetry.ImportDuration.Record(Stopwatch.GetElapsedTime(started).TotalSeconds);

Note activity?.: StartActivity returns null when nothing is listening (in unit tests, for example), so always use the null-conditional operator.

Carrying the trace across the outbox

HTTP calls propagate trace context automatically: HttpClient adds a traceparent header and ASP.NET Core reads it. A database row doesn't. The outbox is a gap where the trace would normally break: the API commits a row, and seconds later a different process publishes it. Ledgerly closes the gap by storing the W3C trace context with the row.

📄 sample/src/Ledgerly.Core/Messaging/Outbox.cs

    // W3C trace context of the request that created the message, so the trace continues in the worker.
    public string? TraceParent { get; init; }
    // ...
        db.OutboxMessages.Add(new OutboxMessage
        {
            Type = typeof(T).Name,
            Destination = destination,
            Payload = JsonSerializer.Serialize(message),
            TraceParent = System.Diagnostics.Activity.Current?.Id,
        });

Activity.Current?.Id is the W3C traceparent string of the current span (version, trace id, span id, flags). The relay then starts its span as a child of that id:

📄 sample/src/Ledgerly.Worker/OutboxRelay.cs

            // Continue the trace of the HTTP request that created the message.
            using var activity = LedgerlyTelemetry.ActivitySource.StartActivity(
                $"outbox publish {message.Destination}", ActivityKind.Producer, message.TraceParent);

ActivityKind.Producer tells backends this span hands work to a queue. Inside it, the Service Bus SDK sends the message, and its own spans (collected through AddSource("Azure.*")) belong to the same trace. On the consumer side, the processor callback runs inside the SDK's processing span, and import process becomes a child of whatever Activity.Current is at that moment.

Ledgerly relies on the Azure SDK to carry context across the broker. The SDK's exact propagation behavior isn't something this course verified, so if you see the trace split in two at the queue, carry it yourself. The pattern is the same as the outbox:

// illustrative, not in the sample
// Producer: copy the current trace context onto the message.
var outgoing = new ServiceBusMessage(payload) { MessageId = id };
if (Activity.Current?.Id is { } traceParent)
{
    outgoing.ApplicationProperties["traceparent"] = traceParent;
}

// Consumer: continue that trace explicitly.
var parentId = args.Message.ApplicationProperties.TryGetValue("traceparent", out var value) ? value as string : null;
using var activity = LedgerlyTelemetry.ActivitySource.StartActivity("import process", ActivityKind.Consumer, parentId);

💡 Anywhere work crosses a boundary that isn't HTTP (a table, a queue, a file), the trace breaks unless you carry traceparent across it yourself.


🗺️ One import, one trace

This is the trace Maya opened in the story. Every arrow below is a span or a log record with the same trace id, as long as the context is carried across the outbox row and the queue.

sequenceDiagram
    participant B as "Browser"
    participant A as "Api"
    participant D as "SQL"
    participant R as "OutboxRelay"
    participant Q as "Service Bus"
    participant W as "ImportProcessor"
    B->>A: POST import with traceparent
    A->>D: Import row plus outbox row with TraceParent
    A-->>B: 202 Accepted
    R->>D: read pending outbox rows
    R->>R: span outbox publish as child of TraceParent
    R->>Q: send with MessageId equal to outbox id
    Q->>W: deliver message
    W->>W: span import process
    W->>D: trades plus ProcessedMessages in one SaveChanges

When you search Application Insights for the beta tester's import id, you land on the last log line ("Import ... completed_with_errors: 47 imported, 3 skipped"). From there, the trace id leads back to the upload request, and the import's Error field (returned by GET /api/imports/{importId}) lists the first rows that failed to parse. That's the answer Elena wanted the same day.


🧠 What not to send: PII, secrets and cardinality

Observability data is data. It is stored, retained, queried by many people and billed by volume. Three rules keep it safe and affordable.

1. No secrets, no unnecessary personal data. Log ids, not content. Ledgerly logs ImportId, MessageId, Symbol and counts. It never logs the CSV content, the JWT, connection strings or the user's name. Jonas has a short list for PR reviews:

Never log Log instead
Tokens, keys, connection strings, passwords Nothing. If you must, a fixed marker like "token present: true"
File contents and request bodies Ids, sizes, row counts, line numbers of errors
Names, emails, account numbers The internal owner id, or nothing
Full exception messages from third parties that echo input Your own message plus the exception object (and review what the exception contains)

Remember that spans carry attributes too. url.full on an outgoing request includes the query string, so never put secrets in URLs.

2. Mind metric cardinality. Every unique combination of tag values on a metric becomes a separate time series. outcome on ledgerly.imports.processed has three values ("completed", "completed_with_errors", "rejected"), which is perfect. AlertsTriggered is tagged by symbol, which is fine while the simulator knows a short fixed list of symbols, and worth revisiting if users can enter any ticker. A tag like userId or importId on a metric is a mistake: put high-cardinality values on spans and logs, where each event is stored individually, and keep metric tags to small, bounded sets.

Value On a span or log? As a metric tag?
ImportId, MessageId, PortfolioId ✅ Yes ❌ No, unbounded
outcome, HTTP status code, operation name ✅ Yes ✅ Yes, a small set
symbol ✅ Yes ⚠️ Only while the list is bounded
User id or email ⚠️ Internal id only ❌ Never

3. Choose levels on purpose. Information for business facts you want in production ("Import ... completed"). Warning for degraded but handled situations ("Price for AAPL unavailable"). Error for failures that need attention ("Import ... failed on delivery 3"). Debug and Trace for local investigation only. Generated [LoggerMessage] methods cost almost nothing when a level is disabled, so you can leave debug messages in the code without paying for them.

💡 Ids on spans, small enums on metrics, nothing secret anywhere. Cardinality and PII mistakes are both cheap to make and expensive to clean up.


⚠️ Common traps

Trap What happens Do this instead
String interpolation in log calls Every message is unique, nothing is queryable, and the string is built even when the level is off Message templates, ideally [LoggerMessage]
Forgetting AddSource / AddMeter for your own names Custom spans and metrics are silently dropped, and you think the code doesn't run Subscribe to your names in one shared place ("Ledgerly" in the service defaults)
Calling activity.SetTag without ?. NullReferenceException whenever no listener is attached, typically in tests activity?.SetTag(...)
Initializing both UseAzureMonitor() and UseMicrosoftOpenTelemetry() 🔧 Duplicate or conflicting pipelines, per the docs Pick one distro per app
Health probes in traces Thousands of meaningless spans per hour per replica, and an inflated bill Filter /health and /alive out, as ConfigureOpenTelemetry does
User ids or import ids as metric tags Cardinality explosion, slow queries, cost High-cardinality values go on spans and logs only
Assuming the trace survives a queue or a table The worker's span starts a new trace, and you can't connect cause and effect Store traceparent (outbox TraceParent column) and start the next span from it

🧰 Helpers

A [LoggerMessage] starter you can paste into any class:

// illustrative, not in the sample
public sealed partial class SomethingHandler(ILogger<SomethingHandler> logger)
{
    [LoggerMessage(Level = LogLevel.Information, Message = "Order {OrderId} placed with {LineCount} lines")]
    private static partial void LogOrderPlaced(ILogger logger, Guid orderId, int lineCount);

    [LoggerMessage(Level = LogLevel.Error, Message = "Order {OrderId} failed")]
    private static partial void LogOrderFailed(ILogger logger, Guid orderId, Exception ex);
}

OTLP variables worth knowing (Aspire sets these for you locally):

Variable Purpose
OTEL_SERVICE_NAME service.name, shown as the role name in Application Insights
OTEL_RESOURCE_ATTRIBUTES Extra resource attributes, for example deployment.environment=beta
OTEL_EXPORTER_OTLP_ENDPOINT Where to send OTLP
OTEL_EXPORTER_OTLP_PROTOCOL grpc or http/protobuf
OTEL_EXPORTER_OTLP_HEADERS Auth headers for a hosted OTLP backend

Make it permanent:


🧪 Try it yourself (optional, ~25 min)

You need a container runtime for Aspire.

  1. Start everything and open the dashboard link printed in the console: bash cd sample aspire run
  2. Create a token, then use src/Ledgerly.Api/Ledgerly.Api.http to create a portfolio and upload a small broker CSV to the imports endpoint. bash dotnet user-jwts create --project src/Ledgerly.Api
  3. In Structured logs, filter by the worker resource and find "Import ... completed". Expand it and look at the properties: ImportId, Outcome, RowsImported, RowsSkipped, plus the trace id.
  4. Click the trace id. In the waterfall, find the API's POST span, the outbox publish imports span, and the import process span. Check whether they share one trace id or the trace splits at Service Bus. Either answer is a finding worth writing down.
  5. Open Metrics, choose the worker, and find ledgerly.imports.processed. Look at the outcome dimension. Then find http.server.request.duration on the api and check that no /health or /alive requests appear in Traces.
  6. Break something on purpose: in src/Ledgerly.Api/Services/ValuationService.cs, remove ({Reason}) from the LogPriceUnavailable template so the reason parameter is unused. Build with dotnet build Ledgerly.slnx and look for SYSLIB1015; with warnings as errors it fails the build. Revert the change.

✅ Recap

  1. Logs are structured events, not lines in a file: use message templates and [LoggerMessage], and put business ids in properties so you can query them.
  2. OpenTelemetry unifies traces (Activity), metrics (Meter) and logs (ILogger). The service defaults subscribe to "Ledgerly", filter probe noise, and pick OTLP or Azure Monitor from configuration (one distro, never two).
  3. Traces cross HTTP automatically but not tables or queues: Ledgerly stores TraceParent in the outbox and continues it in the relay, and you keep secrets, PII and high-cardinality tags out of telemetry.

Next → Chapter 16: Observability II: Azure Monitor, KQL and SLOs

Chapter 16: Observability II: Azure Monitor, KQL and SLOs

⏱ About 35 min · You'll learn: where Ledgerly's telemetry lands in Azure, how to answer real questions with KQL, and how to turn "is it healthy?" into SLIs, SLOs, error budgets and alerts that don't wake people up for nothing.


📖 Story: prove it with a number

Elena wants to open the beta to five hundred more users next month. In the planning meeting she asks the obvious question: "Is the app healthy enough for that?" You say yes. It feels healthy: the dashboard was green when you last looked, and nobody has complained since the import fix.

Priya puts her coffee down. "Is the app healthy? Prove it with a number." She isn't being difficult. She's the one who will carry the pager when five hundred more people arrive, and she has been burned before by "it feels fine". At her last job, an alert fired on every single exception, and after two weeks everyone had a mail rule that moved those alerts to a folder nobody opened. "I don't want more alerts," she says. "I want fewer, better ones. Tell me what 'healthy' means for Ledgerly, measure it, and page me only when users are actually hurting."

You think back to Nordbank. Health there was SCOM turning a server icon red, and a monthly uptime number someone typed into a slide. You open your notebook under Habits I brought with me and write: "Healthy = the server is up." Then you open the Application Insights resource that monitoring.bicep created and start learning KQL.

All targets, percentages, request counts and budgets in this chapter are illustrative. Product behavior and dates from the docs are marked 🔧.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
SCOM or perfmon watching CPU and disk on named servers Application Insights and a Log Analytics workspace receiving OpenTelemetry from every replica There are no named servers, and CPU doesn't tell you whether users are happy
Grep and Excel to answer a question about last week KQL queries over AppRequests, AppDependencies, AppTraces and friends Telemetry is a queryable dataset, not a pile of text
An email for every exception Alerts on symptoms users feel (error-budget burn rate), routed by action groups Noisy alerts get ignored, and ignored alerts are worse than none
"Uptime 99.9%" typed into a monthly slide SLIs computed from real requests, SLOs agreed with the product owner, error budgets that drive priorities A target you don't measure is a wish
A ping to default.aspx from a monitoring box Availability standard tests from several Azure regions URL ping tests are deprecated 🔧, and one location can't tell you about the internet
Log everything at Debug "just in case" Sampling, log levels, a daily cap and retention chosen on purpose Ingestion is billed by volume 💰

💡 Healthy isn't a feeling or a green icon. It's a number your users would agree with, measured continuously.


🧠 Where the telemetry lands

In chapter 15 you saw the switch in the service defaults: when APPLICATIONINSIGHTS_CONNECTION_STRING is set, UseAzureMonitor() exports traces, metrics and logs to Application Insights. In Azure, Bicep sets that variable for every container app, and the resource it points to is created by one small module.

💻 In Ledgerly: one workspace, one Application Insights

📄 sample/infra/modules/monitoring.bicep

// Observability backbone: one Log Analytics workspace (logs of the platform + apps) and
// one workspace-based Application Insights resource (OpenTelemetry traces, metrics, logs).
// ...
resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' = {
  name: 'log-${name}'
  location: location
  tags: tags
  properties: {
    sku: { name: 'PerGB2018' }
    retentionInDays: 30
    workspaceCapping: { dailyQuotaGb: 1 } // 💰 a hard daily cap keeps a learning subscription safe
  }
}

resource appInsights 'Microsoft.Insights/components@2020-02-02' = {
  name: 'appi-${name}'
  location: location
  tags: tags
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: workspace.id
  }
}

📄 sample/infra/modules/apps.bicep

var commonEnv = [
  { name: 'AZURE_CLIENT_ID', value: identityClientId } // makes Azure SDK credentials pick the user-assigned identity
  { name: 'APPLICATIONINSIGHTS_CONNECTION_STRING', value: appInsightsConnectionString }
]

WorkspaceResourceId is the important line. Classic Application Insights resources are retired; every Application Insights resource is now workspace-based, storing its data in a Log Analytics workspace and billed through Log Analytics (create a workspace-based resource). The Container Apps environment (environment.bicep) also sends its platform and console logs to the same workspace, so one place holds everything.

Two names for the same data

You'll query the same telemetry from two places, and the table names differ. The Application Insights Logs blade shows the classic names; the Log Analytics workspace shows the App* names. This course uses the workspace names, because that's where the data really lives and where cross-resource queries work.

Application Insights name Workspace table What lands there from Ledgerly
requests AppRequests Incoming HTTP requests (server spans)
dependencies AppDependencies Outgoing calls: HttpClient to marketdata, SQL, Service Bus
traces AppTraces ILogger records, including every [LoggerMessage]
exceptions AppExceptions Exceptions recorded on logs and spans
customMetrics AppMetrics Meter instruments, for example ledgerly.imports.processed
availabilityResults AppAvailabilityResults Availability test results

Columns are renamed too: timestamp becomes TimeGenerated and cloud_RoleName becomes AppRoleName (data model). AppRoleName comes from OTEL_SERVICE_NAME, which is why apps.bicep sets it to api, worker and marketdata. Your custom spans also appear as requests or dependencies depending on their kind; check the data model page when a span isn't where you expect it.

The portal experiences sit on top of the same data. Application Map draws api, worker, marketdata, SQL and Service Bus as a graph keyed by role name, with call volume and failure rates (Application Map). Transaction search and the end-to-end transaction view follow one W3C trace across services, which is the cloud version of what you did in the Aspire dashboard. Failures, Performance and Live Metrics cover the rest.

flowchart LR
    subgraph ACA["Container Apps environment"]
        API["api"]
        WRK["worker"]
        MD["marketdata"]
        ENV["Environment<br/>platform and console logs"]
    end
    API -- "OTel via UseAzureMonitor" --> AI["Application Insights<br/>appi-name"]
    WRK -- "OTel via UseAzureMonitor" --> AI
    MD -- "OTel via UseAzureMonitor" --> AI
    ENV -- "appLogsConfiguration" --> LAW["Log Analytics workspace<br/>log-name, 30 days, 1 GB per day cap"]
    AI -- "stores in" --> LAW
    LAW --> KQL["KQL queries"]
    KQL --> WB["Workbooks and Grafana"]
    KQL --> AL["Log search alerts"]
    AL --> AG["Action group"]

💡 One workspace per environment, every service tagged with its role name: that's what makes "show me everything about this request" a single query.


🧠 KQL: the queries you'll actually run

Kusto Query Language (KQL) reads top to bottom: start with a table, then pipe through filters and aggregations. If you know SQL and LINQ, you'll be productive in an hour. The pattern from the docs is the one you'll use most:

AppRequests
| where TimeGenerated > ago(1h)
| summarize count(), p95=percentile(DurationMs, 95) by Name, bin(TimeGenerated, 5m)

The eight queries below are starting points, written for Ledgerly's names. TimeGenerated, AppRoleName, Name and DurationMs come straight from the docs; other columns (Success, Target, Properties, Message, Sum, ItemCount) come from the workspace schema, so check them in the Logs view's schema pane before you build alerts on them 🔧. The full set, with comments, is in kql-queries.md in the toolkit.

1. Failure rate per service, hourly

AppRequests
| where TimeGenerated > ago(24h)
| summarize total = count(), failed = countif(Success == false) by AppRoleName, bin(TimeGenerated, 1h)
| extend failurePct = round(100.0 * failed / total, 2)
| order by TimeGenerated desc

2. p95 latency per operation

AppRequests
| where TimeGenerated > ago(1h)
| where AppRoleName == "api"
| summarize requests = count(), p95 = percentile(DurationMs, 95) by Name
| order by p95 desc

Operation names come from route templates (for example the valuation route with {id:guid}), so all portfolios group into one row. That's the low-cardinality naming from chapter 15 paying off.

3. Market data dependency health

AppDependencies
| where TimeGenerated > ago(6h)
| where Target has "marketdata"
| summarize calls = count(), failed = countif(Success == false), p95 = percentile(DurationMs, 95) by bin(TimeGenerated, 5m)
| render timechart

During Priya's chaos demo in chapter 14, this chart would show a wall of failures and a p95 near the attempt timeout.

4. What users saw: partial valuations

AppTraces
| where TimeGenerated > ago(6h)
| where Message has "partial valuation"
| summarize partialPrices = count() by bin(TimeGenerated, 5m)

This counts LogPriceUnavailable from ValuationService. Query 3 tells you the vendor is failing; query 4 tells you how many prices users didn't get. Elena cares about the second one.

5. Import throughput by outcome

AppMetrics
| where TimeGenerated > ago(24h)
| where Name == "ledgerly.imports.processed"
| extend outcome = tostring(Properties["outcome"])
| summarize imports = sum(Sum) by outcome, bin(TimeGenerated, 1h)
| render columnchart

Metrics arrive pre-aggregated, so you sum the Sum column rather than counting rows. The outcome tag is one of "completed", "completed_with_errors" or "rejected".

6. Imports heading for the dead-letter queue

union AppTraces, AppExceptions
| where TimeGenerated > ago(24h)
| where isnotempty(Properties["DeliveryCount"])
| extend importId = tostring(Properties["ImportId"]), delivery = toint(Properties["DeliveryCount"])
| summarize lastDelivery = max(delivery), lastSeen = max(TimeGenerated) by importId
| where lastDelivery >= 8

LogImportCrashed logs "Import {ImportId} failed on delivery {DeliveryCount}; abandoning for retry". With maxDeliveryCount: 10, an import that reaches delivery 10 and fails is in the dead-letter queue next. The query unions both tables because a log with an exception attached may be stored as an exception. Note the gap Priya found: the ImportNotFound path dead-letters without logging, so this query can't see it. Service Bus also exposes namespace metrics in Azure Monitor (dead-lettered message counts among them, check the docs), and an alert on that metric covers every path.

7. Cold starts per service

AppTraces
| where TimeGenerated > ago(7d)
| where Message has "Application started"
| summarize starts = count() by AppRoleName, bin(TimeGenerated, 1d)
| order by TimeGenerated desc

ASP.NET Core logs "Application started" once per process at Information level, and Ledgerly's appsettings.json keeps that category at Information. With apiMinReplicas = 0 (the default in main.bicepparam), every idle period ends with a cold start for the first user. If this number is high and users complain about the first request, that's the evidence for setting it to 1 💰.

8. Top exceptions

AppExceptions
| where TimeGenerated > ago(24h)
| summarize occurrences = count() by AppRoleName, ExceptionType, OuterMessage
| top 10 by occurrences

💡 Write queries in terms of what users experience (requests, partial results, stuck imports), then add the cause queries (dependencies, exceptions) you'll use to explain it.


🧠 SLIs, SLOs and error budgets

This is the answer to Priya's question. The vocabulary comes from the Google SRE Workbook (implementing SLOs):

Term Meaning Ledgerly example (illustrative)
SLI (indicator) A measured ratio of good events to all events Valuation requests that return 2xx in under 1 s, divided by all valuation requests
SLO (objective) A target for the SLI over a window 99.5% over 28 days
Error budget 1 − SLO: how much badness you're allowed 0.5% of valuation requests per 28 days
Burn rate How fast you're spending the budget compared to "exactly on target" A burn rate of 1 spends the budget in exactly 28 days. A burn rate of 10 spends it in 2.8 days

You and Elena wrote three SLOs, using slo-worksheet.md from the toolkit:

User journey SLI: good events SLO
View portfolio and valuation api requests on portfolio routes that succeed in under 1 s 99.5% over 28 days
Record a trade POST transaction requests that don't return 5xx 99.9% over 28 days
Import a broker file Imports that reach a final status within 5 minutes of upload 99% over 28 days

Two discussions came up that you'll have too:

The math, with illustrative traffic: if the API serves 200,000 valuation requests in 28 days, a 99.5% SLO allows 1,000 bad ones. If a bad deploy causes 2% of requests to fail, the burn rate is 2% ÷ 0.5% = 4, and the whole month's budget is gone in a week.

An SLI and burn rate in KQL

// illustrative SLO: 99.5% of valuation requests succeed in under 1 s
let slo = 0.995;
AppRequests
| where TimeGenerated > ago(1h)
| where AppRoleName == "api" and Name has "valuation"
| summarize total = count(), good = countif(Success == true and DurationMs < 1000)
| extend sli = todouble(good) / total
| extend burnRate = (1.0 - sli) / (1.0 - slo)

Run it over 1 hour and over 6 hours. A high burn rate over a short window means something broke now; a moderate burn rate over a long window means a slow leak. The SRE Workbook recommends alerting on burn rate instead of raw thresholds, and checking a short and a long window together keeps alerts both fast and quiet.

💡 An SLO is a product decision with a number on it. Agree on it with the product owner, publish it, and let the error budget settle the "features or reliability?" argument.


🧠 Dashboards and alerts that don't cry wolf

Workbooks and Grafana

A Workbook is an interactive report in the portal that combines KQL queries, metrics and parameters (a time range, an environment). Ledgerly's "Service health" workbook has four tiles: SLI per journey (the burn-rate query), failure rate per service (query 1), market data health (queries 3 and 4) and import outcomes (query 5). If your team already lives in Grafana, Azure Managed Grafana and the built-in Dashboards with Grafana in Application Insights 🔧 read the same data (Grafana dashboards).

Alert types

Alert type Evaluates Use it for in Ledgerly
Metric alert A platform or custom metric against a threshold, evaluated frequently Service Bus dead-lettered messages above zero (check the metric name in the Service Bus docs)
Log search alert A KQL query on a schedule SLO burn rate, imports at delivery 8 or more
Activity log alert Control-plane events in the subscription Someone deleted or reconfigured a resource
Smart detection Anomalies Application Insights finds by itself A free extra signal, not a paging source

Action groups define who gets told and how: email, SMS, voice, push, webhooks, Logic Apps, Functions and more. They're reused across alerts. Email receivers now require one-time-passcode verification 🔧, so new recipients have a step to complete. Alert processing rules suppress notifications during planned maintenance (alerts overview, action groups).

Priya's alerting policy, which the team adopted:

Severity Condition (illustrative) Goes to
Page Burn rate above 10 over 1 hour and above 10 over 5 minutes On-call phone
Ticket Burn rate above 2 over 6 hours Team channel, handled next working day
Ticket Dead-lettered imports above zero Team channel, with the runbook link
No alert A single exception, a single failed request, high CPU without user impact Dashboards only

The second, short window on the page condition makes the alert stop firing quickly once the problem is fixed. Every alert that pages must link a runbook (runbook-template.md in the toolkit); chapter 19 covers on-call and postmortems.

Availability tests

Burn-rate alerts need traffic. At 3 a.m. there may be none, so an outage can go unnoticed until the first user arrives. An availability test sends synthetic requests from several Azure regions. Use standard tests: they support SSL certificate checks, HTTP verbs and custom headers. URL ping tests are deprecated and retire on 2028-09-30 🔧 (the date was extended, so older articles say 2026). See availability tests. Results land in AppAvailabilityResults.

For Ledgerly, point the test at the API's /health endpoint. It checks that the replica can reach its essential dependencies, it is mapped in every environment, and it reveals nothing beyond one word. If apiMinReplicas is 0, remember that the test itself keeps waking the API, which removes cold starts for users but also costs replica time 💰.

💡 Page on symptoms that burn the error budget, ticket on causes, and put everything else on a dashboard. Fewer alerts, each one meaning "a human must act now".


🗺️ From bad requests to a page

flowchart TB
    R["AppRequests<br/>valuation requests"] --> SLI["SLI query<br/>good divided by total"]
    SLI --> BR{"Burn rate"}
    BR -- "above 10 over 1 h and 5 min" --> PG["Page on-call<br/>action group"]
    BR -- "above 2 over 6 h" --> TK["Ticket<br/>team channel"]
    BR -- "below 2" --> OK["Dashboard only"]
    PG --> RB["Runbook"]
    TK --> RB
    AV["Availability standard test<br/>/health from regions"] -- "fails" --> PG
    DLQ["Dead-lettered imports"] -- "above zero" --> TK

🧠 Sampling and cost 💰

Log Analytics ingestion is often the surprise cost of a cloud-native system. Every request, dependency call, log line and metric point is billed by volume, and a chatty service at Debug level can cost more than the service itself. The levers, from most to least effective:

Lever Where in Ledgerly Trade-off
Don't produce noise Probes filtered out of traces; health endpoints excluded from HTTP metrics; Microsoft.EntityFrameworkCore.Database.Command and Azure.Messaging.ServiceBus at Warning in appsettings.json None, really. Do this first
Sampling The distro's default sampling 🔧 You keep representative traces, not every one
Retention retentionInDays: 30 in monitoring.bicep Longer retention for audits costs more
Daily cap dailyQuotaGb: 1 in monitoring.bicep A hard cap: once it's hit, you're blind until the next day

Sampling. Per the sampling docs, the Microsoft distro defaults to rate-limited sampling of 5 traces per second 🔧, fixed-percentage sampling is the alternative, and trace-based log sampling is on by default in .NET: logs that belong to a trace that wasn't sampled are dropped too. That keeps traces complete (you never get half a trace), but it means a rare log line can disappear at high volume. Two consequences for your queries:

The daily cap is a safety net, not a budget. One GB a day is right for a learning subscription: a logging bug can't produce a shocking bill. For the production environment, the team raised it and set a Cost Management budget with alerts instead (chapter 19), because going blind at 4 p.m. during an incident is worse than paying for a few extra gigabytes.

To see what's filling the workspace, count rows by table and service:

union withsource = TableName AppRequests, AppDependencies, AppTraces, AppExceptions, AppMetrics
| where TimeGenerated > ago(1d)
| summarize rows = count() by TableName, AppRoleName
| order by rows desc

For prices, use the Azure Pricing Calculator; per-GB ingestion and retention prices change and vary by region 🔧.

💡 Cut noise at the source first, sample second, cap last. Metrics are your exact counters; traces and logs are your representative evidence.


⚠️ Common traps

Trap What happens Do this instead
Querying classic names in one place and App* tables in another Queries "don't work" when copied between the Application Insights blade and the workspace Pick the workspace names (AppRequests, TimeGenerated, AppRoleName) and keep a shared query file
Alerting on every exception or a fixed error count Alert fatigue, mail rules, and the one alert that matters gets missed Alert on SLO burn rate over two windows; send causes to dashboards and tickets
Counting rows on sampled data Request totals look far lower than reality Use ratios, sum(ItemCount) 🔧, or metrics for exact counts
Setting a 100% SLO, or none Either nothing can ever ship, or "healthy" stays a feeling Pick a target users would accept, and an error budget policy agreed with the product owner
Relying only on request-based alerts At night there's no traffic, so an outage stays silent until the first user hits it Add an availability standard test (not URL ping, which retires 🔧)
A low daily cap in production Telemetry stops mid-incident, exactly when you need it Use the cap for dev and learning, and budgets plus sampling for production 💰
Assuming every failure path logs ImportNotFound dead-letters silently, and log queries miss it Pair log alerts with platform metrics (dead-lettered message counts)

🧰 Helpers

The SLO one-pager (the full version is slo-worksheet.md in the toolkit):

Journey:            View portfolio valuation
SLI (good events):  api requests on valuation route, Success == true and DurationMs < 1000
SLI (all events):   api requests on valuation route
SLO:                99.5% over 28 days
Error budget:       0.5% of requests
Page:               burn rate > 10 over 1 h and 5 min
Ticket:             burn rate > 2 over 6 h
Budget policy:      when spent, the next sprint prioritizes reliability work
Owner:              Priya (SRE) and Elena (product)

Make it permanent:


🧪 Try it yourself (optional, ~40 min)

These steps need the Azure environment from chapter 12. If you don't have it, run the KQL in your head against the tables above, and do step 1 only.

  1. Locally, start the system with cd sample and aspire run, create some traffic with src/Ledgerly.Api/Ledgerly.Api.http, and look at the Traces and Metrics views in the dashboard. Every concept in this chapter has a local twin there.

🧪 Not run by the author: this step was written from the docs and validated with bicep build only. Expect small differences.

  1. In the Azure portal, open the Application Insights resource appi-<name>, then Application Map. Check that api, worker and marketdata appear with their dependencies.
  2. Open the Log Analytics workspace log-<name>, then Logs. Run queries 1, 2 and 7 from this chapter. Note which columns exist in the schema pane and adjust the queries if your schema differs.
  3. Save the burn-rate query, then create a log search alert from it that fires when burnRate is above 10. Create an action group with your email (complete the verification step 🔧).
  4. Create an availability standard test against https://<api-fqdn>/health. Wait for results, then query AppAvailabilityResults.
  5. Optional: if your environment allows it, temporarily raise market data failures (in Azure, only through configuration, because /chaos exists only in Development) and watch queries 3 and 4 react. Reset it afterwards.
  6. Fill in slo-worksheet.md for one Ledgerly journey you care about.

✅ Recap

  1. Ledgerly's telemetry lands in one workspace-based Application Insights resource backed by Log Analytics; query it with KQL using the App* tables, TimeGenerated and AppRoleName.
  2. "Healthy" is an SLI with an SLO and an error budget agreed with the product owner, and alerts page only when the burn rate says users are hurting, with availability tests covering quiet hours.
  3. Ingestion is the surprise cost 💰: cut noise at the source, rely on sampling and exact metrics, and treat the daily cap as a learning-environment safety net.

Next → Chapter 17: Testing strategy

Chapter 17: Testing strategy

⏱ About 30 min · You'll learn: how to shape a test pyramid for a small cloud-native system, how to run integration tests against a real SQL Server with WebApplicationFactory and Testcontainers, why the .NET 10 SDK pushed Ledgerly onto Microsoft.Testing.Platform, and where end-to-end and contract tests fit.


📖 Story: The 409 on every trade

It is Tuesday morning and Elena forwards a message from a beta tester: "I can't record any trades. Every time I press Save it says the portfolio changed and I should retry. I retried eleven times." You try it on the beta environment and get the same answer, a 409 Conflict with "Portfolio changed while recording the trade. Retry the request."

The unit tests are green. HoldingCalculator is correct, the CSV parser is correct, and the price generator is correct. The bug isn't in any of them. It lives in the seam between your code and the database: a small refactor added the new Transaction through portfolio.Transactions instead of through the DbContext, and EF Core decided the row already existed. Nothing in the test suite sent a real trade to a real database.

Maya pulls up a chair. "At Nordbank, how did you test the data layer?" You think about it. A test database on a shared server that someone restored every few weeks, a few NUnit tests that only ran on the build machine, and a lot of manual clicking before each release. "So mostly in production," she says, kindly. You open your notebook to Habits I brought with me and write: "Unit tests for logic, manual testing for everything else."

"Let's write the test that would have caught it," Maya says. "Then let's make sure it can't be skipped by accident."

Numbers in the Ledgerly story (test durations, tester counts) are illustrative. The 23 tests in the sample (15 unit, 8 integration against real SQL Server) are real.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
MSTest or NUnit tests run by vstest.console on the build server xUnit v3 test projects that are executables, run by Microsoft.Testing.Platform through dotnet test The .NET 10 SDK supports MTP mode natively, and xUnit v3 is built for it
A shared test database restored "every few weeks" A throwaway SQL Server per test run (Testcontainers) or a unique LocalDB database per run Tests that share state fail randomly, and nobody trusts them
The EF in-memory provider, or mocked repositories The real API in memory (WebApplicationFactory<Program>) against a real SQL Server The bugs that hurt are in the seams: SQL translation, keys, concurrency tokens, migrations
DateTime.Now in domain code and Thread.Sleep in tests TimeProvider in code, FakeTimeProvider in tests Time becomes a dependency you control, so tests are fast and deterministic
Manual regression clicking before each release End-to-end smoke tests of the whole system, and contract checks for the SPA Releases are frequent now, so humans can't be the regression suite
Coverage from a separate tool with its own runner dotnet test --coverage with Microsoft.Testing.Extensions.CodeCoverage One runner, one command

💡 A test suite is a portfolio. You spend most on cheap, fast tests and buy a few expensive ones only where the cheap ones can't see.


🧠 Concept 1: A test pyramid for this system

The classic pyramid says: many unit tests, fewer integration tests, very few end-to-end tests. The shape depends on where your bugs live. Ledgerly has a small domain and a lot of plumbing (HTTP, auth, EF Core, SQL, Service Bus, a second service). So the useful layers are:

Layer What it proves in Ledgerly Speed Where it runs
Unit Holding math, alert rules, CSV parsing, price generation Milliseconds Every build, every PR
Integration (in-process API plus real SQL) Endpoints, auth, validation, EF mappings, concurrency, degradation Seconds Every PR (CI has Docker)
Contract The OpenAPI document Leo's SPA depends on hasn't changed by accident Seconds Every PR
End-to-end (whole AppHost) Services find each other, health checks pass, messages flow Minutes Nightly, or before a release

Unit tests in Ledgerly follow two habits that pay off quickly. First, time is injected. PriceGenerator takes a TimeProvider, so the test controls the clock instead of sleeping:

💻 In Ledgerly: a clock you can move

📄 sample/tests/Ledgerly.UnitTests/PriceTests.cs

    [Fact]
    public void Prices_are_stable_within_a_minute_and_move_over_time()
    {
        // TimeProvider makes "time" a dependency you can control in tests. No more DateTime.Now in domain code.
        var clock = new FakeTimeProvider(new DateTimeOffset(2026, 10, 28, 12, 0, 0, TimeSpan.Zero));
        var generator = new PriceGenerator(clock);

        var first = generator.GetQuote("aapl");
        clock.Advance(TimeSpan.FromSeconds(20));
        var sameMinute = generator.GetQuote("AAPL");
        clock.Advance(TimeSpan.FromHours(3));
        var later = generator.GetQuote("AAPL");
        // ...
    }

Three hours pass in a microsecond. FakeTimeProvider comes from the Microsoft.Extensions.TimeProvider.Testing package, and production code gets TimeProvider.System instead.

Second, every production bug becomes a test. The old IIS server had a Lithuanian culture, and "190,10" broke imports. The fix was invariant-culture parsing. The test pins the culture to lt-LT and proves the fix holds:

💻 In Ledgerly: the culture regression test

📄 sample/tests/Ledgerly.UnitTests/BrokerCsvParserTests.cs

    [Fact]
    public void Decimal_parsing_ignores_server_culture()
    {
        // The old IIS box had lt-LT culture and "190,10" broke imports. Invariant culture, always.
        var previous = Thread.CurrentThread.CurrentCulture;
        Thread.CurrentThread.CurrentCulture = new System.Globalization.CultureInfo("lt-LT");
        try
        {
            var result = BrokerCsvParser.Parse($"{Header}\n2026-09-01,buy,AAPL,5,190.10,BRK-1");
            Assert.Equal(190.10m, Assert.Single(result.Trades).Price);
        }
        finally
        {
            Thread.CurrentThread.CurrentCulture = previous;
        }
    }

Note the try/finally. A test that changes shared state (culture, environment variables, a static flag) must put it back, or the next test inherits it and fails for no visible reason.

Ledgerly uses xUnit v3 (xunit.v3 4.0.2 🔧). xUnit v2 is in maintenance mode: critical fixes only, with all new features going to v3. v3 test projects are executables (OutputType Exe), which is what lets them run on Microsoft.Testing.Platform (Concept 3). If your team prefers another framework, MSTest and NUnit also support MTP, and TUnit (1.76.28 🔧) is a newer, MTP-only, source-generated framework that runs tests in parallel by default.

💡 Unit tests are for logic that has no I/O. If a test needs a mock of DbContext to say anything, it probably belongs one layer up.


🧠 Concept 2: Integration tests against a real database

WebApplicationFactory<TEntryPoint> (package Microsoft.AspNetCore.Mvc.Testing, 10.0.14 🔧) starts your real API in memory on a TestServer (integration tests docs). Requests go through the real middleware, routing, authentication, validation and EF Core code. Only the network socket is missing. Because Program.cs uses top-level statements, it ends with public partial class Program; so the test project can see it.

The question is what to put behind it. Ledgerly answers: a real SQL Server, never the EF in-memory provider. The factory looks for one in two places.

💻 In Ledgerly: where the database comes from

📄 sample/tests/Ledgerly.IntegrationTests/LedgerlyApiFactory.cs

public sealed class LedgerlyApiFactory : WebApplicationFactory<Program>, IAsyncLifetime
{
    // ...
    public async ValueTask InitializeAsync()
    {
        var external = Environment.GetEnvironmentVariable("LEDGERLY_TEST_SQL");
        if (!string.IsNullOrWhiteSpace(external))
        {
            var csb = new Microsoft.Data.SqlClient.SqlConnectionStringBuilder(external)
            {
                InitialCatalog = $"LedgerlyTests_{Guid.NewGuid():N}",
            };
            _connectionString = csb.ConnectionString;
        }
        else
        {
            try
            {
                _container = new MsSqlBuilder("mcr.microsoft.com/mssql/server:2022-latest").Build();
                await _container.StartAsync();
                _connectionString = _container.GetConnectionString();
            }
            catch (Exception ex) when (ex is not OutOfMemoryException)
            {
                SkipReason = $"No SQL Server available (set LEDGERLY_TEST_SQL or start Docker): {ex.GetType().Name}";
                return;
            }
        }

        await using var scope = Services.CreateAsyncScope();
        await scope.ServiceProvider.GetRequiredService<LedgerlyDbContext>().Database.MigrateAsync();
    }

There are three ideas packed in here:

The rest of the factory swaps the parts you don't want to be real:

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.UseEnvironment("Testing");

        // UseSetting is applied before Program.cs reads configuration, so Aspire's client integration sees it.
        builder.UseSetting("ConnectionStrings:ledgerlydb", _connectionString ?? "Server=unused;Database=unused");

        builder.ConfigureLogging(logging => logging.SetMinimumLevel(LogLevel.Warning));

        builder.ConfigureTestServices(services =>
        {
            services.RemoveAll<IMarketDataClient>();
            services.AddSingleton<IMarketDataClient>(MarketData);

            services.AddAuthentication(TestAuthHandler.SchemeName)
                .AddScheme<AuthenticationSchemeOptions, TestAuthHandler>(TestAuthHandler.SchemeName, _ => { });
        });
    }

💻 In Ledgerly: the bug that started this chapter

With that factory in place, you write a test that buys 5 shares and checks for 201 Created. It fails with 409. Now you can see the trap, and the fix sits in the endpoint with a comment so nobody "simplifies" it back:

📄 sample/src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs

        // EF trap: the Id is already set (Guid v7, generated in C#). Adding via portfolio.Transactions would make EF
        // assume the row EXISTS and send an UPDATE (0 rows -> DbUpdateConcurrencyException). Add it explicitly.
        db.Transactions.Add(transaction);
        portfolio.UpdatedAt = DateTimeOffset.UtcNow; // bumps rowversion: two parallel "sell all" requests cannot both win

        try
        {
            await db.SaveChangesAsync(ct);
        }
        catch (DbUpdateConcurrencyException)
        {
            return TypedResults.Problem("Portfolio changed while recording the trade. Retry the request.", statusCode: StatusCodes.Status409Conflict);
        }

The entity key is a client-generated Guid.CreateVersion7(). When EF discovers a new object in a tracked navigation collection and its key is already set, it assumes the row exists and sends an UPDATE. Zero rows are affected, and EF reports that as a concurrency conflict. The endpoint's honest 409 handler then made the bug look like a user problem. Chapter 5 covers the EF side. The testing lesson is simpler: no unit test could see this, because it only happens when EF talks to a real database.

💡 Mock what you don't own and can't run (a vendor API). Run what you can (your database, your API, your migrations).


🗺️ How the pieces fit

The first diagram is Ledgerly's test portfolio. Width is roughly "how many tests", and the label says when each layer runs.

flowchart TB
    E2E["End-to-end<br/>Aspire testing builder, whole AppHost<br/>nightly"]
    CON["Contract<br/>OpenAPI document diff for Leo's SPA<br/>every PR"]
    INT["Integration<br/>WebApplicationFactory plus real SQL<br/>8 tests, every PR"]
    UNIT["Unit<br/>holdings, alerts, CSV, prices<br/>15 tests, every PR"]
    E2E -- "fewer, slower, broader" --> CON
    CON --> INT
    INT -- "more, faster, narrower" --> UNIT

The second diagram is one run of the integration test class. The factory is created once per test class (IClassFixture<LedgerlyApiFactory>), so the database and the in-memory server are shared by the eight tests, and each test uses its own user and its own portfolio.

sequenceDiagram
    participant X as "xUnit v3 on MTP"
    participant F as "LedgerlyApiFactory"
    participant S as "SQL Server"
    participant A as "Api in TestServer"
    participant M as "FakeMarketData"
    X->>F: InitializeAsync
    F->>S: LEDGERLY_TEST_SQL or Testcontainers
    F->>S: MigrateAsync on a fresh database
    X->>F: CreateClientFor carol
    X->>A: POST transaction with X-Test-User header
    A->>S: INSERT transaction and bump rowversion
    A-->>X: 201 Created
    X->>A: GET valuation
    A->>M: GetQuotesAsync
    M-->>A: prices or HttpRequestException
    A-->>X: 200 with IsComplete true or false
    X->>F: DisposeAsync drops database or container

🧠 Concept 3: Microsoft.Testing.Platform on the .NET 10 SDK

The first time you ran the new suite with the .NET 10 SDK, dotnet test didn't run a single test. It stopped with:

Testing with VSTest target is no longer supported by Microsoft.Testing.Platform on .NET 10 SDK

Here's what happened. dotnet test has two modes. VSTest mode is the default and was the only mode before .NET 10. MTP mode is new in the .NET 10 SDK 🔧. Microsoft.Testing.Platform is a lightweight runner embedded in the test project itself, so a v3 test project is just an executable you can also start with dotnet run. Running MTP projects through VSTest mode is now legacy, and MTP v2 removes that support when you use the .NET 10 SDK. That's the wall Ledgerly hit with xUnit v3 4.x: the old path was closed.

The fix is one setting at the repository root:

💻 In Ledgerly: opting in to MTP

📄 sample/global.json

{
  "sdk": {
    "version": "10.0.100",
    "rollForward": "latestFeature"
  },
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}

MTP mode requires MTP 1.7 or later 🔧 (dotnet test docs). After the switch, you name the solution with --solution, and MTP options go directly on the command line (in VSTest mode they came after --).

Then you clean up the test projects. The VSTest-era packages (Microsoft.NET.Test.Sdk, xunit.runner.visualstudio, coverlet.collector) are gone. Coverage now comes from an MTP extension:

📄 sample/tests/Ledgerly.IntegrationTests/Ledgerly.IntegrationTests.csproj

  <PropertyGroup>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
    <OutputType>Exe</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="xunit.v3" />
    <PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" />
    <PackageReference Include="Testcontainers.MsSql" />
  </ItemGroup>

There are no versions here because Central Package Management keeps them in Directory.Packages.props (Chapter 3). The CI step uses the same command you run locally, plus coverage and a TRX report:

📄 .github/workflows/ci.yml

      # Integration tests start a throwaway SQL Server container with Testcontainers (Docker is on the runner).
      - name: Test
        run: >
          dotnet test --solution Ledgerly.slnx -c Release --no-build
          --coverage --coverage-output-format cobertura
          --report-xunit-trx

The workflow then uploads sample/TestResults/ as an artifact even when tests fail, so a red build always comes with evidence (Chapter 9).

One MTP behavior surprises people coming from VSTest: when zero tests run, MTP returns exit code 8, and dotnet test fails. Tomás hit it on the first nightly job, where a test filter had a typo and matched nothing. Under the old runner, that job would have stayed green forever. "I ran nothing" is not "everything passed". If a project legitimately runs no tests in some configuration, MTP lets you ignore specific exit codes; check the MTP docs for the current option 🔧, and scope it to that one job.

💡 A runner change is a good moment to delete packages. Every test package you no longer need is one less Dependabot PR and one less audit warning.


🧠 Concept 4: Tests that lie, and tests that hide

A passing test is a claim: "this behavior works". Two things can make the claim false, and Ledgerly met both.

A test that passes for the wrong reason. The first version of the oversell test bought 5 shares, sold 6, and expected 400 Bad Request. It passed. Weeks later, during the 409 investigation, you noticed something odd: the setup buy had been failing too (with the 409), so the portfolio held nothing, and selling 6 of nothing is also a validation error. The test proved a sentence that was true for an unrelated reason. The fix is to assert the setup, not just the outcome:

💻 In Ledgerly: asserting the arrange step

📄 sample/tests/Ledgerly.IntegrationTests/PortfolioApiTests.cs

    [Fact]
    public async Task Overselling_is_a_validation_error()
    {
        var client = ClientFor("carol");
        var portfolio = await CreatePortfolioAsync(client);

        await BuyAsync(client, portfolio.Id, "AAPL", 5);
        var oversell = await PostTradeAsync(client, portfolio.Id, "AAPL", TransactionType.Sell, 6);

        Assert.Equal(HttpStatusCode.BadRequest, oversell.StatusCode);
    }
    // ...
    private static async Task BuyAsync(HttpClient client, Guid portfolioId, string symbol, decimal quantity)
    {
        var response = await PostTradeAsync(client, portfolioId, symbol, TransactionType.Buy, quantity);
        Assert.Equal(HttpStatusCode.Created, response.StatusCode);
    }

BuyAsync now fails loudly if the buy doesn't return 201 Created, and CreatePortfolioAsync calls EnsureSuccessStatusCode(). A helper that hides a failed precondition is a test that lies. Whenever a test expects an error, ask "could this error come from the setup?" And watch every new test fail once before you trust it.

A test that hides. The factory skips integration tests when there's no SQL Server, which is friendly on a laptop. In CI it's a risk: if Docker were ever missing on the runner, eight tests would report "skipped" and the build would stay green, which is exactly how a regression slips through. Two cheap defenses:

// illustrative, not in the sample
catch (Exception ex) when (ex is not OutOfMemoryException)
{
    // Set LEDGERLY_REQUIRE_SQL=true in the CI workflow so a missing database fails the run instead of skipping it.
    if (string.Equals(Environment.GetEnvironmentVariable("LEDGERLY_REQUIRE_SQL"), "true", StringComparison.OrdinalIgnoreCase))
    {
        throw;
    }

    SkipReason = $"No SQL Server available (set LEDGERLY_TEST_SQL or start Docker): {ex.GetType().Name}";
    return;
}

Flaky tests are the third kind of liar: sometimes green, sometimes red, with no code change. In a cloud-native system, the usual causes are shared state, real time and real networks. Ledgerly's tests avoid all three on purpose:

Cause How Ledgerly avoids it
Two tests share data A fresh database per run, a different user per test (alice, bob, carol...), unique portfolio names (P-{Guid})
Shared switches leak between tests FakeMarketData.IsDown is reset in finally, and the culture test restores the culture
Real time FakeTimeProvider instead of Thread.Sleep or DateTime.Now
Real network to a vendor FakeMarketData replaces IMarketDataClient in integration tests
Hanging tests Every call passes TestContext.Current.CancellationToken, so the runner can cancel cleanly

When a test flakes, don't add a retry. Open an issue, skip the test with a reason that links to it, and fix it within the sprint. A suite people re-run "until it's green" has stopped being a test suite.

💡 Distrust a test you've never seen fail. Distrust a green build that skipped things even more.


🧠 Concept 5: The whole system, and the contract with Leo

Integration tests run one service in one process. They can't tell you whether the API can find marketdata through service discovery, whether the worker's health checks pass with a real Service Bus emulator, or whether the migration service finishes before the API starts. The AppHost from Chapter 6 already describes all of that, and the Aspire testing builder (package Aspire.Hosting.Testing) can run it inside a test, with real containers. Ledgerly doesn't have this test yet. A first smoke test could look like this:

// illustrative, not in the sample
// Test project references Aspire.Hosting.Testing and the Ledgerly.AppHost project.
public sealed class SystemSmokeTests
{
    [Fact]
    public async Task Api_is_healthy_when_the_whole_system_runs()
    {
        var builder = await DistributedApplicationTestingBuilder.CreateAsync<Projects.Ledgerly_AppHost>();
        await using var app = await builder.BuildAsync();
        await app.StartAsync();

        // Wait until the "api" resource reports healthy before calling it (see the Aspire testing docs).
        var http = app.CreateHttpClient("api");
        var response = await http.GetAsync(new Uri("/health", UriKind.Relative), TestContext.Current.CancellationToken);

        Assert.Equal(HttpStatusCode.OK, response.StatusCode);
    }
}

This test starts SQL Server, Redis, the Service Bus emulator and four .NET projects. That's real coverage, but it's slow and needs a container runtime, so it belongs in a nightly job. Keep end-to-end tests few and broad ("the system starts and is healthy", "an import gets processed"), never one per business rule.

Contract tests protect the boundary with Leo's React SPA. Leo generates TypeScript types from the API's OpenAPI document (served at /openapi/v1.json, and only in Development in Ledgerly's Program.cs). When you renamed totalMarketValue in a branch last month, his build broke a day later, after the merge. The fix is to make the contract a reviewed file. ASP.NET Core can generate the OpenAPI document at build time with Microsoft.Extensions.ApiDescription.Server. Commit the generated document, and let CI fail when the build produces a different one that nobody committed:

# illustrative, not in the sample
# After adding Microsoft.Extensions.ApiDescription.Server to Ledgerly.Api and pointing its output at sample/contracts/
dotnet build Ledgerly.slnx -c Release
git diff --exit-code -- contracts/

Now a contract change shows up as a diff in the PR, and a CODEOWNERS rule puts Leo on the review. Additive changes (a new optional field) are usually safe. Renames, removals and type changes are breaking. Note that .NET 10 generates OpenAPI 3.1 by default 🔧, which matters if Leo's code generator only understands 3.0. For snapshots of other outputs (JSON responses, generated SQL), Verify (Verify.XunitV3 33.3.4 🔧) does approval testing for xUnit v3.

So where does each test run?

Trigger What runs Why
Every PR Build with warnings as errors, unit, integration (Testcontainers), contract diff, coverage Fast enough to block a merge, and it covers the seams
Nightly Aspire end-to-end smoke tests, longer chaos scenarios from Chapter 14 Slow or needs the whole system, so it shouldn't hold up a PR
After each deploy A smoke call to /health and one read-only endpoint in the target environment Proves this deployment, not just this code (Chapter 12)
Before big launches Load tests (Chapter 18) Answers "how much", which no functional test does

💡 Each layer answers a different question. Unit: "is the rule right?" Integration: "does it work with the real database?" End-to-end: "does the system start and talk?" Contract: "will Leo's app still work?"


⚠️ Common traps

Trap What happens Do this instead
Testing data code with the EF in-memory provider It doesn't translate SQL or behave like SQL Server, so bugs in keys, concurrency or queries pass the tests Run against real SQL Server: Testcontainers in CI, LocalDB or a container locally
Keeping VSTest settings on the .NET 10 SDK with xUnit v3 4.x "Testing with VSTest target is no longer supported..." and nothing runs Opt in to MTP in global.json, then drop Microsoft.NET.Test.Sdk, xunit.runner.visualstudio and coverlet.collector
Helpers that ignore the setup result A test passes because the setup failed (Ledgerly's oversell test) Assert every arrange step, and watch each new test fail once
Skipped tests that nobody reads Integration tests silently stop running and regressions slip through a green build Check skip counts in CI, or make the database required there
One shared database for all test runs Tests depend on each other's data and fail randomly A fresh database per run, a different user per test
Thread.Sleep and DateTime.Now in tests Slow, timing-dependent, flaky tests TimeProvider in code and FakeTimeProvider in tests
End-to-end tests for every rule A slow suite that people skip A few broad end-to-end smoke tests, nightly. Push rules down to unit tests
Silencing exit code 8 globally A typo in a filter means nothing runs, and CI stays green Treat exit code 8 as a failure, except in the one job that really expects no tests

🧰 Helpers

A test checklist for every PR (paste into the PR description or pr-review-checklist.md):

- [ ] New logic has unit tests (no I/O, no mocks of DbContext)
- [ ] New or changed endpoints have an integration test against real SQL
- [ ] Every new test was seen failing once before it passed
- [ ] Setup steps are asserted, not assumed
- [ ] No Thread.Sleep, DateTime.Now or shared static state in tests
- [ ] A bug fix comes with the test that reproduces the bug
- [ ] OpenAPI changes are intended, and Leo is on the review

Run the suite locally:

cd sample
dotnet build Ledgerly.slnx
dotnet test --solution Ledgerly.slnx
# No Docker on this laptop? Point the integration tests at LocalDB instead.
$env:LEDGERLY_TEST_SQL='Server=(localdb)\MSSQLLocalDB;Integrated Security=true;TrustServerCertificate=true'
dotnet test --solution Ledgerly.slnx

Make it permanent:


🧪 Try it yourself (optional, ~25 min)

You need Docker (for Testcontainers) or LocalDB (with LEDGERLY_TEST_SQL) for the integration tests. The unit tests run anywhere.

  1. Run the whole suite and confirm 23 tests pass (15 unit, 8 integration): bash cd sample dotnet build Ledgerly.slnx dotnet test --solution Ledgerly.slnx If you have neither Docker nor LocalDB, notice how the eight integration tests are reported as skipped with the reason "No SQL Server available...".
  2. Reproduce the EF trap. In src/Ledgerly.Api/Endpoints/PortfolioEndpoints.cs, replace db.Transactions.Add(transaction); with portfolio.Transactions.Add(transaction);. Run the tests again. Observe which tests fail and with which status code. The unit tests stay green. Revert the change.
  3. Make a test lie. In tests/Ledgerly.IntegrationTests/PortfolioApiTests.cs, replace the two lines inside BuyAsync with _ = await PostTradeAsync(client, portfolioId, symbol, TransactionType.Buy, quantity); and repeat step 2. Observe that Overselling_is_a_validation_error now passes even though buying is broken. Restore both changes.
  4. Break the culture fix. In src/Ledgerly.Core/Imports/BrokerCsvParser.cs, find the line that parses the price with CultureInfo.InvariantCulture and change it to CultureInfo.CurrentCulture. Run the tests and watch Decimal_parsing_ignores_server_culture catch it. Revert.
  5. See the coverage. Run dotnet test --solution Ledgerly.slnx --coverage --coverage-output-format cobertura and find the .cobertura.xml files in the TestResults output. Which endpoint has the lowest coverage?
  6. Read the runner. Remove the test block from global.json, run dotnet test again, and read the error. Put it back.

✅ Recap

  1. Shape the pyramid around where your bugs live: fast unit tests for rules, integration tests with WebApplicationFactory<Program> against a real SQL Server for the seams, and a few nightly end-to-end tests with the Aspire testing builder.
  2. On the .NET 10 SDK, xUnit v3 runs on Microsoft.Testing.Platform: opt in through global.json, drop the VSTest packages, use --coverage, and treat exit code 8 ("zero tests ran") as a real failure.
  3. Make tests honest: assert the setup, watch new tests fail once, keep state isolated, notice skipped tests, and turn Leo's API contract into a reviewed file.

Next → Chapter 18: Performance and scale

Chapter 18: Performance and scale

⏱ About 35 min · You'll learn: how to find the real bottleneck before you touch code, when to use HybridCache and when to use output caching, and how Container Apps scale rules, EF Core habits and load tests fit together.


📖 Story: The newsletter

On Tuesday morning a popular personal-finance newsletter mentions Ledgerly's public beta. By ten o'clock signups are running at roughly eight times the normal rate (all numbers in this chapter are illustrative). Elena posts a screenshot of the signup graph in the team channel with three rocket emojis. Ten minutes later Priya posts a different graph: the p95 latency of GET /api/portfolios/{id}/valuation has gone from about 180 ms to over 2 seconds.

Your reflex comes straight from Nordbank Insurance: "Let's add a server." Back there, when the quote screen got slow before the annual renewal season, someone raised a ticket for two more VMs and the problem went away for a year. Maya shakes her head. "We can add ten replicas with one parameter. That's the easy part. The question is whether replicas fix it, or whether they just make the real bottleneck hurt faster. Measure first."

You open your notebook to the page titled Habits I brought with me and write: "Slow? Buy a bigger box." Then, under it: "Find out where the time goes."


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Buy a bigger VM or add a server when it's slow Measure with traces, counters and KQL, fix the bottleneck, then scale out with rules Replicas are cheap to add, so adding them without measuring just moves the problem (often to the database)
MemoryCache or HttpRuntime.Cache on each server, with hand-written locking HybridCache: in-process L1, optional Redis L2, stampede protection and tags One API covers both levels, and only one caller rebuilds an expired entry
.Result and .Wait() in controllers "because it's simpler" Async all the way, with CancellationToken passed through Blocking calls starve the thread pool under load, long before CPU is busy
Load test once before go-live with a big vendor tool Small load tests with pass/fail criteria, run on demand or from a pipeline Performance regressions are bugs, so they need a repeatable test
A Stopwatch loop in a console app to compare two methods BenchmarkDotNet in Release mode, with memory diagnostics JIT warm-up, tiering and GC make hand-made timing loops misleading
A fixed number of servers sized for the yearly peak minReplicas, maxReplicas and KEDA scale rules for HTTP traffic and queue length Capacity follows demand, and 💰 you stop paying for the peak all year
"The ORM is slow" EF Core with no-tracking reads, projections, indexes and pooling EF Core sends the SQL you let it send. Your SQL instincts still apply

💡 Performance work is a loop: measure, change one thing, measure again. Skip the first step and you are guessing with production traffic.


🧠 Measure first: counters, traces, load tests and benchmarks

Priya's dashboard told you that the valuation endpoint is slow. It didn't tell you why. Each of these tools answers a different question:

Question Tool What you look at
Which endpoints are slow, and since when? KQL over AppRequests in Application Insights (chapter 16) p95 by operation name in 5-minute bins
Where does one slow request spend its time? A trace in the Aspire dashboard locally, or end-to-end transaction details in Application Insights Long spans: SQL, the market-data HTTP call, the cache
Is the process healthy right now? dotnet-counters Thread-pool queue length, thread count, GC, exceptions, request rate
What is the CPU doing? dotnet-trace A .nettrace file you open in Visual Studio or PerfView
How does the system behave at 10× traffic? A load test (Azure Load Testing, k6) p95 and error rate under load, plus server-side metrics
Is method A faster than method B? BenchmarkDotNet Mean time and allocations per call

Start at the top and go down only as far as you need. Most problems are found in the first two rows. Priya's first query is the one from chapter 16:

AppRequests
| where TimeGenerated > ago(1h)
| summarize count(), p95=percentile(DurationMs, 95) by Name, bin(TimeGenerated, 5m)

The trace for a slow valuation shows the answer. The SQL is fast. The time goes to the market-data vendor, which is slow because every valuation asks for every symbol, and many users hold the same handful of ETFs.

The diagnostic CLI tools

The dotnet-* diagnostic tools attach to a running .NET process and need no profiler installed:

Tool Use it for
dotnet-counters monitor -p <pid> --counters System.Runtime,Microsoft.AspNetCore.Hosting Live metrics: thread pool, GC, request rate and duration
dotnet-trace collect -p <pid> A CPU and events trace (.nettrace) for Visual Studio or PerfView

New in .NET 10, dnx runs a tool without installing it first: dnx dotnet-counters ... 🔧. Sources: dotnet-counters and dotnet-trace.

Load tests and micro-benchmarks

These two get confused, but they answer different questions:

Load test Micro-benchmark
Question "How does the deployed system behave with N users?" "Which of these two methods is faster, and how much does it allocate?"
Scope The whole system: ingress, replicas, database, vendor One method, in-process
Tools Azure Load Testing 🔧 (URL quick tests, JMeter or Locust scripts, fail criteria such as p95 or error %, GitHub Actions integration), or Grafana k6 🔧 (JavaScript scripts with thresholds) BenchmarkDotNet 🔧 ([Benchmark], [MemoryDiagnoser], always in Release)
Where it runs Against a test environment, never against real users On a quiet machine
Cost 💰 Azure Load Testing is billed per virtual-user-hour 🔧, and the target environment scales up during the test Free

Azure Load Testing runs JMeter and Locust scripts, not k6 scripts natively 🔧 (overview). k6 2.x is a new major line 🔧 (k6 releases); see also BenchmarkDotNet releases. The Helpers section has a k6 script and a benchmark class for Ledgerly.

💡 A trace tells you where one request spent its time. A load test tells you which resource gives up first. You need both, in that order.


🧠 Caching: HybridCache for data, output caching for responses

The trace showed the real problem: the same quote is fetched from the vendor hundreds of times a minute. A price that is 30 seconds old is fine for a portfolio overview, so the answer is a cache. Before you write new MemoryCache(...) like you did at Nordbank, look at the two caches ASP.NET Core gives you:

HybridCache (data caching) Output caching (response caching)
What is cached A value your code computes: a quote, a DTO, a list The whole HTTP response for an endpoint
Setup AddHybridCache(), then GetOrCreateAsync(key, factory) AddOutputCache(), UseOutputCache(), then .CacheOutput() on an endpoint
Levels In-process L1, plus an optional IDistributedCache L2 such as Redis In-memory by default; a Redis store is available (Microsoft.AspNetCore.OutputCaching.StackExchangeRedis)
Concurrency Stampede protection: concurrent callers for the same key wait for one factory call Handled by the middleware
Invalidation Expiry, remove by key, or by tag Expiry, or IOutputCacheStore.EvictByTagAsync
Use it when Different responses share expensive building blocks (the same quote appears in many portfolios) Many callers get exactly the same response (a public list of symbols)
Avoid it when The value must be exact, like a balance after a trade The response depends on the user, unless you vary the cache key by user

Microsoft.Extensions.Caching.Hybrid is GA (10.11.0 🔧). Docs: HybridCache and output caching.

💻 In Ledgerly: HybridCache with an optional Redis L2

📄 sample/src/Ledgerly.Api/Program.cs

// ── Caching: HybridCache (in-memory L1). Redis becomes the shared L2 when a "cache" connection exists.
if (!string.IsNullOrEmpty(builder.Configuration.GetConnectionString("cache")))
{
    builder.AddRedisDistributedCache("cache");
}

builder.Services.AddHybridCache(options => options.DefaultEntryOptions = new HybridCacheEntryOptions
{
    Expiration = TimeSpan.FromSeconds(30),
    LocalCacheExpiration = TimeSpan.FromSeconds(10),
});

What matters here:

💻 In Ledgerly: one vendor call per symbol, not per request

📄 sample/src/Ledgerly.Api/Services/ValuationService.cs

    private async Task<decimal?> GetPriceAsync(string symbol, CancellationToken ct)
    {
        try
        {
            return await cache.GetOrCreateAsync(
                $"quote:{symbol}",
                async token =>
                {
                    var quotes = await marketData.GetQuotesAsync([symbol], token);
                    return quotes.TryGetValue(symbol, out var q) ? q.Price : (decimal?)null;
                },
                tags: ["quotes"],
                cancellationToken: ct);
        }
        catch (Exception ex) when (ex is HttpRequestException or BrokenCircuitException or TimeoutRejectedException)
        {
            LogPriceUnavailable(logger, symbol, ex.GetType().Name);
            return null;
        }
    }

Read it the way Maya would, asking "what happens when it fails?":

Situation What happens
300 valuations ask for quote:VWCE in the same second, and the entry has expired Stampede protection: one factory call goes to the vendor, the other callers wait for its result
The vendor knows the symbol The price is stored in L1 and L2, and the next callers never leave the process (or Redis)
The vendor doesn't know the symbol The factory returns null, and null is cached too. Unknown symbols don't hit the vendor on every request
The vendor is down The factory throws, so there is nothing to store. The catch sits outside GetOrCreateAsync, so the null it returns is not cached, and the next request tries again
You need to drop all prices at once Every entry carries the quotes tag, so one remove-by-tag call (RemoveByTagAsync) clears them

💻 In Ledgerly: output caching for a public list

The fake vendor uses the other kind of cache for a response that is the same for everyone:

📄 sample/src/Ledgerly.MarketData/Program.cs

builder.Services.AddOutputCache();
// ...
app.UseOutputCache();

app.MapGet("/symbols", () => PriceGenerator.KnownSymbols)
    .CacheOutput(policy => policy.Expire(TimeSpan.FromMinutes(10)));

The list of known symbols changes rarely and contains nothing personal, so the whole response can be reused for 10 minutes. A valuation is the opposite: it is different for every user, so caching the response would be wrong. Caching its building blocks (quotes) is right.

💡 Cache the expensive, shared parts with HybridCache. Use output caching only for responses that really are identical for every caller.


🗺️ How a valuation finds its prices

flowchart LR
    Req["GET valuation<br/>for one portfolio"] --> Svc["ValuationService<br/>one task per holding"]
    Svc -- "quote:VWCE" --> L1["L1 memory<br/>this replica, 10 s"]
    L1 -- "miss" --> L2["L2 Redis<br/>shared, 30 s, optional"]
    L2 -- "miss" --> Gate["Stampede protection<br/>one factory call per key"]
    Gate -- "HTTP with resilience" --> Vendor["MarketData vendor"]
    Vendor -- "price or unknown" --> Gate
    Gate -- "store in L2 and L1" --> Svc
    Svc -- "vendor down after retries" --> Partial["Partial valuation<br/>IsComplete = false"]

🧠 Async all the way, and why .Result hurts more here

At Nordbank, a .Result in an ASP.NET MVC controller could deadlock, because the classic SynchronizationContext tried to resume on the thread you had just blocked. ASP.NET Core has no such context, so the deadlock is gone. The habit is still harmful, because the failure mode changed: thread-pool starvation.

Each blocked call holds a thread-pool thread while it waits for I/O. Under light load you never notice. Under the newsletter spike, every request blocks a thread for the full duration of the vendor call. The pool injects new threads slowly, requests queue up, and latency explodes while the CPU sits mostly idle. In dotnet-counters this shows up as a growing thread-pool queue length and a climbing thread count.

// illustrative, not in the sample: the old habit
app.MapGet("/api/quotes/{symbol}", (string symbol, IMarketDataClient client) =>
{
    var quotes = client.GetQuotesAsync([symbol], CancellationToken.None).Result; // blocks a pool thread
    return quotes[symbol].Price;
});

// illustrative, not in the sample: the same endpoint, async all the way
app.MapGet("/api/quotes/{symbol}", async (string symbol, IMarketDataClient client, CancellationToken ct) =>
{
    var quotes = await client.GetQuotesAsync([symbol], ct);
    return quotes[symbol].Price;
});
Do Avoid
async Task all the way from the endpoint to the I/O call .Result, .Wait(), GetAwaiter().GetResult() in request paths
Pass the request's CancellationToken down CancellationToken.None in request code
Run independent I/O calls concurrently with Task.WhenAll Concurrent queries on one DbContext (it isn't thread-safe)
Move CPU-heavy work to the worker Task.Run in an endpoint to "make it async"

💻 In Ledgerly: concurrent price lookups, one database query

📄 sample/src/Ledgerly.Api/Services/ValuationService.cs

    public async Task<PortfolioValuation?> GetAsync(Guid portfolioId, string ownerId, CancellationToken ct)
    {
        var portfolio = await db.Portfolios.AsNoTracking()
            .Include(p => p.Transactions)
            .SingleOrDefaultAsync(p => p.Id == portfolioId && p.OwnerId == ownerId, ct);

        if (portfolio is null)
        {
            return null;
        }

        var holdings = HoldingCalculator.Calculate(portfolio.Transactions);
        var prices = await Task.WhenAll(holdings.Select(h => GetPriceAsync(h.Symbol, ct)));
        // ...
    }

The database is queried once, then the price lookups run concurrently. That is safe because GetPriceAsync never touches the DbContext. Eight holdings cost one round trip for the slowest price, not eight in a row.

Don't expect the build to catch every blocking call for you. Analyzers flag some async mistakes, but a .Result buried in a helper method usually survives until a load test or a trace finds it. That is why "is every I/O call awaited?" belongs in code review.

💡 In ASP.NET Core, .Result rarely deadlocks. It does something worse: it works fine in testing and falls over under the first real spike.


🧠 EF Core performance: your SQL instincts still apply

The trace showed the SQL was fast today. It won't stay that way by accident. EF Core sends the SQL you let it send, so the habits you built tuning stored procedures at Nordbank translate directly:

Habit EF Core version In Ledgerly
Read-only queries shouldn't pay for change tracking AsNoTracking() on reads Every read endpoint and ValuationService
Select only the columns you need Project with .Select(...) into a DTO ListNotifications
Avoid N+1 round trips Load related data in the same query with Include, or project it ValuationService.GetAsync loads transactions with the portfolio
Index the columns you filter on HasIndex(...) in OnModelCreating, shipped as a migration Owner, alert and outbox indexes
Don't open a new connection per call Connection pooling plus DbContext pooling The Aspire client integration enables DbContext pooling
Cap the result size Take(n) with a stable OrderBy Notifications are capped at 50
Update many rows without loading them ExecuteUpdateAsync / ExecuteDeleteAsync DeleteAlert uses ExecuteDeleteAsync

💻 In Ledgerly: a no-tracking projection with a cap

📄 sample/src/Ledgerly.Api/Endpoints/AlertEndpoints.cs

    private static async Task<Ok<NotificationResponse[]>> ListNotifications(ClaimsPrincipal user, LedgerlyDbContext db, CancellationToken ct)
    {
        var ownerId = user.GetOwnerId();
        var items = await db.Notifications.AsNoTracking()
            .Where(n => n.OwnerId == ownerId)
            .OrderByDescending(n => n.CreatedAt)
            .Take(50)
            .Select(n => new NotificationResponse(n.Id, n.Message, n.CreatedAt))
            .ToArrayAsync(ct);
        return TypedResults.Ok(items);
    }

The generated SQL selects three columns, filters by owner, sorts and stops at 50 rows. The filter is backed by an index:

📄 sample/src/Ledgerly.Core/Data/LedgerlyDbContext.cs

        modelBuilder.Entity<PriceAlert>(e =>
        {
            // ...
            e.HasIndex(a => new { a.IsActive, a.Symbol });
        });

        modelBuilder.Entity<Notification>(e =>
        {
            // ...
            e.HasIndex(n => n.OwnerId);
        });

An honest note: ValuationService loads every transaction of a portfolio. For 40 trades that is nothing. For ten years of broker history, it is the next thing a trace will show you, and the fix (store calculated holdings) is a design decision for an ADR.

Compiled queries help only for very hot, simple queries where LINQ translation shows up in a CPU trace:

// illustrative, not in the sample
private static readonly Func<LedgerlyDbContext, string, IAsyncEnumerable<Portfolio>> PortfoliosByOwner =
    EF.CompileAsyncQuery((LedgerlyDbContext db, string ownerId) =>
        db.Portfolios.AsNoTracking().Where(p => p.OwnerId == ownerId));

// usage: await foreach (var portfolio in PortfoliosByOwner(db, ownerId)) { ... }

For what is new in EF Core 10, see What's new in EF Core 10.

💡 EF Core doesn't make SQL slow. Unbounded, tracked, unindexed queries do, in any data-access technology.


🧠 Scaling out in Container Apps

Once the vendor calls were cached, Priya's p95 dropped back below 300 ms. The next question was Elena's: "If the next newsletter is twice as big, are we fine?" Now scaling matters.

Container Apps scales with KEDA-based scale rules: HTTP concurrency, TCP, and custom KEDA scalers such as Service Bus queue length. The defaults and limits 🔧:

Setting Default Notes
minReplicas 0 0 means scale to zero: no usage charges when idle, but a cold start on the first request 💰
maxReplicas 10 Configurable up to 1,000 🔧
HTTP scale rule 10 concurrent requests per replica Ledgerly raises it to 50 for the API

Idle replicas (min above 0, not processing) are billed at a reduced idle rate 💰. Docs: scaling and billing.

💻 In Ledgerly: the API scales on HTTP concurrency

📄 sample/infra/modules/apps.bicep

      scale: {
        minReplicas: apiMinReplicas // 0 = no cost when idle but a cold start; 1 = always warm
        maxReplicas: 10
        rules: [
          { name: 'http', http: { metadata: { concurrentRequests: '50' } } }
        ]
      }

When replicas are handling more than about 50 concurrent requests each, Container Apps adds replicas, up to 10. When traffic drops, it removes them again, down to apiMinReplicas.

💻 In Ledgerly: the cold-start decision is a parameter

📄 sample/infra/main.bicep

@description('0 = scale to zero (cheapest), 1 = always warm (no cold start).')
@minValue(0)
param apiMinReplicas int = 0

This one number is a product decision, so Elena should be in the conversation:

apiMinReplicas User experience Cost 💰
0 (sample default) The first request after a quiet period waits for a replica to start No usage charges while idle, and the monthly free grant 🔧 covers a lot of learning traffic
1 Always warm, no cold start One replica billed at the idle rate around the clock

For a learning subscription, 0 is right. For the public beta, the team sets 1 in the parameter file, through a pull request, because "if it's not in git, it doesn't exist."

💻 In Ledgerly: the worker scales on queue length

📄 sample/infra/modules/apps.bicep

      scale: {
        // 1, not 0: the outbox relay and price alerts poll on a timer, so one replica must always run.
        minReplicas: 1
        maxReplicas: 5
        rules: [
          {
            // KEDA watches the queue length and adds replicas when imports pile up.
            name: 'imports-queue'
            custom: {
              type: 'azure-servicebus'
              metadata: {
                namespace: serviceBusNamespace
                queueName: 'imports'
                messageCount: '20'
              }
              identity: identityId
            }
          }
        ]
      }

KEDA reads the length of the imports queue (using the managed identity, no connection string) and adds worker replicas as messages pile up, aiming at roughly 20 waiting messages per replica, up to 5. Check the KEDA scaler details in the scaling docs.

Scaling moves the bottleneck

Adding replicas is never free of side effects. In Ledgerly, three show up right away:

Side effect Why What to do
The database becomes the limit The serverless Azure SQL database in sql.bicep scales up to 2 vCores. Ten API replicas can send more work than that Watch SQL CPU during load tests. Cache reads, fix queries, then raise the max vCores 💰
The import rate limit grows The rate limiter in Program.cs is in memory, per replica: 5 imports per minute per user per replica Accept it as "roughly 5", or move the limit to a shared store if it must be exact
Price alerts are evaluated more than once AlertEvaluator runs inside every worker replica, and its own comment says to run one replica or move it to a scheduled Container Apps job Move the timer work to a scheduled job, or keep the worker at one replica (chapter 19 shows what this costs at 2 a.m.)

💡 Scale rules are easy to add. The hard part is knowing which shared resource the extra replicas will hit next.


🗺️ Who scales on what

flowchart TB
    Users["Beta users"] -- "HTTPS" --> Ingress["Ingress"]
    subgraph Env["Container Apps environment"]
        Api["api<br/>min apiMinReplicas, max 10"]
        Worker["worker<br/>min 1, max 5"]
        MD["marketdata<br/>min 0, max 3"]
    end
    Ingress -- "HTTP rule: about 50 concurrent per replica" --> Api
    Api -- "cached quotes" --> MD
    Api -- "reads and outbox rows" --> Sql["Azure SQL serverless<br/>up to 2 vCores"]
    Worker -- "outbox relay sends" --> Queue["Service Bus queue<br/>imports"]
    Queue -- "messages" --> Worker
    Keda["KEDA scaler"] -- "watches queue length" --> Queue
    Keda -- "about 20 messages per replica" --> Worker
    Worker -- "imports and alerts" --> Sql

⚠️ Common traps

Trap What happens Do this instead
Scaling out before measuring Ten replicas hammer the same slow vendor or a 2-vCore database, and latency gets worse Read one slow trace and the counters first, then decide between caching, fixing and scaling
.Result or .Wait() in a request path Fine in testing. Under load the thread pool starves and latency jumps while CPU looks idle Async all the way, with the request's CancellationToken passed down
Output-caching a per-user response One user can see another user's data, or a stale balance Output-cache only public, identical responses. Use HybridCache with user-scoped keys for the rest
Assuming L1 is shared Each replica has its own memory cache, so replicas can disagree for up to LocalCacheExpiration Keep L1 short, add Redis as L2 when consistency matters, and evict by tag on writes
Load testing a scaled-to-zero app and blaming the code The first minute is dominated by cold starts and scale-out, not by your code Warm up first, or set min replicas for the test, and read p95 after the ramp

🧰 Helpers

Watch a local process (after aspire run): dnx dotnet-counters ps to find the PID, then dnx dotnet-trace collect -p <pid> for a CPU trace.

Find the slow endpoints in Application Insights (the toolkit's kql-queries.md has more):

AppRequests
| where TimeGenerated > ago(1h)
| summarize count(), p95=percentile(DurationMs, 95) by Name, bin(TimeGenerated, 5m)

A k6 load test for the valuation endpoint:

// illustrative, not in the sample (k6 2.x 🔧: check the current script API)
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 50,
  duration: '2m',
  thresholds: {
    http_req_duration: ['p(95)<500'], // fail the run if p95 is 500 ms or more
    http_req_failed: ['rate<0.01'],   // or if more than 1% of requests fail
  },
};

export default function () {
  const res = http.get(`${__ENV.API}/api/portfolios/${__ENV.PORTFOLIO}/valuation`, {
    headers: { Authorization: `Bearer ${__ENV.TOKEN}` },
  });
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

Run it with k6 run -e API=https://... -e PORTFOLIO=<id> -e TOKEN=<token> valuation.js against a test environment, never against production users.

A micro-benchmark for the holdings calculation:

// illustrative, not in the sample: a separate console project referencing Ledgerly.Core
// run with: dotnet run -c Release
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using Ledgerly.Core.Domain;

BenchmarkRunner.Run<HoldingBenchmarks>();

[MemoryDiagnoser]
public class HoldingBenchmarks
{
    private Transaction[] _trades = [];

    [Params(100, 10_000)]
    public int Count { get; set; }

    [GlobalSetup]
    public void Setup() => _trades = [.. Enumerable.Range(0, Count).Select(i => new Transaction
    {
        Symbol = i % 2 == 0 ? "AAPL" : "MSFT",
        Type = TransactionType.Buy,
        Quantity = 1,
        Price = 100 + i,
        TradeDate = new DateOnly(2026, 1, 1).AddDays(i % 300),
    })];

    [Benchmark]
    public int Calculate() => HoldingCalculator.Calculate(_trades).Count;
}

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

  1. Start the system (you need Docker Desktop or Podman): bash cd sample aspire run
  2. Create a token with dotnet user-jwts create --project src/Ledgerly.Api. Use src/Ledgerly.Api/Ledgerly.Api.http to create a portfolio and record a few trades in different symbols (for example VWCE, AAPL and MSFT).
  3. Call the valuation request twice within a few seconds. In the Aspire dashboard, open Traces. Observe: the first trace has HTTP spans to marketdata, the second has none, because the quotes came from the cache.
  4. Make the vendor slow. Find the marketdata URL in the dashboard, then: bash curl -X POST <marketdata-url>/chaos -H "Content-Type: application/json" -d '{"failureRate": 0, "latencyMs": 2000}' In PowerShell: Invoke-RestMethod -Method Post -Uri <marketdata-url>/chaos -ContentType 'application/json' -Body '{"failureRate": 0, "latencyMs": 2000}'. Call the valuation again a few times over a minute. Observe: one slow call, then fast calls while the cache holds, then slow again after the entries expire.
  5. While the chaos is on, watch the API with dnx dotnet-counters monitor -p <pid> --counters System.Runtime,Microsoft.AspNetCore.Hosting (find the PID with dnx dotnet-counters ps).
  6. Reset the vendor with the same request and {"failureRate": 0, "latencyMs": 0}.
  7. Read sample/infra/modules/apps.bicep and answer: with 10 API replicas, what is the real import limit per user? Which worker component should not run on 5 replicas, and why?

✅ Recap

  1. Measure first: KQL for which endpoint, a trace for where the time goes, counters for process health, load tests for the system and BenchmarkDotNet for single methods.
  2. Cache shared building blocks with HybridCache (stampede protection, L1 plus optional Redis L2, tags), keep requests async all the way, and give EF Core the same care you gave your SQL.
  3. Container Apps scales the API on HTTP concurrency and the worker on queue length with KEDA. Every new replica moves pressure to a shared resource, and apiMinReplicas trades cold starts against 💰.

Next → Chapter 19: Running production

Chapter 19: Running production

⏱ About 35 min · You'll learn: how to run on-call with actionable alerts and runbooks, how to use feature flags, revisions and dead-letter replay to recover safely, and how to learn from an incident with a blameless postmortem, ADRs and a cost review.


📖 Story: The 2 a.m. page

It is your first week on the on-call rotation. Priya is your secondary. At 01:52 your phone buzzes: "Sev 2: imports dead-letter count above 0 and queue length rising for 10 minutes." You open the laptop on the kitchen table. The market-data vendor has been returning errors since about 01:38 (all times and numbers in this chapter are illustrative). Valuations are degrading gracefully, exactly as designed in chapter 14. Imports are not.

Two weeks ago, a pull request added "value at import time" to CSV imports: the worker now asks the vendor for a closing price for every imported trade. (This change is part of the story; it is not in the sample.) It had no fallback and no flag. Now every import message waits for the resilience timeout or hits an open circuit breaker, gets abandoned, comes back, fails again, and after ten attempts lands in the dead-letter queue. KEDA sees the queue growing and scales the worker to five replicas, which doesn't help at all.

At Nordbank, this would have been the moment you opened RDP to three servers and started reading log files. Here there is a runbook link in the alert, a dashboard, and a calm message from Priya: "Mitigate first. Debug in the morning."


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Every error email goes to the whole team, and everybody ignores most of them An on-call rotation with a primary and a secondary, paged only by alerts that need a human now Noisy alerts train people to ignore the one that matters
A wiki page "how to restart the service", last updated years ago A runbook in the repo, linked from the alert, updated after every incident The runbook is read at 2 a.m. by a tired person, so it must be short, current and findable
A hotfix copied to the servers in the middle of the night Turn a feature flag off, or revert the change and let the pipeline roll forward The pipeline is the tested path. A manual change at 3 a.m. is the untested path
A rollback plan of "restore last week's VM snapshot" Immutable revisions, a smoke test after each deploy, and small changes that are easy to revert Small deploys are easy to undo. Big ones are not
Failed messages in a poison table nobody looked at A dead-letter queue with an alert, a known cause per reason, and a replay tool Dead-lettered messages are never deleted automatically, so someone has to own them
"Root cause: human error" A blameless postmortem with a timeline, contributing factors and owned actions Blame hides the system problem. The next person will make the same mistake
The monthly invoice as the first cost signal Tags, budgets with alerts, and a cost review after incidents and every month 💰 Autoscaling means cost follows load, including the load an incident creates

💡 Running production is a practiced skill, not a heroic one. The goal is a boring 2 a.m.: a clear alert, a short runbook and a safe switch.


🧠 On-call that people can live with: alerts and runbooks

Priya's rule for alerts is simple: "If nobody needs to do anything right now, it's not a page." Everything else is a ticket, a dashboard or a weekly report.

A good page A noisy page
Fires on a symptom users feel: failed imports, error rate, SLO burn rate Fires on a cause that may not matter: one CPU spike, one exception
Has a clear owner and a severity Goes to a shared mailbox
Links to a runbook with the first three steps Says "Error in worker" and nothing else
Has a threshold and a time window that ignore one-off blips Fires and resolves five times an hour
Can be silenced during planned maintenance Wakes people during a deploy everyone knew about

Azure Monitor gives you the building blocks:

Building block What it does in Ledgerly
Metric alerts Dead-letter count and queue length on the imports queue
Log search alerts (KQL) Failed requests or dependency failures above a threshold, or SLO burn rate (chapter 16)
Action groups Who gets notified and how: email (🔧 email receivers now need OTP verification), SMS, voice, push, webhook, Logic App and more. Reused across alerts
Alert processing rules Suppress notifications during a maintenance window

Alert on burn rate for SLOs rather than raw thresholds: the error budget is 1 − SLO, and the question that matters is "how fast are we spending it?" See alerts overview, action groups and the SRE Workbook on SLOs.

The runbook

A runbook is not documentation of the system. It is a checklist for a specific alert. The toolkit's runbook-template.md has this shape:

Section What goes in it Example for "imports dead-lettering"
Symptom What the alert means for users Imports stay "Pending". Users don't see new trades
Impact check One query or dashboard that sizes the problem Queue length, DLQ count and import failures in the last hour
Likely causes The usual suspects, most common first Vendor down, SQL unavailable, a bad deploy
Mitigation Safe, reversible steps, in order Check vendor dependency failures, flip the flag, revert the last deploy
Escalation When and whom to wake up No improvement in 20 minutes: page the secondary, then Maya
Recovery What to do after mitigation Replay the DLQ, check for duplicates, post an update

The first query in that runbook looks at dependencies, because the trace context from chapter 15 tells you which service is failing:

AppDependencies
| where TimeGenerated > ago(1h)
| summarize calls = count(), failed = countif(Success == false) by Target, AppRoleName, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

💡 An alert without a runbook is a riddle. Write the runbook on the day you create the alert, not the morning after the first page.


🧠 Feature flags as kill switches

At 02:10, Priya asks the obvious question: "Can we just turn it off?" For the import enrichment, the answer was no. The pull request shipped without a flag. For price alerts, there is a flag, and that is where you learn the most important lesson of the night.

💻 In Ledgerly: the PriceAlerts flag

📄 sample/src/Ledgerly.Api/appsettings.json

  "FeatureManagement": {
    "PriceAlerts": true
  },

📄 sample/src/Ledgerly.Api/Endpoints/AlertEndpoints.cs

        var alerts = app.MapGroup("/api/alerts")
            .WithTags("Alerts")
            .RequireAuthorization()
            .AddEndpointFilter(RequireFeature(Features.PriceAlerts));
        // ...

    /// <summary>Feature flag gate: when the flag is off the endpoints behave as if they do not exist.</summary>
    private static Func<EndpointFilterInvocationContext, EndpointFilterDelegate, ValueTask<object?>> RequireFeature(string feature) =>
        async (context, next) =>
        {
            var features = context.HttpContext.RequestServices.GetRequiredService<IFeatureManager>();
            return await features.IsEnabledAsync(feature) ? await next(context) : TypedResults.NotFound();
        };

The flag is read with IFeatureManager.IsEnabledAsync, registered by builder.Services.AddFeatureManagement() in the API's Program.cs. When it is off, every alert endpoint returns 404, the same behavior [FeatureGate] has for controllers.

Now look at what it does not do. AlertEvaluator in the worker keeps polling the vendor every 30 seconds, on every worker replica, because the worker never registers feature management and never checks the flag. Turning PriceAlerts off would have hidden the alert screens from users and changed nothing about the load on the vendor. A kill switch only kills the code that checks it.

To make the flag a real kill switch, the worker needs the same check:

// illustrative, not in the sample: the worker honoring the same flag
// Program.cs: builder.Services.AddFeatureManagement();
// AlertEvaluator: inject IFeatureManager features, then at the top of EvaluateAsync:
if (!await features.IsEnabledAsync("PriceAlerts"))
{
    return;
}

How fast can you flip it?

Today the flag lives in configuration. Changing it in Azure means setting the environment variable FeatureManagement__PriceAlerts on the container app. An environment variable change creates a new revision, so flipping the flag is effectively a deploy. That is the comment in Program.cs coming due: "from configuration now, Azure App Configuration later".

Where the flag lives How you flip it How long it takes
appsettings.json / environment variables (Ledgerly today) Change the value through Bicep and the pipeline A full deploy and a new revision
Azure App Configuration with refresh Toggle it in App Configuration About the refresh interval (30 seconds by default) 🔧, no redeploy

The App Configuration version, from the docs (Microsoft.Azure.AppConfiguration.AspNetCore 8.6.1 🔧, with the App Configuration Data Reader role for the managed identity):

// illustrative, not in the sample
builder.Configuration.AddAzureAppConfiguration(o =>
    o.Connect(new Uri(builder.Configuration["AppConfig:Endpoint"]!), new DefaultAzureCredential())
     .UseFeatureFlags());          // loads flags with no label, refreshed every 30 s by default
builder.Services.AddAzureAppConfiguration();
builder.Services.AddFeatureManagement();
// ...
app.UseAzureAppConfiguration();    // enables dynamic refresh

Microsoft.FeatureManagement.AspNetCore (4.8.0 🔧) also has percentage, time-window and targeting filters, which turn the same mechanism into gradual rollouts. Docs: feature management reference and FeatureManagement-Dotnet.

Flag type Use it for Remove it when
Kill switch Any feature that calls a dependency you don't control Rarely: it is permanent operational tooling
Release flag Shipping code dark and turning it on later The feature is fully on and stable
Rollout flag (percentage, targeting) Trying a change on a few users first The rollout reaches 100%

💡 Every new call to a dependency you don't control should ship behind a flag, and every service that makes that call must check it.


🧠 Safe deploys, rollback and roll forward

At 02:20 the team chose the safest fix available: revert the enrichment pull request and let the normal pipeline ship the revert. Nobody touched Azure by hand.

💻 In Ledgerly: one deploy at a time, and proof it answers

📄 .github/workflows/deploy.yml

# Never run two deployments at once, and never cancel one halfway.
concurrency:
  group: deploy-production
  cancel-in-progress: false
# ...
      # The deployment "succeeding" only means Azure accepted it. This proves the new revision answers.
      - name: Smoke test
        run: |
          for i in $(seq 1 30); do
            if curl -fsS "${{ steps.apps.outputs.apiUrl }}/alive"; then echo; echo "API is up"; exit 0; fi
            sleep 10
          done
          echo "::error::API did not become healthy"; exit 1

Three properties made the revert safe at 2 a.m.:

Property Where it comes from Why it matters during an incident
Deploys never overlap or stop halfway concurrency with cancel-in-progress: false A second urgent merge waits instead of corrupting the first deploy
Every image is tagged with the commit SHA SHA in deploy.yml You know exactly which commit each revision runs
Migrations run before the apps change Phases 2 and 3 of the deploy Old code never runs against a schema it doesn't understand, as long as migrations stay backward compatible

Notice the limit of the smoke test: it proves the API answers /alive, nothing about the worker or imports. After the incident, "check a worker signal after deploy" went on the production readiness checklist.

Revisions: single mode today, traffic splitting next

Every app in apps.bicep uses activeRevisionsMode: 'Single'. A revision is an immutable snapshot (image, environment variables, scale rules), and in single mode the new revision replaces the old one with zero downtime.

Option How it works Use it when
Revert and roll forward (Ledgerly today) git revert, merge, and the pipeline deploys a new revision Always available, always tested, and it keeps git as the truth
Multiple revision mode Several revisions stay active, and you split traffic by percentage You want canary releases, or an instant switch back to the previous revision
Labels A revision gets a stable URL You want to smoke-test a new revision before it receives real traffic

Docs: revisions and traffic splitting.

A database never "rolls back" with the code, so a revert only works if the schema change was additive. The team's rule: migrations add, later releases remove (if ever).

💡 The fastest safe rollback is the one your pipeline already does every day. Practice it before you need it.


🗺️ The night, as a sequence

sequenceDiagram
    participant V as "Vendor"
    participant W as "Worker replicas"
    participant Q as "imports queue"
    participant D as "Dead-letter queue"
    participant M as "Azure Monitor"
    participant Y as "You on call"
    V->>W: 503 on every price request
    W->>Q: Abandon message, delivery N fails
    Q->>W: Redeliver up to MaxDeliveryCount 10
    Q->>D: Move to DLQ after the 10th failure
    M->>Y: Page with runbook link
    Y->>Y: Check dependency failures and decide to mitigate
    Y->>W: Revert merged, pipeline deploys new revision
    W->>Q: Imports complete, queue drains
    Y->>D: Replay dead letters in the morning

🧠 Dead letters: handle them, then replay them

By 02:50 the queue was draining, but 37 messages sat in the dead-letter queue. Those are 37 users whose imports say "Pending". Dead-lettered messages are never deleted automatically, so they are your job.

💻 In Ledgerly: how a message ends up in the DLQ

📄 sample/infra/modules/servicebus.bicep

  properties: {
    lockDuration: 'PT1M'
    maxDeliveryCount: 10 // after 10 failed attempts the message moves to imports/$deadletterqueue
    requiresDuplicateDetection: true // the same MessageId within the window is dropped
    duplicateDetectionHistoryTimeWindow: 'PT10M'
    deadLetteringOnMessageExpiration: true
  }

📄 sample/src/Ledgerly.Worker/ImportProcessor.cs

        if (import is null)
        {
            // Poison message: retrying will never help. Park it in the DLQ for a human to look at.
            await args.DeadLetterMessageAsync(args.Message, "ImportNotFound", $"No import for message {messageId}", ct);
            return;
        }
        // ...
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            activity?.AddException(ex);
            activity?.SetStatus(ActivityStatusCode.Error);
            LogImportCrashed(logger, import.Id, args.Message.DeliveryCount, ex);
            await args.AbandonMessageAsync(args.Message, cancellationToken: ct); // Service Bus will redeliver
        }

There are two kinds of dead letters, and they need different handling:

Dead-letter reason Meaning What to do
ImportNotFound (set by Ledgerly) Poison: the message points at an import that doesn't exist. Retrying will never help Investigate, then delete or fix by hand
MaxDeliveryCountExceeded (set by Service Bus) Transient: it failed 10 times, here because the vendor was down Fix the cause, then replay

Replaying safely

Replay means: receive from imports/$deadletterqueue, send a copy to imports, complete the dead-lettered original. Two details make it safe in Ledgerly:

// illustrative, not in the sample: replay transient dead letters after the cause is fixed
await using var client = new ServiceBusClient(fullyQualifiedNamespace, new DefaultAzureCredential());
await using var receiver = client.CreateReceiver("imports", new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
await using var sender = client.CreateSender("imports");

while (true)
{
    var batch = await receiver.ReceiveMessagesAsync(maxMessages: 50, maxWaitTime: TimeSpan.FromSeconds(5));
    if (batch.Count == 0)
    {
        break;
    }

    foreach (var message in batch)
    {
        if (message.DeadLetterReason != "MaxDeliveryCountExceeded")
        {
            continue; // poison: leave it, it returns to the DLQ when its lock expires
        }

        await sender.SendMessageAsync(new ServiceBusMessage(message)); // keeps body, properties and MessageId
        await receiver.CompleteMessageAsync(message);
    }
}

The identity running this needs both the Data Sender and Data Receiver roles on the namespace. Docs: dead-letter queues and duplicate detection.

💡 A dead-letter queue is an inbox with your name on it. Alert on it, know what each reason means, and keep a tested replay tool.


🗺️ The life of an import message

stateDiagram-v2
    [*] --> Active: "Outbox relay sends"
    Active --> Locked: "Worker receives"
    Locked --> Completed: "Trades saved and message completed"
    Locked --> Active: "Abandoned, delivery count plus 1"
    Locked --> DeadLettered: "ImportNotFound or 10 deliveries"
    DeadLettered --> Active: "Replay after the fix"
    DeadLettered --> Deleted: "Poison, removed by a human"
    Completed --> [*]
    Deleted --> [*]

🧠 After the incident: postmortem, ADRs and the cost review

At 03:05 the vendor recovered. Then one more thing went wrong: several users got two notifications for the same price alert. The worker was still at three replicas after the scale-out, each running AlertEvaluator, and each found the same active alerts in the same 30-second tick. The class comment had warned about this ("Run the worker with ONE replica (or move this to a scheduled Container Apps job)"), and the scale rule had quietly broken that assumption.

The blameless postmortem

The next afternoon Priya runs the review. The rule: describe what the system allowed, not who made a mistake. The person who merged the enrichment PR is in the room and explains their reasoning, because that reasoning is the most useful data in the review. The toolkit's postmortem-template.md gives the structure: summary, impact, timeline, contributing factors, what went well, actions with owners.

The timeline (illustrative):

Time What happened Signal
01:38 Vendor starts returning 503 for all price requests Dependency failures in AppDependencies
01:40 Valuations return IsComplete = false. No page, by design Partial-valuation warnings in logs
01:52 DLQ count above 0 and queue length rising for 10 minutes. You are paged Metric alert, action group
01:58 You acknowledge and open the runbook. Dependency query shows the vendor at 100% failures KQL from the runbook
02:05 KEDA has scaled the worker to 5 replicas. It doesn't help: every import calls the vendor Replica count
02:10 You page Priya. The team finds that PriceAlerts doesn't stop the worker's vendor calls Code reading
02:20 Revert of the enrichment PR merged with an expedited review Pull request
02:34 Deploy finished, smoke test passed, imports complete again Deploy run, queue length falling
02:50 Queue back to normal. 37 messages in the DLQ Metric
03:05 Vendor recovers. Duplicate alert notifications go out from 3 worker replicas User reports in the morning
09:30 DLQ replayed. All 37 imports complete, no duplicate trades Import status, idempotency log lines

Contributing factors, not causes:

Factor Action Owner
A new vendor call shipped without a flag or fallback PR checklist item: "new dependency call ships behind a kill switch with a fallback" You
The PriceAlerts flag didn't reach the worker Add feature management to the worker, and check the flag in AlertEvaluator You
Timer work ran on every scaled-out replica Move alert evaluation to a scheduled Container Apps job (record it as an ADR) Maya
Flipping a flag requires a deploy Evaluate Azure App Configuration for flags Tomás
The smoke test only proves the API is alive Add a worker check to the production readiness checklist Priya

The ADR

Moving alert evaluation into a scheduled job changes the architecture, so Maya writes it down with the toolkit's adr-template.md: context (timer work in a scaled worker), decision (a Container Apps job with a Schedule trigger, see jobs), alternatives (keep the worker at one replica, or add a distributed lock), and consequences (one more deployable, but scaling the worker becomes safe). The next person who asks "why is this a job?" finds the answer in git.

The cost review 💰

Elena asks her usual question: "What did the night cost?" Autoscaling means an incident has a price, so the team looks at three things:

Cost driver What happened that night Lever
Worker replicas 5 replicas for over an hour instead of 1 Max replicas in Bicep; the scale rule now only scales import work
Log ingestion Five replicas logging failures every few seconds Log Analytics ingestion is often the surprise cost. Sampling, log levels and a daily cap
Vendor calls Retries multiplied by replicas Caching, circuit breakers, and the kill switch

📄 sample/infra/modules/monitoring.bicep

  properties: {
    sku: { name: 'PerGB2018' }
    retentionInDays: 30
    workspaceCapping: { dailyQuotaGb: 1 } // 💰 a hard daily cap keeps a learning subscription safe
  }

A hard cap is right for a learning subscription. In production, a cap reached during an incident stops the telemetry you need most, so many teams use a budget alert instead.

📄 sample/infra/main.bicep

var tags = {
  app: appName
  'managed-by': 'bicep'
}

Every resource gets these tags, so cost analysis can group by app. Budgets add the alarm:

Budget fact Detail
Scopes and resets Subscription, resource group, management group or billing scope, reset monthly, quarterly or annually
Thresholds On actual or forecasted cost, notifying by email or an action group
What budgets don't do Stop resources. They only notify, unless you wire automation to the action group
Delay Cost data lands within 8 to 24 hours, and budgets are evaluated every 24 hours

So a budget will not catch a runaway incident within the hour. That is the job of maxReplicas and your alerts. See create a budget and the pricing calculator.

The production readiness checklist

The last action has the longest life. The toolkit's production-readiness-checklist.md lists what a service needs before real users: health probes, alerts with runbooks, kill switches, a DLQ owner, scale limits, budgets and tags, and a tested rollback. After this night it gained two lines.

💡 A postmortem is worth exactly as much as its actions. Give every action an owner and put the permanent ones in a checklist.


⚠️ Common traps

Trap What happens Do this instead
Paging on causes instead of symptoms People get woken for one CPU spike, then start ignoring pages Page on what users feel (errors, SLO burn, stuck imports). Everything else is a ticket
A flag that only one service checks You flip it at 2 a.m. and the load doesn't change Every service that runs the feature checks the same flag. Test the "off" path
Fixing production by hand at night The fix is lost on the next deploy, or breaks something untested Revert and roll forward through the pipeline. Change Bicep, not the portal
Replaying the DLQ without idempotency Replayed messages create duplicate trades Keep the ProcessedMessages check, and replay only reasons you understand
Scaling out a worker that runs timers Duplicate notifications when the dependency recovers Timer work runs once: a scheduled job, or a single instance
A postmortem that names a person People hide mistakes, and the system problem stays Blameless format, contributing factors and actions with owners
Treating a budget as a safety net The alert arrives a day after the money is spent Hard limits (maxReplicas, caps) for safety, budgets for trends

🧰 Helpers

Incident first steps (paste into your runbook):

  1. Acknowledge the page and post "Investigating" in the incident channel.
  2. Size the impact: which users, which feature, since when.
  3. Check the last deploy and the dependency failures query.
  4. Mitigate with the safest reversible step: flag off, revert, or scale limit.
  5. Escalate if there is no improvement in 20 minutes.
  6. When stable, write down the timeline while you still remember it.

Exceptions by service (KQL, next to the dependency query from the runbook section):

AppExceptions
| where TimeGenerated > ago(1h)
| summarize count() by AppRoleName, ExceptionType, bin(TimeGenerated, 5m)

Templates in the toolkit: runbook-template.md, postmortem-template.md, adr-template.md, production-readiness-checklist.md and kql-queries.md. See the toolkit.

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

  1. Start the system: bash cd sample aspire run
  2. Create a token with dotnet user-jwts create --project src/Ledgerly.Api, then use src/Ledgerly.Api/Ledgerly.Api.http to create a portfolio, a trade and a price alert.
  3. Break the vendor. Find the marketdata URL in the Aspire dashboard, then: bash curl -X POST <marketdata-url>/chaos -H "Content-Type: application/json" -d '{"failureRate": 1, "latencyMs": 0}' In PowerShell: Invoke-RestMethod -Method Post -Uri <marketdata-url>/chaos -ContentType 'application/json' -Body '{"failureRate": 1, "latencyMs": 0}'.
  4. In the dashboard, open Structured logs for the worker. Observe: "Alert evaluation failed; will retry on next tick" every 30 seconds. The loop survives. Call the valuation request and observe isComplete: false.
  5. Turn the flag off for the API: stop the AppHost, set "PriceAlerts": false in src/Ledgerly.Api/appsettings.json, and run aspire run again. Call GET /api/alerts. Observe: 404. Then check the worker logs. Observe: it still evaluates alerts. That is the kill-switch lesson.
  6. Restore the flag and reset the vendor with {"failureRate": 0, "latencyMs": 0}.
  7. Write a five-line runbook for "vendor down" using the shape in this chapter, and one action you would add to the production readiness checklist.

✅ Recap

  1. Page only on symptoms users feel, and give every alert a runbook with safe, reversible first steps.
  2. Recover through tested paths: feature flags that every affected service checks, revert and roll forward through the pipeline, and an idempotent dead-letter replay.
  3. Learn without blame: a timeline, contributing factors, owned actions, an ADR for design changes, and a cost review, all ending up in your checklists.

Next → Chapter 20: AI-assisted .NET development

Chapter 20: AI-assisted .NET development

⏱ About 30 min · You'll learn: which GitHub Copilot surface to use for which job, how to give AI the context it needs with custom instructions, prompt files and MCP, and how to review AI output with the same bar and guardrails as any pull request.


📖 Story: Use AI like a senior

On Monday you find two pull requests from the weekend. The first one, from a junior contractor, "adds retry logic" to the worker: 400 lines, a hand-written retry loop around every Service Bus call, a new package nobody asked for, and tests that assert the mocks were called. The description says "generated with AI, all tests pass". The second one, from Priya, is 60 lines: a new KQL alert in Bicep and a runbook update. Her description says: "Copilot drafted the query, I checked it against a week of real data and fixed the time window."

Maya reviews both in the stand-up. "Same tool, different results. The tool didn't write the first PR badly. Nobody owned it." Then she says the sentence that ends up on the first page of your notebook's last section: "Use AI like a senior, not like a junior." A junior accepts what the tool produces. A senior decides what to ask for, gives it the right context, and checks the result against the bar they would apply to any colleague.

You have 15 years of .NET judgment. That turns out to be exactly the skill that makes these tools useful. At Nordbank you never had them. Here, the question is not whether to use AI, but where your judgment has to stay in the loop.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Searching forums for an API example, then adapting it by hand Asking Copilot in the IDE, with your code as context, then verifying against the docs The draft is faster. Verification is still yours
Writing boilerplate (DTOs, mappings, test setup) by hand Letting agent mode draft it, then reviewing it like a PR Boilerplate is where AI is strongest and the risk is lowest
Tribal knowledge about "how we do things here" A short .github/copilot-instructions.md that the tools read The model can't guess your conventions. Write the ones that matter down
Reviewing code by trusting the author One bar for every change, human or AI: tests, analyzers, warnings as errors, human review "The AI wrote it" is not a reason to lower the bar, or to raise it unfairly
The upgrade project nobody wanted to start An agent drafting the mechanical parts of an upgrade, step by step, behind a green build The boring 80% gets cheaper. The tricky 20% still needs you
Copying production data into a test to reproduce a bug Synthetic data, and no secrets or customer data in prompts Prompts are data leaving your machine. Treat them like any other data flow

💡 AI changes how fast you produce a draft. It doesn't change who is responsible for what gets merged.


🧠 The Copilot surfaces, and which one to use when

GitHub Copilot is not one tool. It shows up in several places, and each is good at a different size of job. Everything in this table is volatile 🔧: names, plans and features change often, so check Copilot plans and Copilot on GitHub before you rely on a detail.

Surface 🔧 What it does Use it for
Completions and chat in the IDE Inline suggestions and a chat that sees your open files Small, local edits: a method, a test case, "explain this LINQ query"
Agent mode (VS Code and Visual Studio) An autonomous loop in the IDE that edits several files, builds and runs tests A contained multi-file change you watch while it happens
Copilot CLI A terminal agent (it needs Node.js 22 or later) Work that lives in the terminal: scripts, git, build failures, running commands
Copilot cloud agent (formerly "Copilot coding agent") You assign an issue to Copilot, or start a task from the agents panel. It works in an ephemeral GitHub Actions environment and opens a draft pull request Well-specified issues you would give to a new teammate. Paid plans only
Copilot code review Copilot as a reviewer on a PR, requested by hand or automatic through repo or org settings or rulesets A first pass on every PR, before a human looks

The plans are Free, Student, Pro, Pro+, Max, Business and Enterprise 🔧. Premium usage is billed in "GitHub AI Credits" 💰, and each plan includes a monthly allowance. For Business and Enterprise, an org admin must enable the cloud agent. Elena's question applies here too: before you turn on automatic agents and reviews for a whole org, look at what the usage will cost.

Picking the right size

The surfaces differ in one thing that matters more than features: how long you are out of the loop.

Surface How long it runs without you What you must provide up front
Completions A line or a block Nothing: your code is the context
Chat and agent mode Minutes, and you watch it A clear goal and the files that matter
Copilot CLI Minutes, with your approval for commands A goal and permission boundaries
Cloud agent Until it opens a PR A written issue with acceptance criteria, as for a new colleague

The longer it runs alone, the better your specification must be. An issue that says "fix imports" produces a guess. An issue that says "when the CSV has a header row with a byte-order mark, the parser skips the first column; add a failing test in BrokerCsvParserTests, then fix the parser" produces a reviewable PR.

💡 Match the surface to the size of the job, and the size of the job to how good your specification is.


🧠 Giving AI context: custom instructions, prompt files and MCP

The model sees your code, but it doesn't know your rules. It can't tell that Ledgerly uses central package management, TimeProvider, [LoggerMessage] and the outbox pattern unless something tells it. Three mechanisms provide that context.

Custom instructions

A file called .github/copilot-instructions.md in the repository (or AGENTS.md) is read as extra context for Copilot requests in that repo 🔧. It is the team's "how we do things here", written for a reader that is fast, literal and new every time.

💻 In Ledgerly: the instructions template

The sample repo doesn't ship one in .github/, but the toolkit has a template written for this codebase. An excerpt:

📄 toolkit/copilot-instructions.md

## C# style

- C# 14 with nullable reference types on. File-scoped namespaces, primary constructors and records where they fit.
- `async` all the way with `CancellationToken` passed through. Never `.Result`, `.Wait()` or `async void`.
- Inject `TimeProvider` instead of using `DateTime.Now` or `DateTime.UtcNow`.
<!-- ... -->

## Messaging

- Never publish to Service Bus directly from a request. Write an outbox message in the same `SaveChanges` call; the worker relays it.
- Consumers must be idempotent: check `ProcessedMessages` before doing work.
<!-- ... -->

## When you're not sure

- Say so, and point to the file or doc you would check. Don't invent APIs or package versions.
- Prefer small, reviewable changes with a test that fails without the change.

What makes a good instructions file:

Do Don't
Write rules the model can't infer from the code ("never publish from a request, use the outbox") Restate C# basics the model already knows
Name exact commands (dotnet test --solution Ledgerly.slnx) Write a 20-page style guide nobody maintains
Say what is forbidden and why, in one line Paste secrets, URLs of internal systems or customer names
Review changes to it in PRs, like code Let each developer keep a private version

The weekend contractor's PR would have looked very different with this file in place: "consumers must be idempotent" and the resilience conventions are exactly what the hand-written retry loop ignored.

Prompt files

Prompt files are reusable, task-shaped prompts stored in the repository, such as "add an endpoint following our conventions" or "write a migration that is backward compatible". 🔧 The file location, format and which surfaces support them change, so check the current GitHub Copilot docs before you set them up. The idea is what matters: if you type the same careful prompt three times, save it in git, where the team can review and improve it.

MCP servers

The Model Context Protocol (MCP) lets an AI tool call other tools: read your issues, query a database schema, look at a running system. Copilot supports MCP servers 🔧. One you already have: aspire mcp is an MCP server for AI agents built into the Aspire CLI, so an agent can see the resources, logs and traces of the app you started with aspire run, instead of guessing from source code.

Jonas has one rule for MCP: every MCP server is a new integration with access to something. Ask the same questions as for any dependency: who wrote it, what can it read, what can it change, and where do its credentials come from? Prefer read-only access, and never point an agent at production data with write permissions.

💡 Context is the difference between a generic answer and a Ledgerly answer. Put the rules in git, and treat tool access as a security decision.


🗺️ The senior loop

flowchart LR
    A["You write the goal<br/>and acceptance criteria"] --> B["Context:<br/>instructions, files, MCP"]
    B --> C["Copilot drafts<br/>code and tests"]
    C --> D["Build: analyzers<br/>warnings as errors"]
    D -- "fails" --> C
    D -- "passes" --> E["Tests: do they test<br/>behavior or mocks?"]
    E -- "weak" --> A
    E -- "good" --> F["Human review<br/>same bar as any PR"]
    F -- "changes requested" --> C
    F -- "approved" --> G["Merge and deploy<br/>through the pipeline"]

🧠 Reviewing AI output: one bar, enforced by machines first

Maya's review rule: an AI-written PR gets exactly the same review as a human-written one. Not lower, because the tool sounds confident. Not higher out of suspicion, because then people stop saying which PRs used AI. What changes is where you look harder.

Tests first, and read the tests

The single best guardrail is a test you wrote (or at least read carefully) before accepting the implementation. A test that encodes the business rule tells you whether the draft is right. A test generated together with the code mostly tells you the code does what the code does.

💻 In Ledgerly: a test that encodes a rule

📄 sample/tests/Ledgerly.UnitTests/HoldingCalculatorTests.cs

    [Fact]
    public void Selling_keeps_the_average_cost_of_the_rest()
    {
        var holdings = HoldingCalculator.Calculate(
        [
            Trade("AAPL", TransactionType.Buy, 10, 100m, 1),
            Trade("AAPL", TransactionType.Sell, 4, 150m, 2),
        ]);

        var aapl = Assert.Single(holdings);
        Assert.Equal(6, aapl.Quantity);
        Assert.Equal(100m, aapl.AverageCost);

This test states a real rule of average-cost accounting: selling doesn't change the average cost of what's left. If an AI "refactor" of HoldingCalculator breaks that rule, this test fails, no matter how plausible the new code looks. Compare it with the contractor's tests, which verified that a mock was called three times. Those would pass for almost any implementation.

Let the build say no

The second guardrail is the build itself. Ledgerly's build is strict, and that strictness applies to every author:

📄 sample/Directory.Build.props

    <!-- Quality gates: warnings fail the build, analyzers and code style run in every build (also in CI). -->
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <AnalysisLevel>latest-recommended</AnalysisLevel>
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>

📄 .github/workflows/ci.yml

      # Warnings are errors, analyzers + code style run, NuGet audit fails on vulnerable packages.
      - name: Build
        run: dotnet build Ledgerly.slnx -c Release --no-restore

Nullable warnings, unused usings, code style and vulnerable packages all fail the build before a human spends a minute on review. Agent mode runs the build in its loop, so with these settings it fixes many problems itself. One thing to watch: a tool under pressure to make the build green may add a #pragma warning disable or a NoWarn entry. The template instructions say "never suppress them without a comment explaining why", and your review should check for it.

Where to look harder in an AI-written PR

Look for Why AI gets it wrong Ledgerly example
Invented APIs or package versions The model fills gaps with plausible names A HybridCache method that doesn't exist
New dependencies A package is the easy answer A retry library next to the standard resilience handler
Ignored conventions The model doesn't know your rules Publishing to Service Bus from an endpoint instead of the outbox
Missing failure paths Happy paths dominate training data No CancellationToken, no fallback when market data is down
Security shortcuts Examples online often use keys A connection string with a key instead of managed identity
Tests that test mocks Generated together with the code Asserting call counts instead of results

The toolkit's pr-review-checklist.md has the full list, and it applies to every PR, whoever wrote it. Copilot code review can take a first pass, but a human approves.

💡 Make the machines say no first (tests, analyzers, warnings as errors), so human review can spend its time on design and failure paths.


🧠 Where AI shines for you, where it doesn't, and what never goes in a prompt

You are a senior .NET developer moving from .NET Framework to cloud native. Here is an honest map, based on the work in this course:

Task How well AI helps Your job
Upgrading .NET Framework code (the changes from chapter 2) Strong on mechanical changes: project files, System.Web replacements, async signatures Plan the order, keep the build green at each step, and decide what to redesign instead of port
Writing tests Strong at setup and test data. Weak at deciding what to test Write the test names (the rules) first, then let it fill in bodies
Drafting KQL Good first drafts of summarize and join queries Run it against real data, and check time windows and bins
Drafting Bicep Good at resource shapes and properties az bicep lint, az deployment group what-if, and check identities and roles
Explaining unfamiliar code or errors Very good, especially stack traces and config Verify against the docs before acting
Architecture decisions It produces confident options. It doesn't know your constraints Decide, and write the ADR yourself
Security-sensitive code (auth, crypto, permissions) Plausible and sometimes subtly wrong Use the framework's built-in features, and get Jonas to review
Production incidents Can draft queries and summaries Mitigate with the runbook first. Don't experiment on production

For Bicep, the what-if step deserves its own line. It previews create, modify and delete operations before anything changes:

az deployment group what-if -g <rg> -f main.bicep -p main.bicepparam

An AI draft that would delete and recreate a database shows up there as a delete, in plain text. See Bicep what-if.

What never goes into a prompt

Jonas's rules, which apply to every AI tool, including approved ones:

Never Because Instead
Secrets: keys, connection strings, tokens Prompts and logs are data that leaves your machine Managed identity, and placeholder values in examples
Customer data or production records Privacy and compliance obligations follow the data Synthetic test data
Code from a project your license doesn't allow you to share The license applies to the code, not to the tool Check with your organization first
Accepting large generated blocks you don't understand You can't maintain or license-check what you don't understand Ask for smaller pieces, read each one

Your organization's Copilot policies (which features are enabled, what data handling applies, how suggestions that match public code are treated) are set by admins 🔧. Know what yours are before you rely on a feature.

💡 AI is strongest where the result is easy to check. Invest your attention where checking is hard: design, security and failure paths.


🗺️ The cloud agent's pull request

sequenceDiagram
    participant Y as "You"
    participant I as "GitHub issue"
    participant A as "Copilot cloud agent"
    participant CI as "CI workflow"
    participant R as "Reviewers"
    Y->>I: Write goal, acceptance criteria and a failing test idea
    Y->>A: Assign the issue to Copilot
    A->>A: Work in an ephemeral Actions environment
    A->>R: Open a draft pull request
    CI->>R: Build with warnings as errors, tests, audit
    R->>A: Copilot code review and your comments
    A->>R: Push fixes to the same PR
    R->>Y: Human approval required to merge

🧠 Building AI features into Ledgerly: a short pointer

Using AI to write code is one thing. Putting AI inside your app is another, and .NET has standard building blocks for it.

Library What it is When you'd use it
Microsoft.Extensions.AI (GA, 10.11.0 🔧) Common abstractions IChatClient and IEmbeddingGenerator<TInput,TEmbedding>, implemented by provider libraries (OpenAI, Azure OpenAI, Ollama, Foundry and others), plus middleware Any AI call from a .NET app. Apps reference Microsoft.Extensions.AI, libraries only Microsoft.Extensions.AI.Abstractions
Microsoft Agent Framework (Microsoft.Agents.AI, GA, 1.26.0 🔧) Agents and multi-agent workflows for .NET and Python, built on IChatClient, with graph-based workflows, checkpointing, human-in-the-loop and a DevUI. It succeeds Semantic Kernel and AutoGen When one prompt isn't enough: multi-step agents with tools and approvals

The middleware is the part a cloud-native developer will like: UseOpenTelemetry() puts AI calls into the same traces as everything else (chapter 15), UseLogging() logs them, UseDistributedCache() caches identical requests, and UseFunctionInvocation() lets the model call your C# functions.

A small, realistic first feature: explain a failed CSV import in plain language.

// illustrative, not in the sample
// Program.cs: the inner client comes from a provider package (OpenAI, Azure OpenAI, Ollama, Foundry, ...)
builder.Services.AddChatClient(innerClient)
    .UseOpenTelemetry()
    .UseLogging();

public sealed class ImportErrorExplainer(IChatClient chat)
{
    public async Task<string> ExplainAsync(string errorSummary, CancellationToken ct)
    {
        ChatResponse response = await chat.GetResponseAsync(
            $"Explain this CSV import error to a non-technical user in two sentences: {errorSummary}",
            cancellationToken: ct);
        return response.Text;
    }
}

Everything from earlier chapters still applies, and Maya's question comes first: "What happens when it fails?" An AI provider is an external dependency like the market-data vendor. It needs a timeout, a fallback (show the original error message), a kill-switch flag (chapter 19), cost monitoring 💰 and a decision about what user data may be sent. Docs: Microsoft.Extensions.AI and Microsoft Agent Framework.

💡 An AI model in your app is just another unreliable, metered dependency. Wrap it in the same resilience, flags and telemetry as the rest.


⚠️ Common traps

Trap What happens Do this instead
Accepting a large AI diff because "the tests pass" The tests were generated with the code and assert mocks, so the bug ships Write or read the tests first. Ask for small PRs
No instructions file Every suggestion ignores your conventions, and reviewers fix the same things every week A short .github/copilot-instructions.md, reviewed in PRs
Assigning vague issues to the cloud agent The draft PR solves a different problem Acceptance criteria and the files involved, as for a new colleague
Letting AI silence warnings #pragma warning disable hides the problem the build found Treat any suppression as a review finding that needs a reason
Pasting a production error with customer data into chat Personal data leaves your control Redact, or reproduce with synthetic data
Installing MCP servers without review An agent gets access you never approved Review MCP servers like dependencies, read-only first
Trusting AI on versions and APIs Invented packages, wrong method names Check NuGet and the docs. The build and NuGet audit catch some, not all

🧰 Helpers

A good issue for the cloud agent (template):

## Goal
One sentence: what should be true when this is done.

## Acceptance criteria
- [ ] A failing test in <test file> that shows the problem
- [ ] The fix, with the test passing
- [ ] No new packages, no suppressed warnings

## Context
Files involved: <paths>. Conventions: see .github/copilot-instructions.md.

## Out of scope
What not to change.

Prompts that work like a senior:

Verifying a Bicep draft:

az bicep lint --file main.bicep
az deployment group what-if -g <rg> -f main.bicep -p main.bicepparam

In the toolkit: copilot-instructions.md (copy it to .github/ and edit) and pr-review-checklist.md. See the toolkit.

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

  1. In your own fork of the course repo, copy toolkit/copilot-instructions.md to .github/copilot-instructions.md.
  2. Open sample/tests/Ledgerly.UnitTests/HoldingCalculatorTests.cs. Write only the name of a new test for a rule you know, for example "selling everything removes the holding". Then ask Copilot chat or agent mode to write the body.
  3. Run the tests: bash cd sample dotnet test --solution Ledgerly.slnx Read the generated test. Does it test the rule, or does it just repeat the implementation?
  4. Ask Copilot to "add retry logic to the market data client". Compare its answer with what the sample already does in src/Ledgerly.Api/Program.cs (the standard resilience handler from chapter 14). Did it notice? Would your instructions file have helped?
  5. Ask it to draft a KQL query for "failed imports per hour in the last day". Read every line and decide which table and columns you would check before trusting it.
  6. Build to see the guardrail: dotnet build Ledgerly.slnx. Add an unused using to any file and build again. Observe: the build fails. Remove it.

✅ Recap

  1. Pick the Copilot surface by how long it runs without you, and give it context through committed instructions, prompt files and carefully reviewed MCP servers.
  2. Review AI output with the same bar as any PR: tests that encode rules, analyzers and warnings as errors first, then human judgment on design, security and failure paths.
  3. Use AI where results are easy to check, keep secrets and customer data out of prompts, and treat AI inside your app as one more unreliable, metered dependency.

Next → Chapter 21: Your toolkit

Chapter 21: Your toolkit

⏱ About 25 min · You'll learn: what is in the toolkit and when to reach for each file, the whole habits notebook in one table, a day-one checklist for a new service, and what to learn next (and when you would actually need it).


📖 Story: Closing the notebook

It is a Friday afternoon, a few months after your first day. Ledgerly's beta is stable, the 2 a.m. page has a runbook, and the last three deploys were so boring that nobody mentioned them at stand-up. Maya asks you to lead the next service: a small "statements" worker that builds monthly PDF statements for users. Elena wants it in the beta next quarter. Tomás has already created the repository, and the first thing he put in it was a link to the toolkit.

You open your "Habits I brought with me" notebook one last time. The first page still has your notes from day one: "No servers to RDP into? Logs are queries?" Every habit since then has a line crossed out and a new practice written next to it. Some crossings-out were easy (DateTime.Now). Some took an incident (the flag that didn't reach the worker). You realize the notebook is no longer a list of things you used to do wrong. It is a checklist for the next service.

So this chapter is the handover: from the course to your own work. Here is everything you can copy, the whole notebook in one table, and an honest map of what to learn next.


🏚️ Old way → 🚀 New way

What you did before What you do now Why it changed
Starting a new service by copying the last project and deleting what you don't need Starting from a known-good toolkit: build settings, service defaults, CI, deploy and checklists The defaults carry every lesson you already paid for
Knowledge lived in senior people's heads Knowledge lives in git: instructions, ADRs, runbooks, checklists People change teams. The repository stays
Learning the next big technology because it's popular Learning what the next problem needs, when you hit it Every new platform has a running cost in complexity
"We'll add monitoring and alerts before go-live" Telemetry, health probes, alerts and a runbook from day one Retrofitting observability is slower and always incomplete
A big upgrade project every few years Small, regular updates: Dependabot, NuGet audit, an LTS plan Small steps are cheap. Big jumps are risky and expensive
A personal notebook of lessons A shared checklist the whole team uses and improves A lesson that only you know protects only your code

💡 The toolkit isn't the end of the course. It's the starting point for every service you build after it.


🧠 The toolkit, file by file

Everything below is in the toolkit/ folder. The toolkit README explains how to copy each file into a new repository. Treat the files as starting points: edit them for your team, then keep them under review like code.

File What it is When you use it
README.md The index of the toolkit, with copy instructions First, when you start a new repository
cheat-sheet-old-to-new.md The .NET Framework habit next to its modern replacement, on one page When you catch yourself typing an old pattern, or when you onboard a colleague with the same background
production-readiness-checklist.md What a service must have before real users touch it Before the first production deploy, and after every incident that adds a line
pr-review-checklist.md The review bar for every PR, human or AI-written Every review, especially large or AI-assisted ones
kql-queries.md Ready-made KQL for requests, dependencies, exceptions and SLOs During an incident, and when you build alerts and workbooks
slo-worksheet.md A worksheet to choose SLIs, SLO targets and error budgets When a service gets its first users, and when Elena asks "is it healthy?"
runbook-template.md The structure of a short runbook for one alert The same day you create an alert
postmortem-template.md A blameless postmortem: timeline, factors, owned actions After every incident that hurt users, within a few days
adr-template.md An architecture decision record: context, decision, consequences When you make a choice that someone will later ask "why?" about
copilot-instructions.md A template for repository custom instructions for GitHub Copilot Copy it to .github/copilot-instructions.md in every new repo and edit it
pull_request_template.md A PR description template with the questions reviewers need Copy it to .github/ so every PR starts with it
Directory.Build.props Shared build settings: net10.0, nullable, warnings as errors, analyzers, code style At the root of every new solution, on day one
Directory.Packages.props Central package management: one place for all package versions At the root of every new solution, before the first package is added
.editorconfig Code style and analyzer severities Next to the props files, so the IDE and CI agree
ServiceDefaults.Extensions.cs AddServiceDefaults(): OpenTelemetry, health checks, service discovery, resilience In every service, as the first line after the builder is created
ci.yml The CI workflow: build with warnings as errors, tests, infrastructure lint In .github/workflows/ of every new repo
deploy.yml The CD workflow: OIDC login, images tagged with the commit, migrations job, smoke test When the service gets its first Azure environment
dependabot.yml Dependabot updates for NuGet packages and GitHub Actions In .github/ on day one, so updates stay small
main.bicep (pointer) A pointer to the sample's Bicep entry point and modules When you write the infrastructure for a new service

💻 In Ledgerly: one line that carries ten chapters

The toolkit's ServiceDefaults.Extensions.cs comes from this file. Look at how much one method gives every service:

📄 sample/src/Ledgerly.ServiceDefaults/Extensions.cs

    public static TBuilder AddServiceDefaults<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
    {
        builder.ConfigureOpenTelemetry();
        builder.AddDefaultHealthChecks();

        builder.Services.AddServiceDiscovery();

        builder.Services.ConfigureHttpClientDefaults(http =>
        {
            // Retries with jittered back-off, per-attempt + total timeouts, circuit breaker, rate limiter.
            http.AddStandardResilienceHandler();

            // Resolves "https+http://marketdata" to a real address (Aspire locally, config or DNS in Azure).
            http.AddServiceDiscovery();
        });

        return builder;
    }

Telemetry (chapters 15 and 16), health checks for Container Apps probes (chapter 14), service discovery (chapter 6) and resilience for every HttpClient (chapter 14). A new service that calls builder.AddServiceDefaults() starts where Ledgerly ended, not where it began.

💡 Copy the toolkit on day one, not "when we have time". Defaults are cheapest before the first line of business code.


🗺️ The course in one picture

mindmap
  root(("Cloud-native .NET"))
    "Foundations"
      "Modern C# and .NET 10"
      "Solution set up like a pro"
    "Build"
      "APIs, data, messaging"
      "Aspire locally"
    "Ship"
      "GitHub, CI and CD"
      "Bicep and identity"
    "Run"
      "Reliability and observability"
      "Performance and production"
    "Practice"
      "Testing, AI and the toolkit"

🧠 The habits notebook, closed

Here is the whole notebook in one table. Each row is a habit you brought from Nordbank, the practice that replaced it, and where it happened.

Old habit New practice Chapter
Servers you log in to, logs you open as files Disposable containers, logs as queries, boring deploys 1
C# 5 style: DateTime.Now, sync I/O, null checks everywhere Modern C#: records, nullable types, TimeProvider, async all the way 2
"Works on my machine" builds with versions in every project One-command builds, central package management, warnings as errors, NuGet audit 3
Controllers returning whatever, errors as strings Minimal APIs, validation, ProblemDetails, OpenAPI, rate limits 4
Hand-run SQL scripts and "last write wins" EF Core migrations as a deploy step, concurrency tokens 5
A wiki page of local setup steps An Aspire AppHost: F5 starts the whole system 6
Windows Services and SQL Agent jobs Queues, the outbox, idempotent consumers, dead-letter handling 7
Long-lived branches and email reviews GitHub flow, rulesets, CODEOWNERS, small PRs 8
A build server someone configured by hand CI as code in GitHub Actions, Dependabot, CodeQL 9
"We'll get a bigger VM" Choosing managed services by need and cost 💰 10
Clicking resources together in the portal Bicep in git, what-if before every change 11
Publish profiles and a deploy password in a secret OIDC, environments with approvals, build once and promote 12
Connection strings with passwords in config Managed identity everywhere, JWT for users 13
"The vendor is never down" Timeouts, retries, circuit breakers, graceful degradation 14
Console.WriteLine and log files Structured logs, OpenTelemetry traces and metrics 15
"It seems fine" KQL, SLOs, error budgets and alerts with numbers 16
Manual testing before release A test pyramid with real dependencies in containers 17
Guessing at performance, caching "everything" Measure first, HybridCache, async, scale rules with limits 18
Heroic nights and "human error" Runbooks, kill switches, roll forward, blameless postmortems 19
Either ignoring AI or trusting it blindly AI as a drafting tool, held to the same review bar 20

A pattern runs through the right-hand column: almost every new practice moves something from a person's memory into a file in git. The build settings, the infrastructure, the pipeline, the alert, the runbook, the instructions for the AI. That is what Tomás meant on your first week: "If it's not in git, it doesn't exist."

The toolkit's cheat-sheet-old-to-new.md is the code-level version of this table: the specific API or pattern you used to write, and what you write now.

💡 You didn't stop being a senior .NET developer. You moved your judgment into files the whole team can use.


🧠 The day-one checklist for a new service

Maya hands you a single page for the statements service. It is the production readiness checklist turned around: what to do on day one, so the list before go-live is short.

💻 In Ledgerly: a whole service on one screen

The worker is a good model for a new service. Everything it needs is wired in a few lines:

📄 sample/src/Ledgerly.Worker/Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();
builder.AddSqlServerDbContext<LedgerlyDbContext>("ledgerlydb");
builder.AddAzureServiceBusClient("messaging");
// ...
builder.Services.AddSingleton(TimeProvider.System);
// ...
// Give in-flight messages time to finish when Container Apps stops a replica (SIGTERM).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));

var app = builder.Build();
app.MapDefaultEndpoints();
app.Run();

And it joins the local system with one registration in the AppHost:

📄 sample/src/Ledgerly.AppHost/AppHost.cs

builder.AddProject<Projects.Ledgerly_Worker>("worker")
    .WithReference(db)
    .WithReference(messaging)
    .WithReference(marketData)
    .WaitFor(messaging)
    .WaitForCompletion(migrations)
    .WithHttpHealthCheck("/health");

The checklist

Repository (hour one)

Code (day one)

Azure (before the first deploy)

Running it (before real users)

💡 Most of this list is copying files and adding one line each. That's the point: the expensive lessons are already encoded.


🗺️ A new service, from empty repo to first user

flowchart TB
    A["Empty repository"] --> B["Toolkit files:<br/>props, editorconfig, CI, Dependabot"]
    B --> C["Service with AddServiceDefaults<br/>and MapDefaultEndpoints"]
    C --> D["Registered in the AppHost<br/>aspire run works"]
    D --> E["Tests: one rule,<br/>one integration path"]
    E --> F["Bicep module<br/>identity, probes, scale limits"]
    F --> G["Deploy workflow<br/>OIDC and smoke test"]
    G --> H["SLO, alert and runbook"]
    H --> I["Production readiness<br/>checklist passes?"]
    I -- "no" --> J["Fix the gap,<br/>write an ADR if it's a decision"]
    J --> I
    I -- "yes" --> K["First real users"]

🧠 What to learn next, and when you'd need it

The cloud-native world has far more technology than any one team needs. Here is an honest map. "When you'd need it" matters more than "what it is": learn each one when a real problem asks for it.

Topic What it is When you'd need it When you wouldn't
Kubernetes and AKS Full Kubernetes API control on Azure, as AKS Standard or AKS Automatic. Container Apps runs on Kubernetes but doesn't expose its API You need custom operators, special networking or scheduling, or a platform team already runs Kubernetes Your services fit Container Apps. Ledgerly does
Dapr A set of building blocks (state, pub/sub, service invocation) as a sidecar. Optional per app in Container Apps Many services in different languages need the same building blocks A .NET-only system using Aspire integrations and the Azure SDKs directly. Nothing in this course needs it
Event-driven architecture Service Bus topics and subscriptions: publish once, each subscription gets a copy, with SQL or correlation filters A second consumer wants the same events, like the statements service listening for completed imports One producer, one consumer: a queue is simpler
Platform engineering A team that builds paved roads (templates, pipelines, environments) for other teams Several teams build services the same way and keep solving the same setup problems One team, a few services. The toolkit is your paved road
.NET 11 🔧 The next release, STS (2 years of support). At RC3 as of late October 2026, GA expected at .NET Conf in November 2026 You want a specific new feature and can upgrade again within two years You prefer the 3-year LTS cycle. .NET 10 is supported until 2028-11-14
aspire deploy 🔧 The AppHost provisions Azure resources and deploys containers, now the recommended Aspire deploy path New projects or prototypes where the AppHost model is enough You need full Bicep control and review, as Ledgerly's infra/ does
Azure App Configuration Central configuration and feature flags with refresh, no redeploy Flags must flip in seconds, or many services share settings One service with a few settings in environment variables
Microsoft Agent Framework 🔧 Agents and multi-agent workflows on IChatClient, GA Your product needs multi-step AI with tools and approvals One prompt with IChatClient is enough

Release dates and support windows change, so check the .NET support policy and what's new in .NET 11. For compute choices, Microsoft's container options comparison is the place to start. For events, see queues, topics and subscriptions.

How to learn the next thing

The method that worked for this course works for the next topic too:

  1. Start from a problem, not a technology: "two services need the same import events" leads to topics, not the other way round.
  2. Build the smallest version in the sample, locally with aspire run, before it goes near production.
  3. Write the ADR: what you chose, what you didn't, what it costs to run 💰 and to operate.
  4. Add the lesson to the toolkit, so the next service gets it for free.

💡 The best next technology is the one your next real problem needs. Everything else can wait in a bookmark.


⚠️ Common traps

Trap What happens Do this instead
Copying the toolkit once and never updating it Each repository drifts, and lessons from new incidents never reach the old services Treat the toolkit as a repository with PRs, and update it after postmortems
Leaving the readiness checklist until go-live week Alerts, runbooks and identities get rushed or skipped Use the day-one checklist, so go-live is a short final check
Adopting Kubernetes or Dapr "to be ready" Weeks of platform work and a new on-call surface, with no user benefit Wait for a problem that needs it, and write an ADR when it comes
Upgrading to every new STS release without a plan Frequent upgrade work, or running an unsupported version Choose LTS or STS on purpose, and put the end-of-support date in your calendar
Keeping the notebook private Your team repeats the mistakes you already learned from Turn lessons into checklist lines, instructions and templates in git
Treating checklists as bureaucracy People tick boxes without thinking Keep them short, explain each line, delete lines that never catch anything

🧰 Helpers

Starting a new service, in order:

  1. Read the toolkit README and copy the build, style and workflow files.
  2. Add ServiceDefaults and register the service in the AppHost.
  3. Write the first rule as a test, then the code.
  4. Write the Bicep module and the deploy workflow before the first user story is finished.
  5. Walk through production-readiness-checklist.md with Priya before real users arrive.

Questions to ask in any design review (from the whole team):

Make it permanent:


🧪 Try it yourself (optional, ~30 min)

  1. Run the finished sample one more time, and walk through the dashboard as if you were showing it to a colleague from your old team: bash cd sample aspire run
  2. Build and test it the way CI does, and notice how little you have to remember: bash dotnet build Ledgerly.slnx dotnet test --solution Ledgerly.slnx
  3. Open the toolkit README. For each file, decide whether you would copy it unchanged, edit it, or skip it for your next service. Write down why.
  4. Take the day-one checklist above and fill it in for a service you know from your own work. Which lines would have prevented an incident you remember?
  5. Open your own notebook (or start one) and add one habit this course didn't cover, with its new practice. Then turn it into a line in a checklist.

✅ Recap

  1. The toolkit turns every lesson of this course into files you copy on day one: build settings, service defaults, workflows, templates and checklists.
  2. Almost every new habit moved knowledge from someone's memory into git, where the whole team can use and improve it.
  3. Learn the next technology when a real problem needs it, write the decision down, and feed the lesson back into the toolkit.

Back to → Start

Toolkit

Copy-ready files from the course. The checklists and templates are written for your next real service, not for Ledgerly. The config files come straight from the working sample, so you know they build.

🔧 Versions, defaults and product names in these files were correct in October 2026. Check them when you copy a file into a new repo.

Checklists and cheat sheets

File What it is When to use it Chapter
Cheat sheet: old to new One table that maps every .NET Framework / IIS habit to its modern replacement Keep it open for the first months All
Production readiness checklist What must be true before a service takes real traffic Before the first production deploy, then once a quarter 19
PR review checklist What a reviewer checks beyond "does it compile" Every pull request, yours and AI-written ones 8, 20
KQL queries Starting-point queries for requests, dependencies, logs, custom metrics and imports When you build dashboards, alerts or investigate an incident 16, 19
SLO worksheet Fill-in worksheet: user journeys, SLIs, SLOs, error budgets, alerts When you define what "healthy" means 16

Templates

File What it is Where it goes Chapter
Runbook template One page per alert: what it means, how to check, how to fix Next to the alert, linked from the alert text 19
Postmortem template Blameless incident review A docs/postmortems/ folder or your wiki 19
ADR template Architecture decision record docs/adr/NNNN-title.md 1, 19
Copilot instructions Repository custom instructions for GitHub Copilot .github/copilot-instructions.md 20
Pull request template Pre-filled PR description .github/pull_request_template.md 8

Config files from the sample

These files are not shown on this page because they are not Markdown. Open them on GitHub and copy them.

File What it gives you Where it goes Chapter
Directory.Build.props Target framework, nullable, warnings as errors, analyzers, code style in build Solution root 3
Directory.Packages.props Central package management with transitive pinning Solution root 3, 9
.editorconfig Code style and analyzer severities Solution root 3
ServiceDefaults.Extensions.cs OpenTelemetry, health endpoints, resilience and service discovery in one call A ServiceDefaults project shared by every service 6, 14, 15
ci.yml Build, test with coverage, container images, Bicep lint .github/workflows/ci.yml 9
deploy.yml OIDC login, three-phase Bicep deploy, migration job, smoke test .github/workflows/deploy.yml 12
dependabot.yml Grouped weekly updates for NuGet and GitHub Actions .github/dependabot.yml 8, 9
main.bicep The whole Azure environment (lives in sample/infra, with its modules) infra/ 10, 11

Adapting the config files

File Change before you use it
Directory.Packages.props Remove the packages you don't use. Keep CentralPackageTransitivePinningEnabled.
ci.yml, deploy.yml Paths (sample/, Ledgerly.slnx), the project list in the container matrix, and the resource names.
deploy.yml Set the repository variables AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID and AZURE_RESOURCE_GROUP (chapter 12). Until AZURE_CLIENT_ID exists, the workflow skips itself.
dependabot.yml The directory value and the package groups.
main.bicep appName, the JWT authority and audience, and the CORS origins in main.bicepparam.

How the pieces fit

flowchart LR
    subgraph Repo["Your repository"]
        P["Directory.Build.props<br/>Directory.Packages.props<br/>.editorconfig"]
        G[".github/<br/>workflows, dependabot,<br/>PR template, Copilot instructions"]
        I["infra/<br/>main.bicep"]
        S["ServiceDefaults"]
    end
    subgraph Docs["Team docs"]
        A["ADRs"]
        R["Runbooks"]
        M["Postmortems"]
    end
    subgraph Azure["Azure"]
        K["Application Insights<br/>KQL queries, SLOs"]
    end
    G -- "deploys" --> I
    S -- "telemetry" --> K
    K -- "alerts link to" --> R
    R -- "incidents become" --> M
    M -- "decisions become" --> A

Back to → Chapter 21: Your toolkit

ADR template

An architecture decision record (ADR) captures one significant decision: the context, the options, the choice and its consequences. Six months later, nobody remembers why you picked Container Apps over App Service. The ADR does.

Write one when a decision is hard to reverse, affects more than one team, or has been argued about twice.

Copy everything below the line into docs/adr/NNNN-short-title.md (number them in order and never reuse a number). Delete the hints in italics. To change a decision, write a new ADR that supersedes the old one; don't edit history.


ADR NNNN: decision title, as a short statement

Status Proposed / Accepted / Superseded by ADR NNNN / Deprecated
Date
Deciders
Consulted

Context

What problem are we solving? What constraints apply (team skills, budget, deadlines, compliance)? What forces are in tension? Keep it factual.

Options considered

Option Pros Cons
A
B
C: do nothing

Decision

We will ... because ...

Consequences

When to revisit

The signal that would make us reconsider, e.g. "if we need more than 30 services" or "if the monthly bill passes X".


Example (illustrative): ADR 0003, host the services on Azure Container Apps

Status Accepted
Deciders Maya Okafor, Tomás Varga
Consulted Priya Raman, Jonas Berg, Elena Rossi

Context. Four small services (API, worker, market data, migrations), a team of six, a beta in 12 weeks. Nobody on the team runs Kubernetes today. The worker must scale on queue length. Traffic during the beta is low and spiky.

Options considered.

Option Pros Cons
App Service Closest to the IIS model, mature Queue-based scaling of the worker is harder, no scale to zero
Container Apps Scale to zero, KEDA rules for Service Bus, jobs for migrations, no cluster to run Cold starts at zero replicas, fewer knobs than Kubernetes
AKS Full control, huge ecosystem We'd spend the beta learning to operate a cluster

Decision. We will run all services on Azure Container Apps in one environment, because it gives us queue-based scaling and run-to-completion jobs without operating a cluster.

Consequences. Good: low idle cost, one deploy model for every service. Bad: cold starts on the API unless we keep one replica (a cost decision for Elena). Follow-up: the migrations job (chapter 12), scale rules (chapter 18).

When to revisit. If we need service mesh features, or more than about 20 services.

Cheat sheet: old to new

Each row maps an old .NET Framework or IIS habit to its modern replacement, with the chapter that covers it.

Runtime and code

You used to Now you Chapter
Target .NET Framework 4.8, installed on the server Target net10.0 (LTS). The runtime ships inside your container image. 2
Global.asax, Startup, OWIN WebApplication.CreateBuilder(args) in Program.cs. One composition root. 2
Unity, Autofac or Ninject for DI Built-in IServiceCollection. Add a container only if you really need its extra features. 2
ConfigurationManager.AppSettings["x"] IConfiguration plus the options pattern (IOptions<T>), with environment variables overriding JSON 2
web.config transforms per environment appsettings.{Environment}.json, environment variables and Key Vault 2, 13
new HttpClient() per call, or one static client IHttpClientFactory with typed clients and a resilience handler 2, 14
.Result and .Wait() on tasks async all the way, passing CancellationToken 2, 18
DateTime.Now TimeProvider (inject it, fake it in tests) 2, 17
Guid.NewGuid() for clustered keys Guid.CreateVersion7() (time-ordered) 2, 5
Classes with 30 lines of properties record types, primary constructors and required members 2
string.Format log messages Message templates plus [LoggerMessage] source generation 15

Solution and build

You used to Now you Chapter
packages.config, a version per project Central package management in Directory.Packages.props 3
.sln files that merge-conflict .slnx (XML solution format) 3
Compiler warnings nobody reads TreatWarningsAsErrors, analyzers and code style enforced in build 3
A wiki page called "Developer setup" global.json, a tool manifest and one aspire run 3, 6
Package sources configured per machine nuget.config in the repo with <clear/> and package source mapping 3, 13

API

You used to Now you Chapter
Web API 2 controllers Minimal APIs with route groups and TypedResults (controllers still work) 4
Custom error JSON per team ProblemDetails (RFC 9457 shape) everywhere 4
A Word document describing the API An OpenAPI document generated at build/run time, browsed with Scalar 4
Last write wins Optimistic concurrency with rowversion, returning 409 4, 5
No limits until the server falls over Rate limiting middleware returning 429 4, 14

Data

You used to Now you Chapter
EF6 or hand-written ADO.NET EF Core 10 5
SQL scripts run by hand on release night EF migrations applied by a separate run-to-completion process 5, 12
SQL logins with passwords in web.config Microsoft Entra authentication with managed identities, no passwords 13
A shared dev database A local SQL Server container per developer, started by Aspire 6

Background work

You used to Now you Chapter
A Windows Service A BackgroundService in a worker container 7
A SQL Agent job PeriodicTimer in a worker, or a scheduled Container Apps job 7
MSMQ Azure Service Bus queues with peek-lock and dead-lettering 7
"Send the message, then save" (and hope) The transactional outbox and an idempotent consumer 7

Source control and delivery

You used to Now you Chapter
TFVC or a shared drive Git and GitHub flow: short branches, pull requests, protected main 8
A build server someone configured in 2014 GitHub Actions workflows in the repo 9
"Works on my machine" CI builds and tests every pull request 9
Web Deploy or xcopy over RDP on Thursday night CD with OIDC to Azure, after every green build on main 12
A service account password in the build server OIDC federated credentials: no secret stored in GitHub 12
Rolling back by copying the old folder back Redeploying the previous image tag, or activating a previous revision 12, 19

Hosting

You used to Now you Chapter
IIS on a VM you patch Azure Container Apps (or App Service, which is closer to the IIS model) 10
Clicking in IIS Manager or the portal Bicep in git, linted in CI 11
App pool identity User-assigned managed identity 13
Load balancer health check on /ping.aspx Startup, liveness and readiness probes on /alive and /health 14
"Add another VM" Scale rules: HTTP concurrency, queue length (KEDA), min and max replicas 18

Observability and operations

You used to Now you Chapter
log4net/NLog text files on each server Structured logs exported with OpenTelemetry to Application Insights 15
RDP and grep KQL queries across every instance 16
Performance counters on one box OpenTelemetry metrics (Meter) and dashboards 15, 16
Correlating logs by timestamp Distributed traces with W3C trace context, even through queues 15
"The site is up" SLIs, SLOs and error budgets 16
An alert for every exception Few, actionable alerts with a runbook each 16, 19
A blame meeting after the outage A blameless postmortem with action items 19
Comment out code to switch a feature off Feature flags as kill switches 19

Testing

You used to Now you Chapter
MSTest/NUnit with VSTest xUnit v3 on Microsoft.Testing.Platform 17
Integration tests against a shared test database WebApplicationFactory<Program> plus a throwaway SQL Server (Testcontainers) 17
Mocking DateTime.Now with a wrapper FakeTimeProvider 17

Copilot instructions

This repository contains a .NET 10 (LTS) backend: ASP.NET Core minimal APIs, EF Core 10 on Azure SQL, Azure Service Bus and Aspire for local orchestration. It deploys to Azure Container Apps with Bicep and GitHub Actions.

Build and test

Packages

C# style

API conventions

Data

Messaging

Infrastructure and workflows

When you're not sure

KQL queries

Starting-point queries for a Ledgerly-style system in Application Insights (workspace-based). Adapt the names, thresholds and time ranges, then save the good ones as workbook tiles or log search alerts.

🔧 These use the Log Analytics table names (AppRequests, AppDependencies, AppTraces, AppExceptions, AppMetrics) and their columns (TimeGenerated, AppRoleName, DurationMs). If you run them from the Application Insights Logs blade, the classic names also work there: requests, dependencies, traces, exceptions, customMetrics, with timestamp and cloud_RoleName. Check the column names in your workspace schema before you build alerts on them.

AppRoleName is the OpenTelemetry service.name. In the sample it comes from OTEL_SERVICE_NAME: api, worker and marketdata.

1. Failure rate per service (5-minute buckets)

AppRequests
| where TimeGenerated > ago(6h)
| summarize total = count(), failed = countif(Success == false) by AppRoleName, bin(TimeGenerated, 5m)
| extend failureRatePct = round(100.0 * failed / total, 2)
| order by TimeGenerated desc

Use it for: the "is it broken right now?" tile. 🔧 How Success treats 4xx responses depends on the SDK and exporter, so check a few rows first. If needed, define failure yourself with toint(ResultCode) >= 500: a user's validation error is not your outage.

2. p50, p95 and p99 latency per endpoint

AppRequests
| where TimeGenerated > ago(24h) and AppRoleName == "api"
| summarize requests = count(),
            p50 = percentile(DurationMs, 50),
            p95 = percentile(DurationMs, 95),
            p99 = percentile(DurationMs, 99) by Name
| order by p95 desc

Use it for: finding the slow endpoints. Name is the route template (for example GET /api/portfolios/{id:guid}/valuation), so IDs don't explode the result.

3. Calls to the market-data vendor: failures and latency

AppDependencies
| where TimeGenerated > ago(6h)
| where AppRoleName == "api" and Target has "marketdata"
| summarize calls = count(),
            failed = countif(Success == false),
            p95 = percentile(DurationMs, 95) by bin(TimeGenerated, 5m)
| extend failureRatePct = round(100.0 * failed / calls, 2)
| render timechart

Use it for: telling "our API is slow" apart from "the vendor is slow". Retries from the resilience handler appear as separate dependency calls under the same operation.

4. Partial valuations (graceful degradation in action)

AppTraces
| where TimeGenerated > ago(24h)
| where Message has "returning partial valuation"
| extend Symbol = tostring(Properties.Symbol), Reason = tostring(Properties.Reason)
| summarize count() by Symbol, Reason, bin(TimeGenerated, 15m)
| order by TimeGenerated desc

Use it for: seeing what users experienced while the vendor was failing. The message comes from the [LoggerMessage] in ValuationService. Template placeholders become properties, so you can group by them. 🔧 The exact property names depend on the exporter. Run AppTraces | take 10 and look at Properties first.

5. Import throughput and outcomes (custom metric)

AppMetrics
| where TimeGenerated > ago(24h)
| where Name == "ledgerly.imports.processed"
| extend outcome = tostring(Properties.outcome)
| summarize imports = sum(Sum) by outcome, bin(TimeGenerated, 1h)
| render columnchart

Use it for: Elena's question "how many imports ran today, and how many had errors?" The counter is tagged with outcome (completed or completed_with_errors). Metrics arrive pre-aggregated, so you sum(Sum) instead of count().

6. Import duration (custom histogram)

AppMetrics
| where TimeGenerated > ago(24h)
| where Name == "ledgerly.imports.duration"
| summarize totalSeconds = sum(Sum), imports = sum(ItemCount), slowest = max(Max) by bin(TimeGenerated, 1h)
| extend avgSeconds = round(totalSeconds / imports, 2)

Use it for: spotting imports getting slower as files grow. Histograms are exported as aggregates (sum, count, min, max), so you get an average and a maximum, not exact percentiles.

7. Messages being retried or dead-lettered

AppTraces
| where TimeGenerated > ago(24h) and AppRoleName == "worker"
| where Message has "abandoning for retry" or Message has "already processed"
| extend kind = iff(Message has "abandoning", "retry", "duplicate")
| summarize count() by kind, bin(TimeGenerated, 15m)
| render timechart

Use it for: an early warning before messages reach the dead-letter queue. Repeated retries for the same import mean a poison message. The dead-letter count itself is a Service Bus metric on the queue: alert on it with a metric alert, not with KQL.

8. Exceptions grouped by problem

AppExceptions
| where TimeGenerated > ago(24h)
| summarize count(), firstSeen = min(TimeGenerated), lastSeen = max(TimeGenerated),
            sample = take_any(OuterMessage) by ProblemId, AppRoleName
| order by count_ desc

Use it for: the morning check. A new ProblemId that appeared after the last deploy is your first suspect.

9. Follow one request end to end

let op = "<paste an OperationId here>";
union AppRequests, AppDependencies, AppTraces, AppExceptions
| where OperationId == op
| project TimeGenerated, itemType = Type, AppRoleName, Name, Message, DurationMs, Success
| order by TimeGenerated asc

Use it for: the bug report with a timestamp. The OperationId is the W3C trace ID, so this includes the worker's processing of an import that the API accepted, because the sample carries trace context through the outbox and Service Bus. The portal's end-to-end transaction view shows the same data as a timeline.

10. Replica starts (scale-outs, restarts, cold starts)

AppTraces
| where TimeGenerated > ago(24h)
| where Message has "Application started"
| summarize starts = count() by AppRoleName, bin(TimeGenerated, 1h)
| render columnchart

Use it for: correlating latency spikes with replicas starting. Many starts for api with minReplicas 0 means users are hitting cold starts. Unexpected starts for worker (min 1) mean crashes or failed liveness probes.

11. SLI: share of good valuation requests (for the SLO worksheet)

AppRequests
| where TimeGenerated > ago(28d) and AppRoleName == "api"
| where Name has "valuation"
| summarize total = count(), good = countif(Success == true and DurationMs < 500)
| extend sliPct = round(100.0 * good / total, 3)

Use it for: the monthly SLO report. The 500 ms threshold is illustrative; use the one from your SLO worksheet.

Turning a query into an alert

Step What to decide
1 Is it a symptom users feel (failure rate, latency) or a cause (CPU)? Alert on symptoms.
2 Window and frequency: a 5-minute window evaluated every 5 minutes is a sane start.
3 Threshold: start from the SLO, not from a gut feeling.
4 Action group: who gets paged, who only gets an email.
5 Runbook: put the link in the alert description (see the runbook template).

💰 Log ingestion is often the surprise cost of observability, and alert rules have their own charges (check the Azure Monitor pricing page). Prefer a few well-chosen alerts over many noisy ones, and set sampling and a daily cap on purpose.

Postmortem template

A postmortem is blameless: it asks how the system and the process allowed the failure, not who made the mistake. People who fear blame hide information, and then the same incident happens again.

Copy everything below the line into docs/postmortems/YYYY-MM-DD-short-title.md. Write it within 5 working days of the incident while memories are fresh. Delete the hints in italics.


Postmortem: short title

Date of incident
Authors
Status Draft / In review / Final
Severity
Duration from first impact to full recovery
Error budget spent e.g. 40% of the monthly budget for the valuation SLO

Summary

Three sentences: what happened, what users experienced, how it was fixed.

Impact

Users affected number or share
What they saw errors, slowness, wrong data, nothing
Data lost or wrong yes/no, details
Money or contractual impact

Timeline

All times in UTC. Include detection, decisions and dead ends, not only the fix.

Time (UTC) What happened
First impact (found afterwards in the data)
Alert fired / customer reported
On-call acknowledged
Hypothesis 1 tried, ruled out because...
Mitigation applied
Full recovery confirmed

Detection

Root causes and contributing factors

Ask "why" until you reach something you can change in the system or process. There is usually more than one cause.

  1. Trigger: the change or event that started it.
  2. Why it caused impact: missing timeout, no idempotency, alert not actionable...
  3. Why it took long to fix: runbook missing, rollback unclear, access problems...

What went well

What went badly

Where we got lucky

Action items

Each item has an owner, a due date and an issue link. Prefer items that prevent a whole class of failures over items that fix only this one.

Action Type Owner Due Issue
Prevent / Detect / Mitigate / Process

Lessons for other teams

One or two sentences worth sharing outside the team.

PR review checklist

Use it for every pull request, including ones written with AI help. The bar is the same. CI already checks formatting, warnings, tests and known vulnerabilities, so spend your attention on what tools can't see.

Before you read the code

Correctness

Failure

Data

Security

Observability

Tests

AI-assisted changes

Review etiquette

Production readiness checklist

Go through this before a service takes real traffic, and again once a quarter. Every "no" is either fixed or written down as an accepted risk, with an owner.

Copy this into an issue (one checkbox per line) and work through it with the team. It takes about an hour for a small service.

1. Build and supply chain

2. Tests

3. Deploy and rollback

4. Identity and secrets

5. Reliability

6. Observability

7. Data

8. Cost

9. People

What and why

How to verify

Risk

Observability

Checklist

Runbook template

One runbook per alert. Link it from the alert description so that the person woken at 2 a.m. lands here in one click. Write it for someone who is tired and has never seen this service.

Copy everything below the line into docs/runbooks/<alert-name>.md. Delete the hints in italics.


Runbook: alert name, exactly as it appears in the alert

Service e.g. Ledgerly worker
Owner team and channel
Severity Sev 1 (page now) / Sev 2 (page in working hours) / Sev 3 (ticket)
Last reviewed date, by whom

What this alert means

One or two sentences in plain language. What is the user impact? Example: "Imports are not being processed. Users who upload a CSV see 'pending' forever. Nothing is lost: messages wait in the queue."

Is it real? (first 5 minutes)

  1. Open this dashboard: link.
  2. Run this query: paste the KQL here.
  3. Check the dependency status page: link.
  4. If X is normal, it's a false alarm: resolve the alert and note it in the alert review.

Likely causes

Signal Likely cause Go to
Dependency failures to marketdata above 50% Vendor outage Fix A
Worker replicas restarting Crash loop after deploy Fix B
Messages in the dead-letter queue Poison message Fix C

Fixes

Fix A: dependency outage

  1. Turn off the feature with the kill switch: exact flag name and where to change it.
  2. Confirm users see the degraded experience, not errors.
  3. Post in the status channel using the template below.

Fix B: bad deploy

  1. Find the last good image tag: where to look.
  2. Roll back: exact command or workflow to run.
  3. Confirm the smoke test passes.

Fix C: poison message

  1. How to read the dead-letter queue: tool and permissions.
  2. How to decide: fix data and replay, or discard.
  3. How to replay: exact steps.

Escalation

When Who How
Not mitigated in 30 minutes Tech lead Phone
Data loss suspected Security and product Incident channel

Status message template

We are seeing [symptom] since [time]. [What users experience]. We are [action]. Next update in 30 minutes.

After the incident

SLO worksheet

Fill this in with your product manager and your SRE, not alone. The goal is a few numbers everybody agrees on: what "healthy" means, and when reliability work beats feature work.

The Ledgerly values below are illustrative examples. Replace them with your own.

Step 1: List the user journeys that matter

Pick 3 to 5. If a journey breaks and nobody notices for a day, it doesn't belong here.

# Journey Who notices when it breaks Ledgerly example
1 See my portfolio valuation
2 Record a trade
3 Import a broker CSV
4 Get a price alert

Step 2: Choose an SLI for each journey

An SLI (service level indicator) is a ratio: good events / valid events, measured where the user would notice.

Journey SLI type Good event Valid event Where it's measured Ledgerly example
Availability Valuation requests that return non-5xx
Latency Valuation requests faster than 500 ms
Freshness Imports finished within 5 minutes of upload
Correctness Valuations with isComplete = true

Rules of thumb:

Step 3: Set the SLO

An SLO (service level objective) is a target for the SLI over a window. Start with what you achieve today, minus a little margin, not with "five nines".

SLI SLO target Window Ledgerly example
28 days 99.5% of valuation requests succeed
28 days 95% of valuation requests under 500 ms
28 days 99% of imports finish within 5 minutes

Step 4: Calculate the error budget

Error budget = 1 − SLO. It is the amount of unreliability you are allowed to spend on deploys, experiments and bad luck.

SLO Error budget Over 28 days, that is
99% 1% about 6.7 hours of total failure, or 1 in 100 requests
99.5% 0.5% about 3.4 hours, or 1 in 200 requests
99.9% 0.1% about 40 minutes, or 1 in 1,000 requests

Your numbers:

SLO Valid events per 28 days (estimate) Allowed bad events

Step 5: Agree what happens when the budget runs out

Write this down before you need it. Example policy:

Budget left What the team does
More than 50% Ship normally. Experiments are fine.
25% to 50% Review the top error sources in the weekly meeting.
Less than 25% Reliability work goes first in the next sprint. Risky deploys need a second reviewer.
Exhausted Feature freeze for the service until the budget recovers, except fixes.

Signed off by (product, engineering, SRE): ____________________

Step 6: Alert on burn rate, not on single errors

Burn rate = how fast you are spending the budget compared to "evenly over the window". A burn rate of 1 uses exactly the whole budget in 28 days.

Alert Burn rate Long window (short window) Action
Fast burn 14.4 1 hour (5 minutes) Page on-call
Medium burn 6 6 hours (30 minutes) Page on-call
Slow burn 1 3 days (6 hours) Ticket for the next working day

These values come from the multiwindow, multi-burn-rate example in the Google SRE Workbook, written for a 30-day window. The alert fires only when both the long and the short window exceed the rate, so it stops quickly after recovery. Tune them for your traffic: low-traffic services need longer windows, or a single failed request will page someone.

Step 7: Wire it up

Task Done
SLI query written and tested (see the KQL queries file, query 11)
Workbook tile showing SLI vs SLO for the window
Fast-burn alert with action group and runbook link
Slow-burn alert as a ticket or email
Monthly SLO review in the team calendar

Review questions (every month)

  1. Did we meet the SLO? If we beat it by a lot, is it too loose?
  2. What spent the budget: deploys, a dependency, load, or bugs?
  3. Did any outage happen that the SLI didn't see? Then the SLI is wrong.
  4. Did any alert fire that nobody needed to act on? Then the alert is wrong.

Glossary

Terms used in the course, in plain language, with the chapter that explains them. 🔧 marks names that change often.

Term Meaning Chapter
ACA Azure Container Apps: a serverless platform that runs containers with scale rules, revisions and jobs, without you running a Kubernetes cluster. 10
ACR Azure Container Registry: a private registry for your container images. 11, 12
Activity / ActivitySource .NET's API for OpenTelemetry spans. An ActivitySource creates Activity objects; each one is a span in a trace. 15
ADR Architecture decision record: a short document capturing one significant decision, its context and consequences. 1, 19
AKS Azure Kubernetes Service: managed Kubernetes. Full control, more to operate. 10
Analyzer A Roslyn component that checks code during the build and reports diagnostics, such as IDE0005 or CA2007. 3
AppHost The Aspire project that describes the whole system in C# and starts it locally. 6
Application Insights The Azure Monitor feature for application telemetry: requests, dependencies, traces, exceptions and metrics. Stores data in a Log Analytics workspace. 16
Aspire 🔧 Microsoft's stack for building and running distributed apps: the AppHost, service defaults, integrations and the dashboard. Formerly ".NET Aspire". 6
Aspire dashboard A local web UI that shows resources, console logs, structured logs, traces and metrics from everything the AppHost runs. 6, 15
Availability (SLI) The share of valid requests that succeed. 16
Azure Managed Redis 🔧 The current managed Redis offering in Azure. Use it for new work instead of Azure Cache for Redis, which is being retired. 10
azd The Azure Developer CLI: provisions and deploys an app from a template with a few commands. 11
Bicep A declarative language for Azure resources that compiles to ARM templates. 11
Blameless postmortem An incident review that looks for system and process causes, not people to blame. 19
Build once, deploy many Build one immutable image per commit and promote the same image through every environment. 12
Burn rate How fast an error budget is being spent, relative to spending it evenly over the SLO window. 16
Central package management (CPM) One Directory.Packages.props file holds the versions of every NuGet package in the repo. 3
Circuit breaker A resilience strategy that stops calling a failing dependency for a while so it can recover and your threads aren't wasted. 14
CODEOWNERS A GitHub file that maps paths to the people or teams who must review changes to them. 8
CodeQL GitHub's static analysis engine for finding security bugs. 9
Cold start The delay when a new replica starts to serve a request, for example after scaling from zero. 18
Container Apps environment The boundary around a group of container apps: shared network, logging and scale. 10, 11
Container Apps job A container that runs to completion, started manually, on a schedule or by an event. 7, 12
Correlation / trace context IDs (W3C traceparent) passed between services so their telemetry joins into one trace. 15
CORS Cross-origin resource sharing: browser rules that decide which other origins (like a React SPA) may call your API. 4, 13
Dead-letter queue (DLQ) A sub-queue where Service Bus moves messages that can't be processed, for inspection and replay. 7, 19
DefaultAzureCredential An Azure.Identity credential that tries several sources in order (environment, managed identity, developer tools). Convenient locally; prefer ManagedIdentityCredential in production. 13
Dependabot GitHub's bot that opens pull requests to update dependencies and GitHub Actions. 8, 9
Duplicate detection A Service Bus feature that drops messages with a MessageId already seen within a time window. 7
EF Core Entity Framework Core, the modern .NET ORM. Not the same codebase as EF6. 5
Entra ID Microsoft Entra ID, formerly Azure Active Directory: Microsoft's identity platform for users and workloads. 13
Environment (GitHub) A named deploy target in GitHub with its own protection rules, variables and secrets. 12
Error budget 1 minus the SLO: how much unreliability is allowed in the window. 16
Execution strategy EF Core's retry policy for transient database errors. 5, 14
Expand and contract A migration approach where you add new schema first, deploy code that uses it, then remove the old schema later. Keeps deploys zero-downtime. 12
Feature flag A runtime switch that turns behavior on or off without a deploy. Used as a kill switch during incidents. 4, 19
Federated credential A trust in Entra ID that accepts tokens from an external issuer (like GitHub Actions) for a specific subject, instead of a secret. 12
GitHub Actions GitHub's CI/CD system: YAML workflows in .github/workflows. 9
GitHub flow Branch from main, open a pull request, review, merge, deploy. 8
Graceful degradation Returning a reduced but useful answer when a dependency fails, instead of an error. 14
Graceful shutdown Finishing in-flight work when the platform sends a stop signal, within the grace period. 7, 14
Health probe A platform check of your app. Startup: has it started? Liveness: is the process alive (restart if not)? Readiness: can it take traffic? 14
HybridCache A .NET caching API that combines an in-memory cache with an optional distributed cache and protects against cache stampedes. 18
Idempotency Doing the same operation twice has the same effect as doing it once. Required for message consumers. 7, 14
IHttpClientFactory The .NET factory that manages HttpClient lifetimes and lets you add handlers like resilience. 2, 14
KEDA Kubernetes Event-driven Autoscaling. Container Apps uses KEDA scalers, for example to scale on Service Bus queue length. 18
Kestrel The cross-platform web server built into ASP.NET Core. Replaces IIS as the process that serves requests. 2
Key Vault Azure's service for secrets, keys and certificates. 13
KQL Kusto Query Language, used to query logs in Azure Monitor and Log Analytics. 16
Least privilege Every identity gets only the permissions it needs, on the smallest scope. 11, 13
Log Analytics workspace The Azure Monitor store for logs and Application Insights data, queried with KQL. 16
Managed identity An identity for an Azure resource, managed by Entra ID. Your code gets tokens without any secret. User-assigned identities are separate resources you can share and grant roles before the app exists. 13
Message template A log message with named placeholders ("Import {ImportId} done") whose values are stored as structured properties. 15
Meter .NET's API for OpenTelemetry metrics: counters, histograms and gauges. 15
Microsoft.Testing.Platform (MTP) The modern .NET test runner that replaces VSTest. Opt in through global.json on the .NET 10 SDK. 17
Minimal APIs ASP.NET Core's lightweight way to define HTTP endpoints with lambdas or static methods instead of controllers. 4
Migration (EF) Generated code that changes the database schema from one model version to the next. 5
OIDC OpenID Connect. In CI/CD, GitHub issues a short-lived token that Azure trusts through a federated credential, so no cloud secret is stored. 12
OpenAPI A standard machine-readable description of an HTTP API. ASP.NET Core 10 generates OpenAPI 3.1 by default. 4
OpenTelemetry (OTel) The vendor-neutral standard and SDKs for traces, metrics and logs. 15
OTLP The OpenTelemetry protocol used to send telemetry to a collector or backend, such as the Aspire dashboard. 15
Optimistic concurrency Detecting conflicting updates with a version column (rowversion) and rejecting the second writer (409) instead of locking. 5
Outbox (transactional) Writing the message to publish into the same database transaction as the business change, then relaying it to the broker. No lost or phantom messages. 7
Peek-lock The Service Bus receive mode where a message is locked, not removed, until the consumer completes, abandons or dead-letters it. 7
ProblemDetails The standard JSON shape for HTTP API errors (RFC 9457). 4
Rate limiting Limiting how many requests a client can make in a window. Should return 429 Too Many Requests. 4, 14
RBAC Role-based access control. In Azure, role assignments grant an identity a role on a scope. 11, 13
Replica One running instance of a container app. 10, 18
Revision An immutable snapshot of a container app version in Azure Container Apps. 12
Ruleset GitHub's newer, layered way to protect branches and tags, which can apply across repositories. 8
Run-to-completion A process that does one job and exits, like the migration service. 5, 12
Sampling Keeping only a share of traces to control volume and cost. 15, 16
Scale rule A Container Apps rule that adds or removes replicas based on HTTP concurrency, queue length or other signals. 18
Scalar An interactive UI for browsing and calling an OpenAPI document. 4
Service Bus Azure's enterprise message broker: queues and topics with sessions, dead-lettering and duplicate detection. 7
Service defaults A shared Aspire project that wires OpenTelemetry, health checks, resilience and service discovery into every service with one call. 6
Service discovery Resolving a logical name like https+http://marketdata to a real address from configuration. 6
SLI Service level indicator: a measured ratio of good events to valid events. 16
SLO Service level objective: a target for an SLI over a time window. 16
Span One timed operation in a trace, with attributes and a parent. 15
Standard resilience handler AddStandardResilienceHandler(): a ready-made pipeline of rate limiter, timeouts, retries and circuit breaker for HttpClient. 14
Structured logging Logs as events with named properties, not just text lines, so you can query them. 15
Testcontainers A library that starts real dependencies (like SQL Server) in throwaway containers for tests. 17
TimeProvider The .NET abstraction for the current time and timers. Fake it in tests with FakeTimeProvider. 2, 17
Trace The tree of spans for one request as it flows through every service. 15
Transitive pinning A CPM setting that lets you force the version of a package you don't reference directly, for example to fix a vulnerability. 3
TypedResults ASP.NET Core helpers that return strongly typed HTTP results, which also feed the OpenAPI document. 4
What-if An Azure deployment preview that shows which resources would change, before you deploy. 11
Workload profile The compute type of a Container Apps environment, such as Consumption. 10
xUnit v3 The current major version of the xUnit test framework, running on Microsoft.Testing.Platform. 17
↑ Start