Skip to content

Scheduled Jobs in .NET with Hosted Services and SQL Leases

 · 13 min read

I started my .NET journey three or four years ago at my current company. Before that I had been mostly away from Microsoft’s ecosystem (except for some academic experience 10 years ago), so a lot of what the framework gives you out of the box was new to me. One of the things I ended up using everywhere is the hosted service: a piece of code that lives inside your application, starts with it, stops with it and runs in the background. Sooner or later every backend needs something that runs every few hours: cleaning up expired tokens, syncing data from a third party, sending a daily report. This post is about how I do that with hosted services, and how a single small SQL table makes it safe to run on more than one instance.

Hosted services in two sentences

An IHostedService is anything the generic host starts when the application starts and stops when it shuts down. BackgroundService is the base class most people use, it gives you a single ExecuteAsync method with a cancellation token that fires on shutdown.

The naive version of “run every six hours” looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

public sealed class CleanupService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await DoCleanupAsync(stoppingToken);
            await Task.Delay(TimeSpan.FromHours(6), stoppingToken);
        }
    }
}

It works on your machine and it has two problems in production. First, the schedule lives in memory: every deploy or restart runs the job immediately and resets the six hour countdown, so if you deploy three times a day, the job runs three extra times. Second, every replica runs its own copy. Scale the API to four pods and the cleanup runs four times every six hours, at slightly different moments, possibly on the same rows.

Both problems have the same fix: move the schedule out of the process and into the database, and let the database decide who runs the job.

The lease table

One row per job. It stores who currently holds the job, until when, and when it last completed successfully:

1
2
3
4
5
6
7

CREATE TABLE dbo.JobLeases
(
    JobName         NVARCHAR(100) NOT NULL PRIMARY KEY,
    LeaseOwner      NVARCHAR(200) NULL,
    LeasedUntil     DATETIME2     NULL,
    LastCompletedAt DATETIME2     NULL
);

A lease is a lock with an expiry date. Whoever holds it is allowed to run the job, and if the holder dies without giving it back, the lease simply runs out and somebody else can take it. This matters more than it seems: a pod killed by the OOM killer or a node that disappears never gets to run its finally block, and a plain lock would stay taken forever.

Acquiring the lease is a single conditional UPDATE:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

MERGE dbo.JobLeases WITH (HOLDLOCK) AS t
USING (SELECT @JobName AS JobName) AS s ON t.JobName = s.JobName
WHEN NOT MATCHED THEN INSERT (JobName) VALUES (s.JobName);

UPDATE dbo.JobLeases
SET LeaseOwner  = @Owner,
    LeasedUntil = DATEADD(SECOND, @LeaseSeconds, SYSUTCDATETIME())
WHERE JobName = @JobName
  AND (LeasedUntil IS NULL OR LeasedUntil <= SYSUTCDATETIME())
  AND (LastCompletedAt IS NULL
       OR LastCompletedAt <= DATEADD(SECOND, -@IntervalSeconds, SYSUTCDATETIME()));

SELECT @@ROWCOUNT;

The MERGE makes sure the row exists, so a new job doesn’t need a migration of its own. HOLDLOCK is what makes it safe when two instances start at the same time and both try to insert the same row. The UPDATE then only matches if nobody holds a valid lease and the job is actually due. If it returns 1, we own the job. If it returns 0, either somebody else is running it or it already ran recently enough, and in both cases there is nothing to do.

Completing and releasing are just as small:

1
2
3
4
5
6
7
8
9

-- completed
UPDATE dbo.JobLeases
SET LeaseOwner = NULL, LeasedUntil = NULL, LastCompletedAt = SYSUTCDATETIME()
WHERE JobName = @JobName AND LeaseOwner = @Owner;

-- released without completing (failure or shutdown)
UPDATE dbo.JobLeases
SET LeaseOwner = NULL, LeasedUntil = DATEADD(SECOND, @RetrySeconds, SYSUTCDATETIME())
WHERE JobName = @JobName AND LeaseOwner = @Owner;

The release keeps LeasedUntil in the future on purpose. When a job fails, I don’t want every instance to retry it again a minute later, so the release blocks the job for a retry delay. On shutdown the delay is zero and the next instance can pick it up right away. The LeaseOwner = @Owner condition makes sure an instance can only complete or release a lease it still holds.

I use SQL Server at work, so that is what the examples use. The same idea works on PostgreSQL with INSERT ... ON CONFLICT DO NOTHING instead of the MERGE and now() instead of SYSUTCDATETIME().

The C# side is a thin wrapper, here with Dapper:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

public interface IJobLeaseStore
{
    Task<bool> TryAcquireAsync(string job, string owner, TimeSpan interval, TimeSpan lease, CancellationToken ct);
    Task CompleteAsync(string job, string owner, CancellationToken ct);
    Task ReleaseAsync(string job, string owner, TimeSpan retryAfter, CancellationToken ct);
}

public sealed class SqlJobLeaseStore(string connectionString) : IJobLeaseStore
{
    public async Task<bool> TryAcquireAsync(string job, string owner, TimeSpan interval, TimeSpan lease, CancellationToken ct)
    {
        await using var connection = new SqlConnection(connectionString);
        var acquired = await connection.ExecuteScalarAsync<int>(new CommandDefinition(
            Sql.Acquire,
            new
            {
                JobName = job,
                Owner = owner,
                LeaseSeconds = (int)lease.TotalSeconds,
                IntervalSeconds = (int)interval.TotalSeconds,
            },
            cancellationToken: ct));

        return acquired == 1;
    }

    public async Task CompleteAsync(string job, string owner, CancellationToken ct)
    {
        await using var connection = new SqlConnection(connectionString);
        await connection.ExecuteAsync(new CommandDefinition(
            Sql.Complete, new { JobName = job, Owner = owner }, cancellationToken: ct));
    }

    public async Task ReleaseAsync(string job, string owner, TimeSpan retryAfter, CancellationToken ct)
    {
        await using var connection = new SqlConnection(connectionString);
        await connection.ExecuteAsync(new CommandDefinition(
            Sql.Release,
            new { JobName = job, Owner = owner, RetrySeconds = (int)retryAfter.TotalSeconds },
            cancellationToken: ct));
    }
}

Sql.Acquire, Sql.Complete and Sql.Release are constants holding the statements above.

The base class

Every job I write has the same shape: check every minute, try to take the lease, run if we got it, record the result. That belongs in a base class, so the jobs themselves only say what they are called, how often they run and what they do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

public abstract class ScheduledJob(IServiceScopeFactory scopeFactory, IJobLeaseStore leases, ILogger logger)
    : BackgroundService
{
    private readonly string _owner = $"{Environment.MachineName}:{Guid.NewGuid():N}";

    protected ILogger Logger { get; } = logger;

    protected abstract string Name { get; }
    protected abstract TimeSpan Interval { get; }
    protected virtual TimeSpan LeaseDuration => TimeSpan.FromMinutes(30);
    protected virtual TimeSpan RetryDelay => TimeSpan.FromMinutes(5);
    protected virtual TimeSpan PollInterval => TimeSpan.FromMinutes(1);

    protected abstract Task RunAsync(IServiceProvider services, CancellationToken ct);

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(PollInterval);
        do
        {
            try
            {
                await TryRunAsync(stoppingToken);
            }
            catch (Exception ex) when (!stoppingToken.IsCancellationRequested)
            {
                Logger.LogError(ex, "Scheduler for {Job} failed", Name);
            }
        }
        while (await timer.WaitForNextTickAsync(stoppingToken));
    }

    private async Task TryRunAsync(CancellationToken ct)
    {
        if (!await leases.TryAcquireAsync(Name, _owner, Interval, LeaseDuration, ct))
            return;

        try
        {
            Logger.LogInformation("Running {Job}", Name);
            await using var scope = scopeFactory.CreateAsyncScope();
            await RunAsync(scope.ServiceProvider, ct);
            await leases.CompleteAsync(Name, _owner, CancellationToken.None);
        }
        catch (OperationCanceledException) when (ct.IsCancellationRequested)
        {
            await leases.ReleaseAsync(Name, _owner, TimeSpan.Zero, CancellationToken.None);
            throw;
        }
        catch (Exception ex)
        {
            Logger.LogError(ex, "{Job} failed, retrying in {Delay}", Name, RetryDelay);
            await leases.ReleaseAsync(Name, _owner, RetryDelay, CancellationToken.None);
        }
    }
}

A few details are worth pointing out. Hosted services are singletons, while a DbContext or most of your application services are scoped, so each run gets its own scope and resolves what it needs from there. PeriodicTimer is the cleanest way to tick in modern .NET, it doesn’t overlap ticks and it honours the cancellation token. The owner is the machine name plus a random suffix; in Kubernetes the machine name is the pod name, which makes the table readable when you look at it during an incident. And the outer try keeps the loop alive when the database is briefly unreachable, the job will just try again on the next tick.

Note that the poll interval and the job interval are two different things. The job runs every six hours, but every instance asks every minute whether it is time. That one minute is the precision of the scheduler, and the cost is one tiny query per job per instance per minute.

Implementing jobs

With the base class in place, a job is a name, an interval and a method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

public sealed class ExpiredTokensCleanupJob(
    IServiceScopeFactory scopeFactory,
    IJobLeaseStore leases,
    ILogger<ExpiredTokensCleanupJob> logger)
    : ScheduledJob(scopeFactory, leases, logger)
{
    protected override string Name => "expired-tokens-cleanup";
    protected override TimeSpan Interval => TimeSpan.FromHours(6);

    protected override async Task RunAsync(IServiceProvider services, CancellationToken ct)
    {
        var db = services.GetRequiredService<AppDbContext>();
        var deleted = await db.RefreshTokens
            .Where(t => t.ExpiresAt < DateTime.UtcNow)
            .ExecuteDeleteAsync(ct);

        Logger.LogInformation("Deleted {Count} expired refresh tokens", deleted);
    }
}

A job that takes longer overrides the lease, so it doesn’t expire while the work is still going on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

public sealed class DailyInvoiceReportJob(
    IServiceScopeFactory scopeFactory,
    IJobLeaseStore leases,
    ILogger<DailyInvoiceReportJob> logger)
    : ScheduledJob(scopeFactory, leases, logger)
{
    protected override string Name => "daily-invoice-report";
    protected override TimeSpan Interval => TimeSpan.FromHours(24);
    protected override TimeSpan LeaseDuration => TimeSpan.FromHours(2);

    protected override Task RunAsync(IServiceProvider services, CancellationToken ct) =>
        services.GetRequiredService<IInvoiceReportService>().SendDailyReportAsync(ct);
}

The job itself doesn’t contain business logic. It calls a service from the application layer, the same one an endpoint or a test would call. That is also why the next part matters.

A project of their own

I usually keep all hosted services in their own project, something like MyApp.Jobs, next to the usual suspects:

1
2
3
4
5

MyApp.Domain
MyApp.Application
MyApp.Infrastructure
MyApp.Jobs          -> references Application and Infrastructure
MyApp.Api           -> references Jobs

The jobs project can reference everything it needs from the solution and consume it through DI: application services, the DbContext, HTTP clients, options. Nothing in the domain or application layers needs to know that something runs on a schedule. The project is a plain class library, it only needs the hosting abstractions:

1
2
3

<ItemGroup>
  <PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.0" />
</ItemGroup>

It exposes a single extension method that registers the lease store and every job:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

public static class JobsServiceCollectionExtensions
{
    public static IServiceCollection AddScheduledJobs(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddSingleton<IJobLeaseStore>(
            new SqlJobLeaseStore(configuration.GetConnectionString("Default")!));

        services.AddHostedService<ExpiredTokensCleanupJob>();
        services.AddHostedService<DailyInvoiceReportJob>();

        return services;
    }
}

And the host only has to call it:

1

builder.Services.AddScheduledJobs(builder.Configuration);

Keeping them separate also leaves the door open. Today the jobs run inside the API. If one day they get heavy enough to compete with request traffic, I can create a small worker host with Host.CreateApplicationBuilder, call the same AddScheduledJobs there, deploy it as its own Kubernetes deployment, and remove the line from the API. No job changes.

Why this works with many instances

The database is the only thing that decides. Every instance runs the same loop and asks the same question, but the conditional UPDATE can only succeed for one of them: SQL Server takes an update lock on the row, the first statement wins and changes LeasedUntil, and every statement after it no longer matches the WHERE clause. There is no leader election, no instance is special, and adding replicas doesn’t change how often the job runs.

All the time comparisons use SYSUTCDATETIME(), the clock of the database server, never the clock of the pod. Two machines are never perfectly in sync, and with the database as the single clock, clock skew between instances simply doesn’t matter.

The schedule survives restarts. LastCompletedAt is stored once, for everyone, so a deploy doesn’t reset anything: the new pods come up, ask whether the job is due, and the answer is the same one the old pods would have gotten.

Crashes are covered by the expiry. If the instance running the job dies, nobody releases the lease, but after LeaseDuration it no longer counts and the next instance that polls takes over. Failures are covered by the retry delay, and graceful shutdowns hand the job over immediately.

There is one honest caveat. A lease is not a perfect mutex: if a job runs longer than its lease, a second instance can take over while the first is still working. Pick a lease comfortably larger than the worst run you have seen, or renew it from a background loop while the job runs (UPDATE ... SET LeasedUntil = ... WHERE JobName = @JobName AND LeaseOwner = @Owner every few minutes). And write jobs so that running them twice does no harm. Deleting expired rows is naturally safe; sending an email should check whether it was already sent. That is good advice for any scheduler, as you’ll see below.

Why not cron or a Kubernetes CronJob

A crontab on a VM is the oldest answer, and it doesn’t fit a distributed setup. Put it on one machine and that machine becomes a single point of failure for every scheduled task. Put it on every machine and the job runs once per machine. It also runs outside the application, so it needs its own way to load configuration and secrets, or it ends up calling an internal endpoint over HTTP, which is the same code with an extra network hop and an endpoint to protect.

A Kubernetes CronJob is better, and for some cases it is the right tool. It still comes with costs that a hosted service doesn’t have. Every run starts a new pod: image pull, container start and the whole .NET startup, for a job that might take two seconds. It needs its own entrypoint in your image, its own manifest, and the same config maps and secrets wired up a second time. Logs and metrics are spread over short lived pods. And it is not exactly once either; the Kubernetes documentation states that a CronJob creates a job approximately once per schedule and that in some circumstances two jobs might be created, or none, so the jobs should be idempotent anyway. The same rule from the previous section, without the convenience.

A hosted service runs in a process that is already warm. It shares the DI container, configuration, logging and telemetry with the rest of the application, it is tested and deployed with it, and it scales with it without running more often. The tradeoff is that it shares CPU and memory with your requests, which is exactly why the separate project and the option of a dedicated worker host are there. A long, heavy batch that should run in isolation, with its own resource limits, is where I would still reach for a CronJob.

Alternatives

Writing this yourself makes sense when your needs look like “every x hours, once across the cluster”. When they grow, there are good libraries:

  • Quartz.NET, the full scheduler, with clustering support. So far I’ve only used it for cron jobs whose schedule is configured by our customers.
  • Hangfire, background and recurring jobs with persistent storage and a dashboard.
  • Coravel, a lightweight fluent scheduler.
  • DistributedLock is not a scheduler but a set of distributed lock implementations, including one on top of SQL Server’s sp_getapplock. Combined with a hosted service, it replaces the locking half of this post, though you still need to store the last run somewhere.

Wrapping up

A base class of around sixty lines, a table with four columns and three SQL statements are enough to run jobs every x hours across any number of instances, with the schedule surviving deploys and crashed instances handing over by themselves. Each job is a name, an interval and a call into the application layer, and they all live in one project that can be hosted wherever it makes sense. When I need cron expressions or a dashboard, Quartz.NET or Hangfire are one package away, but for most of what a backend needs to do every few hours, this is all I have needed so far.