SQL Azure Database performance oddness

For a few years I have worked on a midsize Azure application for managing promotional vouchers which are delivered via SMS or email and redeemed via software integrated into the point of sale terminal software of several local vendors.

A promotion has a batch of vouchers which are available for allocation to consumers and I had noticed from the logs and performance counters that the process of getting the next voucher slowed down significantly as the number of available vouchers decreased.

Voucher batches range in size from 100’s to 100,000’s of vouchers and the performance of a small promotion for a local wine shop was where I initially noticed how much the duration increased. The promotion had roughly 850 vouchers the allocation of the first vouchers took 10’s of mSec each but the last 10 vouchers sometimes took more than 1000mSec each.

I took a copy of the live database and removed the customer data so I could explore the performance of my TSQL in a controlled environment. I initially downloaded a copy of the database to one of my development servers and tried to simulate the problem while monitoring performance using SQL Profiler and other tools but the allocation time was fast and consistent.

The database performance appeared to be a SQL Azure specific issue so I built a cut back web role test harness which called the underlying stored procedure so I could closely monitor the performance. The test harness could make a specified number of calls recording the duration of each call and the overall duration. I then de allocated all the vouchers in the wine shop promotion and allocated them in chunks(all durations are in mSec and are the average time it takes to make a single call)

100 vouchers at a time (0 – 800 of 843 vouchers)

43, 88, 136, 191, 260, 305, 358, 379

10 vouchers at a time (800-840 of 843 vouchers)

431, 440, 404, 412

1 voucher at a time (840-843 of 843 vouchers, last one is failure)

400, 423,404, 390

After some debugging, progressive removal of code and looking at query plans I identified the problematic TSQL statement.

UPDATE TOP(1) Voucher SET
Voucher.AllocatedByActivity = @ActivityUID,
Voucher.AllocatedAtUTC = @RequestDateTimeUTC,
Voucher.AllocationReference = @ClientReference,
@VoucherCode = Voucher.Code
WHERE (( Voucher.VoucherBatchUID = @VoucherBatchUID ) AND ( Voucher.AllocatedByActivity IS NULL ))

This update statement uses a single statement transaction to get the next random un-allocated voucher code in the specified voucher batch.

After some conversations with a SQL Azure support engineer at Microsoft (who was very helpful) he figured out that in SQL Azure the query processor needed a query hint to tell it to use an existing index to make it perform consistently.

UPDATE TOP(1) Voucher SET
Voucher.AllocatedByActivity = @ActivityUID,
Voucher.AllocatedAtUTC = @RequestDateTimeUTC,
Voucher.AllocationReference = @ClientReference,
@VoucherCode = Voucher.Code
FROM Voucher WITH (INDEX(IX_VoucherByVoucherBatchUIDAllocatedByActivity))
WHERE (( Voucher.VoucherBatchUID = @VoucherBatchUID ) AND ( Voucher.AllocatedByActivity IS NULL ))

100 vouchers at a time (0 – 800 of 843 vouchers)

20, 8, 13, 9, 32, 7, 15, 9

10 vouchers at a time (800-840 of 843 vouchers)

17, 9, 14, 13

1 voucher at a time (840-843 of 843 vouchers, last one is failure)

12,11,13, 5

I always have plenty of performance counters and logging (using the enterprise library) on my Azure web and worker roles but I was also lucky that I noticed something odd in the logs while checking on the progress of another promotion. I actively monitor the performance of my Azure applications as over time the performance characteristics of the underlying hardware will change as fixes and enhancements are released.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s