Fix for Site Instability is Live

This is a follow-up from last week’s article which covered an issue we are running into at the end of each NASCAR Cup Series race. This morning I pushed an update which will hopefully resolve the issue moving forward. In this post, I’m going to dive into the specifics with what was occurring, and how we are trying to resolve it.

The Bug 🐛

At the end of each Cup race, the site becomes unresponsive and users are unable to see the league race results and standings. This causes a delay in users being able to see the finishing order of their fantasy race, along with the updating points standings for their league.

You may have noticed this was becoming progressively worse with each week. This past Sunday was the longest delay yet. I’m going to dive into the details below on why that happened. My goal is to be fully open with challenges we run into, how we are trying to fix them, and what we’re learning from them.

The Root Cause

The heart of the issue came down to how league results were being calculated, and how the same computations were used to calculate league standings. League results and standings were never stored in the database. Instead, they were computed on-the-fly at page load.

My initial plan was to heavily cache these league results, to allow them to load quick for users and be calculated for league standings, without having to store them in the database. Turns out this was the wrong approach!

To get league results, we had to:

1. Fetch all league members, along with their rosters, which included driver data, schemes, and more
2. Fetch the driver results
3. Calculate driver points based on league settings, calculate total fantasy points, ensure starts validation, and more

On it’s own, that isn’t too bad. The league race results page only needs to load this data for the current league and race. However, if a league has 20+ members, this becomes a complex calculation for a single request.

To top it off, league standings also relied on this same league results computation, making this exponentially worse. The steps above were repeated for every race. If 11 races have been completed, 11 league results calculations were required to generate the standings for a single league, along with calculating points, wins, and stats.

At the start of the season, this wasn’t a bad issue, as the standings only had to calculate a few races. Now that we are 11 races in, the standings calculation is becoming heavier, which is causing the extended delay each week.

What Happens at the End of the Race

When the race is marked complete, it clears the cache for league standings, to allow users to see the updated standings. If hundreds of users are viewing the site simultaneously at the end of the race, the site attempts to provide fresh data to all active users without the opportunity to cache the data first.

This creates hundreds of league standings computations being done at once. Rightfully so, the database is not able to keep up with the amount of data being fetched, and begins timing out. This is when the league results and standings become available.

The Fix

Instead of computing all previous league results via a complex query, I setup a new table in the database to store all league results, with each member, their points, rosters, and other data from each race. When a race is completed, a background function begins adding the latest league race results into this new table.

For reference, 38,962 rows have been backfilled into this new database table to store all previous league results.

When viewing previous league results, we are now able to fetch this single table to display all results data on the site. Better yet, the league standings now only needs to look to this table to tally up the total points and stats for the season.

For the live fantasy results during the race, these continue to be calculated in real-time to allow the fantasy results to update every minute. You will still be able to watch the live fantasy leaderboard during the race!

My hope is this allows the site to remain fully functional at the end of the race.

A before and after of how league results are calculated

One Downside

Previously, league standings updated on-the-fly during a race. You were able to see the updated standings based on the current running order of the fantasy race.

With this update, since the standings are now reading from the new league results database table, and since the results table is only updated immediately after a race, the standings will only reflect completed races, and not the active race.

Once a race is marked complete, the only delay should be waiting less than a minute for the league results to write to the database, to have the updated league standings become available.

I’m hoping this is a fair tradeoff to allow the site to remain available after each race.

My Honest Expectations for This Weekend

After reviewing the updates and comparing them to the live site, all the results and standings appear to lineup across leagues with default settings, and those with custom scoring. Overall, I feel confident the existing data is correct.

I tested live race updates, and confirmed the fantasy results still update in real-time. Once a race is marked complete, I confirmed the league standings reflect the latest race after the league results are added to the database.

However, at the end of the day, this update completely overhauls the way league results and standings are being handled on the site. I wouldn’t be surprised if a new bug or two appears due to these updates. If this is the case, I will be working to immediately sort out any new issues. The true test is on race day, with live users and data flowing across the site for hours. After the race is complete, the automation will trigger to update the league results. This is a key area I’ll be focusing on this Sunday.

Hell, maybe these updates won’t change anything at all and the site will still crash. If that is the case (which I hope it is not), I will be down, but not out, and will go back to the drawing board to figure this out.

Sneak Preview of Segment Champion Award Badges

If you made it this far, thank you for taking the time to read all of this and having an interest in what I’ve been working on. On a lighter note, here are the award badges for the Season Segment Champions.

Watkins Glen this weekend marks the last race of the Spring segment. The points leader in each league will earn the segment championship award and badge at the end of each segment.

If you compete in the fantasy O’Reilly Auto Parts Series, and won the Spring segment which has already completed, you might be asking: “Where’s my award?”. The truth is, I haven’t implemented the segment champion awards yet on the site. Sorting out the site instability issue has delayed a number of other fun projects on the site. If I don’t have the awards update in place by this weekend, I will be backfilling these awards for all Spring segment champions for each league!

Thank You 🏁

Thanks again for your patience, and being a member with us. Also - shout out to Tyler, who keeps the active drivers, schemes, and schedule up to date each week for all series throughout the season. And John, who helps contribute to features on the site, such as the optimal roster and league-standings-by-week chart.

We’re a small crew, but passionate about the sport, and providing a place to play fantasy NASCAR each week. One day, it would be awesome if we can call this a job. But until then, it remains a fun project I get to share with you all.

- Jason