Hey everyone, its been a while since I really posted an update to the site. I've been tied up with other projects so I just haven't had the time to write new code for C8Tracker. My focus has mainly just been on keeping it running.
Over the last week or two I have been working on rewriting the code that downloads and processes all of the window stickers. I am just about done with the work. Everything looks like it is working as it should. The big difference is a drastic performance improvement.
The old scripts were a long and slow process. The script that pulls the data to populate the Daily Dashboard would take about 15-20 minutes to do its thing. The script that handles the big updates every day (downloading all of the new window stickers and adding them to the database) generally took about 2-3 hours to do its thing. It was a long painful process.
The new scripts are running about 90% faster. The new Daily Dashboard script takes about 2 minutes to complete. The big update script took about 8 minutes when I tested it earlier.
My goal here was a bit selfish... I wanted C8Tracker to the the first place on the internet to have the latest data. The hope was that this would simply increase the usage of the site haha. But either way, it was a fun exercise and I learned quite a bit.
A secondary side effect is the new code is better at processing the Canadian window stickers. Right now it is able to pull all of the options data. The only part I can't seem to grab is the MSRP on the Canadian window stickers. I'll work on that some more. But once this code is done and live I plan to write a script to populate the database with all of the missing Canadian option data.
Testing of the new code is going well. Hopefully starting next week it will be live on the site. I'll make a post when that happens. I just wanted to let everyone know what is on the way.