In Excel:
1. More complex formula in cell means more time to calculate the result for this cell. This time also depends on type of (sub)formulas used.;
2. Total time needed for calculations in table is sum of calculation times for every formula in table.

This taken into account, there is no way for fast calculations with your current setup and nearly 100000 records in table.

I'd consider creating a SQL Server database , and a Job there, which:
1. E.g. every night reads info from your table and refreshes a copy of it in SQL Server database;
2. After that, the Job runs a stored procedure in SQL Server database, which calculates all needed statistics and stores them in separate table in SQL Server database.

Now, in your Excel table, you create a query which reads the statistics table from SQL Server database, and returns the result as table in another sheet. So long as you are content with results at 'Yesterday', you get them whenever you open the workbook.

When you don't have full SQL Server, you can get SQL Server Express for free. The caveat is, no Jobs in Express - you have to use Windows Task Scheduler to run task which runs stored procedures in SQL Server database to read data, to do calculations, and to update statistics table.