It's simple! Use rules you used for conditional formatting as filter conditions for average calculations.

The easies wayt will be to use (some) hidden column(s), where some numeric values are calculated depending on one specific conditional formatting rule, or on set of conditional formatting rules. And then you can use hidden column(s) to define conditional formatting rules too - like when value in hidden column is x1, then rule 1 is applied.