He’s getting nervous because modern tech is making the LLMs easier to run at home. MoEs with intelligent routing can be amazing on well equipped gaming machines. On my AMD 7800XT with 64GB RAM (all purchased before the RAMpocalypse) using Strata I can run Qwen 3.8 Flash Next a 90GB 125b parameter model with 256k context size and get between 30 and 40 tokens per second. Strata and probably similarly FreeToken have been absolute game changers.
He’s getting nervous because modern tech is making the LLMs easier to run at home. MoEs with intelligent routing can be amazing on well equipped gaming machines. On my AMD 7800XT with 64GB RAM (all purchased before the RAMpocalypse) using Strata I can run Qwen 3.8 Flash Next a 90GB 125b parameter model with 256k context size and get between 30 and 40 tokens per second. Strata and probably similarly FreeToken have been absolute game changers.
i have a similar build except with a 9070xt, running 64k context window
how well is 256k window working for you?
also which model did you pick? i picked swift 1.5 iq3 xxs