Mixtral is pretty good at almost a thing I’ve thrown at it. It’s still mostly worse than GPT4, but it’s so much better than any other model I can run locally.
I have a niche question about modelling using some called SHACL that most models except GPT4 got right. Bard, Gemini, Llama all got it wrong. Gemini Ultra gets it right. And Mixtral also gets it right!
One weakness of Mixtral for me is its support for Norwegian. GPT4 is fluent, but Mixtral mixes it up with Danish and is generally poor at performing tasks on Norwegian text. Even summarising Norwegian text is pretty bad. This is obviously just an issue for a few million people in Norway, it’s not that I’m expecting a general model that I can run locally to be good in Norwegian.
Yeah Mixtral is between GPT3.5 and GPT4 in perf. Better than 3.5, but trailing behind 4.
> One weakness of Mixtral for me is its support for Norwegian.
I recently added grammar correction and summarization feature to my app (which uses different system prompts based on the language). And one of the Norwegian speaking users on discord told me the same thing. He also told me that
the smaller Nous-Hermes-2-SOLAR-10.7B seems to do be better at Norwegian than Mixtral does. Perhaps you should try that model.
Tried it out a bit this evening and I must say that I’m astounded. I asked it to summarise some news articles in a list with 5 bullet points and it did an amazing job. I’m sure GPT4 is better, but this is more than good enough and leagues ahead of the other models I’ve tried locally. Thanks again for the tip!
I have a niche question about modelling using some called SHACL that most models except GPT4 got right. Bard, Gemini, Llama all got it wrong. Gemini Ultra gets it right. And Mixtral also gets it right!
One weakness of Mixtral for me is its support for Norwegian. GPT4 is fluent, but Mixtral mixes it up with Danish and is generally poor at performing tasks on Norwegian text. Even summarising Norwegian text is pretty bad. This is obviously just an issue for a few million people in Norway, it’s not that I’m expecting a general model that I can run locally to be good in Norwegian.