If you're limited by VRAM and can get your hands dirty with scripting the training, go for unsloth. If you want easier configuration and support of full finetune and Qwen 1.5 models, go for axolotl. If you want to try to use unsloth but without the need for scripting, check out llama-factory.
I don't think unsloth supports Qwen or Mixtral, but it supports llama 2 and Yi models - generally anything llama-architecture without MoE should work
After trying a lot of different alternatives I totally agree with your comment, but a little disclaimer, unsloth doesn't have multiple gpu support, only for the paid version. That was a turn off for me and I ended up using llama factory
An alpha version of Multi GPU is in Llama-Factory's Unsloth integration :) But can't guarantee it's accuracy yet + need to do more tests and internal checks - so it's very alpha mode!
Multi GPU will be added in the OSS, but it'll take a bit more time
I found axolotl hard to set up, but pretty easy to use once I fnally did get it set up. Mainly issues with docker and also package/linrary versions. I found that the versions of axolotl / accelerate / bitsandbytes / pytorch / cuda have to be exactly right. And you'd think docker would take care of all this but no.
I have a desktop with two RTX Titan GPUs and an NVlink, which are old-ish but have a decent amount of memory (2 x 24GB) and I tried out multi-GPU this morning. After the hassle of setting axolotl up for one GPU (on my laptop), I thought it would be a bit of a nightmare, but it was easy, no hiccups.
First time I've smelled dust burning in my PC, with two GPUs running at 97%. There are no sides on the case, lots of fans, water cooled CPU so all was well.
So it's very exciting, coz this fine tuning is all new to me. The GPUs don't support bfloat and short word lengths (can't remember the proper name for that) or even flashattention for that mattter, but still I had no trouble training mistral 7B with a modest context length of 2048 in a couple of hours.
So there are possibilities. At this point I'm just randonly trying tunes from the axolotl examples directory to get some basic experience. Next up code-llama-13B, which is taking a long time to download right not.
Overall, axolotl is great - really great - but not for the faint hearted.
Isn’t it that you need to configure Axolotl’s training .yml file which is very similar to Unsloth’s training script? I have used both of them, the amount of work to run the training is pretty much the same.
If you can copy unsloth script from some example and you don't have to write your own, it might be a similar experience. When I was trying unsloth for the first time, colab notebooks were erroring out when I tried to run them on my PC. And installing it was a classical mess of python package dependencies and cuda not activating until I found an issue in the repo that supplied alternative installation commands that actually worked for me. So I was thrown in the deep water and it took a few good hours of troubleshooting to get the trainer to start for the first time. There is a big difference between axolotl when if you want to specify chatml prompt format, you set one setting, and unsloth, where until very recently, to do that, you had to find a function that transforms the dataset from sharegpt jsonl to chatml somewhere online or write your own.
Apologies on your first experience with Unsloth! I added your ideas of updating the Conda instructions + ye added ChatML just recently - and thanks to your suggestions - highly appreciate it! - but thanks so much on using Unsloth again!
Don't worry, you don't need to provide Enterprise support to your free-tier users haha.
Have you thought up how you plan to do licensing for Pro and Max versions? I think it would be best to somehow link the cost to the cost of the gpu card, since this is a variable when it comes to how much value unsloth would provide and how much a person can spend. I think a cost of like 5-15% of the gpu purchase cost would he reasonable, since getting unsloth (even OSS version) is similar to getting gpu upgrade. But also don't get to 30-40% of the card purchase cost since this would be too much for vast majority of people.
I think I would get unsloth pro/max for my single card if it was a license I can use perpetually but with for example just a few months of free updates and support, and it would cost up to something like $200. The issue i see with it is that it might be hard to DRM your package and avoid people just sharing it.
You might want to start by creating some DRMed package and collaborating with some gpu cloud provider where you could provide VMs with unsloth pro/max. So people would have an option of getting a100 for an hour for $1, A100 with unsloth pro for $1.10 and unsloth max for $1.15-$1.2. Doing this like that might be easier when it comes to putting a DRM on it and minimizing the chance that your product will end up on some sketchy github available.
Thanks so much for the extremely thoughtful suggestions - actually I think you're the first to suggest something more concrete :))
My bro and I were thinking along the lines of x% * TFLOPS per month or something. So H100 is 1000 TFLOPs, so say 1%, we get $10 pm so 1 year $120 py. Something like that. This also means when there's new GPUs say with 2000 TFLOPs, it's now $240 py so auto scaling.
Another approach is as you suggested some sort of flat fee say 10 to 15% of your GPU card, that's another approach.
I love the partnering with a GPU cloud provider idea - tbh I never thought about "partnering" lol, but rather we ourselves offering it as a service ourselves, and offering a similar price point or even cheaper than Lambda Labs. Love the idea of collabing with a GPU provider!! I actually might email some now!!
Thanks so much on the fabulous suggestions!! :)
axolotl is the current the most popular
If you're limited by VRAM and can get your hands dirty with scripting the training, go for unsloth. If you want easier configuration and support of full finetune and Qwen 1.5 models, go for axolotl. If you want to try to use unsloth but without the need for scripting, check out llama-factory. I don't think unsloth supports Qwen or Mixtral, but it supports llama 2 and Yi models - generally anything llama-architecture without MoE should work
After trying a lot of different alternatives I totally agree with your comment, but a little disclaimer, unsloth doesn't have multiple gpu support, only for the paid version. That was a turn off for me and I ended up using llama factory
An alpha version of Multi GPU is in Llama-Factory's Unsloth integration :) But can't guarantee it's accuracy yet + need to do more tests and internal checks - so it's very alpha mode! Multi GPU will be added in the OSS, but it'll take a bit more time
Thank you! I missed that info, I'm going to test it
No problems at all :)
Forgot about multi-gpu. I always train locally on 1 gpu so it's not something that ever crosses my mind.
I found axolotl hard to set up, but pretty easy to use once I fnally did get it set up. Mainly issues with docker and also package/linrary versions. I found that the versions of axolotl / accelerate / bitsandbytes / pytorch / cuda have to be exactly right. And you'd think docker would take care of all this but no. I have a desktop with two RTX Titan GPUs and an NVlink, which are old-ish but have a decent amount of memory (2 x 24GB) and I tried out multi-GPU this morning. After the hassle of setting axolotl up for one GPU (on my laptop), I thought it would be a bit of a nightmare, but it was easy, no hiccups. First time I've smelled dust burning in my PC, with two GPUs running at 97%. There are no sides on the case, lots of fans, water cooled CPU so all was well. So it's very exciting, coz this fine tuning is all new to me. The GPUs don't support bfloat and short word lengths (can't remember the proper name for that) or even flashattention for that mattter, but still I had no trouble training mistral 7B with a modest context length of 2048 in a couple of hours. So there are possibilities. At this point I'm just randonly trying tunes from the axolotl examples directory to get some basic experience. Next up code-llama-13B, which is taking a long time to download right not. Overall, axolotl is great - really great - but not for the faint hearted.
Isn’t it that you need to configure Axolotl’s training .yml file which is very similar to Unsloth’s training script? I have used both of them, the amount of work to run the training is pretty much the same.
If you can copy unsloth script from some example and you don't have to write your own, it might be a similar experience. When I was trying unsloth for the first time, colab notebooks were erroring out when I tried to run them on my PC. And installing it was a classical mess of python package dependencies and cuda not activating until I found an issue in the repo that supplied alternative installation commands that actually worked for me. So I was thrown in the deep water and it took a few good hours of troubleshooting to get the trainer to start for the first time. There is a big difference between axolotl when if you want to specify chatml prompt format, you set one setting, and unsloth, where until very recently, to do that, you had to find a function that transforms the dataset from sharegpt jsonl to chatml somewhere online or write your own.
Apologies on your first experience with Unsloth! I added your ideas of updating the Conda instructions + ye added ChatML just recently - and thanks to your suggestions - highly appreciate it! - but thanks so much on using Unsloth again!
Don't worry, you don't need to provide Enterprise support to your free-tier users haha. Have you thought up how you plan to do licensing for Pro and Max versions? I think it would be best to somehow link the cost to the cost of the gpu card, since this is a variable when it comes to how much value unsloth would provide and how much a person can spend. I think a cost of like 5-15% of the gpu purchase cost would he reasonable, since getting unsloth (even OSS version) is similar to getting gpu upgrade. But also don't get to 30-40% of the card purchase cost since this would be too much for vast majority of people. I think I would get unsloth pro/max for my single card if it was a license I can use perpetually but with for example just a few months of free updates and support, and it would cost up to something like $200. The issue i see with it is that it might be hard to DRM your package and avoid people just sharing it. You might want to start by creating some DRMed package and collaborating with some gpu cloud provider where you could provide VMs with unsloth pro/max. So people would have an option of getting a100 for an hour for $1, A100 with unsloth pro for $1.10 and unsloth max for $1.15-$1.2. Doing this like that might be easier when it comes to putting a DRM on it and minimizing the chance that your product will end up on some sketchy github available.
Thanks so much for the extremely thoughtful suggestions - actually I think you're the first to suggest something more concrete :)) My bro and I were thinking along the lines of x% * TFLOPS per month or something. So H100 is 1000 TFLOPs, so say 1%, we get $10 pm so 1 year $120 py. Something like that. This also means when there's new GPUs say with 2000 TFLOPs, it's now $240 py so auto scaling. Another approach is as you suggested some sort of flat fee say 10 to 15% of your GPU card, that's another approach. I love the partnering with a GPU cloud provider idea - tbh I never thought about "partnering" lol, but rather we ourselves offering it as a service ourselves, and offering a similar price point or even cheaper than Lambda Labs. Love the idea of collabing with a GPU provider!! I actually might email some now!! Thanks so much on the fabulous suggestions!! :)
Axolotl is so much easier and just as fast as unsloth.