torch cuda is available
Steps to Install Torch CUDA in C++
To install Torch CUDA in C++, you can follow the steps below:
- Open the "x64 native tools command prompt" as an administrator.
- Activate conda by running the following command, replacing USERNAME with your username:
powershell -ExecutionPolicy ByPass -NoExit -Command "& 'C:\Users\USERNAME\miniconda3\shell\condabin\conda-hook.ps1' ; conda activate 'C:\Users\USERNAME\miniconda3' "
[1]. - Create a new conda environment with Python 3.10.9 by running the command:
conda create -n textgen python=3.10.9
. - Activate the newly created conda environment by running:
conda activate textgen
. - Install CUDA using the following command:
conda install cuda -c nvidia/label/cuda-11.3.0 -c nvidia/label/cuda-11.3.1
[1]. - Clone the text-generation-webui repository from GitHub by running:
git clone https://github.com/oobabooga/text-generation-webui
. - Navigate to the cloned repository by running:
cd text-generation-webui
. - Install the required Python packages by running:
pip install -r requirements.txt
. - Install Torch with CUDA support by running:
pip install torch==1.12+cu113 -f https://download.pytorch.org/whl/torch_stable.html
. - Create a new directory named "repositories" by running:
mkdir repositories
. - Navigate to the "repositories" directory by running:
cd repositories
. - Clone the GPTQ-for-LLaMa repository from GitHub, specifically the "cuda" branch, by running:
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa --branch cuda --single-branch
[1]. - Navigate to the cloned GPTQ-for-LLaMa directory by running:
cd GPTQ-for-LLaMa
. - Reset the repository to a specific commit by running:
git reset --hard c589c5456cc1c9e96065a5d285f8e3fac2cdb0fd
. - Install the Ninja build system by running:
pip install ninja
. - Set the environment variable
DISTUTILS_USE_SDK
to 1 by running:$env:DISTUTILS_USE_SDK=1
. - Run the setup script for CUDA by running:
python setup_cuda.py
.
These steps will guide you through the installation process of Torch CUDA in C++. Make sure to follow each step carefully to ensure a successful installation.