torch cuda is available

Steps to Install Torch CUDA in C++

To install Torch CUDA in C++, you can follow the steps below:

  1. Open the "x64 native tools command prompt" as an administrator.
  2. Activate conda by running the following command, replacing USERNAME with your username: powershell -ExecutionPolicy ByPass -NoExit -Command "& 'C:\Users\USERNAME\miniconda3\shell\condabin\conda-hook.ps1' ; conda activate 'C:\Users\USERNAME\miniconda3' "[1].
  3. Create a new conda environment with Python 3.10.9 by running the command: conda create -n textgen python=3.10.9.
  4. Activate the newly created conda environment by running: conda activate textgen.
  5. Install CUDA using the following command: conda install cuda -c nvidia/label/cuda-11.3.0 -c nvidia/label/cuda-11.3.1[1].
  6. Clone the text-generation-webui repository from GitHub by running: git clone https://github.com/oobabooga/text-generation-webui.
  7. Navigate to the cloned repository by running: cd text-generation-webui.
  8. Install the required Python packages by running: pip install -r requirements.txt.
  9. Install Torch with CUDA support by running: pip install torch==1.12+cu113 -f https://download.pytorch.org/whl/torch_stable.html.
  10. Create a new directory named "repositories" by running: mkdir repositories.
  11. Navigate to the "repositories" directory by running: cd repositories.
  12. Clone the GPTQ-for-LLaMa repository from GitHub, specifically the "cuda" branch, by running: git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa --branch cuda --single-branch[1].
  13. Navigate to the cloned GPTQ-for-LLaMa directory by running: cd GPTQ-for-LLaMa.
  14. Reset the repository to a specific commit by running: git reset --hard c589c5456cc1c9e96065a5d285f8e3fac2cdb0fd.
  15. Install the Ninja build system by running: pip install ninja.
  16. Set the environment variable DISTUTILS_USE_SDK to 1 by running: $env:DISTUTILS_USE_SDK=1.
  17. Run the setup script for CUDA by running: python setup_cuda.py.

These steps will guide you through the installation process of Torch CUDA in C++. Make sure to follow each step carefully to ensure a successful installation.