[chimerax-users] Specify GPU id for ChimeraX

Shasha Feng shaalltime at gmail.com
Tue Nov 24 18:55:30 PST 2020 

Hi Tom,

I myself also checked out the "NVIDIA X Server Settings" utility on my
ubuntu 20.04 and spent some time tweaking the setting. The binding between
rendering X screen and OpenGL service makes it difficult to separate,
unless I have a second screen loaded on GPU 1 and run ChimeraX there. I was
not sure how the attachment would work in mail list archive, so I opted to
use the old forum post to refer to the setting.

I initially intended to hack this through, as the ability to manipulate on
which GPU to run ChimeraX was really desired. Now discovering how ChimeraX
works with OpenGL, I do agree that choosing the GPU for a compute job is
much easier. Thanks for the discussion and for sharing your insights.

Best,
Shasha


On Tue, Nov 24, 2020 at 9:39 PM Tom Goddard <goddard at sonic.net> wrote:

> Hi Shasha,
>
>   That Ubuntu forum post from 7 - 8 years ago may not be recent enough to
> describe how it works now. Here is another post still pretty old 4 - 5
> years discussing how to tell the X server which GPU to use for a screen.
>
> https://askubuntu.com/questions/787030/setting-the-default-gpu
>
> I'm a bit puzzled by what configurations are possible.  If you have your
> display physically plugged into GPU_0 with say a display port cable, then
> definitely that GPU is doing part of the job of rendering to the display
> since it is sending electrical the signal.  If it is possible to configure
> things so ChimeraX uses GPU_1 to render and then to get it to appear on
> your GPU_0 screen it will have to send that rendering from GPU_1 to GPU_0
> (probably by way of the CPU).  That could be slow and you end up
> interrupting both GPUs to render graphics.
>
>   If X windows uses just one GPU for a screen, say GPU_0, I think it would
> make a lot more sense for you to run compute jobs on GPU_1 and leave all
> graphics rendering on GPU_0 and not try to change which GPU ChimeraX runs
> on.  It is probably much more common and easier to choose the GPU to use
> for a compute job.
>
> Tom
>
>
> On Nov 24, 2020, at 6:12 PM, Shasha Feng <shaalltime at gmail.com> wrote:
>
> Hi Tom,
>
> Thanks for the tips. Tristan, the ISOLDE developer, also mentioned to me
> that ISOLDE GPU selection can be specified by "isolde set gpuDeviceIndex
> {n}" on chimerax cmd line.
>
> After digging into the QOpenGLContext and your description of GPU switches
> on macOS, I realize that it has something to do with OpenGL interaction
> with nvidia-settings. As shown in the snapshots in this thread [
> https://askubuntu.com/questions/280972/how-to-understand-nvidia-settings-save-configuration-options],
> on ubuntu there is an 'NVIDIA X Server Settings' utility. The OpenGL is
> bound to X server/screen, which is a Samsung screen that is loaded on GPU
> 0.
>
> So it looks that I would need a second screen, then when I drag the
> chimerax program there, the job would automatically appear on the second
> GPU. This is not a smart solution, but still a solution... For the time
> being, I will shift the existing jobs to the second GPU.
>
> Thanks,
> Shasha
>
>
> On Tue, Nov 24, 2020 at 1:47 PM Tom Goddard <goddard at sonic.net> wrote:
>
>> Hi Shasha,
>>
>>   ChimeraX does not use CUDA.  It only uses the graphics card with OpenGL
>> for graphics rendering, not for non-graphical calculations.  There is one
>> exception to that, the ISOLDE plugin to ChimeraX can use CUDA if you tell
>> it to.
>>
>>   So I think the environment variable you would need to use is
>> NVIDIA_VISIBLE_DEVICES.  I don't know why that would not work.  ChimeraX is
>> using Qt to create a QOpenGLContext().  That Python code is in your
>> distribution in file
>>
>> chimera/lib/python3.7/site-packages/chimerax/graphics/opengl.py
>>
>>         # Create context
>>         from PyQt5.QtGui import QOpenGLContext
>>         qc = QOpenGLContext()
>>         qc.setScreen(self._screen)
>>
>> The Qt window toolkit has no capabilities to choose the GPU as far as I
>> know.  I don't have a multi-GPU nvidia system to test on, but I tried
>> starting ChimeraX from a bash shell with
>>
>> NVIDIA_VISIBLE_DEVICES=1 chimerax
>>
>> and put in code to print the environment variables before the
>> QOpenGLContext is created and the environment is printed and set.  I was
>> worried that ChimeraX might remove some environment variables but that does
>> not happen.  So I cannot explain why the environment variable does not work.
>>
>>   I know nothing about Nvidia-SMI but am surprised that it can choose
>> between different graphics cards while rendering to the same screen.  I am
>> more familiar with macOS with an external GPU and two displays.  With that
>> operating system if I run ChimeraX on the iMac and MacBook laptop display
>> it uses the computer's graphics, and if I run ChimeraX on an external
>> display attached to the external GPU it runs it using the external GPU --
>> in other words the display you run on controls which GPU is used.  In fact,
>> on macOS it remarkably switches which GPU is being used if I simply drag
>> the ChimeraX window from one display to the other.  Of course Ubuntu is
>> entirely different and it seems like NVIDIA_VISIBLE_DEVICES=1 should work.
>>
>> Tom
>>
>>
>> On Nov 24, 2020, at 9:21 AM, Shasha Feng <shaalltime at gmail.com> wrote:
>>
>> Hi Guillaume, and Eric
>>
>> Thanks for the tip. The temporary assignment of visiable GPU devices is
>> exactly what I want to get. Though it looks like the recipe of using
>> 'CUDA_VISIBLE_DEVICES=1' does not work at least on my ubuntu 20.04 with
>> chimerax 1.0. I also tried Eric's suggestion just now.
>>
>> sf at sf-MS-7C35:~$ echo $CUDA_VISIBLE_DEVICES
>>
>> sf at sf-MS-7C35:~$ export CUDA_VISIBLE_DEVICES=1
>> sf at sf-MS-7C35:~$ echo $CUDA_VISIBLE_DEVICES
>> 1
>> sf at sf-MS-7C35:~$ chimerax &
>> [1] 673010
>> sf at sf-MS-7C35:~$ nvidia-smi
>> Tue Nov 24 12:09:28 2020
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 450.66       Driver Version: 450.66       CUDA Version: 11.0
>>     |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util
>>  Compute M. |
>> |                               |                      |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> |   0  GeForce RTX 207...  Off  | 00000000:2D:00.0  On |
>>  N/A |
>> | 60%   74C    P2   191W / 215W |    763MiB /  7974MiB |     99%
>>  Default |
>> |                               |                      |
>>  N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>> |   1  GeForce RTX 207...  Off  | 00000000:2E:00.0 Off |
>>  N/A |
>> |  0%   34C    P8    14W / 215W |     14MiB /  7982MiB |      0%
>>  Default |
>> |                               |                      |
>>  N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>>      |
>> |  GPU   GI   CI        PID   Type   Process name                  GPU
>> Memory |
>> |        ID   ID                                                   Usage
>>      |
>>
>> |=============================================================================|
>> |    0   N/A  N/A      1343      G   /usr/lib/xorg/Xorg
>> 35MiB |
>> |    0   N/A  N/A      2338      G   /usr/lib/xorg/Xorg
>>  174MiB |
>> |    0   N/A  N/A      2463      G   /usr/bin/gnome-shell
>>  233MiB |
>> |    0   N/A  N/A    671633      G   ...AAAAAAAAA= --shared-files
>> 45MiB |
>> |    0   N/A  N/A    672504      C   /opt/conda/bin/python
>> 229MiB |
>> |    0   N/A  N/A    673010      G   chimerax
>> 33MiB |
>> |    1   N/A  N/A      1343      G   /usr/lib/xorg/Xorg
>>  4MiB |
>> |    1   N/A  N/A      2338      G   /usr/lib/xorg/Xorg
>>  4MiB |
>>
>> +-----------------------------------------------------------------------------+
>>
>> After setting the environment variable and running chimerax in the same
>> session, it still runs on GPU 0.
>> I also tried a recipe that defines
>> "export NVIDIA_VISIBLE_DEVICES=1,
>> export CUDA_VISIBLE_DEVICES=0" shared here [
>> https://stackoverflow.com/a/58445444]. It does not work either.
>>
>> To ChimeraX developers,
>> I wonder how ChimeraX is exposed to CUDA. I have basis in CUDA computing
>> and using CUDA in Python. If you can give some clues, that would be great.
>>
>> Best,
>> Shasha
>>
>> On Tue, Nov 24, 2020 at 12:18 PM Eric Pettersen <pett at cgl.ucsf.edu>
>> wrote:
>>
>>> To supplement Guilaume's very helpful answer, you could make an alias to
>>> reduce the typing involved, and you could put the alias in your shell
>>> startup file.  For the bash shell, the syntax for making an alias named
>>> 'cx' for the command would be:
>>>
>>> alias cx="CUDA_VISIBLE_DEVICES=1 chimerax"
>>>
>>> Other shells have similar (but not necessarily identical) syntaxes.
>>>
>>> --Eric
>>>
>>> Eric Pettersen
>>> UCSF Computer Graphics Lab
>>>
>>>
>>> On Nov 24, 2020, at 12:09 AM, Guillaume Gaullier <guillaume at gaullier.org>
>>> wrote:
>>>
>>> Hello,
>>>
>>> You can restrict which of your GPUs ChimeraX will be able to detect by
>>> starting it from the shell like so:
>>>
>>> CUDA_VISIBLE_DEVICES=1 chimerax
>>>
>>> replace 1 with the device number you want, this is the same one as
>>> reported by nvidia-smi. This will work until you close ChimeraX, next time
>>> you run it you still need to add the environment variable before the
>>> "chimerax" command.
>>>
>>> You can also make this environment variable stay around until you close
>>> the shell session like so:
>>>
>>> export CUDA_VISIBLE_DEVICES=1
>>>
>>> then you can open ChimeraX from the same shell session, close it, and
>>> reopen with only the "chimerax" command and it should still only see the
>>> GPU you indicated.
>>>
>>> When you close and restart your shell, you will have to export the
>>> environment variable again. I don’t recommend adding the export to your
>>> ~/.bashrc or other shell initialization script, because then all your shell
>>> sessions will have this environment variable set, so all the commands you
>>> run will only see this GPU, which is probably not what you want. It is less
>>> likely to get in your way down the road if you only set this environment
>>> variable for the duration of a shell you opened specifically to run
>>> ChimeraX from.
>>>
>>> I hope this helps,
>>>
>>> Guillaume
>>>
>>>
>>> On 24 Nov 2020, at 01:51, Shasha Feng <shaalltime at gmail.com> wrote:
>>>
>>> Hi Tom,
>>>
>>> Sorry about not clarifying my operating system. I am using ubuntu 20.04
>>> with two NVIDIA GPU cards.
>>> Do I need to change OpenGL setting or reconfigure the nvidia setting?
>>>
>>> Thanks,
>>> Shasha
>>>
>>>
>>>
>>> On Mon, Nov 23, 2020 at 6:58 PM Tom Goddard <goddard at sonic.net> wrote:
>>>
>>>> Hi Shasta,
>>>>
>>>> ChimeraX has no way to select which GPU it uses.   The operating system
>>>> or opengl driver decides.  You didn't mention which operating system you
>>>> are using.  Here is an example of how to set the default OpenGL GPU in
>>>> Windows.
>>>>
>>>>
>>>> https://www.techadvisor.co.uk/how-to/pc-components/how-set-default-graphics-card-3612668/
>>>>
>>>>   Tom
>>>>
>>>>
>>>> On Nov 23, 2020, at 2:38 PM, Shasha Feng <shaalltime at gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Is there any way to specify which GPU device for ChimeraX to run on?
>>>> Currently, it uses the default GPU 0, which can disturb the existing jobs.
>>>> Thanks.
>>>>
>>>> Best,
>>>> Shasha
>>>>
>>>> _______________________________________________
>>>> ChimeraX-users mailing list
>>>> ChimeraX-users at cgl.ucsf.edu
>>>> Manage subscription:
>>>> https://www.rbvi.ucsf.edu/mailman/listinfo/chimerax-users
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> ChimeraX-users mailing list
>>> ChimeraX-users at cgl.ucsf.edu
>>> Manage subscription:
>>> https://www.rbvi.ucsf.edu/mailman/listinfo/chimerax-users
>>>
>>>
>>> _______________________________________________
>>> ChimeraX-users mailing list
>>> ChimeraX-users at cgl.ucsf.edu
>>> Manage subscription:
>>> https://www.rbvi.ucsf.edu/mailman/listinfo/chimerax-users
>>>
>>>
>>>
>> _______________________________________________
>> ChimeraX-users mailing list
>> ChimeraX-users at cgl.ucsf.edu
>> Manage subscription:
>> https://www.rbvi.ucsf.edu/mailman/listinfo/chimerax-users
>>
>>
>>
>
> _______________________________________________
> ChimeraX-users mailing list
> ChimeraX-users at cgl.ucsf.edu
> Manage subscription:
> https://www.rbvi.ucsf.edu/mailman/listinfo/chimerax-users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://plato.cgl.ucsf.edu/pipermail/chimerax-users/attachments/20201124/8b706276/attachment-0001.html>

More information about the ChimeraX-users mailing list