crash in `NEO::DrmAllocation::makeBOsResident` or in `checkAllocationsForOverlapping` when using more than one opencl block in gnuradio gr-clenabled

[stef]

not sure if this is a gr-clenabled or a intel opencl icd issue, thus a crosspost notifying you of this report: ghostop14/gr-clenabled#13 (comment)

happy to supply more information on request. also happy to copy paste the issue itself here in case that is more convenient to process.

[HoppeMateusz]

Hi,

problems with checkAllocationsForOverlapping() or with this->fragmentsStorage.fragmentStorageData[0].residency are related and may originate from overlapping memory regions used: a. to create multiple buffers / b. as host_ptr to read/write calls.

To get a better idea what might be happening please check if following is the case:

1. Is flag CL_USE_HOST_PTR used to create buffers?
   Are such buffers mapped (clEnqueueMapBuffer) and returned pointers used to create other buffers? Or passed as a ptr to EnqueueReadWriteBuffer/Image() calls?

This is forbidden by spec.

2. Is pointer returned from Map used in an overlapping manner?
   Offset pointer used to create buffers :
   auto ptr = clEnqueueMapBuffer(buffer, mapSize);
   auto offsetPtr = ptr + 20100;
   clCreateBuffer(CL_MEM_USE_HOST_PTR, size, offsetPtr);

3. API log from CLIntercept would be helpful, or igdrcl.log after setting debug variables with env: NEOReadDebugKeys=1 LogApiCalls=1
   It should be possible to see if the same or offset host_ptr is used to multiple calls

4. You may also try below debug keys to check if they resolve the problem
   NEOReadDebugKeys=1 EnableHostPtrTracking=0

Mateusz

[stef]

1. no CL_USE_HOST_PTR is not found anywhere in the sources of gr-clenabled.
2. i did not see any call to clEnqueueMapBuffer nor clCreateBuffer in gr-clenabled sources.
3. i set these variables (and googled where i could find this igdrcl.log) but no result. i also rerun the whole test with strace -f but there was no call to getenv anywhere in the strace output.
4. see 3. and lack of calls to getenv in straced test.

[HoppeMateusz]

1. / 2. Ok, so only images are used ( clCreateImage / clEnqueueRead/Write/MapImage )?
   Is clEnqueueMapImage used? is returned pointer used to Read/Write calls ?

2. Do you see libigdrcl.so to be opened in strace log?
   it is strange that getenv is not there - do you use recent driver, getenv should be called just after opening libigdrcl.so ? What version of compute runtime do you use?

[stef]

/ 2. Ok, so only images are used ( clCreateImage / clEnqueueRead/Write/MapImage )? Is clEnqueueMapImage used? is returned pointer used to Read/Write calls ?

sorry if i am annoying, but that's because i'm a noob and have no clue about cl - but i'm super excited :)

i checked, and neither clCreateImage nor clEnqueue strings are found in the code. buffers are created with c++ new operator calling this signature: `cl::Buffer (const Context &context, cl_mem_flags flags, size_type size)```

this is the specific file that i am testing with: https://github.com/ghostop14/gr-clenabled/blob/master/lib/clMathOp_impl.cc - in case it helps to look at this.

Do you see libigdrcl.so to be opened in strace log?

18934 openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so", O_RDONLY|O_CLOEXEC) = 4
18934 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260Z\2\0\0\0\0\0"..., 832) = 832

looks ok to me.

it is strange that getenv is not there - do you use recent driver, getenv should be called just after opening libigdrcl.so ? What version of compute runtime do you use?

i use 20.44.18297-1 - the latest version - from debian unstable.

[HoppeMateusz]

Thanks for checks,

this project is using C++ wrappers, it is not possible to tell from the code if overlapping memory regions are used or not.

Driver version you are using should support NEOReadDebugKeys - strace won;t show getenv calls, you need to use ltrace:
ltrace ./your_app

Can you try collecting log with:
NEOReadDebugKeys=1 PrintDebugSettings=1 LogApiCalls=1 ./your_app ? It will help root cause the problem.

Log - igdrcl.log file - should be produced in current working directory from where you execute application. Please make sure you have correct rights.
You should also see modified, non-default debug keys in the output.

[stef]

Thanks for checks,

thanks for your continued engagement!

this project is using C++ wrappers, it is not possible to tell from the code if overlapping memory regions are used or not.

the c++ class being created seems to be standard though: https://github.khronos.org/OpenCL-CLHPP/classcl_1_1_buffer.html - and we do not pass any extra parameters to it. why is it not possible to tell from the code if overlapping regions are used, is this not possible in the khronos c++ api?

Driver version you are using should support NEOReadDebugKeys - strace won;t show getenv calls, you need to use ltrace:
ltrace ./your_app

oh, of course, getenv is no kernel syscall. indeed with ltrace i see getenv calls. sadly my testcase is multithreaded, and ltrace refuses to trace the other threads for some reason/

Can you try collecting log with:
NEOReadDebugKeys=1 PrintDebugSettings=1 LogApiCalls=1 ./your_app ? It will help root cause the problem.

yes. i actually had to drop the python frontend and implemented the testcase in c++ https://gist.github.com/stef/e7818dc85b48c06d98a333e01f3d526f

debug settings were indeed dumped, i put them up in a gist here: https://gist.github.com/stef/d6a6f14a945387c5fedddec7ea23564b

Log - igdrcl.log file - should be produced in current working directory from where you execute application. Please make sure you have correct rights.

i have the correct rights, yet no logfile at all is generated. i have this slight suspicion that something is missing, is there anything i need to install for this clintercept to work, or should it be supported out of the box by the debian intel opencl icd package?

You should also see modified, non-default debug keys in the output.

indeed. have a look a the the gist i linked above.

[HoppeMateusz]

Hi,

It is not possible to deduce overlapping from the code as pointers are passed around and i do not know where input_items/ output_items passed to enqueueReadBuffer/WriteBuffer comes from.

Unfortunately, internal logging with LogApiCalls is only available with Debug build of compute runtime, so to collect API trace, clIntercept must be used. It requires special setup, please take a look at :
https://github.com/intel/opencl-intercept-layer

[ghostop14]

Hi folks, everything from above looks fine. There are no overlapping or host pointers used in the code anywhere. And thread locks are used to ensure there's no race conditions.

[stef]

i patched up clMathOp_impl.c (linked above), and added this code after each new cl::Buffer operation:

std::cout << "clMathOp aBuffer ptr: " << aBuffer <<
             " host ptr: " << aBuffer->getInfo<CL_MEM_HOST_PTR>() <<
             " size: " << aBuffer->getInfo<CL_MEM_SIZE>() <<
             " type: " << aBuffer->getInfo<CL_MEM_TYPE>() <<
             " flags: " << aBuffer->getInfo<CL_MEM_FLAGS>() <<
             std::endl;

when running my test case now (in which i also enabled debug for the two clMathOp instances), the output is the following:

OpenCL: using Intel(R) OpenCL HD Graphics
clMathOp aBuffer ptr: 0x5570fe3bc160 host ptr: 0 size: 65536 type: 4336 flags: 4
clMathOp bBuffer ptr: 0x5570feb8b370 host ptr: 0 size: 65536 type: 4336 flags: 4
clMathOp cBuffer ptr: 0x5570fef54bb0 host ptr: 0 size: 65536 type: 4336 flags: 1
OpenCL INFO: Math Op - too many items for constant memory.  Building kernel with __global params...
OpenCL: using Intel(R) OpenCL HD Graphics
clMathOp aBuffer ptr: 0x5570febc9870 host ptr: 0 size: 65536 type: 4336 flags: 4
clMathOp bBuffer ptr: 0x5570fe860c30 host ptr: 0 size: 65536 type: 4336 flags: 4
clMathOp cBuffer ptr: 0x5570fe840680 host ptr: 0 size: 65536 type: 4336 flags: 1
OpenCL INFO: Math Op - too many items for constant memory.  Building kernel with __global params...

i guess the host ptr being 0 is not very useful and that would be the info that would be most important. is there any other way to get the address of the buffer than what i'm noobing around here?

[ghostop14]

A host ptr is a different kind of pointer (and not what's used here), so I'm not surprised it's returning zero. Everything else around constant memory checks looks correct to me.

[ghostop14]

If you run clinfo from the command-line (you may need to apt install it), and post the output here, it may contain some clues.

[stef]

A host ptr is a different kind of pointer (and not what's used here), so I'm not surprised it's returning zero. Everything else around constant memory checks looks correct to me.

sure. that's clear, the question is, is there any way to figure out if the buffers are somehow overlapping?

[stef]

If you run clinfo from the command-line (you may need to apt install it), and post the output here, it may contain some clues.

there you go sir: https://gist.github.com/stef/ecb32caaceba2c90891c64f065dee554