OpenWrt/LEDE Project

  • Status New
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Medium
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Stijn Tintel - 08.05.2020

FS#3070 - kmod-cryptodev: WARNING: possible circular locking dependency detected

OpenWrt master r13174-73fa1aba94 on octeon with kernel 5.4

[   70.971145] 
[   70.972662] ======================================================
[   70.978843] WARNING: possible circular locking dependency detected
[   70.985029] 5.4.39 #0 Not tainted
[   70.988348] ------------------------------------------------------
[   70.994531] unbound/2895 is trying to acquire lock:
[   70.999414] 800000041baeb868 (&pcr->fcrypt.sem){+.+.}, at: crypto_get_session_by_sid+0x40/0x12b8 [cryptodev]
[   71.009267] 
[   71.009267] but task is already holding lock:
[   71.015103] 800000041b911468 (&ses_new->sem){+.+.}, at: crypto_get_session_by_sid+0xbc/0x12b8 [cryptodev]
[   71.024691] 
[   71.024691] which lock already depends on the new lock.
[   71.024691] 
[   71.032866] 
[   71.032866] the existing dependency chain (in reverse order) is:
[   71.040347] 
[   71.040347] -> #1 (&ses_new->sem){+.+.}:
[   71.045773]        lock_acquire+0xe0/0x220
[   71.049888]        __mutex_lock+0x94/0x628
[   71.054001]        crypto_get_session_by_sid+0xbc/0x12b8 [cryptodev]
[   71.060365]        crypto_get_session_by_sid+0x1e4/0x12b8 [cryptodev]
[   71.066806]
[   71.066806] -> #0 (&pcr->fcrypt.sem){+.+.}:
[   71.072485]        check_noncircular+0x1a8/0x260
[   71.077110]        __lock_acquire+0x12f8/0x19f8
[   71.081648]        lock_acquire+0xe0/0x220
[   71.085754]        __mutex_lock+0x94/0x628
[   71.089862]        crypto_get_session_by_sid+0x40/0x12b8 [cryptodev]
[   71.096226]        crypto_get_session_by_sid+0x6cc/0x12b8 [cryptodev]
[   71.102666]
[   71.102666] other info that might help us debug this:
[   71.102666]
[   71.110668]  Possible unsafe locking scenario:
[   71.110668]
[   71.116588]        CPU0                    CPU1
[   71.121119]        ----                    ----
[   71.125650]   lock(&ses_new->sem);
[   71.129058]                                lock(&pcr->fcrypt.sem);
[   71.135242]                                lock(&ses_new->sem);
[   71.141165]   lock(&pcr->fcrypt.sem);
[   71.144834]
[   71.144834]  *** DEADLOCK ***
[   71.144834]
[   71.150756] 1 lock held by unbound/2895:
[   71.154681]  #0: 800000041b911468 (&ses_new->sem){+.+.}, at: crypto_get_session_by_sid+0xbc/0x12b8 [cryptodev]
[   71.164700]
[   71.164700] stack backtrace:
[   71.169067] CPU: 1 PID: 2895 Comm: unbound Not tainted 5.4.39 #0
[   71.175075] Stack : ffffffff82790000 0000000000000000 0000000010108ce0 ed34220aed96997f
[   71.183091]         ed34220aed96997f 0000000000000000 800000041bb976f8 ffffffff837d0000
[   71.191106]         0000000000000000 0000000000000001 0000000000000000 ffffffff811ac4bc
[   71.199118]         6e626f756e64204e 0000000000000000 ffffffffffffffff 0000000000000010
[   71.207132]         0000000000000000 ffffffff81b40000 fffe000000000000 ffffffff81bc0000
[   71.215145]         0000000000000000 0000000000000000 ffffffff81b40000 0000000000000000
[   71.223158]         800000041f2e6200 0000000000000000 ffffffff81597628 1e00000010734ac7
[   71.231173]         0000000000000001 800000041bb94000 800000041bb976f0 47500c0a872e7996
[   71.239186]         ffffffff81865d8c 0000000000000000 800000041bb97828 ed34220aed96997f
[   71.247198]         ffffffff81b40bf7 ffffffff81865c54 ffffffff8111d4c8 ffffffff81a42a50
[   71.255213]         ...
[   71.257668] Call Trace:
[   71.260136] [<ffffffff8111d4c8>] show_stack+0x40/0x128
[   71.265288] [<ffffffff81865d8c>] dump_stack+0xe4/0x150
[   71.270444] [<ffffffff8119fb58>] check_noncircular+0x1a8/0x260
[   71.276289] [<ffffffff811a2d38>] __lock_acquire+0x12f8/0x19f8
[   71.282046] [<ffffffff811a3d80>] lock_acquire+0xe0/0x220
[   71.287374] [<ffffffff8188532c>] __mutex_lock+0x94/0x628
[   71.292706] [<ffffffffc02a7638>] crypto_get_session_by_sid+0x40/0x12b8 [cryptodev]
[   71.300290] [<ffffffffc02a7cc4>] crypto_get_session_by_sid+0x6cc/0x12b8 [cryptodev]
n8v8R commented on 12.06.2020 17:53

Maybe something device specific or unbound flavour (light vs heavy)? Does not reproduce on

{"kernel":"5.4.45","hostname":"OpenWrt","system":"ARMv7 Processor rev 1 (v7l)","model":"Turris Omnia","board_name":"cznic,turris-omnia","release":{"distribution":"OpenWrt","version":"SNAPSHOT","revision":"r13552-cd09f26660","target":"mvebu/cortexa9","description":"OpenWrt SNAPSHOT r13552-cd09f26660"}}

with

* kmod-cryptodev 5.4.45+1.10-mvebu-2
* unbound-daemon-heavy 1.10.1-2
* libunbound-heavy 1.10.1-2

michael-dev commented on 02.03.2021 21:15

I'm seeing the same on a custom OpenWRT build based onfae69177db52e14b7b966d73603ee269f6ef02c7

[   45.669944] ======================================================
[   45.676115] WARNING: possible circular locking dependency detected
[   45.682292] 5.4.99 #0 Not tainted
[   45.685598] ------------------------------------------------------
[   45.691772] nrpe/3560 is trying to acquire lock:
[   45.696382] ef68f23c (&pcr->fcrypt.sem){+.+.}, at: crypto_get_session_by_sid+0x38/0xc4 [cryptodev]
[   45.705360]
[   45.705360] but task is already holding lock:
[   45.711184] ef68fa3c (&ses_new->sem){+.+.}, at: crypto_get_session_by_sid+0xa8/0xc4 [cryptodev]
[   45.719884]
[   45.719884] which lock already depends on the new lock.
[   45.719884]
[   45.728052]
[   45.728052] the existing dependency chain (in reverse order) is:
[   45.735526]
[   45.735526] -> #1 (&ses_new->sem){+.+.}:
[   45.740942]        __mutex_lock+0x94/0x84c
[   45.745037]        crypto_get_session_by_sid+0xa8/0xc4 [cryptodev]
[   45.751213]        cryptodev_ioctl+0xa4/0x9c8 [cryptodev]
[   45.756609]        do_vfs_ioctl+0xc4/0x948
[   45.760698]        ksys_ioctl+0x50/0xbc
[   45.764535]        ret_from_syscall+0x0/0x38
[   45.768797]
[   45.768797] -> #0 (&pcr->fcrypt.sem){+.+.}:
[   45.774467]        __lock_acquire+0x1004/0x2100
[   45.778992]        lock_acquire+0xe0/0x1ec
[   45.783085]        __mutex_lock+0x94/0x84c
[   45.787179]        crypto_get_session_by_sid+0x38/0xc4 [cryptodev]
[   45.793356]        cryptodev_ioctl+0x538/0x9c8 [cryptodev]
[   45.798835]        do_vfs_ioctl+0xc4/0x948
[   45.802925]        ksys_ioctl+0x50/0xbc
[   45.806756]        ret_from_syscall+0x0/0x38
[   45.811017]
[   45.811017] other info that might help us debug this:
[   45.811017]
[   45.819012]  Possible unsafe locking scenario:
[   45.819012]
[   45.824923]        CPU0                    CPU1
[   45.829444]        ----                    ----
[   45.833965]   lock(&ses_new->sem);
[   45.837360]                                lock(&pcr->fcrypt.sem);
[   45.843534]                                lock(&ses_new->sem);
[   45.849446]   lock(&pcr->fcrypt.sem);
[   45.853102]
[   45.853102]  *** DEADLOCK ***
[   45.853102]
[   45.859015] 1 lock held by nrpe/3560:
[   45.862668]  #0: ef68fa3c (&ses_new->sem){+.+.}, at: crypto_get_session_by_sid+0xa8/0xc4 [cryptodev]
[   45.871803]
[   45.871803] stack backtrace:
[   45.876156] CPU: 1 PID: 3560 Comm: nrpe Not tainted 5.4.99 #0
[   45.881894] Call Trace:
[   45.884342] [ef5b7ba8] [c06844a8] dump_stack+0xb0/0x124 (unreliable)
[   45.890695] [ef5b7bc8] [c0092404] check_noncircular+0x16c/0x1f4
[   45.896611] [ef5b7c08] [c0094b5c] __lock_acquire+0x1004/0x2100
[   45.902439] [ef5b7cc8] [c00964ac] lock_acquire+0xe0/0x1ec
[   45.907838] [ef5b7d08] [c06a429c] __mutex_lock+0x94/0x84c
[   45.913236] [ef5b7d78] [f13d53e4] crypto_get_session_by_sid+0x38/0xc4 [cryptodev]
[   45.920716] [ef5b7d98] [f13d59a8] cryptodev_ioctl+0x538/0x9c8 [cryptodev]
[   45.927500] [ef5b7ec8] [c01c5028] do_vfs_ioctl+0xc4/0x948
[   45.932894] [ef5b7f18] [c01c58fc] ksys_ioctl+0x50/0xbc
[   45.938030] [ef5b7f38] [c0012298] ret_from_syscall+0x0/0x38
[   45.943599] --- interrupt: c01 at 0xfb2aa74

Though, no actual deadlock occurs.

michael-dev commented on 02.03.2021 21:19

Looking at crypto_get_session_by_sid, I also cannot see how this should ever happen.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing