ops issues
https://lab.civicrm.org/infra/ops/-/issues
2019-01-03T17:08:54Z
https://lab.civicrm.org/infra/ops/-/issues/843
padthai: faulty disk is causing performance issues
2019-01-03T17:08:54Z
bgm
padthai: faulty disk is causing performance issues
Currrent status:
* [x] Replace all 3 disks in padthai.c.o
* [x] Re-install padthai OS from scratch
* [x] Configure ~~test-ubu1204-5.c.o~~ test-1.c.o
* [x] (moved to #863) Configure ~~test-ubu1604-1.c.o~~ test-2.c.o
* [x] Restore botdyla...
Currrent status:
* [x] Replace all 3 disks in padthai.c.o
* [x] Re-install padthai OS from scratch
* [x] Configure ~~test-ubu1204-5.c.o~~ test-1.c.o
* [x] (moved to #863) Configure ~~test-ubu1604-1.c.o~~ test-2.c.o
* [x] Restore botdylan.c.o
* [x] Restore www-test.c.o
* [ ] Ensure that backups are correctly configured on padthai/test-1/test-2, and that relevant files are backed up (i.e. anything other than `/etc`).
Initial ticket: What seems to be a faulty disk is causing performance issues.
```
root@padthai:~# zpool status
pool: zpadthai
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-9P
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zpadthai ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sda5 ONLINE 0 0 0
sdb5 ONLINE 0 0 0
sdc5 ONLINE 3 0 0
errors: No known data errors
```
```
# dmesg | grep sd
[54960868.851963] sd 0:0:2:0: [sdc] tag#3 CDB: Read(10) 28 00 1d 3a f4 90 00 00 f8 00
[54960868.859534] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[54960868.860477] sd 0:0:2:0: [sdc] tag#3 Sense Key : Medium Error [current]
[54960868.861381] sd 0:0:2:0: [sdc] tag#3 Add. Sense: Unrecovered read error
[54960868.862272] sd 0:0:2:0: [sdc] tag#3 CDB: Read(10) 28 00 1d 3a f4 90 00 00 f8 00
[54960868.863147] blk_update_request: critical medium error, dev sdc, sector 490402960
[54962799.895947] sd 0:0:2:0: [sdc] tag#1 CDB: Read(10) 28 00 0b 6a ec 40 00 00 18 00
[54962799.903136] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[54962799.904018] sd 0:0:2:0: [sdc] tag#0 CDB: Read(10) 28 00 0b 6a f3 90 00 00 08 00
[54962799.904884] blk_update_request: I/O error, dev sdc, sector 191558544
[54962799.905730] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[54962799.906568] sd 0:0:2:0: [sdc] tag#2 CDB: Read(10) 28 00 0b 6a f9 50 00 00 10 00
[54962799.907392] blk_update_request: I/O error, dev sdc, sector 191560016
[54962799.908224] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[54962799.909032] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current]
[54962799.909828] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[54962799.910607] sd 0:0:2:0: [sdc] tag#1 CDB: Read(10) 28 00 0b 6a ec 40 00 00 18 00
[54962799.911373] blk_update_request: critical medium error, dev sdc, sector 191556672
```