Consider activating "Lockdown" a mainlined patch that does some hardening measures that including some grsec formerly did. Package currently available only in Sid.
What it does:
* disables loading/removing kernel modules after boot
* disables live kernel patching (kexec)
* disables Berkeley packet filter (BPF)
(*) No unsigned modules and no modules for which can’t validate the signature.
(*) No use of ioperm(), iopl() and no writing to /dev/port.
(*) No writing to /dev/mem or /dev/kmem.
(*) No hibernation.
(*) Restrict PCI BAR access.
(*) Restrict MSR access.
(*) No kexec_load().
(*) Certain ACPI restrictions.
(*) Restrict debugfs interface to ASUS WMI.
While idsabling modules might interfere with non buil-in devices, they can be whitelisted on demand or an extended timeout specified so the needed components are loaded during boot.