# NetBSD 10: 3 Years in the Making ## Nia Alarie, EuroBSDCon 2022 [Notes for this talk](./releng-notes.txt) ``` \-__,------,___. \ __,---` NetBSD/evbarm efiboot (arm64) \ `---,_. Revision 2.10 (Thu Aug 19 12:13:37 UTC 2021) \-,_____,.---` Memory: 512 MB \ \ \ ``` --- # Release flow (not to scale!) ``` HEAD------------------------------------------------------------[...] | | | | | | | netbsd-10 ---| | 10.0 | netbsd-9 ------|----------|---------|-------------[...] 9.0 9.1 9.2 [...]--|--------------------------|--------------------fin 8.1 8.2 ``` --- # Quality assurance tools ``` build.sh | | ----------------------| / | Static analysis | (-Werror, lint, Coverity...)| | Anita (9+ archs - QEMU/NVMM, gexmul...) | ------------------ / \ Rump kernels Sanitizers | (KUBSAN, KASAN, KMSAN...) | and KASSERT() ATF | 10,000+ tests | Kernel fuzzers (syzbot) ``` --- # Late 2019 Andrew Doran begins reworking critical subsystems of the NetBSD kernel. ``` scheduler: Improve performance and interactivity. [ad 20191201] vfs: Reduce false sharing and lock overhead during normal operation. [ad 20191201] uvm: Replace global counters with per-CPU counters, significantly reducing system time on workloads that are VM-system heavy. [ad 20191216] scheduler: Make the scheduler topology aware. It understands and makes good use of HyperThreading/SMT, spreads the load evenly across different physical CPU packages, and can effectively run asymmetric systems with slow and fast CPUs [ad 20200113] ``` By November 2020, performance on multi-core systems almost doubles. --- # Trouble a'brewing current-users@, early 2020: - 'diagnostic assertion "uvm_page_owner_locked_p(new_pg, false)" failed' - 'panic: _pmap_enter: cannot allocate L3 table (aarch64)' - ... and other fun kernel panics Many bugs found with syzbot, then fixed. --- # Lessons learned? For finding and fixing bugs early? - User participation in testing the unstable branch (current-users@) _critical_ - Kernel fuzzers (syzbot) and sanitizers _critical_ - Use of KASSERT() in kernel code _critical_ --- # January 2020 - Great Driver Cleanup - Want wide improvement to kernel APIs / networking stack. - Some non-Ethernet networks fallen out of use entirely. - Feature removals in NetBSD quite rare. --- # Driver Cleanup Results Gone: all FDDI, all HIPPI, all Token Ring, SMBFS, strip(4), de(4), uyurex(4), uyap(4) Kept: sl(4) (serial line IP), se(4) (SCSI Ethernet adapter), pf(4) (the deprecated packet filter) It went well compared to previous removal proposals! --- # Summer 2020 "We're all excited about the fabulous progress that -current has made, especially in performance but it's also quite obvious that it has serious issues and we are far from being able to branch netbsd-10" --- # Summer 2020: random saga begins - Early 2020: Entropy estimator removed from kernel - machines with trusted RNG block never, others until manual intervention. - Some hardware lacks any good random source - NetBSD supports a lot of hardware not built for modern encryption needs. - Disagreements about how to "trust" randomness, whether blocking or never blocking is okay. - _getrandom_ adopted from Linux due to compromise behavior. - See Taylor's EuroBSDCon 2021 talk... Unforseen problems: "Entropy error blocks lang/python38 installation" --- # 2020 in Summary - New AArch64 security features (Pointer Authentication, Branch Target Identification) - Massive improvements to Xen support (PVH, fast MPSAFE networking) - ARM EFI bootloader improvements (boot.cfg, modules...) - Raspberry Pi 4 support - ... with EDK II UEFI firmware required - wg(4) imported - _Mr. Donenfeld displeased_ - Retrocomputing: Major SMP improvements on Alpha - Default package database location moved... - _Rise of the "split brain" problem!_ --- # May 2021: CFARGS audit and kernel APIs NetBSD 9.x: - Pile of functions: `config_found`, `config_found_sm_loc`, `config_found_ia`... - State of kernel device configuration: some code copy and pasted, some code misusing functions. Verdict: - Simplify writing device configuration code for the NetBSD kernel - Challenge: NetBSD 9.x had (some) level of compile-time error checking. Preserve this and make it easier to use. --- # August 2021: CFARGS solution Old example: c->c_dev = config_found(sc->sc_dev, &pa, pciprint, CFARG_SUBMATCH, config_stdsubmatch, CFARG_LOCATORS, locs, CFARG_DEVHANDLE, devhandle, CFARG_EOL); New example: c->c_dev = config_found(sc->sc_dev, &pa, pciprint, CFARGS(.submatch = config_stdsubmatch, .locators = locs, .devhandle = devhandle)); --- # DRM update - Long term problem: Users sad about our 4-year-old GPU support, lack of AMDGPU. - Riastradh to current-users@: "HEADS UP: Merging drm update" - GPU drivers synced with Linux 5.6, much newer hardware support. --- # DRM update challenges - Some errors difficult to debug (e.g. "strange colors on screen"). - Many users interested testing the branch before merge without the technical skills. - Users confused at our priority to update without regressions ("where's amdgpu?") - Matching Linux semantics difficult. - Lack of assertions in code ported from Linux. Once again, KASSERT critical to get useful information from users for debugging... _How can we help users test large kernel changes?_ --- # 2021 in Summary - arm: Support for Apple M1, Amlogic G12 (ODROID-N2+), Rockchip RK3288, Allwinner V3s, performance improvements... - compat_linux(8): add various new system calls, support AArch64 - Major VirtIO improvements (support for newer device standards) - regex: Add NLS support and gnu regex extensions (off by default). _Some GNU regexes that were accepted before are no longer!_ - Various kernel API audits completed, many bugs found and fixed - New password hashing and disk encryption keygen method (Argon2id) - Start of UDF file system improvements - fsck_udf(8) by 2022 - GPU driver update, synced with Linux 5.6. - Switched to GCC 10 --- # Early 2022 - Rush of changes to kernel interfaces before they have to freeze for -10 - we keep kernel modules compatible within a stable branch. - opencrypto(9) cleanup, usbenet(9) overhaul, preparation for making more kernel data structures opaque... - Version reaching as high as 9.99.99 - never been done before Lessons learned from pkgsrc: - A rush before freezing for a new branch just delays the branch ;) --- # Mid 2022 Foot placed down by releng@. - File system corruption after NetBSD 10 file system shared with NetBSD 9. - "No more major changes until FFS ACL compatibility issue resolved" - "FFSv2ea" file system type introduced as a fix. - [Mailing list thread](https://marc.info/?l=netbsd-current-users&m=165336763203336&w=2) --- # Unfinished business - Wi-Fi renewal - we need more volunteers/hardware - PF deprecation - DRM update - again? --- # Conclusions NetBSD 10.0: - _Massively better performance_ (especially on multi-processor systems). - _Device drivers:_ 19+ new, 13+ extended, 13+ removed. Support for new GPUs, Apple M1, Raspberry Pi 4, PINE64 Quartz64, ODROID-N2+, Asus Tinker Board, VMware ESXI-Arm, Xen PVH... - _Userspace programs:_ 5+ new (aiomixer, realpath, ioctlprint...), 20+ significantly improved (sh, script, vmstat, inetd...), 1 removed. - _2000+ new test cases_, kernel memory sanitizer, kernel concurrency sanitizer, various kernel-wide audits. - _Compatibillity with WireGuard_, faster better disk encryption, new ARMv8 security features. - _Enhanced UDF filesystem_, compat_linux(8), FFS access control lists, newer FUSE... _WORTH IT._ It's been an amazing journey! Thanks everyone!