Is 4.26 worth the upgrade?

In my opinion, yes.
For me, at least, (with my potato computer) performance is better and project loading times are shorter. The ability to edit 3d meshes in-editor is really cool and allows for a more streamlined experience when designing levels.
Also, I personally mostly use 3d assets rather than plug-ins because I much rather implement my own code in my projects. If you are worried about bp plugins/projects fitting your project, I'd recommend learning how to implement them yourself (Not to mention a lot of the bp assets available are not actually that good).
If it is CPP code you are worried about, and you don't know how to code (or don't want to), you can mostly use 4.25 code plugins compiled in 4.26 (the code is still the same).
Additionally, soon you will have to migrate your project to UE5, which has some really interesting features that are a fundamental improvement from UE4. So you might as well do it, before your project becomes big and bloated and it takes forever to migrate.