The person you're talking too can be on a different one and their admin can see them too.

That very much depends on the protocol and type of federation, but a good point indeed

Also, I wouldn't want to be able to access content from any user - it's a "no trust needed" thing.

Sure, but e2ee also comes with lots of trade-offs and strings attached, that almost only ever make sense in case of extreme centralization (i.e. in a non-federation, where trust in the faceless provider is not an option). PFS means that setting up a new device is a PITA because you can't access your full messages history on new devices without off-band synchronization, no server-side search means that clients are either limited in this area or have to carry large histories and inefficiently search themselves, MITM/server-mediated attacks are only mitigated with verification (on top of encryption), which is a UX disaster for users non-versed into crypto (and this complexity is imposed upon such users no matter what), etc, etc.

Of course I'm not advocating against e2ee in the general case (and quite the opposite at that), but if you self host (topic of this community) for yourself and few family members, the downsides quickly outweigh the benefits and so I believe that e2ee should be left at the discretion of the users.