It’s my understanding that something is only part of the Fediverse if it can communicate with the ActivityPub protocol. Something that the Matrix protocol cannot.

Matrix is federated though, but with another protocol. Element X uses Matrix.