For web client I recommend that you implement your inheritor of CubaAuthProvider and set FQN of this class to cuba.web.externalAuthenticationProviderClass. Then enable cuba.web.externalAuthentication application property.
In fact CubaAuthProvider is a javax.servlet.Filter with additional methods. If you want to implement SSO using SAML you can implement doFilter and perform security checks there. As an example of CubaAuthProvider with SSO you can see IdpAuthProvider.
If you want to simply authenticate your users against SAML server in LoginWindow you just override authenticate method. You can find such an implementation in LdapAuthProvider.
At the moment, we do not support custom authentication for portal and rest-api. There is an issue that addresses this problem: https://youtrack.cuba-platform.com/issue/PL-8477
See also: https://doc.cuba-platform.com/manual-6.4/ldap.html