Feature/Sylvester solver

[stekien]

Summary

Adds sylvester_solver(A, B, C), which solves the Sylvester equation $AX + XB = C$ for $X$. Addresses #3396 (motivated by this Discourse thread).

A is n x n, B is m x m and C is n x m. Each argument can be real or complex, in any combination. The result is always a complex n x m matrix.

Primitive (stan/math/prim/fun/sylvester_solver.hpp). Bartels–Stewart with complex Schur decompositions $A = USU^H$ and $B = VTV^H$ (Eigen::ComplexSchur). This reduces the problem to $SY + YT = U^H C V$ with upper-triangular $S$ and $T$, which is solved one column at a time by triangular back substitution. Then $X = UYV^H$. The cost is $O(n^3 + m^3 + nm(n + m))$. The scalar type is templated, so forward mode (fvar) goes through this implementation.

Reverse mode (stan/math/rev/fun/sylvester_solver.hpp). The forward pass runs the same algorithm on values and keeps the Schur factors in the arena. With $G$ the adjoint of $X$ (adjoint of $\mathrm{Re},X$ plus $i$ times the adjoint of $\mathrm{Im},X$), the reverse pass solves the adjoint equation $A^H Z + Z B^H = G$ and accumulates

$$\nabla_A = -ZX^H, \qquad \nabla_B = -X^H Z, \qquad \nabla_C = Z.$$

The derivation is in the issue thread (#3396). The adjoint equation reuses the forward-pass Schur factors: it becomes $S^H W + W T^H = U^H G V$, solved column by column from last to first, so no second decomposition is needed. Complex inputs receive the real and imaginary parts of their gradient, and real inputs receive the real part. Inputs can be Matrix<var>, var_value<Matrix> or Matrix<std::complex<var>>, mixed freely with double inputs. The returned Matrix<std::complex<var>> is copied out of the arena, so writing into the result does not change what the reverse pass reads.

Lyapunov. The continuous Lyapunov equation $AX + XA^H = Q$ is the special case sylvester_solver(A, A.adjoint(), Q). This PR does not add a separate function for it. I can add a lyapunov_solver wrapper here or in a follow-up if that is preferred.

Questions for reviewers

• The result is complex even when all inputs are real. A real Schur (quasi-triangular) path would return a real matrix for real inputs, at the cost of handling 2 x 2 diagonal blocks. Is a complex result acceptable for a first version?
• A unique solution requires $\lambda_i(A) + \mu_j(B) \neq 0$ for all eigenvalues of A and B. This is not checked. If it fails, the result contains inf or NaN and no exception is thrown. Should the function check the diagonals of the shifted triangular systems and throw?
• Naming: sylvester_solver, or something closer to SciPy's solve_sylvester?

This adds the function to the Math library only. Exposing it in the Stan language needs a separate stanc3 PR.

Tests

test/unit/math/prim/fun/sylvester_test.cpp

• Real and complex systems with a known solution (6 x 5): checks the shape and residual, and that real inputs give an imaginary part at roundoff level.
• Mixed real and complex arguments.
• Diagonal A and B against the closed form $X_{ij} = C_{ij} / (a_i + b_j)$.
• Lyapunov equation $AX + XA^T = -Q$ with stable A and positive definite Q: the solution is symmetric and positive definite.
• Eigen expression arguments give the same result as plain matrices.
• Error handling: non-square A or B, and C with the wrong number of rows or columns, throw std::invalid_argument.
• Zero-size inputs return an empty result of the right shape.

test/unit/math/mix/fun/sylvester_test.cpp

• expect_ad with real inputs (A has a complex eigenvalue pair), complex inputs, and mixed real and complex inputs. This checks values and derivatives in reverse, forward and mixed mode against finite differences.
• Lyapunov case with B = A^T, so the autodiff variables of A enter twice.
• The size-mismatch error is checked for every autodiff type.
• var_value<Matrix> inputs give the same value and adjoints as Matrix<var> inputs. (expect_ad_matvar does not apply because the result has std::complex<var> scalars.)
• Writing into the returned matrix does not affect the reverse pass.

Side Effects

None for existing code. The PR adds two new headers and includes them in stan/math/prim/fun.hpp and stan/math/rev/fun.hpp.

Release notes

Added sylvester_solver(A, B, C), which solves the Sylvester equation AX + XB = C for real or complex matrices, with forward- and reverse-mode autodiff. The continuous Lyapunov equation AX + XA^H = Q can be solved as sylvester_solver(A, A^H, Q).

Checklist

• Copyright holder: Simons Foundation

  The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
  - Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
  - Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

• the basic tests are passing

  • unit tests pass (to run, use: ./runTests.py test/unit)
  • header checks pass, (make test-headers)
  • dependencies checks pass, (make test-math-dependencies)
  • docs build, (make doxygen)
  • code passes the built in C++ standards checks (make cpplint)

• the code is written in idiomatic C++ and changes are documented in the doxygen

• the new changes are tested

[stan-buildbot]

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan	0.22	0.22	1.0	-0.42% slower
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan	0.06	0.06	0.98	-1.81% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan	6.4	6.29	1.02	1.63% faster
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan	0.02	0.02	0.97	-2.83% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan	8.25	8.23	1.0	0.24% faster
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan	4.45	4.44	1.0	0.08% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan	167.82	217.4	0.77	-29.54% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan	0.89	0.9	0.99	-1.32% slower
stat_comp_benchmarks/benchmarks/arma/arma.stan	0.61	0.61	1.0	0.17% faster
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan	43.23	43.38	1.0	-0.35% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan	0.58	0.58	1.0	0.2% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	20.92	20.87	1.0	0.23% faster
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan	0.11	0.11	1.01	0.58% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan	3.19	3.16	1.01	0.71% faster
performance.compilation	372.99	364.9	1.02	2.17% faster
Mean result: 0.984461654245662

Jenkins Console Log
Jenkins Build Stages
Commit hash: 82cce9dc3ba9d84abc98e4e149e89d4de88daaeb

Machine information

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           43 bits physical, 48 bits virtual
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Thread(s) per core:                      2
Core(s) per socket:                      64
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              23
Model:                                   49
Model name:                              AMD EPYC 7742 64-Core Processor
Stepping:                                0
Frequency boost:                         enabled
CPU MHz:                                 1497.104
CPU max MHz:                             3416.0681
CPU min MHz:                             1500.0000
BogoMIPS:                                4491.85
Virtualization:                          AMD-V
L1d cache:                               4 MiB
L1i cache:                               4 MiB
L2 cache:                                64 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-63,128-191
NUMA node1 CPU(s):                       64-127,192-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin