Higgs Decays to Zγ and γγ in the Flavor-Gauged Two Higgs Doublet Model

The discovery of the Higgs boson at the Large Hadron Collider (LHC) by the ATLAS and CMS collaborations in 2012 marked a significant triumph for the SM [1−3]. Despite the significant progress made in particle physics, many fundamental questions remain unanswered. For example, the matter-antimatter asymmetry in the universe, the origin of neutrino mass, the nature of dark matter, and so on. These unsolved mysteries underscore the limitations of the SM. Since then, the quest for new physics (NP) beyond the SM (BSM) has been one of the central endeavors in elementary particle physics.

One of the attractive extensions is to add to the SM with an additional Higgs doublet, which was first proposed by T. D. Lee to introduce the spontaneous CP violation [4], and is now dubbed as the two Higgs doublet model (2HDM). However, a major drawback of the generic 2HDM is that it typically introduce the tree-level flavor-changing neutral currents (FCNCs) in Yukawa interactions that are unexpected in a model construction. To suppress the generation of FCNCs, various modified 2HDMs are raised to control the two Yukawa interactions including by, imposing different $ Z_2 $ charges for the two scalars [5, 6], aligning the two Yukawa matrices [7−9], or making the two Yukawa interactions proportional to off-diagonal elements of the Cabibbo–Kobayashi–Maskawa (CKM) matrix [10, 11]. The last one is usually called Branco-Grimus-Lavoura (BGL) model, named after the authors of this model [12]. A flavor gauged version of BGL model, i.e., FG2HDM, was first proposed in Ref. [13], and was further developed in Refs. [14−16]. The FG2HDM extends the original 2HDM by incorporating a scalar singlet field and an additional $ U(1)' $ flavor gauge symmetry. Compared to the SM particle spectrum, the FG2HDM features five additional physical scalars and one neutral gauge boson $ Z' $.

Recently, the ATLAS and CMS collaborations reported initial evidence for the $ h\to Z\gamma $ decay. The observed signal strength ¹ $ \mu_{Z\gamma} $ was measured at the $ 68$% confidence level (CL) to be $ 2.0_{-0.9}^{+1.0} $ by ATLAS [17], $ 2.4\pm{0.9} $ by CMS [18], and $ 2.2\pm0.7 $ in their combined analysis [19]. Such a signal strength deviates from the SM prediction $ \mu_{Z\gamma}^\mathrm{SM}=1 $ by $ 1.9\sigma $, motivating many possible explanations from two distinct perspectives. First, as $ h\to Z\gamma $ is a loop-induced process, higher-order corrections to the leading-order (LO) amplitudes from one-loop amplitudes are crucial. ² These include two-loop QCD corrections [21−24], providing a $ 0.3$% increase, and two-loop electroweak corrections [25, 26], yielding a $ 7$% enhancement. Clearly, these next-to-leading-order (NLO) corrections alone cannot account for the $ 1.9\sigma $ excess. Alternatively, the deviation may stem from BSM contributions, a possibility extensively explored in the literature (e.g., [27−44]). However, integrating the most recent ATLAS measurement ($ \mu_{Z\gamma}=0.9^{+0.7}_{-0.6} $) [45] with previous results yields a revised signal strength of $ \mu_{Z\gamma}=1.3^{+0.6}_{-0.5} $, bringing the observation into agreement with SM expectations. While this reduces the immediate tension, analysing $ h\to Z\gamma $ within specific BSM frameworks still requires a simultaneous consideration of the $ h\to\gamma\gamma $ channel. This is due to the inherent correlation between the two, as both are mediated by the same charged loop-particles [46, 47]. Given that the current experimental value $ \mu_{\gamma\gamma}=1.10\pm0.06 $ [48], highly accords with the SM, it imposes stringent constraints on the FG2HDM parameter space, effectively narrowing the regions permitted by the $ \mu_{Z\gamma} $ data.

Building upon our previous work [49], where specific Yukawa matrix textures in the FG2HDM were proposed to explain the long-standing B anomalies, the present study investigates whether the model's parameter space remains viable under current experimental constraints on $ \mu_{Z\gamma} $ and $ \mu_{\gamma\gamma} $. While the $ h\to Z\gamma $ and $ h\to\gamma\gamma $ channels have been explored in other frameworks like the complex 2HDM (C2HDM) [50] using earlier data [51, 52] the recent updates from ATLAS and CMS [17−19, 45] necessitate a more refined numerical and theoretical analysis. In the FG2HDM, both decay modes receive loop-level contributions from charged Higgs bosons; furthermore, $ h \to Z\gamma $ is modified by corrections to the $ f\bar{f}Z $ vertex. Consequently, this work aims to identify the permitted FG2HDM parameter space for $ \mu_{Z\gamma} $ and $ \mu_{\gamma\gamma} $, while accounting for the constraints on the $ f\bar{f}Z $ vertex from top-quark measurements and the $ b\to s\ell^+\ell^- $ FCNC process.

The rest of the paper is organized as follows. In Sec. II, we provide a brief overview of the key features of the FG2HDM. In Sec. III, we calculate the one-loop amplitudes for $ h\to Z\gamma $ and $ h\to\gamma\gamma $ decays in the unitary gauge. The numerical analysis and subsequent discussions are detailed in Sec. IV. Our conclusion is presented in Sec. V. For convenience, the relevant Lagrangian and the corresponding Feynman rules are prepared in Appendix A. Additionally, the explicit expressions for the scalar functions obtained from one-loop calculations are listed in Appendix B, along with the kinematics for Higgs decays in Appendix C.

The FG2HDM is a specific 2HDM comprising two $ SU(2)_L $ scalar doublets, $ \Phi_1 $ and $ \Phi_2 $, and a scalar singlet S, and endowing with an additional $ U(1)' $ flavor gauge symmetry. Here, the $ U(1)' $ charges, denoted as $ X_i $ ($ i=L_L,\cdots,S $), are assigned to ensure the $ U(1)' $ gauge invariance. The detailed relationships among these $ U(1)' $ charges for various fields are elaborated in Appendix A of Ref. [49]. In the presence of the new $ U(1)' $ gauge symmetry, one should further consider the gauge anomaly constraints. Note that a variety of anomaly-free FG2HDMs have been surveyed and studied in Refs. [14−16].

For convenience, we collect the quantum representations for various fields in FG2HDM under the $ SU(2)_L\times U(1)_Y\times U(1)' $ symmetries in Table 1. With these assignments, the total Lagrangian of the FG2HDM can be expressed as [49]

where subscripts S, G, and F denote respectively for the scalar, gauge, and fermion components of the total Lagrangian: ³

where $ L_L=(\nu_L,e_L)^T $ and $ Q_L=(u_L,d_L)^T $ represent separately for the left-handed lepton and quark $ SU(2)_L $ doublets, while $ e_R $, $ u_R $, and $ d_R $ denote the right-handed lepton, up-type quark, and down-type quark $ SU(2)_L $ singlets, respectively. The physical significance of each component is elaborated as follows.

The scalar potential in Eq. (2) is given by [49]

where $ m_{11,22,S} $, $ \lambda_{1,2,3,4,S} $, and $ \kappa_{1,2,3} $ denote the coupling constants of various interaction terms. The two doublets and the singlet in the flavor basis are parametrized as:

where the vacuum expectation values (VEVs) $ v_1 $ and $ v_2 $ combine to form the SM Higgs VEV $ v=1/(\sqrt{2}G_F)^{1/2}=\sqrt{v_1^2+v_2^2}\simeq246 $ GeV, with $ G_F $ being the Fermi constant. The singlet VEV $ v_S $ is introduced to spontaneously break the $ U(1)' $ symmetry. Rotating to the mass basis yields:

where $ U_{1} $, $ U_{2} $, and $ U_{3} $ are the rotation matrices defined as:

Here, α and β are the rotation angles, and $ U_\gamma $ and $ \tan2\alpha $ are respectively given by:

where

After spontaneous symmetry breaking, the four undesired Goldstone bosons $ G^\pm $, $ G^0 $ and $ G^{\prime0} $ in Eq. (7) are 'eaten' to give masses to the $ W^\pm $, Z, and $ Z' $ gauge bosons, respectively. This leaves us with six physical scalars: two charged Higgs $ H^\pm $, three neutral scalars H, h, and $ H_S $, and one neutral pseudoscalar $ H_A $. Their masses are separately given by

Among the four neutral particles, h, H, and $ H_S $ are CP-even, while $ H_A $ is CP-odd. Specially, the scalar h is identified as the Higgs boson discovered at the LHC in 2012 [1−3].

In Eqs. (2) and Eq. (4), the covariant derivatives for the $ SU(2)_L $ doublets and singlets are defined respectively as follows:

where $ \tau^a $ ($ a=1,2,3 $) is the Pauli matrix, and $ B_\mu $, $ \hat{Z}'_\mu $, and $ W^a_\mu $ the $ U(1)_Y $, $ U(1)' $ and $ SU(2)_L $ gauge bosons, respectively. Here, $ g_1 $, $ g' $, and $ g_2 $ are the corresponding coupling constants, while $ Y_i $ and $ X_{i} $ denote separately the hypercharge and $ U(1)' $ charge of the fields. After spontaneous symmetry breaking, the gauge bosons acquire their masses. The transformation from the flavor basis to the mass basis is achieved through the mixing matrix:

where $ \theta_W $ is the Weinberg angle, and $ \theta_2^\prime $ the mixing angle between Z and $ Z' $. In the limit where $ \theta_2^\prime $ approaches zero, the mixing restores to the case in the SM. This arrangement introduces the tree-level FCNCs in the down-type quark sector, which can account for the anomalies observed in B physics [49]. Additionally, it generates off-diagonal interactions, with Higgs and Z boson couple to charged Higgs and W boson, and new corrections to the $ f\bar{f}Z $ couplings, rendering that FG2HDM contribute more to $ h\to Z\gamma $ rather than to $ h\to\gamma\gamma $.

The interactions between fermions and gauge bosons are derived from the covariant derivative terms in the first line of Eq. (4). These terms encapsulate how fermions couple to the gauge bosons associated with the symmetries of the model. Meanwhile, the remaining terms in Eq. (4) depict the Yukawa interactions between fermions and the scalar doublets. ⁴ After spontaneous symmetry breaking, the scalar fields acquire VEVs, which give masses to the fermions.

Armed with the Lagrangian of the FG2HDM, we can derive the relevant Feynman rules. In this context, we concentrate solely on the crucial terms indispensable for computing the one-loop amplitudes of the $ h\to Z\gamma $ and $ h\to\gamma\gamma $ decays. These vital terms, together with their corresponding Feynman rules are cataloged in Appendix A.

Following a general Lorentz decomposition while enforcing photon gauge invariance, as detailed in Ref. [23], we can express the amplitudes for the processes $ h(p)\to Z(p_1)\gamma(p_2) $ and $ h(p)\to\gamma(p_1)\gamma(p_2) $ in the following form:

where $ \epsilon_{\mu(\nu)} $ represents the polarization vector of the Z boson or photon, and $ T_{X} $ denotes the one-loop functions to be determined. In the presence of NP, the function $ T_{X} $ can be written as $ T_{X}=T_{X}^\mathrm{SM}+T_{X}^\mathrm{NP} $, and the signal strength $ \mu_{X} $ is defined as

Utilizing the Feynman rules for vertices and propagators outlined in Figure A1 of Appendix A, we can determine the one-loop Feynman diagrams for the decays $ h\to Z\gamma $ and $ h\to\gamma\gamma $, as depicted in Figures 1 and 2. We have omitted the diagrams containing off-diagonal fermions in the loops, given the strong suppression of the FCNCs. Upon examining the Feynman diagrams for $ h\to Z\gamma $ and $ h\to\gamma\gamma $, an additional class of diagrams with off-diagonal couplings to the Higgs and Z boson is observed for $ h\to Z\gamma $ (i.e., the last row of Figure 1), ⁵ which suggests a greater contribution of the FG2HDM to $ h\to Z\gamma $ relative to $ h\to\gamma\gamma $. Furthermore, the novel corrections to the $ f\bar{f}Z $ couplings offer an alternative pathway to enhance the contribution to $ h\to Z\gamma $, as previously discussed in Sec. 1. The one-loop functions $ T_{X} $ can then be extracted by matching onto the calculations of the one-loop Feynman diagrams. In this work, we will calculate these diagrams with the help of the Package-X [54], details are as follows.

Figure 1.  One-loop Feynman diagrams for $ h\to Z\gamma $.

Figure 2.  One-loop Feynman diagrams for $ h\to\gamma\gamma $.

Figure A1.  Feynman rules for the relevant vertices and propagators in the unitary gauge.

The one-loop Feynman diagrams that contribute to the $ h\to Z\gamma $ process are depicted in Figure 1. We have intentionally omitted the external legs self-energy diagrams for this decay mode because they lack the Lorentz structure specified in Eq. (15), and the divergences present in the calculations of the Feynman diagrams in each row of Figure 1 are found to cancel each other out (a pattern that also applies to $ h\to\gamma\gamma $). These diagrams are categorized into four distinct classes (one row, one class), each corresponding to a type of particle involved in the internal lines. Consequently, the total one-loop function $ T_{Z\gamma} $ is decomposed of the following four distinct components:

corresponding to contributions from fermions, W boson, charged Higgs, and mixed W boson and charged Higgs loops, respectively. Specifically,

In Eq. (18), we have summed over the contributions from all the fermions, and the color factor $ N_C $ is 3 for quarks and 1 for leptons. The symbols $ \mathcal{Q}_{Lf} $, $ \mathcal{Q}_{fR} $, $ \mathcal{Q}_{1} $, $ \mathcal{Q}_{2} $ denote the $ U(1)' $ charge of left-handed fermions, right-handed fermions, $ \Phi_{1} $, and $ \Phi_{2} $, respectively. The definitions of $ M_f $, $ N_f $ (Eq. (18)) and the coupling $ \lambda_{hH^+H^-} $ (Eq. (20)) are provided in Appendix A. The analytical expressions for the scalar functions $ \Lambda(m_a^2;m_b,m_c) $, $ C_0(m_Z^2,0,m_h^2,m_i^2,m_i^2,m_i^2) $, and $ C_0(0,m_h^2,m_Z^2,m_i^2,m_i^2,m_j^2) $ in Eqs. (18)-(21) are listed in Appendix B. Note that similar expressions for $ T_X $ can also be found in Ref. [50]. It is worth emphasizing the novel contribution $ T^{WH}_{Z \gamma} $, which arises from the $ W^\pm H^\mp Z $ coupling and is uniquely present in the FG2HDM, distinguishing it from other conventional 2HDMs.

The one-loop Feynman diagrams responsible for the $ h\to\gamma\gamma $ process are illustrated in Figure 2. In the depiction of the first and second diagrams in each row, distinct momenta $ p_1 $ and $ p_2 $ are assigned to differentiate the two identical photons in the final state. Paralleling the approach for $ h\to Z\gamma $, the total one-loop function $ T_{\gamma\gamma} $ is decomposed into three distinct parts:

with each term explicitly given by:

The analytical expressions for the scalar function $ C_0(0,0,m_h^2,m_i^2,m_i^2,m_i^2) $ in Eqs. (23)-(25) can also be found in Appendix B. Unlike in the $ h \to Z \gamma $ process, there is no $ T^{WH}_{Z \gamma} $ contribution because the $ W^\pm H^\mp\gamma $ interaction is forbidden in FG2HDM, as it is in other conventional 2HDMs and the SM.

In this section, we perform a numerical analysis to assess whether the parameter spaces of FG2HDM can simultaneously accommodate the current measured $ \mu_{Z\gamma} $ and $ \mu_{\gamma\gamma} $. For ease of reference, the input parameters used in the numerical analysis throughout this work are summarized in Table 2. It should be noted that the fine structure constant, $ \alpha\equiv e^2/4\pi $, is scale dependent and is related to the electroweak gauge coupling $ g_2 $ via the relation $ e=g_2\sin\theta_W $. Additionally, we do not adopt the experimental value for $ \Gamma_h $ from the particle data group (PDG), which has a relatively large uncertainty ($ \Gamma_h^\mathrm{PDG}=3.7_{-1.4}^{+1.9} $ MeV) [48], but rather, we cite the value with smaller uncertainty from Ref. [55]. Our strategy in this section initially involves calculating the SM LO contributions for the decays $ h\to Z\gamma $ and $ h\to\gamma\gamma $, the numerical values of which provide valuable insights for NP model buildings. We then incorporate the corrections from the FG2HDM in details.

The expressions in Eqs. (18)-(21) and (23)-(25) revert to the SM formulas upon setting $ \theta'_2\to 0 $ and $ \sin(\alpha-\beta)\to 1 $. The numerical results for $ T_{Z\gamma} $ (in the unit of GeV$ ^{-1} $) in the SM are then given by:

Eq. (26) (and also Eq. (27)) clearly indicates that the dominant contributions arise from the W loops, and the secondary contributions come from the interference between the W and top quark loops. Contributions from the s, d, u, μ, and e loops are not presented here, as they are several orders of magnitude smaller than those we have displayed and therefore can be neglected safely. Likewise, the SM contributions to $ T_{\gamma\gamma} $ (in the unit of GeV$ ^{-1} $) are

With the kinematics formula derived in Appendix C, the SM predictions for $ h\to Z\gamma $ and $ h\to\gamma\gamma $ are given separately by

which are respectively in good agreements with those given in Ref. [55]: $ \mathcal{B}(h\to Z\gamma)=(1.5\pm0.1)\times10^{-3} $, and $ \mathcal{B}(h\to\gamma\gamma)=(2.27\pm0.07)\times10^{-3} $.

The $ h\to\gamma\gamma $ decay imposes stringent constraints on the FG2HDM parameters, as the SM prediction of $ \mu_{\gamma\gamma} $ aligns well with its measurement. This implies that we should have $ \sin(\alpha-\beta)\approx 1 $ (or equivalently, $ \cos(\alpha-\beta)\approx 0 $) to ensure that $ T_{\gamma\gamma}^f $ and $ T_{\gamma\gamma}^W $ are close to their SM values, and the contribution from the charged Higgs loops should be minimal. Additionally, we should have $ \cos\theta'_2\approx 1 $ (or equivalently, $ \sin\theta'_2\approx 0 $) to ensure that the $ ZW^+W^- $ coupling does not deviate too much from its SM value.

With the aforementioned approximations, we can first determine the parameter regions for the charged Higgs interactions. They can be classified into two parts, i.e., the pure charged Higgs diagrams and the $ W^\pm $-$ H^\pm $ mixing diagrams. Note that the latter only contribute to $ h\to Z\gamma $, which is actually negligible compared to the pure one. The analysis is as follows.

To estimate the magnitude of $ T_{Z\gamma}^{WH} $ given in Eq. (21), we need to endow values for $ \cos(\alpha-\beta) $ (or equivalently, $ \sin(\alpha-\beta) $), $ g' $, $ \mathcal{Q}_2-\mathcal{Q}_1 $, $ \cos\beta $ (or equivalently, $ \sin\beta $), and $ \sin\theta'_2 $. However, as we do not have the exact values for these parameters, we can only make some rough but reasonable estimates. As discussed in the first paragraph of this subsection, to ensure that $ T_{\gamma\gamma}^f $ and $ T_{\gamma\gamma}^W $ do not deviate too much from their SM predictions, we should have $ \cos(\alpha-\beta)\approx 0 $ and $ \sin\theta'_2\approx 0 $. What is more, the constraints from the B observables in our previous work suggest that $ \tan\beta<28 $ and $ g'>10^{-2} $ [49]. The large upper limit of $ \tan\beta $ suggests that $ \cos\beta $ should be small, whereas $ g' $ is expected to remain sufficiently small to preserve the perturbative nature of the $ U(1)' $ gauge theory. Furthermore, it is reasonable to assume that $ \mathcal{Q}_2-\mathcal{Q}_1 $ is of $ \mathcal{O}(1) $ under the consideration of naturalness. While satisfying these conditions and simultaneously hoping that $ T_{Z\gamma}^{WH} $ to be as large as possible, we assume that $ \cos(\alpha-\beta)\sim \cos\beta\sim \sin\theta'_2 \sim \mathcal{O}(10^{-1}) $ and $ g'\sim(\mathcal{Q}_2-\mathcal{Q}_1)\sim\mathcal{O}(1) $. With these assumptions, we can conservatively estimate:

To make a maximum estimate on $ T_{Z\gamma}^{WH} $, let us assume that $ |a| $ is close to the margin of $ \mathcal{O}(10^{-2}) $, say, $ |a|=0.01 $. With this entry, we plot the magnitude of $ T_{Z\gamma}^{WH} $ as a function of $ m_{H^\pm} $ in the left panel of Figure 3. Obviously, $ T_{Z\gamma}^{WH} $ decreases as $ m_{H^\pm} $ increases, and the magnitude of which can only reach $ \mathcal{O}(10^{-8}) $ GeV$ ^{-1} $, comparable to that of the b quark loops in the SM (cf. the value of $ T_{Z\gamma}^{b,\text{SM}} $ in Eq. (26)), which is one and two orders of magnitude smaller than that of the t quark loops and W loops, respectively. Therefore, the contribution from the $ W^\pm $-$ H^\pm $ mixing diagrams is negligible. Below, we merely consider the contributions from the pure charged Higgs diagrams.

Figure 3.  (color online) Left: The magnitude of $ T_{Z\gamma}^{WH} $ as a function of $ m_{H^\pm} $ with input $ |a|=0.01 $. Right: the $ 1\sigma $ and $ 2\sigma $ allowed regions for $ m_{H^\pm} $ and $ \lambda_{hH^+H^-} $ with constraints from $ \mu_{Z\gamma} $ (blue) and $ \mu_{\gamma\gamma} $ (red).

In the right panel of Figure 3, we present the $ 1\sigma $ and $ 2\sigma $ allowed regions for the parameters $ m_{H^\pm} $ and $ \lambda_{hH^+H^-} $, with constraints derived from $ \mu_{Z\gamma} $ (blue) and $ \mu_{\gamma\gamma} $ (red). It is clear from this plot that in the parameter regions where $ m_{H^\pm}>200 $ GeV and $ \lambda_{hH^+H^-}<0 $, both the measured $ \mu_{Z\gamma} $ and $ \mu_{\gamma\gamma} $ can be accommodated within the $ 1\sigma $ level. Interestingly, the bound on the charged Higgs mass obtained in this work is also consistent with that from the constraints of B observables in Ref. [49], see the right panel of Figure 1 of that reference. As discussed above, since $ \mu_{\gamma\gamma} $ is better than $ \mu_{Z\gamma} $ on the consistency between the SM prediction and measurement, the overlapped parameter regions allowed by the two observables in the right panel of Figure 3 are mainly narrowed down by the former.

The remaining contribution comes from the fermion loop diagrams, in which the FG2HDM influence $ h\to Z\gamma $ decay through correcting the $ f\bar{f}Z $ couplings. As shown in the Feynman diagrams depicted in Figures 1 and 2, such corrections only influence $ \mu_{Z\gamma} $ and leave $ \mu_{\gamma\gamma} $ entirely unaffected, as the latter does not involve any $ f\bar{f}Z $ vertex at all. By the way, although the $ f\bar{f}Z $ vertex corrections also influence $ h\to ZZ^\ast $ decay, yet since there involves two $ f\bar{f}Z $ vertices in such a process, the impacts to which are small with respect to that to $ h\to Z\gamma $ decay, as the latter contains only one $ f\bar{f}Z $ vertex. Therefore, we do not invoke $ h\to ZZ^\ast $ in the numerical constraints.

Note that the total $ f\bar{f}Z $ coupling in Eq. (42) has been projected into the left-handed part $ C_L^f $ and right-handed part $ C_R^f $, with

In the limit $ \cos\theta'_2\sim 1 $ ($ \sin\theta'_2\sim 0 $), the above expressions restore to the ones in the SM. Given the dominant role of the top quark in contributions from fermion loops, particular emphasis is placed on the correction from the $ t\bar{t}Z $ vertex. Using $ I_t^3=1/2 $ and $ Q_t=+2/3 $ for the top quark, along with the values of $ s_W $ and $ \alpha(m_Z) $ from Table 2, we obtain $ C_L^{t,\mathrm{SM}}\simeq0.256 $ and $ C_R^{t,\mathrm{SM}}\simeq-0.114 $. Notice that in Eq. (18) the contribution from top loops depends on the sum of $ C_L^t $ and $ C_R^t $, i.e., the vector coupling $ C_V^t\equiv(C_L^t+C_R^t)/2 $. Therefore, the deviation $ \Delta C_L^t\equiv(\Delta C_L^t+\Delta C_R^t)/2 $, where

measures deviation from the SM, i.e., the contribution from FG2HDM. The measurement of $ \mu_{Z\gamma} $ can impose a constraint on $ \Delta C_V^t $, which subsequently can be translated into the bounds on the parameters of FG2HDM.

Besides $ \mu_{Z\gamma} $, $ \Delta C_V^t $ also subjects to constraints from the following two primary sources: i) top quark observables, and ii) the FCNC process $ b\to s\ell^+\ell^- $. Regarding the former, both the CMS [56] and ATLAS [57, 58] collaborations have conducted precise measurements of the differential and/or inclusive cross section of top quark pair production in association with a Z boson at the LHC. ATLAS has also measured the production of single top quark and anti-top quark via the t-channel exchange of a W boson [59]. These measurements align well with their SM predictions and can be leveraged to establish limits on the Wilson coefficients of the SM effective field theory (SMEFT) [60−62]. By invoking the matching between $ \Delta C_V^t $ and the SMEFT Wilson coefficients [63]:

where $ C_{H u} $, $ C_{H Q}^{(1)} $, and $ C_{H Q}^{(3)} $ denote respectively the Wilson coefficients of the following SMEFT operators,

we can delineate the permissible range for $ \Delta C_V^t $. Adopting the $ 95$% CL intervals provided by ATLAS [58]:

which are obtained with $ \Lambda=1 $ TeV and when one coefficient at a time is assumed to be non-zero, we obtain $ \Delta C_V^t\in[-0.076,0.126] $. Regarding the $ b \to s \ell^+ \ell^- $ constraint, the anomalous coupling $ \Delta C_V^t $ contributes to this FCNC process via the insertion of the $ t\bar{t}Z $ vertex into the Z-penguin diagram. This modification affects the Wilson coefficients $ C_9 $ and $ C_{10} $, which are associated with the operators $ \mathcal{O}_9 \equiv (\bar{s}\gamma_\mu P_L b)(\bar{\ell}\gamma^\mu\ell) $ and $ \mathcal{O}_{10} \equiv (\bar{s}\gamma_\mu P_L b)(\bar{\ell}\gamma^\mu\gamma_5\ell) $ ($ P_L = \dfrac{1-\gamma_5}{2} $), respectively. Beyond these loop-level effects, the FG2HDM introduces additional tree-level FCNC amplitudes that contribute to $ C_9 $ and $ C_{10} $. These contributions depend on the same set of model parameters as $ \Delta C_V^t $. Consequently, our numerical strategy begins with a comprehensive parameter scan to identify the allowed ranges of the model parameters. These results are then used to determine the viable range for $ \Delta C_V^t $. The total corrections to $ C_9 $ and $ C_{10} $ are expressed as [64]:

where $ N = 2\sqrt{2}G_F V_{tb}V_{ts}^\ast \alpha / (4\pi) $, and $ m_Z $ ($ m_{Z'} $) denotes the mass of the Z ($ Z' $) boson, representing the contributions from Z-exchange ($ Z' $-exchange) diagrams. The functions $ \mathcal{A}_{L} $ and $ \mathcal{B}_{(5)} $ depend on the FG2HDM parameter set $ \{g', \mathcal{Q}_{\mu_R}, \mathcal{Q}_{d_R}, \sin\theta_2'\} $, for which detailed expressions are provided in Ref. [49]. By fixing $ \sin\theta_2' = 0.1 $, requiring $ m_{Z'} \lt 450 $ GeV, and incorporating the established constraints, our scan reveals that $ \Delta C_V^t $ is constrained within $ [-0.05, 0.05] $ at the $ 3\sigma $ level.

Fixing $ g'=1 $ and $ \sin\theta'_2=0.1 $, ⁶ and designating $ \mathcal{Q}_{tL} $ and $ \mathcal{Q}_{tR} $ as the two independent parameters that need to be restricted, the constraints on which from $ \mu_{Z\gamma} $ (blue), top quark observables (orange), and $ b\to s\ell^+\ell^- $ (purple) are shown in Figure 4. It is clear from this plot that there is an overlapped region for $ \mathcal{Q}_{tL} $ and $ \mathcal{Q}_{tR} $ to simultaneously reconcile the three observables. However, due to the relatively large uncertainty, the constraint from $ \mu_{Z\gamma} $ is weaker than that from top quark observables and $ b\to s\ell^+\ell^- $. It is also observed that up to date the most stringent bound in this respect is taken from low-energy process $ b\to s\ell^+\ell^- $ decay.

Figure 4.  (color online) The $ \mathcal{Q}_{tL} $-$ \mathcal{Q}_{tR} $ allowed regions obtained from $ \mu_{Z\gamma} $ (blue), top quark observables (orange), and $ b\to s\ell^+\ell^- $ (purple), with $ g'=1 $ and $ \sin\theta'_2=0.1 $.

By utilizing the parameters derived above, we evaluate the contributions of different T functions to $ \mu_{Z\gamma} $ and $ \mu_{\gamma\gamma} $. The results show that $ T_{Z\gamma}^W $ and $ T_{\gamma\gamma}^W $ are of the order $ 10^{-5} $ GeV$ ^{-1} $, consistent with the SM, whereas $ T_{Z\gamma}^H $, $ T_{\gamma\gamma}^H $, and $ T_{Z\gamma}^t $ are around $ 10^{-6} $ GeV$ ^{-1} $. Due to this clear hierarchy, the W-boson contributions dominate the amplitudes. Interference terms involving W are sub-dominant, and all remaining cross-terms are effectively negligible. This justifies our approach of analysing the constraints from each T function separately rather than accounting for their combined interference effects.

In this paper, we investigated the $ h\to Z\gamma $ and $ h\to \gamma\gamma $ decays within the framework of the FG2HDM. The FG2HDM is a BSM model based on the $ SU(3)_C\times SU(2)_L\times U(1)_Y\times U(1)' $ gauge symmetry, extending the SM particle spectrum with five additional physical scalars and one neutral gauge boson, $ Z' $. We derived the Feynman rules for the model and calculated the one-loop amplitudes for both decay channels. We found that while both channels are influenced by the charged Higgs bosons, $ h \to Z\gamma $ is uniquely modified by corrections to $ f\bar{f}Z $ vertex. These vertex corrections are also subject to constraints from top-quark observables and the FCNC process $ b\to s\ell^+\ell^- $. Our numerical analysis demonstrated that the amplitude contributions from $ W^\pm $-$ H^\pm $ mixing diagrams are comparable to the SM b-quark loop contributions, which are known to be negligible. Consequently, we focused our analysis on the contributions from pure charged Higgs diagrams. We showed that a parameter region with $ m_{H^\pm}>200 $ GeV and $ \lambda_{hH^+H^-}<0 $ can simultaneously accommodate both $ \mu_{Z\gamma} $ and $ \mu_{\gamma\gamma} $ measurements at the $ 1\sigma $ level. This region is primarily constrained by the latter due to its relatively small uncertainty. regarding the constraints on the $ f\bar{f}Z $ vertex corrections, we considered only the dominant contribution from the top quark. By fixing $ g'=1 $ and $ \sin\theta'_2=0.1 $, we identified a viable region that simultaneously satisfies constraints from $ \mu_{Z\gamma} $, top-quark observables, and $ b\to s\ell^+\ell^- $, with the strongest constraints coming from $ b\to s\ell^+\ell^- $. In summary, due to the currently large uncertainty associated with $ \mu_{Z\gamma} $, its constraining power is limited compared to other precision observables. Nevertheless, as discussed in the previous section, the HL-LHC with a projected integrated luminosity of $ 3\; \rm{ab}^{-1} $ is expected to reduce the relative uncertainty of $ \mu_{Z\gamma} $ to $ 14$% [65]. This improvement would significantly strengthen the constraints on the FG2HDM.

Note Added. All the authors contribute equally and they are co-first authors, while F. Xu is the corresponding author.

Since there involves at least one photon in both $ h\to Z\gamma $ and $ h\to\gamma\gamma $ decays, the propagators of the one-loop diagrams shall always be charged particles, see Figures 1 and 2. Therefore, to calculate the amplitudes we should first find out all the Lagrangian that describe the interactions between the charged particles and the Higgs, Z boson, and photon. The formalisms of Feynman rules depend on the choice of gauge. In this paper we work in the unitary gauge, in which the Goldstone bosons disappear and only physical particles participate the interactions. After scrutinizing the total Lagrangian provided in Ref. [49], we list the most relevant terms to calculate the amplitudes of $ h\to Z\gamma $ and $ h\to\gamma\gamma $ decays as follows,

The first Lagrangian $ \mathcal{L}_{S} $, which describes the interaction between the neutral and charged Higgs, is obtained by expressing the scalar potential (cf. Eq. (5)) in terms of physical fields, and their coupling reads

The second Lagrangian $ \mathcal{L}_{SF} $ is the Yukawa interactions, with $ M_f $ ($ f=u,d,\ell $) being the diagonal mass matrices: $ M_u=\text{diag}(m_u,m_c,m_t) $, $ M_d=(m_d,m_s,m_b) $, and $ M_\ell=(e,\mu,\tau) $. Here, $ M_f $ is diagonalized by a biunitary transformation:

where $ Y_1^f $ and $ Y_2^f $ are Yukawa matrices, and $ U_{fL} $ and $ U_{fR} $ are unitary matrices. For convenience, we have also introduced the auxiliary matrices $ N_f $, which are defined as

To investigate the B-anomalies reported in Ref. [49], we employ specific $ U(1)' $ charge assignments for the fields. This configuration ensures that tree-level FCNCs emerge specifically in the down-type quark sector. With such assignments, the Yukawa matrices have the following textures:

where '$ * $' denotes an arbitrary non-zero number. Then the auxiliary matrices $ N_f $ have the following explicit forms:

where $ V_{\rm CKM} $ denotes the Cabibbo-Kobayashi-Maskawa (CKM) matrix [10, 11]. The third Lagrangian $ \mathcal{L}_{SG} $ describes the interactions between the scalars and the gauge bosons, which is obtained by expanding the scalar kinetic terms with gauge bosons contained in the covariant derivatives. The quantities $ \mathcal{Q}_1 $ and $ \mathcal{Q}_2 $ denote respectively the $ U(1)' $ charges of $ \Phi_1 $ and $ \Phi_2 $, and $ s_W $ and $ c_W $ are separately short for $ \sin\theta_W $ and $ \cos\theta_W $. The fourth Lagrangian $ \mathcal{L}_{G} $ describes the interactions between the gauge bosons, which results from the vector field strength terms. The last Lagrangian $ \mathcal{L}_{FG} $ contains the interactions between the fermions and Z boson and photon, where $ I_f^3 $ stands for the third component of the weak isospin of a fermion doublet, $ Q_f $ denote the electric charge of a given fermion f, and $ \mathcal{Q}_{fL} $ and $ \mathcal{Q}_{fR} $ label the $ U(1)' $ charge of a left-handed and right-handed fermion, respectively. It is also clear from Eq. (A5) that due to the $ Z-Z' $ mixing there is an additional term that has the same Lorentz structure as the SM one, which therefore provides a correction to the original couplings. For more details of FG2HDM Lagrangian in different sectors, readers are referred to Ref. [49].

With the Lagrangian listed in Eqs. (A1)-(A5) at hand, obtaining the Feynman rules for the vertices is straightforward. Besides, the Feynman rules for the propagators of the charged Higgs, W boson, and fermions can be derived directly from the free Lagrangian that we do not show. Working in the unitary gauge, we summarize all the relevant vertices as well as propagators of FG2HDM that are necessary to calculate the amplitudes of $ h\to Z\gamma $ and $ h\to\gamma\gamma $ decays in Figure A1.

In this appendix, we show the analytical expressions for the scalar functions appearing in Sec. III A and III B.

where $ \Delta_{ij}=m_i^2-m_j^2 $, $ \lambda(a,b,c)=a^2+b^2+c^2-2ab-2ac-2bc $ is the usual Källén function, and $ \mathrm{DiLog}[a,b] $ is a function defined in Package-X [54].

Both $ h(p)\to Z(p_1)\gamma(p_2) $ and $ h(p)\to \gamma(p_1)\gamma(p_2) $ are processes of two-body decays, the differential decay width of which reads

where $ \mathcal{M}_X $ denotes the amplitude, and $ d\Phi_2 $ is the two-body decay phase space given by

Here, $ |\vec{p}_1|=\lambda^{1/2}(m_h^2,m_1^2,m_2^2)/2m_h $, which is equal to $ (m_h^2-m_Z^2)/2m_h $ for $ h\to Z\gamma $ and $ m_h/2 $ for $ h\to\gamma\gamma $, respectively; $ d\Omega=d\phi_1 d\cos\theta_1 $ being the solid angle of particle $ 1 $ in the final state, which, after integration, yields $ 4\pi $. To calculate the decay width or branching ratio of the two processes, one has to sum over the spins of Z boson and photon in the final state:

For $ X=Z\gamma $, one has

Substituting this into Eqs. (C3) and (C1) and dividing by the total Higgs decay width $ \Gamma_h $ yields the branching fraction for $ h\to Z\gamma $,

Similarly, for $ X=\gamma\gamma $, one has

and the branching fraction for $ h\to\gamma\gamma $ reads

In deriving Eq. (C7), since there are two identical photons in the final state, an extra $ 1/2 $ factor has been taken into account.