This paper presents an efficient algorithm to calculate the primary basis functions (PBFs) of the characteristic basis function method (CBFM) for the scattering from a dielectric object. The use of the Poggio-Miller-Chang-Harrington-Wu (PMCHW) integral equation discretized by the Galerkin method of moments (MoM) with Rao-Wilton-Glisson basis functions leads to solving a linear system. For a collection of incident waves and for a given block, the CBFM needs to invert the whole PMCHW self-impedance matrix to calculate the PBFs. By decomposing the PMCHW impedance matrix into four sub-matrices of halved sizes, related to the electric and magnetic surface currents and their coupling, the computation of the PBFs is accelerated by using the impedance matrix derived from the electric field integral equation (EFIE) combined with the physical optics (named POZ) approximation. In addition, the PO developed by Jakobus and Landstorfer [35], named POJ and valid for a perfectly- conducting scatterer, is extended to a dielectric surface. Recently, the MECA (modified equivalent current approximation) Li and Mittra [29]) based on the tangent plane or Kirchhoff approximation has also been applied to expedite the PBF calculation. The presented method, HCBFM-POZ (H means halved), accelerated by the adaptive cross approximation (ACA), is tested and compared with CBFM- MECA and HCBFM-POJ on a cube and on a sphere. The numerical results show that HCBFM-POZ is valid for both the shapes, whereas the CBFM-MECA and HCBFM-POJ are not valid on a sphere.