The interferon-induced transmembrane protein 3 (IFITM3) gene is an interferon-stimulated gene that inhibits the replication of multiple pathogenic viruses in vitro and in vivo. IFITM3 is a member of a large protein superfamily, whose members share a functionally undefined area of high amino acid conservation, the CD225 domain. We performed mutational analyses of IFITM3 and identified multiple residues within the CD225 domain, consisting of the first intramembrane domain (intramembrane domain 1 [IM1]) and a conserved intracellular loop (CIL), that are required for restriction of both influenza A virus (IAV) and dengue virus (DENV) infection in vitro. Two phenylalanines within IM1 (F75 and F78) also mediate a physical association between IFITM proteins, and the loss of this interaction decreases IFITM3-mediated restriction. By extension, similar IM1-mediated associations may contribute to the functions of additional members of the CD225 domain family. IFITM3's distal N-terminal domain is also needed for full antiviral activity, including a tyrosine (Y20), whose alteration results in mislocalization of a portion of IFITM3 to the cell periphery and surface. Comparative analyses demonstrate that similar molecular determinants are needed for IFITM3's restriction of both IAV and DENV. However, a portion of the CIL including Y99 and R87 is preferentially needed for inhibition of the orthomyxovirus. Several IFITM3 proteins engineered with rare single-nucleotide polymorphisms demonstrated reduced expression or mislocalization, and these events were associated with enhanced viral replication in vitro, suggesting that possessing such alleles may impact an individual's risk for viral infection. On the basis of this and other data, we propose a model for IFITM3-mediated restriction.