The early life environment, when the microbiome is established, may dictate Inflammatory Bowel Disease (IBD) risk later in life. E. coli has emerged as a neonatal colonizer across species. To investigate E. coli colonization, phylogeny and virulence in response to different exposures, we analyzed metagenomic data and characterized E. coli isolates from mothers and their children. Taxa and alpha diversity for control mothers was higher than IBD mothers, and the opposite was true for E. coli abundance. An increase in taxa and alpha diversity is seen as infants age from 7 days to 4 years old with a concurrent decrease in E. coli abundance. In control mothers, E. coli were predominantly phylogroups B2 and D, whereas B1 isolates were more common in IBD mothers. In contrast, most infants were colonized by B2 isolates, independent of maternal IBD status. Furthermore, the phylogroup distribution of toddlers resembled that of control mothers. Cytotoxin and genotoxin genes were rare in maternal E. coli but common in infants of IBD mothers and absent in infants of control mothers. Conversely, AIEC-associated genes were more common in maternal E. coli particularly IBD mothers than infant E. coli. The E. coli isolates from toddler’s were similar to the maternal cohort in rarity of genes encoding cytotoxins and genotoxins, but had more genes associated with AIEC. Our findings reveal that mothers and their babies are colonized by a diverse group of non-diarrheagenic E. coli that vary in phylogeny, genotype and virulence, refuting clonal transmission of maternal isolates.