Modern data analysis increasingly relies on large language models (LLMs) to process diverse data modalities, including text, images, audio, and video. Although LLMs offer advanced reasoning capabilities for unstructured data, their high inference costs significantly exceed those of traditional relational operators. This computational overhead often becomes a critical bottleneck, limiting the scalability of LLM-driven analytical systems in real-world applications. This thesis introduces two complementary systems designed to mitigate this challenge and enable cost-efficient data analysis with LLMs. Both systems leverage principles from approximate query processing to balance computational cost and result quality. The first, ThalamusDB, reduces inference costs by minimizing the volume of data processed. Guided by specialized cost and error models, ThalamusDB identifies an optimal data subset for processing while respecting user-defined error constraints. The second system, SpareLLM, reduces costs by automatically selecting the smallest, most efficient LLM that satisfies user-defined equivalence constraints compared to a powerful reference model. Together, these systems present distinct strategies for achieving scalable data analysis with LLMs. Furthermore, this work introduces BitGourmet and AggChecker to address related challenges in specialized data processing and result verification, respectively.