Discourse structures characterize inter-sentential relationships and textual organization, enhancing high-level text comprehension. However, deriving these structures relies on annotated data that are linguistically sophisticated and thus challenging to obtain. In the wake of large language models, we introduce a paradigm shift that views discourse structure through the lens of free-form question answering, aligning with the linguistic framework of Questions Under Discussion (QUD). We construct two tasks under QUD: (1) generating curiosity-driven, open-ended questions and locating their answers in a document; (2) QUD dependency parsing, by considering each sentence as the answer to an implicit question elicited from context. We further discuss the evaluation of QUD question generation in a linguistically informed scheme. We conclude with potential new applications enabled by the QUD framework that highlight its versatility, and natural fit with LLMs and human interactions alike.